Skip to contentSkip to conversation

Parameters for the ingress qdisc?

LARTC Patrick Turley 3 participants 12 messages Aug 5, 2003 – Aug 10, 2003

lore
Share & tools
lore
#0 lore raw

I can't find any documentation on the paramaters for the ingress qdisc. Can someone help me?

I have a number of filters feeding into my ingress qdisc, all of which are rate limited, but I want to place a limit on the aggregate flow as well. I don't want to monitor the sum of the flow rates - I want to place a hard ceiling on the total possible flow that overrides everything else.

Any further details about the internal workings of the ingress qdisc would also be very helpful. I've seen some stuff that suggests its really a TBF in disguise, but I'm not sure. In particular, I'd like to understand if the flows into the ingress qdisc can borrow from the root and/or each other. Also, since the ingress qdisc is classless, how does it work that each of the incoming flows are separately metered? (As I understand it, filters only serve to guide packets into classes)

I know that the IMQ device offers a more sophisticated way to do this but, for now, that device is not available to me.

I need answers to these specific questions, but I very much appreciate any other information anyone would like to volunteer.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 17 lines · Patrick Turley

I can't find any documentation on the paramaters for the ingress qdisc. Can someone help me?

I have a number of filters feeding into my ingress qdisc, all of which are rate limited, but I want to place a limit on the aggregate flow as well. I don't want to monitor the sum of the flow rates - I want to place a hard ceiling on the total possible flow that overrides everything else.

Any further details about the internal workings of the ingress qdisc would also be very helpful. I've seen some stuff that suggests its really a TBF in disguise, but I'm not sure. In particular, I'd like to understand if the flows into the ingress qdisc can borrow from the root and/or each other. Also, since the ingress qdisc is classless, how does it work that each of the incoming flows are separately metered? (As I understand it, filters only serve to guide packets into classes)

Quoted 5 lines

I know that the IMQ device offers a more sophisticated way to do this but, for now, that device is not available to me.

I need answers to these specific questions, but I very much appreciate any other information anyone would like to volunteer.

The ingress qdisc itself has no parameters. The only thing you can do is using the policers. I have a link with a patch to extend this :
http://www.cyberus.ca/~hadi/patches/action/
Maybe this can help.

I have some more info about ingress in my mail files, but I have to sort it out and put it somewhere on docum.org. But I still didn't found the the time to do so.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to Patrick Turley

Thank you very much for the link. That information was very helpful and has given me a lot of insight.

Of course, the more I know, the more questions occur...

I try not to be a nuisance, so I've been doing a lot of research and I thought I'd already found all the important web sites for Linux traffic control. I looked again, and I still can't find anything about "filter policers" anywhere. I didn't find any description of a command line that even suggested such a thing was possible. Can you please point me to some more info about this, if any exists?

The fact that the filters are metering traffic flows implies that they have are stateful. When using filters with egress queue hierarchies, it was my understanding that no state was needed since all they do is direct packets into classes. It sounds like my current understanding is quite wrong.

BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we are VERY reluctant to apply patches of any kind, so we're just going to work with whatever is available in the bits we download. We're considering moving up to 2.4.21 to get IMQ, but that doesn't appear to be available for RedHat 7.3 yet - in fact, this may force us to RedHat 9.0 (not a bad thing, really).

Quoted 10 lines · Stef Coene
Quoted 4 lines · Patrick Turley

I can't find any documentation on the paramaters for the ingress qdisc. Can someone help me?

...

The ingress qdisc itself has no parameters. The only thing you can do is using the policers. I have a link with a patch to extend this :
http://www.cyberus.ca/~hadi/patches/action/
Maybe this can help.

Quoted 5 lines

I have some more info about ingress in my mail files, but I have to sort it out and put it somewhere on docum.org. But I still didn't found the the time to do so.

Stef

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Patrick Turley
Quoted 11 lines · Patrick Turley

Thank you very much for the link. That information was very helpful and has given me a lot of insight.

Of course, the more I know, the more questions occur...

I try not to be a nuisance, so I've been doing a lot of research and I thought I'd already found all the important web sites for Linux traffic control. I looked again, and I still can't find anything about "filter policers" anywhere. I didn't find any description of a command line that even suggested such a thing was possible. Can you please point me to some more info about this, if any exists?

There also some limited example scripts in the iproute2 source.

Quoted 5 lines

The fact that the filters are metering traffic flows implies that they have are stateful. When using filters with egress queue hierarchies, it was my understanding that no state was needed since all they do is direct packets into classes. It sounds like my current understanding is quite wrong.

I don't think the filters are statefull. If you use policers, you only rate limit the filters. The filters itself have no idea about the traffic flows.

Quoted 6 lines

BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we are VERY reluctant to apply patches of any kind, so we're just going to work with whatever is available in the bits we download. We're considering moving up to 2.4.21 to get IMQ, but that doesn't appear to be available for RedHat 7.3 yet - in fact, this may force us to RedHat 9.0 (not a bad thing, really).

Why don't you download the kernel source 7.3 2.4.20-18.7 from RH and the config file? So you have the same source as the kernel you trust. If you trust that source, you can pacth it with what ever you want.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#4 lore raw
In reply to Patrick Turley
Quoted 5 lines
Quoted 4 lines

I still can't find anything about "filter
policers" anywhere. I didn't find any description of a command line that even suggested such a thing was possible. Can you please point me to some more info about this, if any exists?

There also some limited example scripts in the iproute2 source.

OK - I'll have a look at those scripts to see how one can associate policers with filters.

Quoted 7 lines
Quoted 5 lines

The fact that the filters are metering traffic flows implies that they have are stateful. When using filters with egress queue hierarchies, it was my understanding that no state was needed since all they do is direct packets into classes. It sounds like my current understanding is quite wrong.

I don't think the filters are statefull. If you use policers, you only rate limit the filters. The filters itself have no idea about the traffic flows.

Then there are two further questions (the first of which I actually should have asked before):

1) When working with egress traffic control, filters are attached to classes. The documents I've read so far tell me that the ingress qdisc is classless so, on the face of it, filters shouldn't be useable at all. Please tell me which part of this I have badly misunderstood.

2) Since the filters themselves are, as you say, stateless, then it sounds like a "policer" is a separate object that's being created at the same time as the filter. Is there any other way to create a "policer" object, or do they only exist when attached to filters? When working with egress traffic control, can you attach policers to filters, and how would they interact with the classes?

Quoted 9 lines
Quoted 6 lines

BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we are VERY reluctant to apply patches of any kind, so we're just going to work with whatever is available in the bits we download. We're considering moving up to 2.4.21 to get IMQ, but that doesn't appear to be available for RedHat 7.3 yet - in fact, this may force us to RedHat 9.0 (not a bad thing, really).

Why don't you download the kernel source 7.3 2.4.20-18.7 from RH and the config file? So you have the same source as the kernel you trust. If you trust that source, you can pacth it with what ever you want.

Our problem is not really with "trust". Our problem is that we have very limited staff, and we don't want to open up a whole new area of effort. Using "stock" kernels and not having to compile them means that none of us have to go through the effort of managing all that. We have the expertise, we just don't have the time.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Patrick Turley
Quoted 10 lines · Patrick Turley
Quoted 9 lines
Quoted 5 lines

The fact that the filters are metering traffic flows implies that they have are stateful. When using filters with egress queue hierarchies, it was my understanding that no state was needed since all they do is direct packets into classes. It sounds like my current understanding is quite wrong.

I don't think the filters are statefull. If you use policers, you only rate limit the filters. The filters itself have no idea about the traffic flows.

Quoted 7 lines

Then there are two further questions (the first of which I actually should have asked before):

1) When working with egress traffic control, filters are attached to classes. The documents I've read so far tell me that the ingress qdisc is classless so, on the face of it, filters shouldn't be useable at all. Please tell me which part of this I have badly misunderstood.

Ingress is indeed classless, but you can filters on it. It's not important where the filters point to because there are no classes. But you can add policers to the filters.

Quoted 6 lines

2) Since the filters themselves are, as you say, stateless, then it sounds like a "policer" is a separate object that's being created at the same time as the filter. Is there any other way to create a "policer" object, or do they only exist when attached to filters? When working with egress traffic control, can you attach policers to filters, and how would they interact with the classes?

You have to see the policer as a small tbf qdisc that exists on his own. If you add the policer to a filter, all packets are "queued" in the policer and throttled.
You can also use the same policer on different filters. So the packets from different filters are "queued" in the same policer. To do so, you need the tcindex option.

I will dig in my mail archive and put online on www.docum.org what I can find.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#6 lore raw
In reply to Patrick Turley
Quoted 2 lines · Stef Coene

I will dig in my mail archive and put online on www.docum.org what I can find.

The only thing I found was :
http://www.docum.org/stef.coene/qos/faq/cache/69.html

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#7 lore raw
In reply to Patrick Turley

Hello all,

I played a bit with the ingress qdisc after seeing Patrick and Stef talking about it and came up with a few notes and a few questions.

: The ingress qdisc itself has no parameters. The only thing you can do : is using the policers. I have a link with a patch to extend this :
: http://www.cyberus.ca/~hadi/patches/action/ Maybe this can help. :
: I have some more info about ingress in my mail files, but I have to : sort it out and put it somewhere on docum.org. But I still didn't : found the the time to do so.

Regarding policers and the ingress qdisc. I have never used them before today, but have the following understanding.

About the ingress qdisc:

- ingress qdisc (known as "ffff:") can't have any children classes (hence the existence of IMQ)
- the only thing you can do with the ingress qdisc is attach filters

About filtering on the ingress qdisc:

- since there are no classes to which to direct the packets, the only reasonable option (reasonable, indeed!) is to drop the packets
- with clever use of filtering, you can limit particular traffic signatures to particular uses of your bandwidth

Here's an example of using an ingress policer to limit inbound traffic from a particular set of IPs on a per IP basis. In this case, traffic from each of these source IPs is limited to a T1's worth of bandwidth. Note that this means that this host can receive up to 1536kbit (768kbit + 768kbit) worth of bandwidth from these two source IPs alone.

# -- start of script
#! /bin/ash
#
# -- simulate a much smaller amount of bandwidth than the 100MBit interface #
RATE36kbit
DEV=eth0
SOURCES="10.168.53.2/32 10.168.73.10/32 10.168.28.20/32"

# -- attach our ingress qdisc
#
tc qdisc add dev $DEV ingress

# -- cap bandwidth from particular source IPs
#

for SOURCE in $SOURCES ; do

  tc filter add dev $DEV parent ffff: protocol ip   \
    u32 match ip src $SOURCE flowid :1              \
    police rate $RATE mtu 12k burst 10k drop

done

# -- end of script

Now, if you are using multiple public IPs on your masquerading/SNAT host, you can use "u32 match ip dst $PER_IP" with a drop action to force a particular rate on inbound traffic to that IP.

My entirely unquantified impression is that latency suffers as a result, but traffic is indeed bandwidth limited.

Just a few notes of dissection:

  tc filter add dev $DEV   # -- the usual beginnings
    parent ffff:           # -- the ingress qdisc itself
    protocol ip            # -- more preamble  | make sure to visit
    u32 match ip           # -- u32 classifier | http://lartc.org/howto/
    src $SOURCE            # -- could also be "dst $SOME_LOCAL_IP"
    flowid :1              # -- ??? (but it doesn't work without this)
    police rate $RATE      # -- put a policer here
    mtu 12k burst 10k      # -- ???
    drop                   # -- drop packets exceeding our police params

Maybe a guru or two out there (Stef?, Bert?, Jamal?, Werner?) can explain why mtu needs to be larger than 1k (didn't work for me anyway) and also how these other parameters should be used.

I'll add a few questions:

Why does policing fail entirely without "flowid :1"? (There's no flow!) Why does the policer drop all traffic if mtu is small?
How are mtu and burst used?

Oh yes, and "Ah, that's linux!"

-Martin

Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#8 lore raw
In reply to Patrick Turley

Patrick,

As far as I'm concerned, these are two very good questions.

: 1) When working with egress traffic control, filters are attached to

 :    classes. The documents I've read so far tell me that the ingress
 :    qdisc is classless so, on the face of it, filters shouldn't be
 :    useable at all.  Please tell me which part of this I have badly
 :    misunderstood.

Agreed....it doesn't make sense, to me, except as code reuse. Any takers care to explain why this is?

: 2) Since the filters themselves are, as you say, stateless, then it

 :    sounds like a "policer" is a separate object that's being created at
 :    the same time as the filter. Is there any other way to create a
 :    "policer" object, or do they only exist when attached to filters?

My understand is that the policers only exist attached to filters. They are separate, but not able to be generated without a filter. I think Stef's description isn't so bad....a policer is similar to a token bucket filter (TBF) which is attached to a filter. I don't know what it looks like inside the policing code, so I'll have to assume that this is correct. From the outside of the black box, they are pretty much rate limiters which you can attach to any "tc filter" command.

: When working with egress traffic control, can you attach policers
: to filters,

Yes.

: and how would they interact with the classes?

When a new packet arrives on an output queue prior to outbound traffic control (but after netfilter POSTROUTING), the packet will first traverse the filters attached to qdisc 1:0 to learn of its destination flowid. Here, you could have a number of different filters which classify and/or reclassify the packet into different output classes/qdiscs. You could optionally drop packets with a filter here, just as you could drop packets with filters on the ingress qdisc.

These policers will be VERB (executed? obeyed?) before the packet ever enters the classes and sub-classes.

-Martin

Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#9 lore raw
In reply to Patrick Turley

First of all, thank you Martin - this is fabulously helpful.

Quoted 7 lines · Martin A. Brown

Hello all,

I played a bit with the ingress qdisc after seeing Patrick and Stef talking about it and came up with a few notes and a few questions.

...

Quoted 7 lines

About filtering on the ingress qdisc:

- since there are no classes to which to direct the packets, the only reasonable option (reasonable, indeed!) is to drop the packets

- with clever use of filtering, you can limit particular traffic signatures to particular uses of your bandwidth

That sounds like a correct statement about the filters themselves. But it appears that we don't have to just drop the packets, because the policers we attach to the filters CAN queue packets. The following quote is from one of Stef's earlier replies:

Quoted 3 lines

You have to see the policer as a small tbf qdisc that exists on his own. If you add the policer to a filter, all packets are "queued" in the policer and throttled.

Martin continues...

Quoted 15 lines

Here's an example of using an ingress policer to limit inbound traffic from a particular set of IPs on a per IP basis. In this case, traffic from each of these source IPs is limited to a T1's worth of bandwidth. Note that this means that this host can receive up to 1536kbit (768kbit + 768kbit) worth of bandwidth from these two source IPs alone.

# -- start of script
#! /bin/ash
#
# -- simulate a much smaller amount of bandwidth than the 100MBit interface #
RATE36kbit
DEV=eth0
SOURCES="10.168.53.2/32 10.168.73.10/32 10.168.28.20/32"

Quoted 3 lines

# -- attach our ingress qdisc
#
tc qdisc add dev $DEV ingress

I don't see a handle argument here. I presume that, since the ingress handle for an interface MUST be ffff, that the handle argument is optional.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
In reply to Patrick Turley
Quoted 5 lines · Martin A. Brown

: 2) Since the filters themselves are, as you say, stateless, then it

 :    sounds like a "policer" is a separate object that's being created at
 :    the same time as the filter. Is there any other way to create a
 :    "policer" object, or do they only exist when attached to filters?
Quoted 7 lines

My understand is that the policers only exist attached to filters. They are separate, but not able to be generated without a filter. I think Stef's description isn't so bad....a policer is similar to a token bucket filter (TBF) which is attached to a filter. I don't know what it looks like inside the policing code, so I'll have to assume that this is correct. From the outside of the black box, they are pretty much rate limiters which you can attach to any "tc filter" command.

From what Stef says, the documentation on the TC_INDEX qdisc illuminates the use and structure of policers. I've looked TC_INDEX only very briefly. I mention this here only so that anyone else who is interested will have the pointer.

Quoted 15 lines

: When working with egress traffic control, can you attach policers
: to filters,

Yes.

: and how would they interact with the classes?

When a new packet arrives on an output queue prior to outbound traffic control (but after netfilter POSTROUTING), the packet will first traverse the filters attached to qdisc 1:0 to learn of its destination flowid. Here, you could have a number of different filters which classify and/or reclassify the packet into different output classes/qdiscs. You could optionally drop packets with a filter here, just as you could drop packets with filters on the ingress qdisc.

Quoted 2 lines

These policers will be VERB (executed? obeyed?) before the packet ever enters the classes and sub-classes.

OK - This is very interesting to me. I understand the mechanism whereby filters guide packets to the classes for which they are destined. That all makes sense.

Based on what I've read here, it sounds like you can interpose a policer between the filter itself and the class to which it is directing packets. Given that classes already have access to the full range of qdiscs, with all their marvelous dials and knobs, interposing a policer seems superfluous. Have I misunderstood?

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Patrick Turley

: OK - This is very interesting to me. I understand the mechanism whereby : filters guide packets to the classes for which they are destined. That : all makes sense.
:
: Based on what I've read here, it sounds like you can interpose a : policer between the filter itself and the class to which it is : directing packets. Given that classes already have access to the full : range of qdiscs, with all their marvelous dials and knobs, interposing : a policer seems superfluous. Have I misunderstood?

Well, I wouldn't say superfluous.....and I don't think you have misunderstood, but perhaps you don't yet see the rationale for a policer. Let me suggest a scenario. You have users who send mail.* You wish to guarantee a certain small percentage of your 2Mbit to only the SMTP traffic, so that at least 256kbit is reserved for mail at any time. Additionally, you want to put any SMTP flows above this 256kbit into a general consumption class.

Well, your policer reclassifies any traffic above a given rate into the general consumption class.

That's just one way of using a policer. My understanding is that policers are more commonly used for DiffServ domains and limiting inbound traffic, although there's nothing preventing them from being used in ingress traffic situations.

-Martin

* I know it sounds incredible for me to suggest that a network admin might have users, but suspend your disbelief for now, if you can.

Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/