Skip to contentSkip to conversation

how flexible is ingress traffic policing to bandwidth limit?

LARTC Greg Stark 5 participants 9 messages Jun 9, 2004 – Jun 11, 2004

lore
Share & tools
lore

Greg Stark

author
#0 lore raw
Quoted 4 lines · Damion de Soto

You can create different ingress policers that only match specific ports, and give them different priorities, but that still won't work as well as using IMQ, or if your box is a gateway (and you are only shaping traffic going through it), then you can use egress queues on the LAN interface.

For some reason that hadn't occurred to me. That should work just fine. I guess I should mark the packets in iptables to avoid throttling traffic from gateway itself, or does match see the external ip?

IMQ does seem like a handy tool, but why is there a distinction at all between egress and ingress qdiscs at all? Why not just allow people to attach HTB as an ingress qdisc directly?

I suppose in an ideal world the best thing would be to receive the packets, hand them to user-space, but not mark them as received, ie, not ack them. That avoids introducing any loss from the user point of view but still slows the flow down. But that seems fiddly and would only work for TCP I guess.

Signature · 5 lines
greg

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw

Greg,

Quoted 3 lines

For some reason that hadn't occurred to me. That should work just fine. I guess I should mark the packets in iptables to avoid throttling traffic from gateway itself, or does match see the external ip?

The only (common) time you need to use iptables to mark traffic, is when you're using
egress qdiscs on your outbound (WAN) interface (to shape upload speeds) and want to
filter based on private (LAN) IP address.

Quoted 3 lines

IMQ does seem like a handy tool, but why is there a distinction at all between egress and ingress qdiscs at all? Why not just allow people to attach HTB as an ingress qdisc directly?

because you can't shape inbound traffic. Shaping works by delaying the transmission, and you can't delay packets that haven't arrived yet. Ingress policing just drops packets, and hopes the sender will slow down.

Quoted 4 lines

I suppose in an ideal world the best thing would be to receive the packets, hand them to user-space, but not mark them as received, ie, not ack them. That avoids introducing any loss from the user point of view but still slows the flow down. But that seems fiddly and would only work for TCP I guess.

That's an interesting idea, but yeah... i think it might be a bit hard to implement, and violate far too many RFCs...

regards,

Signature · 10 lines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damion de Soto - Software Engineer  email:     [email protected]
SnapGear - A CyberGuard Company ---    ph:         +61 7 3435 2809
  | Custom Embedded Solutions          fax:         +61 7 3891 3630
  | and Security Appliances            web: http://www.snapgear.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ---  Free Embedded Linux Distro at   http://www.snapgear.org  ---
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#2 lore raw
In reply to Greg Stark
Quoted 8 lines · Damion de Soto

Greg,

Quoted 3 lines

For some reason that hadn't occurred to me. That should work just fine. I guess I should mark the packets in iptables to avoid throttling traffic from gateway itself, or does match see the external ip?

The only (common) time you need to use iptables to mark traffic, is when you're using egress qdiscs on your outbound (WAN) interface (to shape upload speeds) and want to filter based on private (LAN) IP address.

Well I'll be doing the opposite here, putting an egress qdisc on my LAN interface limiting the downstream bandwidth. So I would have the equivalent need, to distinguish traffic originating from outside vs traffic originating from the router so I can still scp kernels over to the router without being limited. Hm, though in practice I hardly ever scp things _from_ the router so perhaps I don't care.

Quoted 7 lines
Quoted 3 lines

IMQ does seem like a handy tool, but why is there a distinction at all between egress and ingress qdiscs at all? Why not just allow people to attach HTB as an ingress qdisc directly?

because you can't shape inbound traffic. Shaping works by delaying the transmission, and you can't delay packets that haven't arrived yet. Ingress policing just drops packets, and hopes the sender will slow down.

Well ultimately all shaping works by dropping packets. Merely delaying transmission isn't going to slow down anything in the long run, just increase the pipeline. You can delay and/or drop them after they've arrived just as easily. Though it would have to be before they're ack'd and delivered to the user. That's basically what IMQ does, I'm just saying perhaps that should just work instead of requiring a fake interface.

Hm, I wonder if I want RED or something similar to ensure packets get dropped fast enough instead of filling HTB queues and then dropping.

Quoted 7 lines
Quoted 4 lines

I suppose in an ideal world the best thing would be to receive the packets, hand them to user-space, but not mark them as received, ie, not ack them. That avoids introducing any loss from the user point of view but still slows the flow down. But that seems fiddly and would only work for TCP I guess.

That's an interesting idea, but yeah... i think it might be a bit hard to implement, and violate far too many RFCs...

My original thought was to just drop an ACK packet, which would clearly not violate any assumptions. But if there's data in the ACK packet that could have negative effects. I suspect this lies well into Bad Idea(tm) realm, I'm just thinking out loud.

Signature · 5 lines
greg

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to Greg Stark
Quoted 1 line · Greg Stark

Damion de Soto <[email protected]> writes:

<snip>

Quoted 11 lines
Quoted 4 lines

because you can't shape inbound traffic. Shaping works by delaying the transmission, and you can't delay packets that haven't arrived yet. Ingress policing just drops packets, and hopes the sender will slow down.

Well ultimately all shaping works by dropping packets. Merely delaying transmission isn't going to slow down anything in the long run, just increase the pipeline. You can delay and/or drop them after they've arrived just as easily. Though it would have to be before they're ack'd and delivered to the user. That's basically what IMQ does, I'm just saying perhaps that should just work instead of requiring a fake interface.

Ultimately, packets from a misbehaving flow can be dropped, but it does not always come to a drop. When you shape on egress, you force applications on the local network to throttle back, believing they're sending as fast as the receiver can receive. As you delay, TCP figures it out. Contrast that with ingress, where the packets you want to delay are already on their way.

Quoted 2 lines

Hm, I wonder if I want RED or something similar to ensure packets get dropped fast enough instead of filling HTB queues and then dropping.

If you're curious about RED, here's a possible example implementation for ingress policing:

http://digriz.org.uk/jdg-qos-script/

<snip>

Signature · 8 lines
Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#4 lore raw
In reply to Greg Stark
Quoted 9 lines · Jason Boxman
Quoted 7 lines · Greg Stark

Well ultimately all shaping works by dropping packets. Merely delaying transmission isn't going to slow down anything in the long run, just increase the pipeline. You can delay and/or drop them after they've arrived just as easily. Though it would have to be before they're ack'd and delivered to the user. That's basically what IMQ does, I'm just saying perhaps that should just work instead of requiring a fake interface.

Quoted 4 lines

Ultimately, packets from a misbehaving flow can be dropped, but it does not always come to a drop. When you shape on egress, you force applications on the local network to throttle back, believing they're sending as fast as the receiver can receive. As you delay, TCP figures it out.

I don't think so. It may look like that's what's happening, but at least for TCP I don't think it works that way. As long as packets aren't being dropped then TCP will just continue to grow the window, interpreting this delay as just a longer pipeline that needs filling. Applications will be slowed down temporarily because it takes time to do this, but they'll eventually be outputting data just as fast as an application without a shaper.

Only when packets get dropped or are delayed so long that the client retransmits does TCP scale back the transmit window. And only when that happens does the client see any reduced bandwidth. So if your shaper isn't dropping packets it's just evening out the flow of data, not actually affecting the net rate the clients can pump out data.

Quoted 2 lines

Contrast that with ingress, where the packets you want to delay are already on their way.

Well on egress the packets are "already on their way" as well, after all. They're just haven't gone as many hops. Even for locally generated traffic the egress qdisc is being run after the data packetized and ready to go.

Quoted 7 lines
Quoted 2 lines

Hm, I wonder if I want RED or something similar to ensure packets get dropped fast enough instead of filling HTB queues and then dropping.

If you're curious about RED, here's a possible example implementation for ingress policing:

http://digriz.org.uk/jdg-qos-script/

This is an interesting script. It looks like a successor to wondershaper. But I'm a bit too deep in my own re-implementation of wondershaper now to start over.

Signature · 5 lines
greg

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw

Re: [LARTC] Re: how flexible is ingress traffic policing to

In reply to Greg Stark
Quoted 7 lines · Greg Stark
Quoted 3 lines · Damion de Soto

because you can't shape inbound traffic. Shaping works by delaying the transmission, and you can't delay packets that haven't arrived yet. Ingress policing just drops packets, and hopes the sender will slow down.

Sorry to interrupt the flow, especially being a newbie, but won´t the sender just retransmit the dropped packets at the same rate? I am not so thorogh with TCP/IP, but is there something in the protocol that speeds or slows the transmission.

Please do explain in the TCP/IP for complete idiots terminology ;-))

With best regards.
Sanjay.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#6 lore raw
In reply to Greg Stark
Quoted 2 lines · Sanjay Arora

Sorry to interrupt the flow, especially being a newbie, but won´t the sender just retransmit the dropped packets at the same rate?

no.

Quoted 2 lines

I am not so thorogh with TCP/IP, but is there something in the protocol that speeds or slows the transmission.

yes.

Quoted 1 line

Please do explain in the TCP/IP for complete idiots terminology ;-))

You could do worse than this:

http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0

Signature · 5 lines
greg

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#7 lore raw
In reply to Greg Stark
Quoted 9 lines · Greg Stark
Quoted 2 lines · Sanjay Arora

Sorry to interrupt the flow, especially being a newbie, but won´t the sender just retransmit the dropped packets at the same rate?

no.

Quoted 2 lines

I am not so thorogh with TCP/IP, but is there something in the protocol that speeds or slows the transmission.

Quoted 7 lines

yes.

Quoted 1 line

Please do explain in the TCP/IP for complete idiots terminology ;-))

You could do worse than this:

http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0

I have to admit, I am thus far thoroughly enjoying:

Jacobson, V., Congestion Avoidance and Control. Proceedings of ACM SIGCOMM '88. August 1988, p. 314-329.

It's also available in Postscript format:

ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z

For those who don't know how TCP handles congestion, this is a rather excellent read.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw

Re: [LARTC] Re: how flexible is ingress traffic policing to bandwidth

In reply to Greg Stark
Quoted 18 lines · Greg Stark
Quoted 9 lines · Jason Boxman
Quoted 6 lines · Greg Stark

Well ultimately all shaping works by dropping packets. Merely delaying transmission isn't going to slow down anything in the long run, just increase the pipeline. You can delay and/or drop them after they've arrived just as easily. Though it would have to be before they're ack'd and delivered to the user. That's basically what IMQ does, I'm just saying perhaps that should just work instead of requiring a fake interface.

Quoted 4 lines

Ultimately, packets from a misbehaving flow can be dropped, but it does not always come to a drop. When you shape on egress, you force applications on the local network to throttle back, believing they're sending as fast as the receiver can receive. As you delay, TCP figures it out.

Quoted 6 lines

I don't think so. It may look like that's what's happening, but at least for TCP I don't think it works that way. As long as packets aren't being dropped then TCP will just continue to grow the window, interpreting this delay as just a longer pipeline that needs filling. Applications will be slowed down temporarily because it takes time to do this, but they'll eventually be outputting data just as fast as an application without a shaper.

Eventually in the egress case either you will run out of buffer and drop, or your queue is big enough for the advertised window (say 32k-64k) and tcp will only add a packet for everyone acked - you decide when the packets go so you have good control. If you drop, the sender reduces it's cwin and only slowly tries to increase until another drop.

Quoted 6 lines

Only when packets get dropped or are delayed so long that the client retransmits does TCP scale back the transmit window. And only when that happens does the client see any reduced bandwidth. So if your shaper isn't dropping packets it's just evening out the flow of data, not actually affecting the net rate the clients can pump out data.

Once you have dropped the rate you release packets does controll the flow and if it's slow the cwin will only grow slowly as it's clocked by acks (I think).

Quoted 10 lines
Quoted 4 lines

Contrast that with ingress, where the packets you want to delay are already on their way.

Well on egress the packets are "already on their way" as well, after all. They're just haven't gone as many hops. Even for locally generated traffic the egress qdisc is being run after the data packetized and ready to go.

I think what Jason was getting at is that on egress you have total control - it doesn't matter what TCP does.

Ingress shaping is harder as packets are already headed for you - in the case of P2P possibly quite alot - it's easier to shape when peering with "real" servers. Though in both cases, the fact that TCP slow (but sort of exponential) start overshoots bandwidth hurts ingress shaping, but not egress.

Andy.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Hm, I wonder if I want RED or something similar to ensure packets get dropped fast enough instead of filling HTB queues and then dropping.

If you're curious about RED, here's a possible example implementation for ingress policing:

Quoted 7 lines
Quoted 1 line

http://digriz.org.uk/jdg-qos-script/

This is an interesting script. It looks like a successor to wondershaper. But I'm a bit too deep in my own re-implementation of wondershaper now to start over.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/