Skip to contentSkip to conversation

Auditing a broken and basic traffic shaping setup - PRIO

DISCUSSION OmegaPhil 2 participants 5 messages Dec 6, 2014 – Aug 23, 2015

lore
Share & tools
lore

OmegaPhil

author
#0 lore raw

Disclaimer: I don't do this very often so there is probably a retard error in here somewhere. I'm not expecting people to do my work for me, I'm just after a better understanding of the problem so I can get more control of the situation.

tldr: Custom priomap + iptables TOS isn't sorting packets correctly, Wireshark won't even filter on TOS...

----

I'm currently attempting to implement a 4 band prio shaper with fq_codel queues on a 100Mbit connection (Debian Testing server):

======================================================================

tc qdisc add dev eth0 root handle 1: htb default 1
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil 12800kibps
tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap 1 3 1 3 2 3 2 3 0 3 0 3 1 3 1 3
tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel
tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel
tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel
tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel

======================================================================

Packets are tagged for the various prio queues via iptables:

======================================================================

# ICMP
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos Minimize-Delay

# TCP control packets
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-tos Minimize-Delay

# TCP ACK packets with no or very little data payload (p2p traffic sets all packets to ACK packets otherwise, source of size: http://phix.me/dm/)
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-Delay

# Band 2 prioritisation
# Torrenting
$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j TOS --set-tos Maximize-Throughput

# Band 3 prioritisation
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1 -j TOS --set-tos Minimize-Cost
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2 -j TOS --set-tos Minimize-Cost

======================================================================

This is based on an otherwise-successful configuration on a local Ubuntu server that admittedly doesn't originate traffic itself, without a custom priomap.

The general idea is:

Band 0: High priority TCP packets, Minimize Delay, Band 1: Normal (nothing targetted here) Band 2: Torrenting, Maximize Throughput Band 3: Special programs, Minimize Monetary Cost

When I let the above run, virtually all packets get dumped into band 1, whereas by far the bulk of the traffic is torrenting - the shaping code is behaving like iptables isn't tagging the packets properly, however 'iptables -v -L -t mangle' is showing a lot of packets going through the TOS rules.

I next captured packets and opened up with Wireshark to see what was going on (it would be nice if I could just capture from the queues directly but I've found no evidence this is possible), however the following expressions fail to return anything:

ip.tos
ip.tos==8
ip.tos==0x8

etc with other values - I then moved to ip.dsfield.dscp, this failed in a different way - ip.dsfield.dscp==2 returned packets with 'Differentiated Services Field: 0x08', ip.dsfield.dscp==2 returned 0x10
- why?

At this point I stopped as I clearly didn't know what I was doing. Any pointers?

Thanks for any help.

#1 lore raw
Quoted 4 lines · OmegaPhil

Disclaimer: I don't do this very often so there is probably a retard error in here somewhere. I'm not expecting people to do my work for me, I'm just after a better understanding of the problem so I can get more control of the situation.

A couple quick notes:

1) strict priority queuing as you do here is generally a hugely bad idea, as the higher classes can completely starve the rest.

DRR with weights or QFQ with weights are better alternatives, or htb if you want to strictly rate limit each class. (and been working on something easier to setup than all that called cake... aint done yet, if you want patches to test, contact me off list).

Here for example, I ran a netperf-wrapper rrul test, and the EF class was completely starved.

http://pastebin.com/WaKRDATx

2) ToS as used here, was obsoleted in *1998* by the ietf and replaced with Diffserv and ECN.

http://en.wikipedia.org/wiki/Type_of_service

CS1 would have been the right thing for minimize-cost in particular.

Quoted 61 lines

tldr: Custom priomap + iptables TOS isn't sorting packets correctly, Wireshark won't even filter on TOS...

----

I'm currently attempting to implement a 4 band prio shaper with fq_codel queues on a 100Mbit connection (Debian Testing server):

===================================

tc qdisc add dev eth0 root handle 1: htb default 1
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil 12800kibps
tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap 1 3 1 3 2 3 2 3 0 3 0 3 1 3 1 3
tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel
tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel
tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel
tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel

===================================

Packets are tagged for the various prio queues via iptables:

===================================

# ICMP
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos Minimize-Delay

# TCP control packets
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-tos Minimize-Delay

# TCP ACK packets with no or very little data payload (p2p traffic sets all packets to ACK packets otherwise, source of size: http://phix.me/dm/)
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-Delay

# Band 2 prioritisation
# Torrenting
$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j TOS --set-tos Maximize-Throughput

# Band 3 prioritisation
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1 -j TOS --set-tos Minimize-Cost
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2 -j TOS --set-tos Minimize-Cost

===================================

Quoted 7 lines

This is based on an otherwise-successful configuration on a local Ubuntu server that admittedly doesn't originate traffic itself, without a custom priomap.

The general idea is:

Band 0: High priority TCP packets, Minimize Delay,

By peeing on the markings here you are messing with the intent of the sender.

and there is no need to fiddle with the lower level tcp flags here at all.

fq_codel automagically recognises sparse flows (like tcp syns) and does the right thing already. IF you are on an asymmetric network you might want to use fq_codel with a lower quantum so to give acks a little more priority that way.

Quoted 2 lines

Band 1: Normal (nothing targetted here)
Band 2: Torrenting, Maximize Throughput

No, this should be Background, CS1. IF you have control over your torrent clients, most support setting the CS1 bit in their configuration....

Quoted 1 line

Band 3: Special programs, Minimize Monetary Cost

totally obsolete bit. dont do that. see ecn.

Quoted 26 lines

When I let the above run, virtually all packets get dumped into band 1, whereas by far the bulk of the traffic is torrenting - the shaping code is behaving like iptables isn't tagging the packets properly, however 'iptables -v -L -t mangle' is showing a lot of packets going through the TOS rules.

I next captured packets and opened up with Wireshark to see what was going on (it would be nice if I could just capture from the queues directly but I've found no evidence this is possible), however the following expressions fail to return anything:

ip.tos
ip.tos=8
ip.tos=0x8

etc with other values - I then moved to ip.dsfield.dscp, this failed in a different way - ip.dsfield.dscp=2 returned packets with 'Differentiated Services Field: 0x08', ip.dsfield.dscp=2 returned 0x10
- why?

At this point I stopped as I clearly didn't know what I was doing. Any pointers?

Thanks for any help.

Signature · 3 lines
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
author
#2 lore raw
In reply to OmegaPhil
Quoted 16 lines · Dave Taht
Quoted 4 lines · OmegaPhil

Disclaimer: I don't do this very often so there is probably a retard error in here somewhere. I'm not expecting people to do my work for me, I'm just after a better understanding of the problem so I can get more control of the situation.

A couple quick notes:

1) strict priority queuing as you do here is generally a hugely bad idea, as the higher classes can completely starve the rest.

DRR with weights or QFQ with weights are better alternatives, or htb if you want to strictly rate limit each class. (and been working on something easier to setup than all that called cake... aint done yet, if you want patches to test, contact me off list).

Quoted 4 lines

Here for example, I ran a netperf-wrapper rrul test, and the EF class was completely starved.

http://pastebin.com/WaKRDATx

Thanks for the reply :) For reference all traffic on this server is mine, and therefore I can do what I want with it.

I do know about strict priority stuff - that is the aim, to make sure that important packets are not affected by those of less importance (e.g. so I don't care that there are 100 torrent UDP streams hammering a connection, my SSH connection always wins and the lag impact of any other traffic is minimal).

I will read into DRR and QFQ - I originally settled in PRIO because of the KISS principle, it sounded exactly what I wanted and should be easy to set up and maintain.

Will email off list - while the remote server wouldn't be a good idea for testing I do have the local Ubuntu server running the default priomap that I can test on - cheers.

Quoted 6 lines

2) ToS as used here, was obsoleted in *1998* by the ietf and replaced with Diffserv and ECN.

http://en.wikipedia.org/wiki/Type_of_service

CS1 would have been the right thing for minimize-cost in particular.

I know that ToS is old, I thought it was 1:1 with the new Diffserv stuff, but since you said that I guess not. I will read into it again (if it isn't then I'll need to look into how iptables is supposed to tag the packets without --set-tos).

Quoted 8 lines

By peeing on the markings here you are messing with the intent of the sender.

and there is no need to fiddle with the lower level tcp flags here at all.

fq_codel automagically recognises sparse flows (like tcp syns) and does the right thing already. IF you are on an asymmetric network you might want to use fq_codel with a lower quantum so to give acks a little more priority that way.

I am the sender, so the ToS stuff is my intent? Its just relevant for the local prioritisation, while deluge can flag ToS I can't properly audit it (the Wireshark issue), and the other programs don't have the functionality.

Right, I see with fq_codel - that was a recent advancement for me, originally the PRIO children were just normal queues. Good, I'll get rid of that complexity.

Quoted 10 lines
Quoted 2 lines

Band 1: Normal (nothing targetted here)
Band 2: Torrenting, Maximize Throughput

No, this should be Background, CS1. IF you have control over your torrent clients, most support setting the CS1 bit in their configuration....

Quoted 1 line

Band 3: Special programs, Minimize Monetary Cost

totally obsolete bit. dont do that. see ecn.

Will read into Diffserv and ECN.

Thanks for your feedback.

Signature · 2 lines
Libre software on Github: https://github.com/OmegaPhil
FSF member #9442
#3 lore raw
In reply to OmegaPhil
Quoted 23 lines · OmegaPhil
Quoted 16 lines · Dave Taht
Quoted 4 lines · OmegaPhil

Disclaimer: I don't do this very often so there is probably a retard error in here somewhere. I'm not expecting people to do my work for me, I'm just after a better understanding of the problem so I can get more control of the situation.

A couple quick notes:

1) strict priority queuing as you do here is generally a hugely bad idea, as the higher classes can completely starve the rest.

DRR with weights or QFQ with weights are better alternatives, or htb if you want to strictly rate limit each class. (and been working on something easier to setup than all that called cake... aint done yet, if you want patches to test, contact me off list).

Quoted 4 lines

Here for example, I ran a netperf-wrapper rrul test, and the EF class was completely starved.

http://pastebin.com/WaKRDATx

Quoted 8 lines

Thanks for the reply :) For reference all traffic on this server is mine, and therefore I can do what I want with it.

I do know about strict priority stuff - that is the aim, to make sure that important packets are not affected by those of less importance (e.g. so I don't care that there are 100 torrent UDP streams hammering a connection, my SSH connection always wins and the lag impact of any other traffic is minimal).

I am all in favor of making sure ssh works. Having other forms of traffic fail completely (instead of just getting slow), is not something I care for.

5% of traffic for other stuff and 95% for ssh is ok (well, I would prefer ssh tunnels did more of the right thing, and ssh does try to use Tos markings to do more of the right thing)

/me has switched to mosh entirely for interactive traffic.

Quoted 15 lines

I will read into DRR and QFQ - I originally settled in PRIO because of the KISS principle, it sounded exactly what I wanted and should be easy to set up and maintain.

Will email off list - while the remote server wouldn't be a good idea for testing I do have the local Ubuntu server running the default priomap that I can test on - cheers.

Quoted 5 lines

2) ToS as used here, was obsoleted in *1998* by the ietf and replaced with Diffserv and ECN.

http://en.wikipedia.org/wiki/Type_of_service

Quoted 7 lines
Quoted 1 line

CS1 would have been the right thing for minimize-cost in particular.

I know that ToS is old, I thought it was 1:1 with the new Diffserv stuff, but since you said that I guess not. I will read into it again (if it isn't then I'll need to look into how iptables is supposed to tag the packets without --set-tos).

I would have liked it if there were multiple classes for background traffic, not just CS1. The other markings (EF, AFxx) all fail the game theory test. But if the sender wants to be background-ish, it would be nice to be able to signal that end to end.

CS1 is the only marking (besides ect(0) and ect(1) (ecn) that comcast preserves e2e, for example.

Quoted 12 lines
Quoted 9 lines

By peeing on the markings here you are messing with the intent of the sender.

and there is no need to fiddle with the lower level tcp flags here at all.

fq_codel automagically recognises sparse flows (like tcp syns) and does the right thing already. IF you are on an asymmetric network you might want to use fq_codel with a lower quantum so to give acks a little more priority that way.

Quoted 4 lines

I am the sender, so the ToS stuff is my intent? Its just relevant for the local prioritisation, while deluge can flag ToS I can't properly audit it (the Wireshark issue), and the other programs don't have the functionality.

you might be able to do it by using fw markings rather than messing with tos.

if you are going to alter tos, try to preserve ecn marks.

Quoted 3 lines

Right, I see with fq_codel - that was a recent advancement for me, originally the PRIO children were just normal queues. Good, I'll get rid of that complexity.

Yea....

I realize that how SFQ works has confused the issues here, and that is why we used DRR instead in fq_codel.

I ranted a bit more here....

http://svn.dd-wrt.com/ticket/3366

Quoted 22 lines
Quoted 5 lines
Quoted 4 lines

Band 1: Normal (nothing targetted here)
Band 2: Torrenting, Maximize Throughput

Quoted 7 lines

No, this should be Background, CS1. IF you have control over your torrent clients, most support setting the CS1 bit in their configuration....

Quoted 1 line

Band 3: Special programs, Minimize Monetary Cost

totally obsolete bit. dont do that. see ecn.

Will read into Diffserv and ECN.

Thanks for your feedback.

--
Libre software on Github: https://github.com/OmegaPhil
FSF member #9442

Signature · 3 lines
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
author
#4 lore raw
In reply to OmegaPhil
Quoted 96 lines · OmegaPhil

Disclaimer: I don't do this very often so there is probably a retard error in here somewhere. I'm not expecting people to do my work for me, I'm just after a better understanding of the problem so I can get more control of the situation.

tldr: Custom priomap + iptables TOS isn't sorting packets correctly, Wireshark won't even filter on TOS...

----

I'm currently attempting to implement a 4 band prio shaper with fq_codel queues on a 100Mbit connection (Debian Testing server):

======================================================================

tc qdisc add dev eth0 root handle 1: htb default 1
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil 12800kibps
tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap 1 3 1 3 2 3 2 3 0 3 0 3 1 3 1 3
tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel
tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel
tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel
tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel

======================================================================

Packets are tagged for the various prio queues via iptables:

======================================================================

# ICMP
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos Minimize-Delay

# TCP control packets
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-tos Minimize-Delay

# TCP ACK packets with no or very little data payload (p2p traffic sets all packets to ACK packets otherwise, source of size: http://phix.me/dm/)
$IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-Delay

# Band 2 prioritisation
# Torrenting
$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j TOS --set-tos Maximize-Throughput

# Band 3 prioritisation
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1 -j TOS --set-tos Minimize-Cost
#$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2 -j TOS --set-tos Minimize-Cost

======================================================================

This is based on an otherwise-successful configuration on a local Ubuntu server that admittedly doesn't originate traffic itself, without a custom priomap.

The general idea is:

Band 0: High priority TCP packets, Minimize Delay, Band 1: Normal (nothing targetted here) Band 2: Torrenting, Maximize Throughput Band 3: Special programs, Minimize Monetary Cost

When I let the above run, virtually all packets get dumped into band 1, whereas by far the bulk of the traffic is torrenting - the shaping code is behaving like iptables isn't tagging the packets properly, however 'iptables -v -L -t mangle' is showing a lot of packets going through the TOS rules.

I next captured packets and opened up with Wireshark to see what was going on (it would be nice if I could just capture from the queues directly but I've found no evidence this is possible), however the following expressions fail to return anything:

ip.tos
ip.tos==8
ip.tos==0x8

etc with other values - I then moved to ip.dsfield.dscp, this failed in a different way - ip.dsfield.dscp==2 returned packets with 'Differentiated Services Field: 0x08', ip.dsfield.dscp==2 returned 0x10
- why?

Quoted 4 lines

At this point I stopped as I clearly didn't know what I was doing. Any pointers?

Thanks for any help.

This answering my own question for others that want a simple strict priority hierarchy with a customisable band count:

I've finally managed to get a custom number of bands PRIO queue on my server working now (no need to maintain a custom kernel, tc etc) - the key was to drop the broken TOS classification and just the iptables CLASSIFY target directly (no need to get involved in complicated tc filter stuff either):

Aim:

Band 0: SSH traffic
Band 1: 'Normal' traffic, anything unclassified including iroffer Band 2: Torrent traffic
Band 3: Darknet traffic

Setup 4 band PRIO qdisc:

=======================================================================

tc qdisc add dev eth0 parent root handle 1: prio bands 4 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

=======================================================================

Handle must be 1+, it doesn't like 0, you end up with a 8000+ number that naturally breaks any later references in iptables. Note that the band number in priomap counts from 0, so the bands are 0, 1, 2 and 3 - the actual qdisc IDs start from 1 (...). Dumping in band 1 (band 2 qdisc ID) across the board acts as the default classification.

Setup usual fq_codel qdiscs:

=======================================================================

tc qdisc add dev eth0 parent 1:1 handle 101: fq_codel tc qdisc add dev eth0 parent 1:2 handle 102: fq_codel tc qdisc add dev eth0 parent 1:3 handle 103: fq_codel tc qdisc add dev eth0 parent 1:4 handle 104: fq_codel

=======================================================================

The child PRIO qdiscs associated with your bands have been created for you already and their ID starts from 1.

Now get iptables to do the classification:

SSH (port 22222 here):

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -p tcp -s "$PUBLIC_IP" --sport 22222 -j CLASSIFY --set-class 1:1

=======================================================================

Torrenting:

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j CLASSIFY --set-class 1:3

=======================================================================

Darknets:

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner debian-tor -j CLASSIFY --set-class 1:4
iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner i2p -j CLASSIFY --set-class 1:4

=======================================================================

Everything else ends up in 1:2 as mentioned previously due to the initial priomap.

For a nice realtime view of how packets are flowing through the qdiscs to prove things are actually doing what you told them to do, use bmon
(https://github.com/tgraf/bmon) - literally the 'bmon' command, then move the left white cursor thing up and down to select the interface or qdisc/class you are interested in.