Skip to contentSkip to conversation

Good qdisc for routers?

DISCUSSION Dennis Jacobfeuerborn 4 participants 6 messages Nov 5, 2013 – Jan 2, 2014

lore
Share & tools
lore
#0 lore raw

Hi,
I'm setting up a linux box as a router/firewall and I'm wondering if there are any best practices with regards to network settings that should be tweaked in this situation.
In particular I'm wondering if the default pfifo_fast is susceptible to a single flow flooding the buffer and causing other flows to lose packets? Is there an alternative that would work better in terms of fairness but doesn't create too much overhead or is overly complicated to configure?
The system has 1Gbit interfaces.

Regards,
Dennis

#1 lore raw
Quoted 12 lines · Remy Mudingay

Hi Dennis,

I highly recommend trying out codel or fq_codel. I've been using fq_codel for well over a year now. I have just over a thousand users going through the router. I use it mainly because I wanted in-queue packet delays statistics but I didn't switch from pfifo_fast because of any perceived problems.
Anyway, You won't need to patch your current Linux install as long as it's version 3.3 or higher.

I use the following two lines for each interface.

Quoted 7 lines

tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor 10242 perturb 300 baseclass 1:1

You can get more in depth information from here
http://www.bufferbloat.net/projects/codel/wiki.

Codel seems more geared towards low latency. The following statement from the wiki doesn't make it sound like this is the right choice here:

"People have tried to run CoDel in very big routers, with hundreds of simultaneous flows, a situation not simulated in advance. There, it isn't controlling the queue the way it should: whether this is a problem with the algorithm, or the implementation, is not yet understood.

It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a data center; it does not react in a timely enough fashion. Whether the modifications of the ideas in CoDel will solve this problem is not yet known. Again, this is an area which CoDel was not designed to solve or it simulated in before publication."

Regards,
Dennis

#2 lore raw
In reply to Dennis Jacobfeuerborn

Hi Dennis,

I read the same article but since it mentioned datacenter I understood this to not apply to Internet bound traffic. However even on a gigabit WAN link codel (in my case fq_codel) could improve the responsiveness of competing flows. A few open source distributions (openwrt,dd-wrt and gargoyle).

By the way, why do you want to move away from the default qdisc?

Cheers,

Remy.

Sent from my iPhone

Quoted 31 lines · Dennis Jacobfeuerborn
Quoted 14 lines · Remy Mudingay

Hi Dennis,

I highly recommend trying out codel or fq_codel. I've been using fq_codel for well over a year now. I have just over a thousand users going through the router. I use it mainly because I wanted in-queue packet delays statistics but I didn't switch from pfifo_fast because of any perceived problems.
Anyway, You won't need to patch your current Linux install as long as it's version 3.3 or higher.

I use the following two lines for each interface.

Quoted 7 lines

tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor 10242 perturb 300 baseclass 1:1

You can get more in depth information from here
http://www.bufferbloat.net/projects/codel/wiki.

Codel seems more geared towards low latency. The following statement from the wiki doesn't make it sound like this is the right choice here:

"People have tried to run CoDel in very big routers, with hundreds of simultaneous flows, a situation not simulated in advance. There, it isn't controlling the queue the way it should: whether this is a problem with the algorithm, or the implementation, is not yet understood.

It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a data center; it does not react in a timely enough fashion. Whether the modifications of the ideas in CoDel will solve this problem is not yet known. Again, this is an area which CoDel was not designed to solve or it simulated in before publication."

Regards,
Dennis

#3 lore raw
In reply to Dennis Jacobfeuerborn

Hi Remy,
from what I understand the default pfifo_fast qdisc isn't particulary fair when a single flow or a few flows flood the interface/buffer with packets so I am wondering if qdiscs like QFQ, SFQ or CHOKe could be improvements compared to the default "dumb" pfifo one.

Regards,
Dennis

Quoted 43 lines · Remy Mudingay

Hi Dennis,

I read the same article but since it mentioned datacenter I understood this to not apply to Internet bound traffic. However even on a gigabit WAN link codel (in my case fq_codel) could improve the responsiveness of competing flows. A few open source distributions (openwrt,dd-wrt and gargoyle).

By the way, why do you want to move away from the default qdisc?

Cheers,

Remy.

Sent from my iPhone

Quoted 31 lines · Dennis Jacobfeuerborn
Quoted 14 lines · Remy Mudingay

Hi Dennis,

I highly recommend trying out codel or fq_codel. I've been using fq_codel for well over a year now. I have just over a thousand users going through the router. I use it mainly because I wanted in-queue packet delays statistics but I didn't switch from pfifo_fast because of any perceived problems.
Anyway, You won't need to patch your current Linux install as long as it's version 3.3 or higher.

I use the following two lines for each interface.

Quoted 7 lines

tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor 10242 perturb 300 baseclass 1:1

You can get more in depth information from here
http://www.bufferbloat.net/projects/codel/wiki.

Codel seems more geared towards low latency. The following statement from the wiki doesn't make it sound like this is the right choice here:

"People have tried to run CoDel in very big routers, with hundreds of simultaneous flows, a situation not simulated in advance. There, it isn't controlling the queue the way it should: whether this is a problem with the algorithm, or the implementation, is not yet understood.

It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a data center; it does not react in a timely enough fashion. Whether the modifications of the ideas in CoDel will solve this problem is not yet known. Again, this is an area which CoDel was not designed to solve or it simulated in before publication."

Regards,
Dennis

#4 lore raw
In reply to Dennis Jacobfeuerborn
Quoted 5 lines · Dennis Jacobfeuerborn

Hi Remy,
from what I understand the default pfifo_fast qdisc isn't particulary fair when a single flow or a few flows flood the interface/buffer with packets so I am wondering if qdiscs like QFQ, SFQ or CHOKe could be improvements compared to the default "dumb" pfifo one.

You are right the pfifo_fast is not fair at all.

Now, unless you already know that you will hit some real use cases of such floods causing damages to the flows I would recommend you to stick to the default pfifo_fast which work great most of the time. Each qdisc comes with advantages and inconveniences.

Of course, if you experiment some network issues the correct thing to do might be to tune some queueing disciplines. Though in practice, this is not even sure that this would be the root cause as network problems can come from a lot of other things (offloading, hardware, drivers, etc).

Most of the time, I think people tune qdiscs because of real issues or a strong policy and I guess this is the good thing to do. Doing it to prevent /potential/ issues looks like avoidable pain and administration overhead, to me.

Have fun,

Signature · 1 line
Nicolas Sebrecht
#5 lore raw
In reply to Dennis Jacobfeuerborn
Quoted 10 lines · Remy Mudingay
Quoted 9 lines

Hi Dennis,

I highly recommend trying out codel or fq_codel. I've been using fq_codel for well over a year now. I have just over a thousand users going through the router. I use it mainly because I wanted in-queue packet delays statistics but I didn't switch from pfifo_fast because of any perceived problems.
Anyway, You won't need to patch your current Linux install as long as it's version 3.3 or higher.

Quoted 6 lines
Quoted 6 lines

I use the following two lines for each interface.

tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor 10242 perturb 300 baseclass 1:1

Unless you *really* care about preserving the pre-natted ip/port information in the hash, the second filter is unneeded. I am not huge on perturbation, in this case it would mean that all the per flow codel state is lost ever 300 seconds. fq_codel basically has an identical hash by default, no perturbation, but it randomizes its default hash on init time so it's unpredictable that way.

I would certainly like it if there was a tc command to "randomize" a custom hash like your second line there for fq_codel.

I generally have no problem leaving ecn on presently. It is rarely used, and even more rarely abused. When used, it helps.

Quoted 4 lines
Quoted 2 lines

You can get more in depth information from here
http://www.bufferbloat.net/projects/codel/wiki.

Codel seems more geared towards low latency. The following statement from the >wiki doesn't make it sound like this is the right choice here:

We are pretty happy with fq_codel in the range 4Mbit to 10Gbit. Why not go for low latency if you can get 100% utilization?

Quoted 1 line

"People have tried to run CoDel in very big routers, with hundreds of >simultaneous flows, a situation not simulated in advance. There, it isn't >controlling the queue the way it should: whether this is a problem with the >algorithm, or the implementation, is not yet understood."

Several ideas are out of context here. I will ask kathie to clarify.

1) The Codel ns2 model has some problems at hundreds of simultaneous full rate flows.
2) fq_codel, which splits up flows into up to 64k queues each managed by codel, does not.
3) "Use fq_codel, not codel. It's an across the board win." - Van Jacobson

http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos

Quoted 1 line

It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a >data center; it does not react in a timely enough fashion. Whether the >modifications of the ideas in CoDel will solve this problem is not yet known. >Again, this is an area which CoDel was not designed to solve or it simulated in >before publication."

And here what kathie meant was that codel was designed for edge networks to "do no harm" to be on by default everywhere, at RTTs ranging from a few ms to hundreds. Within-the-data center speeds are well below that, and the reaction times in *codel* are too slow.

Since the paper, ecn support was added to codel and fq_codel and works well, and by tweaking two variables (target and interval) down to values that make
more sense inside the data center, the aqm stuff works increasingly better, and the "fq" portion of fq_codel helps in all cases at all rates, and improves codel's effective reaction time as well. Yes there is more work to be done inside the data center. We've spent more time trying to improve rates below 4mbit than above 10gig (see nfq_codel)

However:

At this point I'm pretty happy with the new "fq" scheduler for linux hosts in linux 3.12, and fq_codel on linux based hosts and routers, at nearly any speed. Try it, exercise it, you'll like it.

Even at gig+ speeds you get back a more than a few ms of latency.

http://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

http://www.bufferbloat.net/projects/codel/wiki/RRUL_Rogues_Gallery

Signature · 3 lines
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html