Skip to contentSkip to conversation

ANNOUNCE: debloat-testing kernel git tree

DISCUSSION John W. Linville 6 participants 14 messages Feb 25, 2011 – Mar 13, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

git://git.infradead.org/debloat-testing.git

The purpose of this tree is to provide a reasonably stable base for the development and testing of new algorithms, miscellaneous fixes, and maybe a few hacks intended to advance the cause of eliminating or at least mitigating bufferbloat in the Linux world.

Introduction

Bufferbloat is a term coined by Jim Gettys to describe the increasing prevalence of large and (particularly) unmanaged network buffers along the network links that comprise the Internet [2]. If you are not aware of the problems with network latency under load that the Internet is already encountering, we encourage you to visit Jim Gettys' blog [3]. There Jim has begun to fit together enough puzzle pieces to at least frame the issue.

Jim has also made available slides and an audio recording (edited for time) from a presentation on this topic:

	http://mirrors.bufferbloat.net/Talks/BellLabs01192011/

Kernel Bits

The debloat-testing tree is intended to track full and -rc releases from linux-2.6, with interesting patches cherry-picked from net-next and various experimental bits added on top. The current stable of such patches includes the following:

Eric Dumazet (based on original work by Juliusz Chroboczek):
net_sched: SFB flow scheduler

stephen hemminger:
sched: CHOKe flow scheduler

John Fastabend:
net: implement mechanism for HW based QOS
net_sched: implement a root container qdisc sch_mqprio

John W. Linville:
mac80211: implement eBDP algorithm to fight bufferbloat

Nathaniel J. Smith:
iwlwifi: Simplify tx queue management
iwlwifi: Convert the tx queue high_mark to an atomic_t
iwlwifi: Invert the sense of the queue high_mark
iwlwifi: auto-tune tx queue size to minimize latency
iwlwifi: make current tx queue sizes visible in debugfs

Dave Taht:
Bufferbloat reduction for the e1000 driver that started it all Reduce bufferbloated default for e1000e, increase dynamic range Smash bufferbloat in the ath9k driver

Userland Bits

Patches for the userspace tc utility incorporating support for both the CHOKe AQM and the Stochastic Fair Blue scheduler (SFB) are available:

https://github.com/dtaht/iproute2bufferbloat

Contributions

Please send any experimental or research-oriented patches related to bufferbloat to the [email protected] list. Reminders of more mainstream patches that may be relevant and/or interesting for cherry-picking into debloat-testing are welcome there as well.

Obviously, patches that are ready for normal merge consideration should continue to be sent to netdev, linux-wireless, linux-kernel, or whatever other existing list is appropriate for them.

Thanks

Finally, we want to offer a huge thanks to the 130+ new members of the bloat mailing list [4] for leaping into the fray, and to David Woodhouse for hosting the debloat-testing tree at infradead.

Please help us beat the bloat. Good luck, and happy debloating!

Notes

[1] http://bufferbloat.net
[2] http://gettys.wordpress.com/what-is-bufferbloat-anyway/
[3] http://en.wordpress.com/tag/bufferbloat/
[4] https://lists.bufferbloat.net
--
John W. Linville Someday the world will need a hero, and you
[email protected] might be all we have. Be ready.

#1 lore raw
Quoted 87 lines · John W. Linville

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

       git://git.infradead.org/debloat-testing.git

The purpose of this tree is to provide a reasonably stable base for the development and testing of new algorithms, miscellaneous fixes, and maybe a few hacks intended to advance the cause of eliminating or at least mitigating bufferbloat in the Linux world.

Introduction

Bufferbloat is a term coined by Jim Gettys to describe the increasing prevalence of large and (particularly) unmanaged network buffers along the network links that comprise the Internet [2].  If you are not aware of the problems with network latency under load that the Internet is already encountering, we encourage you to visit Jim Gettys' blog [3]. There Jim has begun to fit together enough puzzle pieces to at least frame the issue.

Jim has also made available slides and an audio recording (edited for time) from a presentation on this topic:

       http://mirrors.bufferbloat.net/Talks/BellLabs01192011/

Kernel Bits

The debloat-testing tree is intended to track full and -rc releases from linux-2.6, with interesting patches cherry-picked from net-next and various experimental bits added on top.  The current stable of such patches includes the following:

Eric Dumazet (based on original work by Juliusz Chroboczek):
     net_sched: SFB flow scheduler

stephen hemminger:
     sched: CHOKe flow scheduler

John Fastabend:
     net: implement mechanism for HW based QOS
     net_sched: implement a root container qdisc sch_mqprio

John W. Linville:
     mac80211: implement eBDP algorithm to fight bufferbloat

Nathaniel J. Smith:
     iwlwifi: Simplify tx queue management
     iwlwifi: Convert the tx queue high_mark to an atomic_t
     iwlwifi: Invert the sense of the queue high_mark
     iwlwifi: auto-tune tx queue size to minimize latency
     iwlwifi: make current tx queue sizes visible in debugfs

Dave Taht:
     Bufferbloat reduction for the e1000 driver that started it all Reduce bufferbloated default for e1000e, increase dynamic range Smash bufferbloat in the ath9k driver

Userland Bits

Patches for the userspace tc utility incorporating support for both the CHOKe AQM and the Stochastic Fair Blue scheduler (SFB) are available:

       https://github.com/dtaht/iproute2bufferbloat

Contributions

Please send any experimental or research-oriented patches related to
bufferbloat to the bloat-devel-JXvr2/[email protected] list.  Reminders of more mainstream patches that may be relevant and/or interesting
for cherry-picking into debloat-testing are welcome there as well.

Obviously, patches that are ready for normal merge consideration should continue to be sent to netdev, linux-wireless, linux-kernel, or whatever other existing list is appropriate for them.

Thanks

Finally, we want to offer a huge thanks to the 130+ new members of the bloat mailing list [4] for leaping into the fray, and to David Woodhouse for hosting the debloat-testing tree at infradead.

Please help us beat the bloat.  Good luck, and happy debloating!

Notes

Quoted 8 lines

[1] http://bufferbloat.net
[2] http://gettys.wordpress.com/what-is-bufferbloat-anyway/
[3] http://en.wordpress.com/tag/bufferbloat/
[4] https://lists.bufferbloat.net
--
John W. Linville                Someday the world will need a hero, and you [email protected]                  might be all we have.  Be ready.

Hi,

it should be "localversion-debloat" in the commit-subject in [1] (not "localversion-wireless") :-).
"-db" as suffix is IMHO not very meaningful... Why not add simply a suffix called "-debloat"?
(Anyway, I will revert this patch because I don't want to have any suffix added automatically.)

I have several other questions, but I start compiling first and test this debloat kernel.

Regards,
- Sedat -

[1] "Add localversion-wireless to identify builds from this tree."
http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 8 lines · Sedat Dilek
Quoted 6 lines · John W. Linville

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

git://git.infradead.org/debloat-testing.git

----snip----

Quoted 10 lines

Hi,

it should be "localversion-debloat" in the commit-subject in [1] (not "localversion-wireless") :-). "-db" as suffix is IMHO not very meaningful... Why not add simply a suffix called "-debloat"? (Anyway, I will revert this patch because I don't want to have any suffix added automatically.)

I have several other questions, but I start compiling first and test this debloat kernel.

Excellent. At moment I would recommend building "low latency preempt desktop" kernels with a high HZ value (400 or 1000), enabling highres timers, and compiling in SFB as a module. (I'd like the default for SFB to be "m" rather than "n", too)

Quoted 10 lines

Regards,
- Sedat -

[1] "Add localversion-wireless to identify builds from this tree."
http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
_______________________________________________
Bloat-devel mailing list Bloat-devel-JXvr2/[email protected]
https://lists.bufferbloat.net/listinfo/bloat-devel

Signature · 6 lines
Dave Taht
http://nex-6.taht.net
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
Quoted 41 lines · Dave Täht
Quoted 8 lines · Sedat Dilek
Quoted 6 lines · John W. Linville

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

  git://git.infradead.org/debloat-testing.git

----snip----

Quoted 10 lines

Hi,

it should be "localversion-debloat" in the commit-subject in [1] (not "localversion-wireless") :-).  "-db" as suffix is IMHO not very meaningful... Why not add simply a suffix called "-debloat"?  (Anyway, I will revert this patch because I don't want to have any suffix added automatically.)

I have several other questions, but I start compiling first and test this debloat kernel.

Excellent. At moment I would recommend building "low latency preempt desktop" kernels with a high HZ value (400 or 1000), enabling highres timers, and compiling in SFB as a module. (I'd like the default for SFB to be "m" rather than "n", too)

Quoted 10 lines

Regards,
- Sedat -

[1] "Add localversion-wireless to identify builds from this tree."
http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
_______________________________________________
Bloat-devel mailing list [email protected]
https://lists.bufferbloat.net/listinfo/bloat-devel

Quoted 4 lines

--
Dave Taht
http://nex-6.taht.net

These "debloat guys" are fast :-).
I was just preparing my build-system (which I normally use to debianize linux-next kernels).
Any other recommendation for kernel-config options?
For example:
linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it). Which commits are in debloat-testing GIT but not in linux-next tree? Are you planning debloat feature for 2.6.39?

- Sedat -

#4 lore raw
Quoted 21 lines · Sedat Dilek
Quoted 10 lines · Dave Täht
Quoted 7 lines · Sedat Dilek
Quoted 5 lines · John W. Linville

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

Quoted 9 lines
Quoted 1 line
Quoted 1 line

  git://git.infradead.org/debloat-testing.git

----snip----

Excellent. At moment I would recommend building "low latency preempt desktop" kernels with a high HZ value (400 or 1000), enabling highres timers, and compiling in SFB as a module. (I'd like the default for SFB to be "m" rather than "n", too)

Quoted 4 lines

These "debloat guys" are fast :-). I was just preparing my build-system (which I normally use to debianize linux-next kernels). Any other recommendation for kernel-config options? For example:
linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it).

Enable CHOKe.

The HZ value change is due to my worry that we've smashed latency so much in the driver/mac layer that it's interacting with the higher layers somewhat badly... So we need to add more hooks to the servo loops involved in order to have a normal HZ.

Quoted 1 line

Which commits are in debloat-testing GIT but not in linux-next tree?

The current list was in the release announcement. More on the way (mostly embedded drivers at this point) git pull early and often!

Quoted 1 line

Are you planning debloat feature for 2.6.39?

Depends on how many testers we get and what the results are.

I feel the eBDP stuff will not be ready during this release cycle. SFB and CHOKe are in net-next, so, probably. Various driver patches - particularly those that increase the available dynamic range via ethtool, (e.g lowering the bottommost TX queue limit to, like, 4, especially for home gateways) may make it out if people look harder into the issue.

Quoted 2 lines

- Sedat -

Signature · 2 lines
Dave Taht
http://nex-6.taht.net
#5 lore raw
Quoted 52 lines · Dave Täht
Quoted 21 lines · Sedat Dilek
Quoted 10 lines · Dave Täht
Quoted 7 lines · Sedat Dilek
Quoted 5 lines · John W. Linville

Announcement

The bufferbloat project [1] is pleased to announce the availability of the debloat-testing Linux kernel git tree:

Quoted 9 lines
Quoted 1 line
Quoted 1 line

  git://git.infradead.org/debloat-testing.git

----snip----

Excellent. At moment I would recommend building "low latency preempt desktop" kernels with a high HZ value (400 or 1000), enabling highres timers, and compiling in SFB as a module. (I'd like the default for SFB to be "m" rather than "n", too)

Quoted 4 lines

These "debloat guys" are fast :-).  I was just preparing my build-system (which I normally use to debianize linux-next kernels). Any other recommendation for kernel-config options?  For example:
linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it).

Enable CHOKe.

The HZ value change is due to my worry that we've smashed latency so much in the driver/mac layer that it's interacting with the higher layers somewhat badly... So we need to add more hooks to the servo loops involved in order to have a normal HZ.

Quoted 1 line

Which commits are in debloat-testing GIT but not in linux-next tree?

The current list was in the release announcement. More on the way (mostly embedded drivers at this point) git pull early and often!

Quoted 1 line

Are you planning debloat feature for 2.6.39?

Depends on how many testers we get and what the results are.

I feel the eBDP stuff will not be ready during this release cycle. SFB and CHOKe are in net-next, so, probably. Various driver patches - particularly those that increase the available dynamic range via ethtool, (e.g lowering the bottommost TX queue limit to, like, 4, especially for home gateways) may make it out if people look harder into the issue.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

- Sedat -

--
Dave Taht
http://nex-6.taht.net

OK, thanks for the explanations.

Concerning "more drivers":
What would I have to do to modify ath5k?
I looked into the ath9k patch in debloat-testing GIT and it was to mod some (TX/BUF) values only.
Not sure if ath9k is/was "well" prepared or only a good choice by the testers/committers as they own such a device.

- Sedat -
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 18 lines · Sedat Dilek
Quoted 8 lines · Dave Täht
Quoted 5 lines · Sedat Dilek
Quoted 2 lines · Dave Täht

Sedat Dilek <sedat.dilek-gM/[email protected]> writes:

Are you planning debloat feature for 2.6.39?

Quoted 8 lines

Depends on how many testers we get and what the results are.

I feel the eBDP stuff will not be ready during this release cycle. SFB and CHOKe are in net-next, so, probably. Various driver patches - particularly those that increase the available dynamic range via ethtool, (e.g lowering the bottommost TX queue limit to, like, 4, especially for home gateways) may make it out if people look harder into the issue.

Quoted 6 lines

OK, thanks for the explanations.

Concerning "more drivers":
What would I have to do to modify ath5k?
I looked into the ath9k patch in debloat-testing GIT and it was to mod some (TX/BUF) values only.

Yes, reducing your TX buffer size greatly is the first, best, and easiest patch.

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

http://www.cs.clemson.edu/~jmarty/papers/PID1154937.pdf

I found their data convincing, and there are dozens of other papers that we are sorting out on the bufferbloat.net web site.

(PLEASE Note the key word there is un-managed)

0 would be the best value. :/

In the case of wireless, you also have retries to take into account. I'd argue in those cases, that what I say above is that the number should be FAR less than 32.

Now, whether there is a good compromise between throughput and latency in that range in a DMA TX queue + TXQUEUE, remains to be seen.

Quoted 2 lines

Not sure if ath9k is/was "well" prepared or only a good choice by the testers/committers as they own such a device.

My test network is mostly ath9k - the nanostation M5s and the WNDR5700 router described here:

http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700

There are people looking into the ath6kl, but you're the first to step up with the ath5k. :) Maybe the folk over at #ath6kl on irc can help.

The ath9k patch improves latency under load enormously - I can run voip over it AND do big transfers and stream audio via samba... Which I couldn't before - and DNS, ND, NTP, babel, etc behave much better, but the currently hard coded nature of the TX queue limit does put an upper limit on packet aggregation that the eBDP folk are trying to resolve more generically.

In practice, at "normal" 180Mbit rates, with the new queue depth of 3, I get most of the benefits of packet aggregation without the lag.

I do see higher packet loss than I would like, at present.

Quoted 2 lines

- Sedat -

Signature · 6 lines
Dave Taht
http://nex-6.taht.net
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw
Quoted 3 lines

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

Would it be a good thing to start describing these queues not so much in terms of packets but in terms of delay (or bandwidth X delay)?

Constants tend to live longer than they should, no matter how noble in birth.

rick jones

#8 lore raw
Quoted 9 lines · Rick Jones
Quoted 3 lines

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

Would it be a good thing to start describing these queues not so much in terms of packets but in terms of delay (or bandwidth X delay)?

Constants tend to live longer than they should, no matter how noble in birth.

Yes, it probably would -- that would be more meaningful between different technologies and/or for technologies that might have variable latencies due to shared media or whatnot.

Now, the question becomes what patches to propose to make that happen... :-)

John

Signature · 2 lines
John W. Linville		Someday the world will need a hero, and you
[email protected]			might be all we have.  Be ready.
#9 lore raw
In reply to Rick Jones
Quoted 7 lines · Rick Jones
Quoted 3 lines

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

Would it be a good thing to start describing these queues not so much in terms of packets but in terms of delay (or bandwidth X delay)?

The unit of bandwidth is something like Mbps, that of delay can be second, so bandwidth X delay --> Mb, which is the unit of packet size. So both packets and delay should have the same effect for sizing buffers.

Tianji

Quoted 10 lines

Constants tend to live longer than they should, no matter how noble in birth.

rick jones

_______________________________________________
Bloat-devel mailing list Bloat-devel-JXvr2/[email protected]
https://lists.bufferbloat.net/listinfo/bloat-devel

--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw
Quoted 6 lines · Tianji Li
Quoted 4 lines · Rick Jones
Quoted 3 lines

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

Quoted 7 lines
Quoted 3 lines

Would it be a good thing to start describing these queues not so much in terms of packets but in terms of delay (or bandwidth X delay)?

The unit of bandwidth is something like Mbps, that of delay can be second, so bandwidth X delay --> Mb, which is the unit of packet size. So both packets and delay should have the same effect for sizing buffers.

Yet, not all packets are not created equal - in size. So, 42, 150 byte packets queued is not the same as 42, 1500 byte packets. However, 10, 1500 byte packets is the same as 100, 150 byte packets.

Had you said something like MB and delay should have the same effect I would have agreed with you :)

rick jones

#11 lore raw
In reply to Tianji Li
Quoted 7 lines · Tianji Li
Quoted 4 lines · Rick Jones
Quoted 3 lines

For wireless routers and cable home gateways especially, this research shows that the total un-managed buffers in your system should be less than 32.

Quoted 8 lines
Quoted 3 lines

Would it be a good thing to start describing these queues not so much in terms of packets but in terms of delay (or bandwidth X delay)?

The unit of bandwidth is something like Mbps, that of delay can be second, so bandwidth X delay --> Mb, which is the unit of packet size. So both packets and delay should have the same effect for sizing buffers.

Bandwidth varies greatly with time. On cellphone, you can go from 5KB/sec GPRS to 300KB/sec HSDPA and back...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#12 lore raw
Quoted 2 lines · Pavel Machek

Bandwidth varies greatly with time. On cellphone, you can go from 5KB/sec GPRS to 300KB/sec HSDPA and back...

How quickly/often and does that trigger an event in the host that would allow adjustment of the queue?

rick jones

--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw

Hi!

Quoted 5 lines
Quoted 2 lines

Bandwidth varies greatly with time. On cellphone, you can go from 5KB/sec GPRS to 300KB/sec HSDPA and back...

How quickly/often and does that trigger an event in the host that would allow adjustment of the queue?

I'd expect it to get fast when you leave the building, and back to slow when you re-enter. It all depends how good 3G coverage is, but seeing it change more than once a minute is not that uncommon.

Event... not sure. Linux can ask for current signal strength, and I assume connection type works in similar way...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html