How to optimize routing performance
Quoted 9 lines · Rik van Riel
Quoted 7 lines · On Thu, 15 Mar 2001, [ISO-8859-1] Mårten Wikström wrote:
I've performed a test on the routing capacity of a Linux 2.4.2 box versus a FreeBSD 4.2 box. I used two Pentium Pro 200Mhz computers with 64Mb memory, and two DEC 100Mbit ethernet cards. I used a Smartbits test-tool to measure the packet throughput and the packet size was set to 64 bytes. Linux dropped no packets up to about 27000 packets/s, but then it started to drop packets at higher rates. Worse yet, the output rate actually decreased, so at the input rate of 40000 packets/s
It is a known problem yes. And just as Rik says its has been adressed in 2.1.x by Alexey for first time.
Quoted 5 lines
Quoted 4 lines
almost no packets got through. The behaviour of FreeBSD was different, it showed a steadily increased output rate up to about 70000 packets/s before the output rate decreased. (Then the output rate was apprx. 40000 packets/s).
Quoted 7 lines
Quoted 3 lines
So, my question is: are these figures true, or is it possible to optimize the kernel somehow? The only changes I have made to the kernel config was to disable advanced routing.
There are some flow control options in the kernel which should help. From your description, it looks like they aren't enabled by default ...
CONFIG_NET_HW_FLOWCONTROL enables kernel code for it. But device drivers has to have support for it. But unfortunely very few drivers has support for it.
Also we done experiments were we move the device RX processing to SoftIRQ rather than IRQ. With this RX is in better balance with other kernel tasks and TX. Under very high load and under DoS attacks the system is now manageable. It's in practical use already.
Quoted 5 lines
At the NordU/USENIX conference in Stockholm (this february) I saw a nice presentation on the flow control code in the Linux networking code and how it improved networking performance. I'm pretty convinced that flow control _should_ be saving your system in this case.
Thanks Rik.
This is work/experiments by Jamal and me with support from Gurus. :-) Jamal did this presentation at OLS 2000. At NordU/USENIX I gave an updated presentation of it. The presentation is not yet available form the usenix webb I think.
It can ftp from robur.slu.se:
/pub/Linux/tmp/FF-NordUSENIX.pdf or .ps
In summary Linux is very decent router. Wire speed small packets @ 100 Mbps and capable of Gigabit routing (1440 pkts tested) we used.
Also if people are interested we have done profiling on a Linux production router with full BGP at pretty loaded site. This to give us costs for route lookup, skb malloc/free, interrupts etc.
http://Linux/net-development/experiments/010313
I'm on netdev but not the kernel list.
Cheers.
--ro
Quoted 3 lines · Robert Olsson
CONFIG_NET_HW_FLOWCONTROLenables kernel code for it. But device drivers has to have support for it. But unfortunely very few drivers has support for it.
Isn't it possible to put something like this in the layer just above the driver ?
It probably won't work as well as putting it directly in the driver, but it'll at least keep Linux from collapsing under really heavy loads ...
regards,
Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/
[Sorry for the length]
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Robert Olsson
CONFIG_NET_HW_FLOWCONTROLenables kernel code for it. But device drivers has to have support for it. But unfortunely very few drivers has support for it.Isn't it possible to put something like this in the layer just above the driver ?
There is a dropping point in netif_rx. The problem is that knowledge of congestion has to be pushed back to the devices that is causing this.
Alexey added netdev_dropping for drivers to check. And via netdev_wakeup() the drivers xon_metod can be called when the backlog below a certain threshold.
So from here the driver has do the work. Not investing any resources and interrupts in packets we still have to drop. This what happens at very high load a kind of livelock. For routers routing protocols will time out and we loose conetivity. But I would say its important for all apps.
In 2.4.0-test10 Jamal added sampling of the backlog queue so device drivers get the current congestion level. This opens new possiblities.
Quoted 3 lines
It probably won't work as well as putting it directly in the driver, but it'll at least keep Linux from collapsing under really heavy loads ...
And we have done experiments with controlling interrupts and running the RX at "lower" priority. The idea is take RX-interrupt and immediately postponing the RX process to tasklet. The tasklet opens for new RX-ints. when its done. This way dropping now occurs outside the box since and dropping becomes very undramatically.
As little example of this. I monitored a DoS attack on Linux router equipped with this RX-tasklet driver.
Admin up 6 day(s) 13 hour(s) 37 min 54 sec
Last input NOW
Last output NOW
5min RX bit/s 22.4 M 5min TX bit/s 1.3 M 5min RX pkts/s 44079 <==== 5min TX pkts/s 877 5min TX errors 0 5min RX errors 0
5min RX dropped 49913 <====
Fb: no 3127894088 low 154133938 mod 6 high 0 drp 0 <==== Congestion levels
Polling: ON starts/pkts/tasklet_count 96545881/2768574948/1850259980HW_flowcontrol xon's 0
A bit of explanation. Above is output from tulip driver. We are forwarding 44079 and we are dropping 49913 packets per second! This box has full BGP. The DoS attack was going on for about 30 minutes BGP survived and the box was manageable. Under a heavy attack it still performs well.
Cheers.
--ro
Quoted 5 lines
And we have done experiments with controlling interrupts and running the RX at "lower" priority. The idea is take RX-interrupt and immediately postponing the RX process to tasklet. The tasklet opens for new RX-ints. when its done. This way dropping now occurs outside the box since and dropping becomes very undramatically.
<snip>
Quoted 4 lines
A bit of explanation. Above is output from tulip driver. We are forwarding 44079 and we are dropping 49913 packets per second! This box has full BGP. The DoS attack was going on for about 30 minutes BGP survived and the box was manageable. Under a heavy attack it still performs well.
Nice. Any chance of similar functionality finding its' way outside the Tulip driver, eg. to 3c509 or via-rhine? I'd find those useful, since one or two of my Macs appear to be capable of generating pseudo-DoS levels of traffic under certain circumstances which totally lock a 486 (for the duration) and heavily load a P166 - even though said Macs "only" have 10baseT Ethernet.
OTOH, proper management of the circumstances under which this flooding occurs (it's an interaction bug which occurs when the Linux machine ends up with a zero-sized TCP receive window) would also be rather helpful.
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----
Quoted 6 lines · Jonathan Morton
Nice. Any chance of similar functionality finding its' way outside the Tulip driver, eg. to 3c509 or via-rhine? I'd find those useful, since one or two of my Macs appear to be capable of generating pseudo-DoS levels of traffic under certain circumstances which totally lock a 486 (for the duration) and heavily load a P166 - even though said Macs "only" have 10baseT Ethernet.
I'm not the one to tell. :-)
First its kind of experimental. Jamal has talked about putting together a proposal for enhancing RX-process for inclusion in the 2.5 kernels. There is meeting soon for this.
But why not experiment a bit?
Cheers.
--ro
Quoted 11 lines · Robert Olsson
Quoted 6 lines · Jonathan Morton
Nice. Any chance of similar functionality finding its' way outside the Tulip driver, eg. to 3c509 or via-rhine? I'd find those useful, since one or two of my Macs appear to be capable of generating pseudo-DoS levels of traffic under certain circumstances which totally lock a 486 (for the duration) and heavily load a P166 - even though said Macs "only" have 10baseT Ethernet.
Quoted 8 lines
I'm not the one to tell. :-)
First its kind of experimental. Jamal has talked about putting together a proposal for enhancing RX-process for inclusion in the 2.5 kernels. There is meeting soon for this.
But why not experiment a bit?
I think one of the immediate things usable to drivers is to check the netif_rx() return value and yield the CPU if the system is congested. This is hardware independent. For the Tulip, since it knows how to do mitigation, it infact reduces it's interupt rate. An even simpler thing is to use HW_FLOW_CONTROL where you shutdown rx_interupt based on system congestion (and get worken up later when things get better).
For 2.5 the plan is to work around any hardware dependencies.
cheers,
jamal