bridge latency
Hi list,
we did an investigation of a firewalling Linux bridge in terms of latency and throughput using DAG 3.5E interfaces for precise timestamping of ip packets [1]. The overall results are very good but one fact we can't explain. The following URL points to an image of such an measuring where we push 60MBits/s TCP traffic through the bridge in one direction:
http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_60e_plain_out.png
If you look at the image you'll see the max line showing regular (one per minute) spikes. I've absolutly no idea where these spikes may come from. So I want to ask you, if you could imagine a reason for these spikes? These are only few packets but for us it's important to find out why they get sometimes delayed this much longer.
Some background about the bridging box:
Hardware:
Pentium4 1.7GHz, 1.024MB RAM, 2 x 3Com 3c905 [Boomerang] being br0, 3Com 3c905B as management interface.
Software:
vanilla Linux Kernel 2.6.3 with patched in CONNMARK extension, STP was disabled
Other:
The bridge was running in plain mode wich means no other modules like iptables, ebtables or packet scheduler were loaded. It was running in runlevel 3, multiuser networking without x-system.
For comparison only this link shows what happens if we push as much TCP traffic in one direction (~97.5 MBits/s) through the bridge, in second image having iptables loaded:
http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_120c_delay-0.png
http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_ipp2p_120c_delay-0.png
Thanks in advance for any comments or suggestions!
Cheers,
Eicke
[1] Endace Measurement Systems: http://www.endace.com/
Quoted 14 lines · Eicke Friedrich
Hi list,
we did an investigation of a firewalling Linux bridge in terms of latency and throughput using DAG 3.5E interfaces for precise timestamping of ip packets [1]. The overall results are very good but one fact we can't explain. The following URL points to an image of such an measuring where we push 60MBits/s TCP traffic through the bridge in one direction:
http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_60e_plain_out.pngIf you look at the image you'll see the max line showing regular (one per minute) spikes. I've absolutly no idea where these spikes may come from. So I want to ask you, if you could imagine a reason for these spikes? These are only few packets but for us it's important to find out why they get sometimes delayed this much longer.
Some background about the bridging box:
Hardware:
Pentium4 1.7GHz, 1.024MB RAM, 2 x 3Com 3c905 [Boomerang] being br0, 3Com 3c905B as management interface.Software:
vanilla Linux Kernel 2.6.3 with patched in CONNMARK extension, STP was disabled
Quoted 8 lines
Other:
The bridge was running in plain mode wich means no other modules like iptables, ebtables or packet scheduler were loaded. It was running in runlevel 3, multiuser networking without x-system.For comparison only this link shows what happens if we push as much TCP traffic in one direction (~97.5 MBits/s) through the bridge, in second image having iptables loaded:
http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_120c_delay-0.pnghttp://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_ipp2p_120c_delay-0.png
It could be a NAPI scheduling delay. If that driver uses NAPI, it avoids doing all the receive work at interrupt, by doing the work at softirq. There is an upper bound of packets it will process in one interval, to avoid getting stuck in the softirq. If the mean number of packets per second arriving exceeed the softirq work rate, then you might see something like this. You might try profiling to see if anything shows up.