Dropping Packets in 2.6.17
I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 7 lines · Danial Thom
I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but
Quoted 8 lines
userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?
Hi Danial,
the most likely tunable that will help you is
/proc/sys/vm/min_free_kbytes
For the router kind of device that one usually needs bumping a bit; without the bumping the VM doesn't see enough "normal" activity to tune it's emergency/interrupt handling buffers (and most networking allocations happen in interrupt context), and then ends up failing allocations in interrupt context, which leads to dropped packets.
Greetings,
Arjan van de Ven
Quoted 36 lines · --- Arjan van de Ven
Quoted 1 line · Danial Thom
I'm trying to make a case for using linux as
a
Quoted 2 lines
network appliance, but I can't find any combination of settings that will keep it
from
Quoted 1 line
dropping packets at an unacceptably high
rate.
Quoted 1 line
The test system is a 1.8Ghz Opteron with
intel
Quoted 1 line
gigE cards running 2.6.17. I'm passing about
70K
Quoted 1 line
pps through the box, which is a light load,
but
Quoted 1 line
userland activities (such as building a
kernel)
Quoted 1 line
cause it to lose packets, even with backlog
set
Quoted 1 line
to 20000. I had the same problem with 2.6.12
and
Quoted 2 lines
abandoned the effort. Has anything been done since to give priority to networking? You
can't
Quoted 1 line
have a network appliance drop packets when
some
Quoted 2 lines
application is gathering stats or a user is looking at a graph. What tunings are
available?
Hi Danial,
the most likely tunable that will help you is
/proc/sys/vm/min_free_kbytes
Quoted 10 lines
For the router kind of device that one usually
needs bumping a bit;
without the bumping the VM doesn't see enough
"normal" activity to tune
it's emergency/interrupt handling buffers (and
most networking
allocations happen in interrupt context), and
then ends up failing
allocations in interrupt context, which leads
to dropped packets.
I don't think thats the problem, as I've tracked the problem to packets being dropped because of excessive backlog (ie, they are being dropped gracefully). However with a backlog of 20000, and a traffic level of about 75,000pps, that means almost 1/3 of a second that the system doesn't process packets, which is just unacceptable.
I'll try changing the setting, but running out of memory doesn't seem to be the issue. I think what I need is some mechanism to make interrupts a priority, like it was back in the days when networking was more important then mp3 playback.
Danial
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 7 lines · Danial Thom
I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but
Quoted 8 lines
userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?
For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify and do sophisticated statistical calculations on 625Kpps per interface (1.3 million packets per second). The bottleneck at this point was memory bandwidth. Allowing some drops the average rate went up to the PCI bottleneck of about 850kpps/port. Classification and Computation was done in userspace.
Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI. /proc/interrupts will show your interrupt rate.
If the packets go to userspace, make sure you're using CONFIG_PACKET_MMAP
Pádraig.
In article <[email protected]>,
Quoted 4 lines · Pádraig Brady
Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI.
/proc/interruptswill show your interrupt rate.
The e1000 driver has some more knobs you can turn. I have this in my /etc/modules file:
e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024
(this is for two cards)
and this in /etc/sysctl.conf:
# Tune back swappiness on 2.6
vm.swappiness = 10
# Lots of kernel memory needed for e1000
vm.min_free_kbytes = 65535
This box accepts ~250 mbit/sec, stores that on disk, and streams it out again to multiple peers at ~1300 mbit/sec total. Kernel 2.6.14.2, dual xeon em64t in 64 bits mode, 4 GB memory.
Mike.
Quoted 9 lines · Miquel van Smoorenburg
In article <[email protected]>,
Quoted 4 lines · Pádraig Brady
Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI.
/proc/interruptswill show your interrupt rate.
Quoted 4 lines
The e1000 driver has some more knobs you can turn. I have this in my
/etc/modulesfile:e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024
Yep sorry I should have mentioned those.
I set the descriptors up to the max of 4096
(as the statistical calcs only needed 5ms timestamping accuracy).
I got a large gain also by modifying the e1000 driver to do skb recycling
Pádraig.
Quoted 51 lines · --- Pádraig Brady
Quoted 1 line · Danial Thom
I'm trying to make a case for using linux as
a
Quoted 2 lines
network appliance, but I can't find any combination of settings that will keep it
from
Quoted 1 line
dropping packets at an unacceptably high
rate.
Quoted 1 line
The test system is a 1.8Ghz Opteron with
intel
Quoted 1 line
gigE cards running 2.6.17. I'm passing about
70K
Quoted 1 line
pps through the box, which is a light load,
but
Quoted 1 line
userland activities (such as building a
kernel)
Quoted 1 line
cause it to lose packets, even with backlog
set
Quoted 1 line
to 20000. I had the same problem with 2.6.12
and
Quoted 2 lines
abandoned the effort. Has anything been done since to give priority to networking? You
can't
Quoted 1 line
have a network appliance drop packets when
some
Quoted 2 lines
application is gathering stats or a user is looking at a graph. What tunings are
available?
For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify
and do sophisticated statistical calculations on
625Kpps per interface (1.3 million packets per second).
The bottleneck at this point was memory bandwidth.
Allowing some drops the average rate went up to the
PCI bottleneck of about 850kpps/port. Classification and Computation was done in userspace.Note there is a max interrupt rate of around 80K/s
on x86 at least (not sure about opteron), so make
sure you're using NAPI./proc/interruptswill show your interrupt rate.
Quoted 3 lines
If the packets go to userspace, make sure you're using
CONFIG_PACKET_MMAP
Unfortunately I can do that much with FreeBSD 4.x with 1 2.0Ghz opteron, so its not a very compelling case to have to spend twice as much on hardware to use LINUX. However 2.4 seemed much better than 2.6 in this regard. 2.6 wants to drop a lot more packets. The goal of using 2.6 is to utilize DP better, but it obviously has to perform better than a UP Freebsd box.
What ITR setting are using for the e1000 driver?
Quoted 2 lines
# Lots of kernel memory needed for e1000 vm.min_free_kbytes = 65535
I'm curious as to why a vm setting is useful, as it doesn't seem that the e1000 driver uses virtual memory? Since rings are replenished with sk_buffs, and sk_buffs have to be contiguous, how does vm come into play?
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 8 lines · Danial Thom
Quoted 4 lines · --- Pádraig Brady
For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify and do statistical calculations on 625Kpps per interface (1.3 million pps).
Quoted 8 lines
Unfortunately I can do that much with FreeBSD 4.x with 1 2.0Ghz opteron, so its not a very compelling case to have to spend twice as much on hardware to use LINUX. However 2.4 seemed much better than 2.6 in this regard. 2.6 wants to drop a lot more packets. The goal of using 2.6 is to utilize DP better, but it obviously has to perform better than a UP Freebsd box.
NC.
Quoted 1 line
What ITR setting are using for the e1000 driver?
I didn't use ITR, I used NAPI.
Quoted 9 lines
Quoted 2 lines
# Lots of kernel memory needed for e1000 vm.min_free_kbytes = 65535
I'm curious as to why a vm setting is useful, as it doesn't seem that the e1000 driver uses virtual memory? Since rings are replenished with
sk_buffs, andsk_buffshave to be contiguous, how does vm come into play?
Contiguous? The [tr]x descriptors contain pointers to the skbufs.
Anyway I bypassed the large allocation overhead by using skb recycling.
Pádraig.
Quoted 24 lines · --- Pádraig Brady
Quoted 4 lines · Danial Thom
Quoted 1 line · --- Pádraig Brady
For reference with 2.4.20 on a dual 3.4GHz
xeon
Quoted 5 lines
Quoted 3 lines
and 2 x e1000 cards, I was able to capture,
classify and do statistical calculations
on 625Kpps per interface (1.3 million pps).Unfortunately I can do that much with FreeBSD
4.x
Quoted 2 lines
with 1 2.0Ghz opteron, so its not a very
compelling case to have to spend twice asmuch on
Quoted 1 line
hardware to use LINUX. However 2.4 seemed
much
Quoted 1 line
better than 2.6 in this regard. 2.6 wants to
drop
Quoted 1 line
a lot more packets. The goal of using 2.6 is
to
Quoted 2 lines
utilize DP better, but it obviously has to
perform better than a UP Freebsd box.
Quoted 7 lines
NC.
Quoted 1 line
What ITR setting are using for the e1000
driver?
I didn't use ITR, I used NAPI.
If thats the case then your "system" would have the same problem that I'm encountering, since polling results in buckets of packets being dropped with heavy userland activity.
Since you can control exactly how many interrupts are generated and how often, there should be no advantage to polling, unless LINUX interrupt processing is badly broken, which seems like a strong possibility.
This should work quite naturally without having to use kludges like polling, enormous receive rings or memory cheating.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com