Skip to contentSkip to conversation

Dropping Packets in 2.6.17

DISCUSSION Danial Thom 4 participants 9 messages Jun 22, 2006

lore
Share & tools
lore
#0 lore raw

I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#1 lore raw
Quoted 7 lines · Danial Thom

I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but

Quoted 8 lines

userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?

Hi Danial,

the most likely tunable that will help you is

/proc/sys/vm/min_free_kbytes

For the router kind of device that one usually needs bumping a bit; without the bumping the VM doesn't see enough "normal" activity to tune it's emergency/interrupt handling buffers (and most networking allocations happen in interrupt context), and then ends up failing allocations in interrupt context, which leads to dropped packets.

Greetings,
Arjan van de Ven

#2 lore raw
Quoted 36 lines · --- Arjan van de Ven
Quoted 1 line · Danial Thom

I'm trying to make a case for using linux as

a

Quoted 2 lines

network appliance, but I can't find any combination of settings that will keep it

from

Quoted 1 line

dropping packets at an unacceptably high

rate.

Quoted 1 line

The test system is a 1.8Ghz Opteron with

intel

Quoted 1 line

gigE cards running 2.6.17. I'm passing about

70K

Quoted 1 line

pps through the box, which is a light load,

but

Quoted 1 line

userland activities (such as building a

kernel)

Quoted 1 line

cause it to lose packets, even with backlog

set

Quoted 1 line

to 20000. I had the same problem with 2.6.12

and

Quoted 2 lines

abandoned the effort. Has anything been done since to give priority to networking? You

can't

Quoted 1 line

have a network appliance drop packets when

some

Quoted 2 lines

application is gathering stats or a user is looking at a graph. What tunings are

available?

Hi Danial,

the most likely tunable that will help you is

/proc/sys/vm/min_free_kbytes

Quoted 10 lines

For the router kind of device that one usually
needs bumping a bit;
without the bumping the VM doesn't see enough
"normal" activity to tune
it's emergency/interrupt handling buffers (and
most networking
allocations happen in interrupt context), and
then ends up failing
allocations in interrupt context, which leads
to dropped packets.

I don't think thats the problem, as I've tracked the problem to packets being dropped because of excessive backlog (ie, they are being dropped gracefully). However with a backlog of 20000, and a traffic level of about 75,000pps, that means almost 1/3 of a second that the system doesn't process packets, which is just unacceptable.

I'll try changing the setting, but running out of memory doesn't seem to be the issue. I think what I need is some mechanism to make interrupts a priority, like it was back in the days when networking was more important then mp3 playback.

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#3 lore raw
In reply to Danial Thom
Quoted 7 lines · Danial Thom

I'm trying to make a case for using linux as a network appliance, but I can't find any combination of settings that will keep it from dropping packets at an unacceptably high rate. The test system is a 1.8Ghz Opteron with intel gigE cards running 2.6.17. I'm passing about 70K pps through the box, which is a light load, but

Quoted 8 lines

userland activities (such as building a kernel) cause it to lose packets, even with backlog set to 20000. I had the same problem with 2.6.12 and abandoned the effort. Has anything been done since to give priority to networking? You can't have a network appliance drop packets when some application is gathering stats or a user is looking at a graph. What tunings are available?

For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify and do sophisticated statistical calculations on 625Kpps per interface (1.3 million packets per second). The bottleneck at this point was memory bandwidth. Allowing some drops the average rate went up to the PCI bottleneck of about 850kpps/port. Classification and Computation was done in userspace.

Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI. /proc/interrupts will show your interrupt rate.

If the packets go to userspace, make sure you're using CONFIG_PACKET_MMAP

Pádraig.

#4 lore raw

In article <[email protected]>,

Quoted 4 lines · Pádraig Brady

Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI. /proc/interrupts will show your interrupt rate.

The e1000 driver has some more knobs you can turn. I have this in my /etc/modules file:

e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024

(this is for two cards)

and this in /etc/sysctl.conf:

# Tune back swappiness on 2.6
vm.swappiness = 10

# Lots of kernel memory needed for e1000
vm.min_free_kbytes = 65535

This box accepts ~250 mbit/sec, stores that on disk, and streams it out again to multiple peers at ~1300 mbit/sec total. Kernel 2.6.14.2, dual xeon em64t in 64 bits mode, 4 GB memory.

Mike.

#5 lore raw
Quoted 9 lines · Miquel van Smoorenburg

In article <[email protected]>,

Quoted 4 lines · Pádraig Brady

Note there is a max interrupt rate of around 80K/s on x86 at least (not sure about opteron), so make sure you're using NAPI. /proc/interrupts will show your interrupt rate.

Quoted 4 lines

The e1000 driver has some more knobs you can turn. I have this in my /etc/modules file:

e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024

Yep sorry I should have mentioned those.
I set the descriptors up to the max of 4096
(as the statistical calcs only needed 5ms timestamping accuracy).

I got a large gain also by modifying the e1000 driver to do skb recycling

Pádraig.

#6 lore raw
In reply to Pádraig Brady
Quoted 51 lines · --- Pádraig Brady
Quoted 1 line · Danial Thom

I'm trying to make a case for using linux as

a

Quoted 2 lines

network appliance, but I can't find any combination of settings that will keep it

from

Quoted 1 line

dropping packets at an unacceptably high

rate.

Quoted 1 line

The test system is a 1.8Ghz Opteron with

intel

Quoted 1 line

gigE cards running 2.6.17. I'm passing about

70K

Quoted 1 line

pps through the box, which is a light load,

but

Quoted 1 line

userland activities (such as building a

kernel)

Quoted 1 line

cause it to lose packets, even with backlog

set

Quoted 1 line

to 20000. I had the same problem with 2.6.12

and

Quoted 2 lines

abandoned the effort. Has anything been done since to give priority to networking? You

can't

Quoted 1 line

have a network appliance drop packets when

some

Quoted 2 lines

application is gathering stats or a user is looking at a graph. What tunings are

available?

For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify
and do sophisticated statistical calculations on
625Kpps per interface (1.3 million packets per second).
The bottleneck at this point was memory bandwidth.
Allowing some drops the average rate went up to the
PCI bottleneck of about 850kpps/port. Classification and Computation was done in userspace.

Note there is a max interrupt rate of around 80K/s
on x86 at least (not sure about opteron), so make
sure you're using NAPI. /proc/interrupts will show your interrupt rate.

Quoted 3 lines

If the packets go to userspace, make sure you're using
CONFIG_PACKET_MMAP

Unfortunately I can do that much with FreeBSD 4.x with 1 2.0Ghz opteron, so its not a very compelling case to have to spend twice as much on hardware to use LINUX. However 2.4 seemed much better than 2.6 in this regard. 2.6 wants to drop a lot more packets. The goal of using 2.6 is to utilize DP better, but it obviously has to perform better than a UP Freebsd box.

What ITR setting are using for the e1000 driver?

Quoted 2 lines

# Lots of kernel memory needed for e1000 vm.min_free_kbytes = 65535

I'm curious as to why a vm setting is useful, as it doesn't seem that the e1000 driver uses virtual memory? Since rings are replenished with sk_buffs, and sk_buffs have to be contiguous, how does vm come into play?

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#7 lore raw
Quoted 8 lines · Danial Thom
Quoted 4 lines · --- Pádraig Brady

For reference with 2.4.20 on a dual 3.4GHz xeon and 2 x e1000 cards, I was able to capture, classify and do statistical calculations on 625Kpps per interface (1.3 million pps).

Quoted 8 lines

Unfortunately I can do that much with FreeBSD 4.x with 1 2.0Ghz opteron, so its not a very compelling case to have to spend twice as much on hardware to use LINUX. However 2.4 seemed much better than 2.6 in this regard. 2.6 wants to drop a lot more packets. The goal of using 2.6 is to utilize DP better, but it obviously has to perform better than a UP Freebsd box.

NC.

Quoted 1 line

What ITR setting are using for the e1000 driver?

I didn't use ITR, I used NAPI.

Quoted 9 lines
Quoted 2 lines

# Lots of kernel memory needed for e1000 vm.min_free_kbytes = 65535

I'm curious as to why a vm setting is useful, as it doesn't seem that the e1000 driver uses virtual memory? Since rings are replenished with sk_buffs, and sk_buffs have to be contiguous, how does vm come into play?

Contiguous? The [tr]x descriptors contain pointers to the skbufs.
Anyway I bypassed the large allocation overhead by using skb recycling.

Pádraig.

#8 lore raw
Quoted 24 lines · --- Pádraig Brady
Quoted 4 lines · Danial Thom
Quoted 1 line · --- Pádraig Brady

For reference with 2.4.20 on a dual 3.4GHz

xeon

Quoted 5 lines
Quoted 3 lines

and 2 x e1000 cards, I was able to capture,
classify and do statistical calculations
on 625Kpps per interface (1.3 million pps).

Unfortunately I can do that much with FreeBSD

4.x

Quoted 2 lines

with 1 2.0Ghz opteron, so its not a very
compelling case to have to spend twice as

much on

Quoted 1 line

hardware to use LINUX. However 2.4 seemed

much

Quoted 1 line

better than 2.6 in this regard. 2.6 wants to

drop

Quoted 1 line

a lot more packets. The goal of using 2.6 is

to

Quoted 2 lines

utilize DP better, but it obviously has to
perform better than a UP Freebsd box.

Quoted 7 lines

NC.

Quoted 1 line

What ITR setting are using for the e1000

driver?

I didn't use ITR, I used NAPI.

If thats the case then your "system" would have the same problem that I'm encountering, since polling results in buckets of packets being dropped with heavy userland activity.

Since you can control exactly how many interrupts are generated and how often, there should be no advantage to polling, unless LINUX interrupt processing is badly broken, which seems like a strong possibility.

This should work quite naturally without having to use kludges like polling, enormous receive rings or memory cheating.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com