why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)
Subsystems (3)
Ookhoi wrote (ao):
# Ookhoi wrote (ao):
# > Florian Zwoch wrote (ao):
# > > issue seems to partly solved. the e1000 driver seems to be ok! # > > i reconfigured my kernel and intentionally left out netfilter options. # > > after that my network performance was back to normal.
# > >
# > > netfilter was only compiled in the kernel. it was not used with any rules! # > >
# > > so my wild guess would be that something with the netfilter code (i am # > > not 100% sure it was netfilter.. _maybe_ it was some small odd kernel # > > option i accidently enabled/disabled) is broken since test3 (again # > > uncertified. but i firstly noticed this switching from test3 to test4). #
# > I have netfilter enabled, and will try another -test6 kernel with # > netfilter not compiled in to see if that indeed makes a difference. #
# I can confirm now that disabling netfilter in 2.6.0-test6 makes the nic
# perform oke wrt upload. # I (just like Florian) had no iptables rules active in the former # 2.6.0-test6 kernel, but netfilter was compiled in.
Would somebody like to explain why netfilter (in kernel, but not in use) makes upload go very slow? I am by no means a network guru, but eager to learn :-)
Quoted 3 lines · On Wed, 8 Oct 2003 15:13:20 +0200 [email protected] wrote:
Would somebody like to explain why netfilter (in kernel, but not in use) makes upload go very slow? I am by no means a network guru, but eager to learn :-)
It'll likely happen much quicker if you actually report this to the netfilter lists, which is where the people who can help you are paying attention.
Quoted 11 lines · On Wed, Oct 08, 2003 at 03:13:20PM +0200, [email protected] wrote:
# > I have netfilter enabled, and will try another -test6 kernel with # > netfilter not compiled in to see if that indeed makes a difference. #
# I can confirm now that disabling netfilter in 2.6.0-test6 makes the nic# perform oke wrt upload. # I (just like Florian) had no iptables rules active in the former # 2.6.0-test6 kernel, but netfilter was compiled in.Would somebody like to explain why netfilter (in kernel, but not in use) makes upload go very slow? I am by no means a network guru, but eager to learn :-)
let's get this straight. There are five possible cases
a) CONFIG_NETFILTER disabled. you won't even have the netfilter hooks in the network stack (so certainly no netfilter-using modules loaded)
b) CONFIG_NETFILTER enabled, but _no_ modules (iptable_filter, ip_conntrack, ...) attached to the netfilter hook
c) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o) loaded, NO RULES in the table
d) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o) loaded, RULES in the table
e) CONFIG_NETFILTER enabled and ip_conntrack.o loaded, iptable_filter loaded or not, rules or not
So if you want to give us an idea about where the bottleneck might be, please clearly indicate between which of the two cases you see this performance penalty.
This way we can isolate the culprit.
Signature · 5 lines
- Harald Welte <[email protected]> http://www.netfilter.org/ ============================================================================ "Fragmentation is like classful addressing -- an interesting early architectural error that shows how much experimentation was going on while IP was being designed." -- Paul Vixie
Quoted 9 lines · Harald Welte
Quoted 3 lines
Would somebody like to explain why netfilter (in kernel, but not in use) makes upload go very slow? I am by no means a network guru, but eager to learn :-)
let's get this straight. There are five possible cases
a)
CONFIG_NETFILTERdisabled. you won't even have the netfilter hooks in the network stack (so certainly no netfilter-using modules loaded)
no problem
Quoted 2 lines
b)
CONFIG_NETFILTERenabled, but _no_ modules (iptable_filter,ip_conntrack, ...) attached to the netfilter hook
no problem
Quoted 2 lines
c)
CONFIG_NETFILTERenabled and iptable_filter.o (which pulls ip_tables.o) loaded, NO RULES in the table
no problem
Quoted 2 lines
d)
CONFIG_NETFILTERenabled and iptable_filter.o (which pulls ip_tables.o) loaded, RULES in the table
no problem (as long as i dont load any rules that require ip_conntrack)
Quoted 2 lines
e)
CONFIG_NETFILTERenabled and ip_conntrack.o loaded,iptable_filterloaded or not, rules or not
*boink*
whenever i try to load ip_conntrack the nic performance drops from 5mb/s to 200k/s.
still using 2.6.0-test6.
regards,
Florian
Hi Florian!
I'm Cc'ing all the mailinglists in order to keep them posted about the question you've raised there. All further discussion will move to netfilter-devel, so for those interested: Please continue there.
Quoted 12 lines · Florian Zwoch
Quoted 2 lines
a)
CONFIG_NETFILTERdisabled. you won't even have the netfilter hooks in the network stack (so certainly no netfilter-using modules loaded)no problem
Quoted 2 lines
b)
CONFIG_NETFILTERenabled, but _no_ modules (iptable_filter,ip_conntrack, ...) attached to the netfilter hookno problem
Quoted 2 lines
c)
CONFIG_NETFILTERenabled and iptable_filter.o (which pulls ip_tables.o) loaded, NO RULES in the tableno problem
Quoted 7 lines
Quoted 2 lines
d)
CONFIG_NETFILTERenabled and iptable_filter.o (which pulls ip_tables.o) loaded, RULES in the tableno problem (as long as i dont load any rules that require
ip_conntrack)Quoted 2 lines
e)
CONFIG_NETFILTERenabled and ip_conntrack.o loaded,iptable_filterloaded or not, rules or not*boink*
So It's clearly the connection tracking subsystem. This is on one hand good (because it means it's neither netfilter nor iptables).
Quoted 2 lines
whenever i try to load
ip_conntrackthe nic performance drops from 5mb/s to 200k/s.
On the other hand, this is definitely way worse than you would expect. Can you please tell me more information about:
- number of connections you have? (cat /proc/net/ip_conntrack | wc -l)
- number of buckets and ip_conntrack_max (printed at ip_conntrack loadtime
- your traffic pattern. Are you spraying udp packets with random src/dst? What kind of connections (protocol, application) are you testing with?
- what about the hardware (cpu, memory, smp?)
Even the worst tests we've had so far (random UDP packets) 'only'
reduced the througput by about 50%. Maybe we can do better than 50%
worst case behaviour, but you will always observe a visible impact as soon as you start connection tracking for every single packet (which is what 'insmod ip_conntrack' implies).
Quoted 1 line
still using 2.6.0-test6.
Have you observed this behaviour with other kernel versions? Was there a performance change between 2.4 and 2.6? Or did you always observe this grave performance loss?
Quoted 2 lines
regards,
Florian
Signature · 5 lines
- Harald Welte <[email protected]> http://www.netfilter.org/ ============================================================================ "Fragmentation is like classful addressing -- an interesting early architectural error that shows how much experimentation was going on while IP was being designed." -- Paul Vixie
Re: why does netfilter make upload very slow?
Quoted 7 lines · Harald Welte
So It's clearly the connection tracking subsystem. This is on one hand good (because it means it's neither netfilter nor iptables).
On the other hand, this is definitely way worse than you would expect. Can you please tell me more information about:
- number of connections you have? (cat
/proc/net/ip_conntrack| wc -l)
returns 3 in my 'as is' state. and raises to up to 8 while scp-ing a test file to another machine.
Quoted 2 lines
- number of buckets and
ip_conntrack_max(printed atip_conntrackloadtime
Oct 15 12:25:38 [kernel] ip_conntrack version 2.1 (4095 buckets, 32760 max) - 300 bytes per conntrack
Quoted 3 lines
- your traffic pattern. Are you spraying udp packets with random src/dst? What kind of connections (protocol, application) are you testing with?
no i am not doing anything ugly with this machine. it is my personal workstation where i try to get my work done and learn more about linux and development.
my main tests were scp, but this behaviuor is similar on ftp and smb. i also have problems with irc dcc (more on that later on).
Quoted 2 lines
- what about the hardware (cpu, memory, smp?)
Intel(R) Pentium(R) 4 CPU 2.40GHz on an Asus P4T533 (i850e) with 512MB RDRAM.
no smp/ht whatever. simple single processor with decent hardware.
Quoted 5 lines
Even the worst tests we've had so far (random UDP packets) 'only'
reduced the througput by about 50%. Maybe we can do better than 50%
worst case behaviour, but you will always observe a visible impact as soon as you start connection tracking for every single packet (which is what 'insmodip_conntrack' implies).
sure we do.. but i hope in normal operation it wont choke with 8 connection :-)
Quoted 5 lines
Have you observed this behaviour with other kernel versions? Was there a performance change between 2.4 and 2.6? Or did you always observe this grave performance loss?
when used 2.4 i cant remember to have any problems. i played a bit with netfilter and everything worked as expected. i dont know if there was a secret hickup i dont know of.
then i tried using the 2.6.0 series as of test1. again i think everything was working great. i went through almost all the major test releases (copied .config and checked if nothing serious got broken in menuconfig) and suddenly discovered that my irc dcc sends were almost broken (0.1k/s). what was the time i began to look into that and noticed that problem with the nic's performance. i *think* this dcc bug was introduced in the switch from test3 to test4.
again this is not 100% proven but the best i could remember now. if you need me to test an older kernel version again please let me know. i think i can spend some time here and then helping to fix this problem.
p.s. please put me into cc in your reply as im not yet on the list.
regards,
Signature · 13 lines
Florian Zwoch [email protected] _______________________________ BackendMedia www.backendmedia.com [email protected] Linn Zwoch Smith GbR Pariser Str. 44 D-10707 Berlin Tel +49 30 83 22 50 00 Fax +49 30 83 22 50 07