need advice for high traffic network

Date: Thu, 19 Jul 2007 15:17:00 -0700
From: Konstantin Svist <[email protected]>
To: [email protected]
Subject: need advice for high traffic network

Hi,

I have a network (LAN) consisting of (mostly) gigabit ethernet on a few switches. Most of the traffic is taken up by small HTTP reqests. All computers are running Fedora (all are core 4 through 7).

I've been having some problems with servers not being accessible and just last night noticed that the problems disappear when I turn off the firewall.
What happens is that there are lots of small HTTP requests and apparently at some point the firewall starts dropping or disallowing new connections. This has been verified with both ab (apache benchmark) and plain SSH - a lot of times the connections time out or take a long time to get established. There are ~25 rules total (as listed by 'iptables -L')

As a temporary measure, I've turned off firewalls on more of the servers until I can figure out a better solution - I'd like to have a firewall on each server, but performance is more important.

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

Is there anything else I can do?
Any other firewalls? Tricks with rearranging the rules? etc...

Thanks!

Notes:
* Problems do not seem to be limited to any specific Fedora version or hardware.
* external firewalls are out of the question, unless they're really small & cheap: there are >40 servers in the internal network and the number is growing

I'll bet you are hitting your max connections

check the value of net.ipv4.netfilter.ip_conntrack_max

David Lang

Quoted 34 lines · Konstantin Svist

Date: Thu, 19 Jul 2007 15:17:00 -0700
From: Konstantin Svist <[email protected]>
To: [email protected]
Subject: need advice for high traffic network

Hi,

I have a network (LAN) consisting of (mostly) gigabit ethernet on a few switches. Most of the traffic is taken up by small HTTP reqests. All computers are running Fedora (all are core 4 through 7).

I've been having some problems with servers not being accessible and just last night noticed that the problems disappear when I turn off the firewall.
What happens is that there are lots of small HTTP requests and apparently at some point the firewall starts dropping or disallowing new connections. This has been verified with both ab (apache benchmark) and plain SSH - a lot of times the connections time out or take a long time to get established. There are ~25 rules total (as listed by 'iptables -L')

As a temporary measure, I've turned off firewalls on more of the servers until I can figure out a better solution - I'd like to have a firewall on each server, but performance is more important.

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

Is there anything else I can do?
Any other firewalls? Tricks with rearranging the rules? etc...

Thanks!

Notes:
* Problems do not seem to be limited to any specific Fedora version or hardware.
* external firewalls are out of the question, unless they're really small & cheap: there are >40 servers in the internal network and the number is growing

# cat /proc/sys/net/netfilter/nf_conntrack_max
65536

somehow I doubt I have THAT many connections :)

highest load right now is around 600 requests per second, and ~60% complete within 10ms - the rest complete within 200ms (unless the firewall is turned on - then some start timing out 3s and up)

Quoted 7 lines · Konstantin Svist

# cat /proc/sys/net/netfilter/nf_conntrack_max
65536

somehow I doubt I have THAT many connections :)

highest load right now is around 600 requests per second, and ~60% complete within 10ms - the rest complete within 200ms (unless the firewall is turned on - then some start timing out 3s and up)

600s * 120s ip_conntrack_tcp_timeout_time_wait = 72000 entries

( => http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/ )

You might want to try to reduce those timers or just push up your hash bucket = max entry values to maybe twice that.

How do I reduce those timers?

Quoted 2 lines · Konstantin Svist

How do I reduce those timers?

echo <VALUE> > /proc/sys/net/ipv4/netfilter/<SETTING>

Sorry, I meant:
Which parameters are those and what values would you recommend?

Thanks!

Quoted 8 lines · Thomas Jacob

Quoted 2 lines · Konstantin Svist

How do I reduce those timers?

echo <VALUE> > /proc/sys/net/ipv4/netfilter/<SETTING>

Hmm, not sure really, but lower TIME WAIT settings should keep your conntrack table afloat at least ;-)

I'd rather increase ip_conntrack_max and ip_conntrack_buckets to the values suggested by David,

http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.7

Quoted 18 lines · Konstantin Svist

Sorry, I meant:
Which parameters are those and what values would you recommend?

Thanks!

Quoted 10 lines · Thomas Jacob

Quoted 3 lines · Konstantin Svist

How do I reduce those timers?

echo <VALUE> > /proc/sys/net/ipv4/netfilter/<SETTING>

alright, so far I have:

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1

net.ipv4.netfilter.ip_conntrack_max = 1024000

what would you recommend for the buckets? is default (8192) reasonable?

Quoted 9 lines · Konstantin Svist

alright, so far I have:

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1

AFAIK those values do not influence netfilter performance, just local tcp socket performance.

Quoted 3 lines

net.ipv4.netfilter.ip_conntrack_max = 1024000

Quoted 1 line

what would you recommend for the buckets? is default (8192) reasonable?

At the moment I am always setting this to the value of ip_conntrack_max (on the theory that this should result in constant lookup times), as I can spare the memory. But I haven't run any real performance tests with lower hash bucket counts....

The FAQ says though, that one should use odd hash bucket counts, so you might want to decrease that by one.

you should run the tests. doing a hash across too many buckets ends up costing performance as well.

you want the list per bucket to not be too long, but you also don't want to spend more effort and ram on empty buckets.

setting conntrack_max equal to the number of buckets doesn't mean that you will have one entry in each bucket, it means that you will have a lot of empty buckets and other buckets with several items in them.

Quoted 2 lines

The FAQ says though, that one should use odd hash bucket counts, so you might want to decrease that by one.

it's not unusual for simple (i.e. cheap to use) has algorithims to have pathalogical results for specific sizes. ideally you want the bucket count to be a prime number, if it's not (for example a even power of 2) you can get situations where it only puts things in a very small number of buckets.

The FAQ says

"Please note that due to the nature of the current hashing algorithm, an even hash bucket count (and esp. values of the power of two) are a bad choice."

Is this still true? What is the reason for this? And: Does any odd hashtable size work as well as other odd hash tables sizes in the same order of magnitude?

Quoted 3 lines

Is this still true? What is the reason for this? And: Does any odd hashtable size work as well as other odd hash tables sizes in the same order of magnitude?

changing conntrack_max is not chaning the hash bucket count, that change can only be done in the code or as a module parameter at module load time.

On Thu, Jul 19, 2007 at 05:13:11PM -0700, David Lang wrote:
[..]

Quoted 6 lines

Quoted 3 lines

Is this still true? What is the reason for this? And: Does any odd hashtable size work as well as other odd hash tables sizes in the same order of magnitude?

changing conntrack_max is not chaning the hash bucket count, that change can only be done in the code or as a module parameter at module load time.

Sure, I meant odd hash bucket counts of course, so is this FAQ still accurate?

Quoted 13 lines · Thomas Jacob

On Thu, Jul 19, 2007 at 05:13:11PM -0700, David Lang wrote:
[..]

Quoted 8 lines

Quoted 3 lines

Is this still true? What is the reason for this? And: Does any odd hashtable size work as well as other odd hash tables sizes in the same order of magnitude?

changing conntrack_max is not chaning the hash bucket count, that change can only be done in the code or as a module parameter at module load time.

Quoted 2 lines

Sure, I meant odd hash bucket counts of course, so is this FAQ still accurate?

as far as I know.

Quoted 19 lines · David Lang

Quoted 15 lines · Thomas Jacob

On Thu, Jul 19, 2007 at 05:13:11PM -0700, David Lang wrote:
[..]

Quoted 8 lines

Quoted 3 lines

Is this still true? What is the reason for this? And: Does any odd hashtable size work as well as other odd hash tables sizes in the same order of magnitude?

changing conntrack_max is not chaning the hash bucket count, that change can only be done in the code or as a module parameter at module load time.

Sure, I meant odd hash bucket counts of course, so is this FAQ still accurate?

Quoted 1 line

as far as I know.

None of this is :) hash_buckets can be changed at runtime through the sysfs module parameter and the size can be chosen arbitary, in fact the current -git tree will round it up to the next multiple of PAGE_SIZE / (sizeof(struct hlist_head)) to avoid memory wastage.

Quoted 5 lines · Patrick McHardy

None of this is :) hash_buckets can be changed at runtime through the sysfs module parameter and the size can be chosen arbitary, in fact the current -git tree will round it up to the next multiple of PAGE_SIZE / (sizeof(struct hlist_head)) to avoid memory wastage.

not knowing what that sizeof evaluates to

has anyone checked to make
sure that this doesn't cause pathalogical performance or distribution problems with this hash?

Sorry, I meant:
Which parameters are those and what values would you recommend?

Thanks!

Quoted 8 lines · Thomas Jacob

Quoted 2 lines · Konstantin Svist

How do I reduce those timers?

echo <VALUE> > /proc/sys/net/ipv4/netfilter/<SETTING>

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

Quoted 2 lines

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

nf-HiPAC won't help there if you just have 25 rules
( => http://people.netfilter.org/kadlec/nftest.pdf ), the problem is very likely down to you using the default parameters for the conntrack hash table, just like the other reply indicated.

as I said, the current (and default) value is 65536 what would you suggest changing it to?

Quoted 9 lines · Thomas Jacob

Quoted 2 lines

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

nf-HiPAC won't help there if you just have 25 rules
( => http://people.netfilter.org/kadlec/nftest.pdf ), the problem is very likely down to you using the default parameters for the conntrack hash table, just like the other reply indicated.

Hi,

I have a network (LAN) consisting of (mostly) gigabit ethernet on a few switches. Most of the traffic is taken up by small HTTP reqests. All computers are running Fedora (all are core 4 through 7).

I've been having some problems with servers not being accessible and just last night noticed that the problems disappear when I turn off the firewall.
What happens is that there are lots of small HTTP requests and apparently at some point the firewall starts dropping or disallowing new connections. This has been verified with both ab (apache benchmark) and plain SSH - a lot of times the connections time out or take a long time to get established. There are ~25 rules total (as listed by 'iptables -L')

As a temporary measure, I've turned off firewalls on more of the servers until I can figure out a better solution - I'd like to have a firewall on each server, but performance is more important.

I'l looking at nf-HiPAC right now - will probably try it some time soon. Beyond that, I'm out of ideas for the moment.

Is there anything else I can do?
Any other firewalls? Tricks with rearranging the rules? etc...

Thanks!

Notes:
* Problems do not seem to be limited to any specific Fedora version or hardware.
* external firewalls are out of the question, unless they're really small & cheap: there are >40 servers in the internal network and the number is growing