Limit size of route cache hash table
Subsystems (3)
PATCH 1/1 Limit size of route cache hash table
Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:
IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)
It would be nice to cap this just for memory consumption reasons, but this massive hashtable also causes a significant spike when measuring OS jitter.
With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.
With the patch applied we limit the number of entries to 64k which can still be overriden by using the rt_entries boot option:
IP route cache hash table entries: 65536 (order: 3, 524288 bytes)
With this patch rt_worker_func takes 0.060 ms on the same system.
---
Is 64k a reasonable default for the limit?
net/ipv4/route.c +1 -1
@@ -3397,7 +3397,7 @@ int __init ip_rt_init(void)0 , &rt_hash_log, &rt_hash_mask, - 0); + rhash_entries ? 0 : 64 * 1024);memset (rt_hash_table,0 , (rt_hash_mask +1 ) *sizeof (struct rt_hash_bucket ));rt_hash_lock_init ();
Quoted 5 lines · Anton Blanchard
Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:
IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Quoted 8 lines
It would be nice to cap this just for memory consumption reasons, but this massive hashtable also causes a significant spike when measuring OS jitter.
With a 32MB hashtable and 4 million entries,
rt_worker_funcis taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even thoughrt_worker_funcdoes callcond_sched()to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.
Then boot with rhash_entries = 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval
Quoted 19 lines
With the patch applied we limit the number of entries to 64k which can still be overriden by using the
rt_entriesboot option:IP route cache hash table entries: 65536 (order: 3, 524288 bytes)With this patch
rt_worker_functakes 0.060 ms on the same system.Signed-off-by Anton Blanchard <[email protected]>---
Is 64k a reasonable default for the limit?
net/ipv4/route.c +0
-0
net/ipv4/route.c int __init ip_rt_init(void) · line 3403
Quoted 8 lines
net/ipv4/route.c +1
-1@@ -3397,7 +3397,7 @@ int __init ip_rt_init(void)0 , &rt_hash_log, &rt_hash_mask, - 0); + rhash_entries ? 0 : 64 * 1024);memset (rt_hash_table,0 , (rt_hash_mask +1 ) *sizeof (struct rt_hash_bucket ));rt_hash_lock_init ();
Sorry this limit is too small. Many of my customer machines would collapse.
It would be smart to eventually change ip_rt_gc_interval from 60 to 1 second for such machines ? Dividing 5 ms per 60 gives 83 us, which is correct.
Hi,
Quoted 3 lines
Then boot with
rhash_entries= 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval
Yes we are hardwiring it for now.
Quoted 1 line
Sorry this limit is too small. Many of my customer machines would collapse.
So what would a reasonable upper limit be? Surely we should cap it at some point?
Anton
David Miller
maintainerFrom: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 13:04:33 +1000
Quoted 4 lines
Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:
IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Pretty reasonable size for a machine with that much ram. In fact perhaps not large enough depending upon your workload. :-)
If it's not suitable for you, as Eric Dumazet explained, pass the appropriate option on the kernel command line.
Quoted 7 lines · Anton Blanchard
Hi,
Quoted 3 lines
Then boot with
rhash_entries= 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval
Quoted 7 lines
Yes we are hardwiring it for now.
Quoted 1 line
Sorry this limit is too small. Many of my customer machines would collapse.
So what would a reasonable upper limit be? Surely we should cap it at some point?
A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.
Yes, this was commit :
commit 0ccfe61803ad24f1c0fe5e1f5ce840ff0f3d9660
Author: Jean Delvare <[email protected]> Date: Tue Oct 30 00:59:25 2007 -0700
[TCP]: Saner thash_entries default with much memory.
On systems with a very large amount of memory, the heuristics in alloc_large_system_hash() result in a very large TCP established hash
table: 16 millions of entries for a 128 GB ia64 system. This makes reading from /proc/net/tcp pretty slow (well over a second) and as a result netstat is slow on these machines. I know that /proc/net/tcp is deprecated in favor of tcp_diag, however at the moment netstat only knows of the former.
I am skeptical that such a large TCP established hash is often needed. Just because a system has a lot of memory doesn't imply that it will have several millions of concurrent TCP connections. Thus I believe that we should put an arbitrary high limit to the size of the TCP established hash by default. Users who really need a bigger hash can always use the thash_entries boot parameter to get more.
I propose 2 millions of entries as the arbitrary high limit. This makes /proc/net/tcp reasonably fast on the system in question (0.2 s) while being still large enough for me to be confident that network performance won't suffer.
This is just one way to limit the hash size, there are others; I am not familiar enough with the TCP code to decide which is best. Thus, I would welcome the proposals of alternatives.
[ 2 million is still too large, thus I've modified the limit in the
change to be '512 * 1024'. -DaveM ]
Thanks
David Miller
maintainerFrom: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 15:47:02 +1000
Quoted 4 lines
Quoted 1 line
Sorry this limit is too small. Many of my customer machines would collapse.
So what would a reasonable upper limit be? Surely we should cap it at some point?
32MB is even small for some people.
Like Eric, I think the current cap is just fine. For special situations, tweak settings :)
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Mon, 27 Apr 2009 08:12:21 +0200
Quoted 5 lines
A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.
Yes, this was commit :
Fair enough, I'd be OK with this if you'd be too :)
Eric Dumazet
Acked-byQuoted 10 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Mon, 27 Apr 2009 08:12:21 +0200Quoted 5 lines
A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.
Yes, this was commit :
Fair enough, I'd be OK with this if you'd be too :)
Yes, as setups needing more than 512.000 slots (and 2 million entries with an average of 4 entries per slot) are certainly able to boot with an override.
Anton, please resubmit with this 512 k limit, and my :
thank you.
Quoted 7 lines
Yes, as setups needing more than 512.000 slots (and 2 million entries with an average of 4 entries per slot) are certainly able to boot with an override.
Anton, please resubmit with this 512 k limit, and my :
Acked-by Eric Dumazet <[email protected]>
Thanks Eric, I'll respin with the limit set at 512k.
Anton
Anton Blanchard
Acked-by Eric DumazetRight now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:
IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)
It would be nice to cap this for memory consumption reasons, but a massive hashtable also causes a significant spike when measuring OS jitter.
With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.
With the patch applied we limit the number of entries to 512k which can still be overriden by using the rt_entries boot option:
IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)
With this patch rt_worker_func now takes 0.460 ms on the same system.
---
net/ipv4/route.c +1 -1
@@ -3397,7 +3397,7 @@0 , &rt_hash_log, &rt_hash_mask, - 0); + rhash_entries ? 0 : 512 * 1024);memset (rt_hash_table,0 , (rt_hash_mask +1 ) *sizeof (struct rt_hash_bucket ));rt_hash_lock_init ();
From: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 21:50:07 +1000
Quoted 16 lines
Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:
IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)It would be nice to cap this for memory consumption reasons, but a massive hashtable also causes a significant spike when measuring OS jitter.
With a 32MB hashtable and 4 million entries,
rt_worker_funcis taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even thoughrt_worker_funcdoes callcond_sched()to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.With the patch applied we limit the number of entries to 512k which can still be overriden by using the
rt_entriesboot option:
Quoted 6 lines
IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)With this patch
rt_worker_funcnow takes 0.460 ms on the same system.Signed-off-by Anton Blanchard <[email protected]>Acked-by Eric Dumazet <[email protected]>
Applied, thanks!