Skip to contentSkip to conversation

Limit size of route cache hash table

PATCH Applied Anton Blanchard 3 participants 11 messages Apr 27, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Limit size of route cache hash table

Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:

    IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)

It would be nice to cap this just for memory consumption reasons, but this massive hashtable also causes a significant spike when measuring OS jitter.

With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.

With the patch applied we limit the number of entries to 64k which can still be overriden by using the rt_entries boot option:

    IP route cache hash table entries: 65536 (order: 3, 524288 bytes)

With this patch rt_worker_func takes 0.060 ms on the same system.

---

Is 64k a reasonable default for the limit?

net/ipv4/route.c +1 -1
index c40debe..5064c26 100644
@@ -3397,7 +3397,7 @@ int __init ip_rt_init(void)
 					0,
 					&rt_hash_log,
 					&rt_hash_mask,
-					0);
+					rhash_entries ? 0 : 64 * 1024);
 	memset(rt_hash_table, 0, (rt_hash_mask + 1) * sizeof(struct rt_hash_bucket));
 	rt_hash_lock_init();
 
#1 lore raw
Quoted 5 lines · Anton Blanchard

Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:

    IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)
Quoted 8 lines

It would be nice to cap this just for memory consumption reasons, but this massive hashtable also causes a significant spike when measuring OS jitter.

With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.

Then boot with rhash_entries = 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval

Quoted 19 lines

With the patch applied we limit the number of entries to 64k which can still be overriden by using the rt_entries boot option:

    IP route cache hash table entries: 65536 (order: 3, 524288 bytes)

With this patch rt_worker_func takes 0.060 ms on the same system.

---

Is 64k a reasonable default for the limit?

net/ipv4/route.c +0 -0
index c40debe..5064c26 100644

net/ipv4/route.c int __init ip_rt_init(void) · line 3403

Quoted 8 lines
net/ipv4/route.c +1 -1
@@ -3397,7 +3397,7 @@ int __init ip_rt_init(void)
 					0,
 					&rt_hash_log,
 					&rt_hash_mask,
-					0);
+					rhash_entries ? 0 : 64 * 1024);
 	memset(rt_hash_table, 0, (rt_hash_mask + 1) * sizeof(struct rt_hash_bucket));
 	rt_hash_lock_init();
 

Sorry this limit is too small. Many of my customer machines would collapse.

It would be smart to eventually change ip_rt_gc_interval from 60 to 1 second for such machines ? Dividing 5 ms per 60 gives 83 us, which is correct.

#2 lore raw

Hi,

Quoted 3 lines

Then boot with rhash_entries = 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval

Yes we are hardwiring it for now.

Quoted 1 line

Sorry this limit is too small. Many of my customer machines would collapse.

So what would a reasonable upper limit be? Surely we should cap it at some point?

Anton

maintainer
#10 lore raw
In reply to Anton Blanchard

From: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 13:04:33 +1000

Quoted 4 lines

Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:

    IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)

Pretty reasonable size for a machine with that much ram. In fact perhaps not large enough depending upon your workload. :-)

If it's not suitable for you, as Eric Dumazet explained, pass the appropriate option on the kernel command line.

#3 lore raw
In reply to Anton Blanchard
Quoted 7 lines · Anton Blanchard

Hi,

Quoted 3 lines

Then boot with rhash_entries = 8000 ?
or
echo 1 >/proc/sys/net/ipv4/route/gc_interval

Quoted 7 lines

Yes we are hardwiring it for now.

Quoted 1 line

Sorry this limit is too small. Many of my customer machines would collapse.

So what would a reasonable upper limit be? Surely we should cap it at some point?

A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.

Yes, this was commit :

commit 0ccfe61803ad24f1c0fe5e1f5ce840ff0f3d9660

Author: Jean Delvare <[email protected]>
Date:   Tue Oct 30 00:59:25 2007 -0700

[TCP]: Saner thash_entries default with much memory.

On systems with a very large amount of memory, the heuristics in alloc_large_system_hash() result in a very large TCP established hash
table: 16 millions of entries for a 128 GB ia64 system. This makes reading from /proc/net/tcp pretty slow (well over a second) and as a result netstat is slow on these machines. I know that /proc/net/tcp is deprecated in favor of tcp_diag, however at the moment netstat only knows of the former.

I am skeptical that such a large TCP established hash is often needed. Just because a system has a lot of memory doesn't imply that it will have several millions of concurrent TCP connections. Thus I believe that we should put an arbitrary high limit to the size of the TCP established hash by default. Users who really need a bigger hash can always use the thash_entries boot parameter to get more.

I propose 2 millions of entries as the arbitrary high limit. This makes /proc/net/tcp reasonably fast on the system in question (0.2 s) while being still large enough for me to be confident that network performance won't suffer.

This is just one way to limit the hash size, there are others; I am not familiar enough with the TCP code to decide which is best. Thus, I would welcome the proposals of alternatives.

[ 2 million is still too large, thus I've modified the limit in the
change to be '512 * 1024'. -DaveM ]

Thanks

maintainer
#9 lore raw
In reply to Anton Blanchard

From: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 15:47:02 +1000

Quoted 4 lines
Quoted 1 line

Sorry this limit is too small. Many of my customer machines would collapse.

So what would a reasonable upper limit be? Surely we should cap it at some point?

32MB is even small for some people.

Like Eric, I think the current cap is just fine. For special situations, tweak settings :)

maintainer
#4 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Mon, 27 Apr 2009 08:12:21 +0200

Quoted 5 lines

A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.

Yes, this was commit :

Fair enough, I'd be OK with this if you'd be too :)

Acked-by
#5 lore raw
Quoted 10 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Mon, 27 Apr 2009 08:12:21 +0200

Quoted 5 lines

A similar patch was done for the size of TCP hash table. It was something like 512 * 1024 if I remember well. IMHO this same value would be fine for IP route cache.

Yes, this was commit :

Fair enough, I'd be OK with this if you'd be too :)

Yes, as setups needing more than 512.000 slots (and 2 million entries with an average of 4 entries per slot) are certainly able to boot with an override.

Anton, please resubmit with this 512 k limit, and my :

thank you.

#6 lore raw
Quoted 7 lines

Yes, as setups needing more than 512.000 slots (and 2 million entries with an average of 4 entries per slot) are certainly able to boot with an override.

Anton, please resubmit with this 512 k limit, and my :

Thanks Eric, I'll respin with the limit set at 512k.

Anton

author Acked-by Eric Dumazet
#7 lore raw
In reply to Eric Dumazet

Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:

    IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)

It would be nice to cap this for memory consumption reasons, but a massive hashtable also causes a significant spike when measuring OS jitter.

With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.

With the patch applied we limit the number of entries to 512k which can still be overriden by using the rt_entries boot option:

    IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)

With this patch rt_worker_func now takes 0.460 ms on the same system.

---

net/ipv4/route.c +1 -1
Index: linux-2.6/net/ipv4/route.c
===================================================================
@@ -3397,7 +3397,7 @@
 					0,
 					&rt_hash_log,
 					&rt_hash_mask,
-					0);
+					rhash_entries ? 0 : 512 * 1024);
 	memset(rt_hash_table, 0, (rt_hash_mask + 1) * sizeof(struct rt_hash_bucket));
 	rt_hash_lock_init();
 
maintainer Applied
#8 lore raw

From: Anton Blanchard <[email protected]>
Date: Mon, 27 Apr 2009 21:50:07 +1000

Quoted 16 lines

Right now we have no upper limit on the size of the route cache hash table. On a 128GB POWER6 box it ends up as 32MB:

    IP route cache hash table entries: 4194304 (order: 9, 33554432 bytes)

It would be nice to cap this for memory consumption reasons, but a massive hashtable also causes a significant spike when measuring OS jitter.

With a 32MB hashtable and 4 million entries, rt_worker_func is taking 5 ms to complete. On another system with more memory it's taking 14 ms. Even though rt_worker_func does call cond_sched() to limit its impact, in an HPC environment we want to keep all sources of OS jitter to a minimum.

With the patch applied we limit the number of entries to 512k which can still be overriden by using the rt_entries boot option:

Quoted 6 lines
    IP route cache hash table entries: 524288 (order: 6, 4194304 bytes)

With this patch rt_worker_func now takes 0.460 ms on the same system.

Applied, thanks!