Skip to contentSkip to conversation

Really slow netstat and /proc/net/tcp in 2.4

DISCUSSION Simon Kirby 5 participants 11 messages Oct 11, 2001 – Oct 13, 2001

lore
Share & tools
lore
#0 lore raw

Is there something that changed from 2.2 -> 2.4 with regards to the speed of netstat and /proc/net/tcp? We have some webservers we just upgraded from 2.2.19 to 2.4.12, and some in-house monitoring tools that check /proc/net/tcp have begun to suck up a lot of CPU cycles trying to read that file.

A simple cat or wc -l on the file feels like about on the order of two magnitudes slower ("time" reports around a second when the file has 450 entries). Some servers seem to be worse than others, and it does not appear to be proportional to the number of entries across servers.

netstat -tn just crawls along on these servers. Should I enable profile=1 or something to see what's happening here?

Examples:

2.2.19:

[sroot@marble:/root]# time wc -l /proc/net/tcp
    858 /proc/net/tcp
0.000u 0.010s 0:00.01 100.0%    0+0k 0+0io 112pf+0w

2.4.12:

[sroot@pro:/root]# time wc -l /proc/net/tcp
    463 /proc/net/tcp
0.000u 0.640s 0:00.64 100.0%    0+0k 0+0io 69pf+0w

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#1 lore raw

Hello!

Quoted 2 lines

Is there something that changed from 2.2 -> 2.4 with regards to the speed of netstat and /proc/net/tcp?

Incredibly high size of hash table, I think.
At least here size is ~1MB. And all this is read each 1K of data read via /proc/ :-)

Alexey

#2 lore raw
In reply to Simon Kirby
Quoted 8 lines · On Thu, Oct 11, 2001 at 11:30:25PM +0400, [email protected] wrote:

Hello!

Quoted 2 lines

Is there something that changed from 2.2 -> 2.4 with regards to the speed of netstat and /proc/net/tcp?

Incredibly high size of hash table, I think.
At least here size is ~1MB. And all this is read each 1K of data read via /proc/ :-)

So it's walking the hash table per block read, and the hash table is very large? Hmm. I notice it's a bit faster if I use dd if=/proc/net/tcp of=/dev/null bs=1024k, but not much.

Is it possible to fix this? Was the 2.2 hash table just that much smaller?

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#3 lore raw

Hello!

Quoted 2 lines

Is it possible to fix this? Was the 2.2 hash table just that much smaller?

2.2 did not use hash tables, holding special single list for /proc.

If I understand correctly it was removed because added more data/work and new point of synchronization for main path being useful only for /proc. The approach would be justified, if you had 100000 sockets. In this case both approaches are equally slow. :-) But for 1000 sockets hash table of 100000 entries is sort of overscaled.

Quoted 1 line

Is it possible to fix this?

To fix --- no. To make differently --- yes.

Well, actually, if you are interested drop me a not I can pack for you some my old work on this. It is fully functional, but api is still dirty. It requires some patching kernel, unfortunately.

Alexey

#4 lore raw
In reply to Simon Kirby
Quoted 11 lines · On Fri, Oct 12, 2001 at 08:44:58PM +0400, [email protected] wrote:
Quoted 2 lines

Is it possible to fix this? Was the 2.2 hash table just that much smaller?

2.2 did not use hash tables, holding special single list for /proc.

If I understand correctly it was removed because added more data/work and new point of synchronization for main path being useful only for /proc. The approach would be justified, if you had 100000 sockets. In this case both approaches are equally slow. :-) But for 1000 sockets hash table of 100000 entries is sort of overscaled.

Quoted 7 lines
Quoted 1 line

Is it possible to fix this?

To fix --- no. To make differently --- yes.

Well, actually, if you are interested drop me a not I can pack for you some my old work on this. It is fully functional, but api is still dirty. It requires some patching kernel, unfortunately.

If it involves changing the TCP stack locking stuff and putting the old list back, it's probably not worth the bother. The only thing we're using the file for (other than the occasional admin-run "netstat") is to check to see what ports are listening on the machine without actually attempting to connect to them. We check our services this way more often than actually connecting and requesting a response to reduce log clutter and testing load on the server. Is there an easier way to accomplish this than parsing /proc/net/tcp? We could attempt to bind to the ports we want to check, but that would race with daemons trying to start up.

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#5 lore raw

Hello!

Quoted 1 line

If it involves changing the TCP stack locking stuff

No. It even does not touch the kernel except for exporting 4 new not exported symbols.

Quoted 3 lines

and testing load on the server. Is there an easier way to accomplish this than parsing /proc/net/tcp? We could attempt to bind to the ports we want to check, but that would race with daemons trying to start up.

To syn-flood with single syn using packet socket.

Alexey

#6 lore raw
In reply to Simon Kirby

In article <[email protected]>,

Quoted 10 lines · Simon Kirby
Quoted 8 lines · On Thu, Oct 11, 2001 at 11:30:25PM +0400, [email protected] wrote:

Hello!

Quoted 2 lines

Is there something that changed from 2.2 -> 2.4 with regards to the speed of netstat and /proc/net/tcp?

Incredibly high size of hash table, I think.
At least here size is ~1MB. And all this is read each 1K of data read via /proc/ :-)

Quoted 6 lines

So it's walking the hash table per block read, and the hash table is very large? Hmm. I notice it's a bit faster if I use dd if=/proc/net/tcp of=/dev/null bs=1024k, but not much.

Is it possible to fix this? Was the 2.2 hash table just that much smaller?

The hash table is likely to big anyways; eating cache and not helping that much. If you're interested in some testing
I can send you patches to change it by hand and collect statistics for average hash queue length. Then you can figure out a good size for your workload with some work. Longer time I think the table sizing heuristics are far too aggressive and need to be throttled back; but that needs more data from real servers.

-Andi

#7 lore raw
In reply to Simon Kirby
Quoted 7 lines · Andi Kleen

The hash table is likely to big anyways; eating cache and not helping that much. If you're interested in some testing
I can send you patches to change it by hand and collect statistics for average hash queue length. Then you can figure out a good size for your workload with some work. Longer time I think the table sizing heuristics are far too aggressive and need to be throttled back; but that needs more data from real servers.

Wouldn't just counting the lines in /proc/net/tcp be sufficient to see how many buckets should be used in an ideal hash table distribution scenario? (In which case the size of the hash table depends largely on a machine's work load...)

Most of our web servers seem to have 500-1000 entries in /proc/net/tcp.

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#8 lore raw
In reply to Simon Kirby
Quoted 10 lines · Simon Kirby
Quoted 7 lines · Andi Kleen

The hash table is likely to big anyways; eating cache and not helping that much. If you're interested in some testing
I can send you patches to change it by hand and collect statistics for average hash queue length. Then you can figure out a good size for your workload with some work. Longer time I think the table sizing heuristics are far too aggressive and need to be throttled back; but that needs more data from real servers.

Quoted 4 lines

Wouldn't just counting the lines in /proc/net/tcp be sufficient to see how many buckets should be used in an ideal hash table distribution scenario? (In which case the size of the hash table depends largely on a machine's work load...)

That won't tell you the list length of individual hash buckets. Keeping that number in average slow is the goal of the big hash tables, but I suspect the 2.4 ones are far too big; losing any possible benefit in cache non locality.

I attached a patch. It allows you to get some simple statistics from /proc/net/sockstat (unfortunately costly too). It also adds a new kernel boot argument tcpehashgoal=order. Order is the log2 of how many pages you want to use for the hash table (so it needs 2^order * 4096 bytes on i386) You can experiment with various sizes and check which one gives still reasonable hash distribution under load.

The smallest one you can find is best.

BTW, it seems like the tables are 1/4 too big on SMP systems. the second half reserved for time-wait have per bucket rwlocks too, but they're not used. If established and time-wait were split this wastage could be avoided. This way some memory (but not walk time) could be saved. It would also lower the requirements on continuous memory by half; e.g. useful if tcp/ip was ever turned into a module.

-Andi

ipv4/proc.c renamed from ipv4/proc.c-o +3 -0
@@ -68,6 +68,7 @@
 {
 	/* From  net/socket.c  */
 	extern int socket_get_info(char *, char **, off_t, int);
+	extern int tcp_v4_hash_statistics(char *) ;
 
 	int len  = socket_get_info(buffer,start,offset,length);
 
@@ -82,6 +83,8 @@
 		       fold_prot_inuse(&raw_prot));
 	len += sprintf(buffer+len, "FRAG: inuse %d memory %d\n",
 		       ip_frag_nqueues, atomic_read(&ip_frag_mem));
+	len += tcp_v4_hash_statistics(buffer+len); 
+			   
 	if (offset >= len)
 	{
 		*start = buffer;
ipv4/tcp.c renamed from ipv4/tcp.c-o +15 -2
@@ -2442,6 +2442,15 @@
   	return 0;
 }
 
+static unsigned tcp_ehash_order; 
+static int __init tcp_hash_setup(char *str)
+{
+	tcp_ehash_order = simple_strtol(str,NULL,0); 
+	return 0;
+} 
+
+__setup("tcpehashorder=", tcp_hash_setup); 
+
 
 extern void __skb_cb_too_small_for_tcp(int, int);
 
@@ -2486,8 +2495,12 @@
 	else
 		goal = num_physpages >> (23 - PAGE_SHIFT);
 
-	for(order = 0; (1UL << order) < goal; order++)
-		;
+	if (tcp_ehash_order) 
+		order = tcp_ehash_order;
+	else {	
+		for(order = 0; (1UL << order) < goal; order++)
+			;
+	} 		
 	do {
 		tcp_ehash_size = (1UL << order) * PAGE_SIZE /
 			sizeof(struct tcp_ehash_bucket);
ipv4/tcp_ipv4.c renamed from ipv4/tcp_ipv4.c-o +57 -0

66 lines not on this page · raw message

#9 lore raw
Quoted 7 lines · Andi Kleen

I attached a patch. It allows you to get some simple statistics from /proc/net/sockstat (unfortunately costly too). It also adds a new kernel boot argument tcpehashgoal=order. Order is the log2 of how many pages you want to use for the hash table (so it needs 2^order * 4096 bytes on i386) You can experiment with various sizes and check which one gives still reasonable hash distribution under load.

Wouldn't something like "tcpehashbuckets" make a better boot tunable than "tcpehashorder"? Rounded up to next power of two before used.

I come at this from the PAGE_SIZE angle, rather than the TCP angle:
"order" tunables seem confusing to me (being interested in configurable PAGE_SIZE). And they're confusing to code too: note that the existing calculation of goal from num_physpages gives you more hash buckets for larger PAGE_SIZE (comment says "methodology is similar to that of the buffer cache", but buffer cache gets it right - though for small memory, would do better to multiply mempages by sizeof _before_ shifting right).

Hugh

#10 lore raw

In article <[email protected]>,

Quoted 11 lines · Hugh Dickins
Quoted 7 lines · Andi Kleen

I attached a patch. It allows you to get some simple statistics from /proc/net/sockstat (unfortunately costly too). It also adds a new kernel boot argument tcpehashgoal=order. Order is the log2 of how many pages you want to use for the hash table (so it needs 2^order * 4096 bytes on i386) You can experiment with various sizes and check which one gives still reasonable hash distribution under load.

Wouldn't something like "tcpehashbuckets" make a better boot tunable than "tcpehashorder"? Rounded up to next power of two before used.

[...]

I just hacked something quickly together so that people can test what impact different hash tables sizes have on their workload, and for that using the order was easiest. The goal is of course to do automatic hash table tuning. I don't expect it to be an permanent tunable. My hope is that it'll turn out that smaller hash tables will be good enough.

-Andi