TCP 2MSL on loopback

PATCH In review Howard Chu 5 participants 22 messages Mar 5, 2007 – Mar 7, 2007

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Howard Chu

author

2007-03-05 11:26 UTC #0 lore raw

Why is the Maximum Segment Lifetime a global parameter? Surely the maximum possible lifetime of a particular TCP segment depends on the actual connection. At the very least, it would be useful to be able to set it on a per-interface basis. E.g., in the case of the loopback interface, it would be useful to be able to set it to a very small duration.

As I note in this draft http://www.ietf.org/internet-drafts/draft-chu-ldap-ldapi-00.txt
when doing a connection soak test of OpenLDAP using clients connected through localhost, the entire port range is exhausted in well under a second, at which point the test stalls until a port comes out of TIME_WAIT state so the next connection can be opened.

These days it's not uncommon for an OpenLDAP slapd server to handle tens of thousands of connections per second in real use (e.g., at Google, or at various telcos). While the LDAP server is fast enough to saturate even 10gbit ethernet using contemporary CPUs, we have to resort to multiple virtual interfaces just to make sure we have enough port numbers available.

Ideally the 2MSL parameter would be dynamically adjusted based on the route to the destination and the weights associated with those routes. In the simplest case, connections between machines on the same subnet (i.e., no router hops involved) should have a much smaller default value than connections that traverse any routers. I'd settle for a two-level setting - with no router hops, use the small value; with any router hops use the large value.

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Eric Dumazet

2007-03-05 14:28 UTC #1 lore raw

Quoted 6 lines · Howard Chu

Why is the Maximum Segment Lifetime a global parameter? Surely the maximum possible lifetime of a particular TCP segment depends on the actual connection. At the very least, it would be useful to be able to set it on a per-interface basis. E.g., in the case of the loopback interface, it would be useful to be able to set it to a very small duration.

Hi Howard

I think you should address these questions on netdev instead of linux-kernel.

Quoted 8 lines

As I note in this draft
http://www.ietf.org/internet-drafts/draft-chu-ldap-ldapi-00.txt
when doing a connection soak test of OpenLDAP using clients connected through localhost, the entire port range is exhausted in well under a second, at which point the test stalls until a port comes out of TIME_WAIT state so the next connection can be opened.

Quoted 7 lines

These days it's not uncommon for an OpenLDAP slapd server to handle tens of thousands of connections per second in real use (e.g., at Google, or at various telcos). While the LDAP server is fast enough to saturate even 10gbit ethernet using contemporary CPUs, we have to resort to multiple virtual interfaces just to make sure we have enough port numbers available.

I dont uderstand... doesnt slapd server listen for connections on a given port, like http ? Or is it doing connections like a ftp server ?

Of course, if you want to open more than 60.000 concurrent connections, using 127.0.0.1 address, you might have a problem...

Quoted 7 lines

Ideally the 2MSL parameter would be dynamically adjusted based on the route to the destination and the weights associated with those routes. In the simplest case, connections between machines on the same subnet (i.e., no router hops involved) should have a much smaller default value than connections that traverse any routers. I'd settle for a two-level setting - with no router hops, use the small value; with any router hops use the large value.

Well, is it really a MSL problem ?

I did a small test (linux-2.6.21-rc1) and was able to get 1.000.000 connections on localhost on my dual proc machine in one minute, without an error.

Eric Dumazet

2007-03-05 15:09 UTC #2 lore raw

PATCH 1/1 twcal_jiffie should be unsigned long, not int

Hi David

While browsing include/net/inet_timewait_sock.h, I found this buggy definition of twcal_jiffie.

int twcal_jiffie;

I wonder how inet_twdr_twcal_tick() can really works on x86_64

This seems quite an old bug, it was there before introduction of inet_timewait_death_row made by Arnaldo Carvalho de Melo.

[PATCH] twcal_jiffie should be unsigned long, not int

Signed-off-by Eric Dumazet <[email protected]>

include/net/inet_timewait_sock.h +1 -1

index f7be1ac..09a2532 100644
@@ -66,7 +66,7 @@ #define INET_TWDR_TWKILL_QUOTA 100
 struct inet_timewait_death_row {
 	/* Short-time timewait calendar */
 	int			twcal_hand;
-	int			twcal_jiffie;
+	unsigned long		twcal_jiffie;
 	struct timer_list	twcal_timer;
 	struct hlist_head	twcal_row[INET_TWDR_RECYCLE_SLOTS];

David Miller

maintainer

2007-03-05 21:33 UTC #3 lore raw

Re: [PATCH] twcal_jiffie should be unsigned long, not int

From: Eric Dumazet <[email protected]>
Date: Mon, 5 Mar 2007 16:09:21 +0100

Quoted 5 lines

While browsing include/net/inet_timewait_sock.h, I found this buggy definition of twcal_jiffie.
int twcal_jiffie;

Quoted 8 lines

I wonder how inet_twdr_twcal_tick() can really works on x86_64
This seems quite an old bug, it was there before introduction of inet_timewait_death_row made by Arnaldo Carvalho de Melo.
[PATCH] twcal_jiffie should be unsigned long, not int
Signed-off-by Eric Dumazet <[email protected]>

Grrr, good catch Eric. I'll push this fix to -stable too.

Thanks a lot.

Howard Chu

author

2007-03-06 10:00 UTC #4 lore raw

In reply to Eric Dumazet

Quoted 11 lines · Eric Dumazet

Quoted 6 lines · Howard Chu
Why is the Maximum Segment Lifetime a global parameter? Surely the maximum possible lifetime of a particular TCP segment depends on the actual connection. At the very least, it would be useful to be able to set it on a per-interface basis. E.g., in the case of the loopback interface, it would be useful to be able to set it to a very small duration.
Hi Howard
I think you should address these questions on netdev instead of linux-kernel.

OK, I just subscribed to netdev...

Quoted 7 lines

Quoted 7 lines
As I note in this draft
http://www.ietf.org/internet-drafts/draft-chu-ldap-ldapi-00.txt
when doing a connection soak test of OpenLDAP using clients connected through localhost, the entire port range is exhausted in well under a second, at which point the test stalls until a port comes out of TIME_WAIT state so the next connection can be opened.

Quoted 8 lines

Quoted 6 lines
These days it's not uncommon for an OpenLDAP slapd server to handle tens of thousands of connections per second in real use (e.g., at Google, or at various telcos). While the LDAP server is fast enough to saturate even 10gbit ethernet using contemporary CPUs, we have to resort to multiple virtual interfaces just to make sure we have enough port numbers available.
I dont uderstand... doesnt slapd server listen for connections on a given port, like http ? Or is it doing connections like a ftp server ?

No, you're right, it listens on a single port. There is a standard port (389) though of course you can use any port you want.

Quoted 2 lines

Of course, if you want to open more than 60.000 concurrent connections, using 127.0.0.1 address, you might have a problem...

This is probably not something that happens in real world deployments. I But it's not 60,000 concurrent connections, it's 60,000 within a 2 minute span.

I'm not saying this is a high priority problem, I only encountered it in a test scenario where I was deliberately trying to max out the server.

Quoted 11 lines

Quoted 7 lines
Ideally the 2MSL parameter would be dynamically adjusted based on the route to the destination and the weights associated with those routes. In the simplest case, connections between machines on the same subnet (i.e., no router hops involved) should have a much smaller default value than connections that traverse any routers. I'd settle for a two-level setting - with no router hops, use the small value; with any router hops use the large value.
Well, is it really a MSL problem ?
I did a small test (linux-2.6.21-rc1) and was able to get 1.000.000 connections on localhost on my dual proc machine in one minute, without an error.

It's a combination of 2MSL and /proc/sys/net/ipv4/ip_local_port_range - on my system the default port range is 32768-61000. That means if I use up 28232 ports in less than 2MSL then everything stops. netstat will show that all the available port numbers are in TIME_WAIT state. And this is particularly bad because while waiting for the timeout, I can't initiate any new outbound connections of any kind at all - telnet, ssh, whatever, you have to wait for at least one port to free up. (Interesting denial of service there....)

Granted, I was running my test on 2.6.18, perhaps 2.6.21 behaves differently.

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Eric Dumazet

2007-03-06 10:42 UTC #5 lore raw

Quoted 10 lines · Howard Chu

It's a combination of 2MSL and /proc/sys/net/ipv4/ip_local_port_range - on my system the default port range is 32768-61000. That means if I use up 28232 ports in less than 2MSL then everything stops. netstat will show that all the available port numbers are in TIME_WAIT state. And this is particularly bad because while waiting for the timeout, I can't initiate any new outbound connections of any kind at all - telnet, ssh, whatever, you have to wait for at least one port to free up. (Interesting denial of service there....)

Quoted 2 lines

Granted, I was running my test on 2.6.18, perhaps 2.6.21 behaves differently.

Could you try this attached program and tell me whats happen ?

$ gcc -O2 -o socktest socktest.c -lpthread
$ time ./socktest -n 100000
nb_conn=99999 nb_accp=99999

real    0m5.058s
user    0m0.212s
sys     0m4.844s

(on my small machine, dell d610 :) )

/*
Copyright (C) 2007 Eric Dumazet

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
*/
#include <pthread.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/poll.h>
#include <sys/sendfile.h>
#include <sys/epoll.h>

#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
#include <time.h>
#include <ctype.h>
#include <netdb.h>

int fd __attribute__((aligned(64)));
int port = 9999;
unsigned long nb_acc __attribute__((aligned(64))); unsigned long nb_conn1 __attribute__((aligned(64))); unsigned long nb_conn2 __attribute__((aligned(64))); unsigned long nb_conn3 __attribute__((aligned(64))); int limit = 10000/3;

void *do_accept(void *arg)
{
int s;
struct sockaddr_in sa;
socklen_t addrlen ;
int flags;
char buffer[1024];

while (1) {
	addrlen = sizeof(sa);
	s = accept(fd, (struct sockaddr *)&sa, &addrlen);
	if (s == -1) continue;
	flags = 0;
	recv(s, buffer, 1024, 0);
	send(s, "Answer\r\n", 8, 0);
	close(s);
	nb_acc++;
	}
}

void *do_conn(void *arg)
{
int i;
int on = 1;
struct sockaddr_in sa;
unsigned long *cpt = (unsigned long *)arg;

for (i = 0 ; i < limit ; i++) {
	int s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
	int res;
	setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, 4);
	memset(&sa, 0, sizeof(sa));
	sa.sin_addr.s_addr = htonl(0x7f000001);
	sa.sin_port = htons(port);
	sa.sin_family = AF_INET;
	res = connect(s, (struct sockaddr *)&sa, sizeof(sa));
	if (res == 0) {
		char buffer[1024];
		send(s, "question\r\n", 10, 0);
		recv(s, buffer, sizeof(buffer), 0);
		(*cpt)++;
		}
	else {
		static int errcnt = 0;
		if (errcnt++ < 10) printf("connect error %d\n", errno);
		}
	close(s);
	}
}

int main(int argc, char *argv[])
{
int on = 1;
struct sockaddr_in sa;
pthread_t tid, tid1, tid2, tid3;
int i;
void *res;

while ((i = getopt(argc, argv, "Vn:")) != EOF) {
	if (i == 'n')
		limit = atoi(optarg) / 3;
	}

fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, 4); memset(&sa, 0, sizeof(sa));
sa.sin_port = htons(port);
sa.sin_family = AF_INET;

if (bind(fd, (struct sockaddr *)&sa, sizeof(sa)) == -1) {
	perror("bind");
	return 1;
	}

listen(fd, 30000);
pthread_create(&tid, NULL, do_accept, NULL); pthread_create(&tid1, NULL, do_conn, &nb_conn1); pthread_create(&tid2, NULL, do_conn, &nb_conn2); pthread_create(&tid3, NULL, do_conn, &nb_conn3); pthread_join(tid1, &res);
pthread_join(tid2, &res);
pthread_join(tid3, &res);
printf("nb_conn=%lu nb_accp=%lu\n", nb_conn1 + nb_conn2 + nb_conn3, nb_acc); return 0;
}

Howard Chu

author

2007-03-06 18:45 UTC #6 lore raw

Quoted 20 lines · Eric Dumazet

Quoted 11 lines · Howard Chu
It's a combination of 2MSL and /proc/sys/net/ipv4/ip_local_port_range - on my system the default port range is 32768-61000. That means if I use up 28232 ports in less than 2MSL then everything stops. netstat will show that all the available port numbers are in TIME_WAIT state. And this is particularly bad because while waiting for the timeout, I can't initiate any new outbound connections of any kind at all - telnet, ssh, whatever, you have to wait for at least one port to free up. (Interesting denial of service there....)
Granted, I was running my test on 2.6.18, perhaps 2.6.21 behaves differently.
Could you try this attached program and tell me whats happen ?
$ gcc -O2 -o socktest socktest.c -lpthread
$ time ./socktest -n 100000
nb_conn=99999 nb_accp=99999

Quoted 5 lines

real    0m5.058s
user    0m0.212s
sys     0m4.844s
(on my small machine, dell d610 :) )

On my Asus laptop (2GHz Pentium M) the first time I ran it it completed in about 51 seconds, with no errors. I then copied it to another machine and started it up there, and got connect errors right away. I then went back to my laptop and ran it again, and got errors that time.

This is the laptop run with errors:
viola:~/src> uname -a
Linux viola 2.6.18.2-34-default #1 SMP Mon Nov 27 11:46:27 UTC 2006 i686 i686 i386 GNU/Linux viola:~/src> time ./socktest -n 1000000
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
nb_conn=993757 nb_accp=993757
1.408u 88.649s 1:42.76 87.6% 0+0k 0+0io 0pf+0w

This is my other system, an AMD X2 3800+ (dual core)
mandolin:~/src> uname -a
Linux mandolin 2.6.18.3SMP #9 SMP Sat Nov 25 10:08:51 PST 2006 x86_64 x86_64 x86_64 GNU/Linux mandolin:~/src> gcc -O2 -o socktest socktest.c -lpthread
mandolin:~/src> time ./socktest -n 1000000
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
nb_conn=957088 nb_accp=957088
1.012u 630.991s 5:18.05 198.7% 0+0k 0+0io 0pf+0w

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Eric Dumazet

2007-03-06 20:08 UTC #7 lore raw

Quoted 55 lines · Howard Chu

Quoted 20 lines · Eric Dumazet
Quoted 11 lines · Howard Chu
It's a combination of 2MSL and /proc/sys/net/ipv4/ip_local_port_range - on my system the default port range is 32768-61000. That means if I use up 28232 ports in less than 2MSL then everything stops. netstat will show that all the available port numbers are in TIME_WAIT state. And this is particularly bad because while waiting for the timeout, I can't initiate any new outbound connections of any kind at all - telnet, ssh, whatever, you have to wait for at least one port to free up. (Interesting denial of service there....)
Granted, I was running my test on 2.6.18, perhaps 2.6.21 behaves differently.
Could you try this attached program and tell me whats happen ?
$ gcc -O2 -o socktest socktest.c -lpthread
$ time ./socktest -n 100000
nb_conn=99999 nb_accp=99999
Quoted 5 lines
real    0m5.058s
user    0m0.212s
sys     0m4.844s
(on my small machine, dell d610 :) )
On my Asus laptop (2GHz Pentium M) the first time I ran it it completed in about 51 seconds, with no errors. I then copied it to another machine and started it up there, and got connect errors right away. I then went back to my laptop and ran it again, and got errors that time.
This is the laptop run with errors:
viola:~/src> uname -a
Linux viola 2.6.18.2-34-default #1 SMP Mon Nov 27 11:46:27 UTC 2006 i686 i686 i386 GNU/Linux viola:~/src> time ./socktest -n 1000000
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
nb_conn=993757 nb_accp=993757
1.408u 88.649s 1:42.76 87.6% 0+0k 0+0io 0pf+0w
This is my other system, an AMD X2 3800+ (dual core)
mandolin:~/src> uname -a
Linux mandolin 2.6.18.3SMP #9 SMP Sat Nov 25 10:08:51 PST 2006 x86_64 x86_64 x86_64 GNU/Linux mandolin:~/src> gcc -O2 -o socktest socktest.c -lpthread
mandolin:~/src> time ./socktest -n 1000000
connect error 99
connect error 99
connect error 99
connect error 99

Quoted 8 lines

connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
connect error 99
nb_conn=957088 nb_accp=957088
1.012u 630.991s 5:18.05 198.7% 0+0k 0+0io 0pf+0w

Let me see, any chance you can try the prog on 2.6.20 ?

If not, please send :

grep . /proc/sys/net/ipv4/*

Thank you

Howard Chu

author

2007-03-06 20:34 UTC #8 lore raw

Quoted 1 line · Eric Dumazet

Let me see, any chance you can try the prog on 2.6.20 ?

Not any time soon.

Quoted 4 lines

If not, please send :
grep . /proc/sys/net/ipv4/*

This is the output on the laptop:
/proc/sys/net/ipv4/icmp_echo_ignore_all:0 /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts:1 /proc/sys/net/ipv4/icmp_errors_use_inbound_ifaddr:0 /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses:1 /proc/sys/net/ipv4/icmp_ratelimit:250 /proc/sys/net/ipv4/icmp_ratemask:6168 /proc/sys/net/ipv4/igmp_max_memberships:20 /proc/sys/net/ipv4/igmp_max_msf:10 /proc/sys/net/ipv4/inet_peer_gc_maxtime:120 /proc/sys/net/ipv4/inet_peer_gc_mintime:10 /proc/sys/net/ipv4/inet_peer_maxttl:600 /proc/sys/net/ipv4/inet_peer_minttl:120 /proc/sys/net/ipv4/inet_peer_threshold:65664 /proc/sys/net/ipv4/ip_default_ttl:64 /proc/sys/net/ipv4/ip_dynaddr:0 /proc/sys/net/ipv4/ip_forward:0 /proc/sys/net/ipv4/ipfrag_high_thresh:262144 /proc/sys/net/ipv4/ipfrag_low_thresh:196608 /proc/sys/net/ipv4/ipfrag_max_dist:64 /proc/sys/net/ipv4/ipfrag_secret_interval:600 /proc/sys/net/ipv4/ipfrag_time:30
/proc/sys/net/ipv4/ip_local_port_range:32768 61000
/proc/sys/net/ipv4/ip_nonlocal_bind:0 /proc/sys/net/ipv4/ip_no_pmtu_disc:0 /proc/sys/net/ipv4/tcp_abc:0 /proc/sys/net/ipv4/tcp_abort_on_overflow:0 /proc/sys/net/ipv4/tcp_adv_win_scale:2 /proc/sys/net/ipv4/tcp_app_win:31 /proc/sys/net/ipv4/tcp_base_mss:512 /proc/sys/net/ipv4/tcp_congestion_control:reno /proc/sys/net/ipv4/tcp_dma_copybreak:4096 /proc/sys/net/ipv4/tcp_dsack:1 /proc/sys/net/ipv4/tcp_ecn:0 /proc/sys/net/ipv4/tcp_fack:1 /proc/sys/net/ipv4/tcp_fin_timeout:60 /proc/sys/net/ipv4/tcp_frto:0 /proc/sys/net/ipv4/tcp_keepalive_intvl:75 /proc/sys/net/ipv4/tcp_keepalive_probes:9 /proc/sys/net/ipv4/tcp_keepalive_time:7200 /proc/sys/net/ipv4/tcp_low_latency:0 /proc/sys/net/ipv4/tcp_max_orphans:32768 /proc/sys/net/ipv4/tcp_max_syn_backlog:1024 /proc/sys/net/ipv4/tcp_max_tw_buckets:180000
/proc/sys/net/ipv4/tcp_mem:98304 131072 196608
/proc/sys/net/ipv4/tcp_moderate_rcvbuf:1 /proc/sys/net/ipv4/tcp_mtu_probing:0 /proc/sys/net/ipv4/tcp_no_metrics_save:0 /proc/sys/net/ipv4/tcp_orphan_retries:0 /proc/sys/net/ipv4/tcp_reordering:3 /proc/sys/net/ipv4/tcp_retrans_collapse:1 /proc/sys/net/ipv4/tcp_retries1:3 /proc/sys/net/ipv4/tcp_retries2:15 /proc/sys/net/ipv4/tcp_rfc1337:0
/proc/sys/net/ipv4/tcp_rmem:4096 87380 4194304
/proc/sys/net/ipv4/tcp_sack:1 /proc/sys/net/ipv4/tcp_slow_start_after_idle:1 /proc/sys/net/ipv4/tcp_stdurg:0 /proc/sys/net/ipv4/tcp_synack_retries:5 /proc/sys/net/ipv4/tcp_syncookies:1 /proc/sys/net/ipv4/tcp_syn_retries:5 /proc/sys/net/ipv4/tcp_timestamps:1 /proc/sys/net/ipv4/tcp_tso_win_divisor:3 /proc/sys/net/ipv4/tcp_tw_recycle:0 /proc/sys/net/ipv4/tcp_tw_reuse:0 /proc/sys/net/ipv4/tcp_window_scaling:1
/proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304
/proc/sys/net/ipv4/tcp_workaround_signed_windows:0

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Eric Dumazet

2007-03-06 20:39 UTC #9 lore raw

Quoted 69 lines · Howard Chu

Quoted 1 line · Eric Dumazet
Let me see, any chance you can try the prog on 2.6.20 ?
Not any time soon.
Quoted 4 lines
If not, please send :
grep . /proc/sys/net/ipv4/*
This is the output on the laptop:
/proc/sys/net/ipv4/icmp_echo_ignore_all:0 /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts:1 /proc/sys/net/ipv4/icmp_errors_use_inbound_ifaddr:0 /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses:1 /proc/sys/net/ipv4/icmp_ratelimit:250 /proc/sys/net/ipv4/icmp_ratemask:6168 /proc/sys/net/ipv4/igmp_max_memberships:20 /proc/sys/net/ipv4/igmp_max_msf:10 /proc/sys/net/ipv4/inet_peer_gc_maxtime:120 /proc/sys/net/ipv4/inet_peer_gc_mintime:10 /proc/sys/net/ipv4/inet_peer_maxttl:600 /proc/sys/net/ipv4/inet_peer_minttl:120 /proc/sys/net/ipv4/inet_peer_threshold:65664 /proc/sys/net/ipv4/ip_default_ttl:64 /proc/sys/net/ipv4/ip_dynaddr:0 /proc/sys/net/ipv4/ip_forward:0 /proc/sys/net/ipv4/ipfrag_high_thresh:262144 /proc/sys/net/ipv4/ipfrag_low_thresh:196608 /proc/sys/net/ipv4/ipfrag_max_dist:64 /proc/sys/net/ipv4/ipfrag_secret_interval:600 /proc/sys/net/ipv4/ipfrag_time:30
/proc/sys/net/ipv4/ip_local_port_range:32768 61000
/proc/sys/net/ipv4/ip_nonlocal_bind:0 /proc/sys/net/ipv4/ip_no_pmtu_disc:0 /proc/sys/net/ipv4/tcp_abc:0 /proc/sys/net/ipv4/tcp_abort_on_overflow:0 /proc/sys/net/ipv4/tcp_adv_win_scale:2 /proc/sys/net/ipv4/tcp_app_win:31 /proc/sys/net/ipv4/tcp_base_mss:512 /proc/sys/net/ipv4/tcp_congestion_control:reno /proc/sys/net/ipv4/tcp_dma_copybreak:4096 /proc/sys/net/ipv4/tcp_dsack:1 /proc/sys/net/ipv4/tcp_ecn:0 /proc/sys/net/ipv4/tcp_fack:1 /proc/sys/net/ipv4/tcp_fin_timeout:60 /proc/sys/net/ipv4/tcp_frto:0 /proc/sys/net/ipv4/tcp_keepalive_intvl:75 /proc/sys/net/ipv4/tcp_keepalive_probes:9 /proc/sys/net/ipv4/tcp_keepalive_time:7200 /proc/sys/net/ipv4/tcp_low_latency:0 /proc/sys/net/ipv4/tcp_max_orphans:32768 /proc/sys/net/ipv4/tcp_max_syn_backlog:1024 /proc/sys/net/ipv4/tcp_max_tw_buckets:180000
/proc/sys/net/ipv4/tcp_mem:98304 131072 196608
/proc/sys/net/ipv4/tcp_moderate_rcvbuf:1 /proc/sys/net/ipv4/tcp_mtu_probing:0 /proc/sys/net/ipv4/tcp_no_metrics_save:0 /proc/sys/net/ipv4/tcp_orphan_retries:0 /proc/sys/net/ipv4/tcp_reordering:3 /proc/sys/net/ipv4/tcp_retrans_collapse:1 /proc/sys/net/ipv4/tcp_retries1:3 /proc/sys/net/ipv4/tcp_retries2:15 /proc/sys/net/ipv4/tcp_rfc1337:0
/proc/sys/net/ipv4/tcp_rmem:4096 87380 4194304
/proc/sys/net/ipv4/tcp_sack:1 /proc/sys/net/ipv4/tcp_slow_start_after_idle:1 /proc/sys/net/ipv4/tcp_stdurg:0 /proc/sys/net/ipv4/tcp_synack_retries:5 /proc/sys/net/ipv4/tcp_syncookies:1

Quoted 8 lines

/proc/sys/net/ipv4/tcp_syn_retries:5 /proc/sys/net/ipv4/tcp_timestamps:1 /proc/sys/net/ipv4/tcp_tso_win_divisor:3 /proc/sys/net/ipv4/tcp_tw_recycle:0 /proc/sys/net/ipv4/tcp_tw_reuse:0 /proc/sys/net/ipv4/tcp_window_scaling:1
/proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304
/proc/sys/net/ipv4/tcp_workaround_signed_windows:0

Arf... dont tell me you forgot to do this...

echo 1 >/proc/sys/net/ipv4/tcp_tw_recycle
echo 1 >/proc/sys/net/ipv4/tcp_tw_reuse

Howard Chu

author

2007-03-06 21:11 UTC #10 lore raw

Quoted 4 lines · Eric Dumazet

Arf... dont tell me you forgot to do this...
echo 1 >/proc/sys/net/ipv4/tcp_tw_recycle
echo 1 >/proc/sys/net/ipv4/tcp_tw_reuse

That does not appear to me to be a safe thing to do on a production machine. Tweaks that are only good in a test environment really don't help the testing effort; they just mask a problem that will surface later at deployment time.

We could run our benchmarks this way and get high rates but no one deploying the server for real use would ever get anything like that, which makes the benchmark figure rather pointless.

On the other hand, being able to configure a small MSL for the loopback device is perfectly safe. Being able to configure a small MSL for other interfaces may be safe, depending on the rest of the network layout.

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Rick Jones

2007-03-06 21:25 UTC #11 lore raw

Quoted 1 line

On the other hand, being able to configure a small MSL for the loopback device is perfectly safe. Being able to configure a small MSL for other interfaces may be safe, depending on the rest of the network layout.

A peanut gallery question - I seem to recall prior discussions about how one cannot assume that a packet destined for a given IP address will remain detined for that given IP address as it could go through a module that will rewrite headers etc.

Is traffic destined for 127.0.0.1 immune from that?

rick jones

David Miller

maintainer

2007-03-06 21:35 UTC #12 lore raw

From: Rick Jones <[email protected]>
Date: Tue, 06 Mar 2007 13:25:35 -0800

Quoted 8 lines

Quoted 3 lines
On the other hand, being able to configure a small MSL for the loopback device is perfectly safe. Being able to configure a small MSL for other interfaces may be safe, depending on the rest of the network layout.
A peanut gallery question - I seem to recall prior discussions about how one cannot assume that a packet destined for a given IP address will remain detined for that given IP address as it could go through a module that will rewrite headers etc.

That's right, both netfilter and the packet scheduler actions can do that, that's why this whole idea about changing the MSL on loopback by default is wrong and pointless.

Howard Chu

author

2007-03-06 22:12 UTC #13 lore raw

Quoted 9 lines · David Miller

From: Rick Jones <[email protected]>
Date: Tue, 06 Mar 2007 13:25:35 -0800
Quoted 2 lines
Quoted 1 line
On the other hand, being able to configure a small MSL for the loopback device is perfectly safe. Being able to configure a small MSL for other interfaces may be safe, depending on the rest of the network layout.
A peanut gallery question - I seem to recall prior discussions about how one cannot assume that a packet destined for a given IP address will remain detined for that given IP address as it could go through a module that will rewrite headers etc.
That's right, both netfilter and the packet scheduler actions can do that, that's why this whole idea about changing the MSL on loopback by default is wrong and pointless.

If the headers get rewritten and the packet gets directed elsewhere, then we're no longer talking about a loopback connection, so that's outside the discussion.

If the packet gets munged by multiple filters but still eventually gets to the specified destination, OK. But regardless, if both endpoints of the connection are on the loopback device, then there is nothing wrong with the idea. Those filters can only do so much, they still have to preserve the reliable in-order delivery semantics of TCP, otherwise the system is broken.

It may not have much use, sure, I admitted that much from the outset.

So I'll leave it at this, thanks for the feedback.

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Stephen Hemminger

2007-03-06 22:55 UTC #14 lore raw

Quoted 27 lines · On Tue, 06 Mar 2007 14:07:09 -0800 Howard Chu <[email protected]> wrote:

Quoted 6 lines · David Miller
From: Rick Jones <[email protected]>
Date: Tue, 06 Mar 2007 13:25:35 -0800
Quoted 3 lines
Quoted 3 lines
On the other hand, being able to configure a small MSL for the loopback device is perfectly safe. Being able to configure a small MSL for other interfaces may be safe, depending on the rest of the network layout.
Quoted 8 lines
Quoted 4 lines
A peanut gallery question - I seem to recall prior discussions about how one cannot assume that a packet destined for a given IP address will remain detined for that given IP address as it could go through a module that will rewrite headers etc.
That's right, both netfilter and the packet scheduler actions can do that, that's why this whole idea about changing the MSL on loopback by default is wrong and pointless.
If the headers get rewritten and the packet gets directed elsewhere, then we're no longer talking about a loopback connection, so that's outside the discussion.
If the packet gets munged by multiple filters but still eventually gets to the specified destination, OK. But regardless, if both endpoints of the connection are on the loopback device, then there is nothing wrong with the idea. Those filters can only do so much, they still have to preserve the reliable in-order delivery semantics of TCP, otherwise the system is broken.

Quoted 3 lines

It may not have much use, sure, I admitted that much from the outset.
So I'll leave it at this, thanks for the feedback.

TCP can not assume anything about the path that a packet may take. We have declared a moratorium on loopback benchmark foolishness. Go optimize the idle loop instead ;-)

Signature · 1 line

Stephen Hemminger <[email protected]>

Howard Chu

author

2007-03-06 23:28 UTC #15 lore raw

Quoted 3 lines · Stephen Hemminger

TCP can not assume anything about the path that a packet may take. We have declared a moratorium on loopback benchmark foolishness. Go optimize the idle loop instead ;-)

Sure - A delay loop with fewer instructions is a worthwhile optimization because it has less impact on a CPU's instruction cache...

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

David Miller

maintainer

2007-03-06 18:04 UTC #16 lore raw

In reply to Howard Chu

From: Howard Chu <[email protected]>
Date: Tue, 06 Mar 2007 01:22:18 -0800

Quoted 1 line

OK, I just subscribed to netdev...

Unlike other mailing lists you don't have to subscribe to netdev in order to post to it and ask questions :-)

Rick Jones

2007-03-06 18:46 UTC #17 lore raw

In reply to Howard Chu

Quoted 1 line

This is probably not something that happens in real world deployments. I But it's not 60,000 concurrent connections, it's 60,000 within a 2 minute span.

Sounds like a case of Doctor! Doctor! It hurts when I do this.

Quoted 10 lines

I'm not saying this is a high priority problem, I only encountered it in a test scenario where I was deliberately trying to max out the server.
Quoted 7 lines
Quoted 7 lines
Ideally the 2MSL parameter would be dynamically adjusted based on the route to the destination and the weights associated with those routes. In the simplest case, connections between machines on the same subnet (i.e., no router hops involved) should have a much smaller default value than connections that traverse any routers. I'd settle for a two-level setting - with no router hops, use the small value; with any router hops use the large value.

With transparant bridging, nobody knows how long the datagram may be out there. Admittedly, the chances of a datagram living for a full two minutes these days is probably nil, but just being in the same IP subnet doesn't really mean anything when it comes to physical locality.

Quoted 1 line

It's a combination of 2MSL and /proc/sys/net/ipv4/ip_local_port_range - on my system the default port range is 32768-61000. That means if I use up 28232 ports in less than 2MSL then everything stops. netstat will show that all the available port numbers are in TIME_WAIT state. And this is particularly bad because while waiting for the timeout, I can't initiate any new outbound connections of any kind at all - telnet, ssh, whatever, you have to wait for at least one port to free up. (Interesting denial of service there....)

SPECweb benchmarking has had to deal with the issue of attempted TIME_WAIT reuse going back to 1997. It deals with it by not relying on the client's configured local/anonymous/ephemeral port number range and instead making explicit bind() calls in the (more or less) entire unpriv port range (actually it may just be from 5000 to 65535 but still)

Now, if it weren't necessary to fully randomize the ISNs, the chances of a successful transition from TIME_WAIT to ESTABLISHED might be greater, but going back to the good old days of more or less purly clock driven ISN's isn't likely.

rick jones

Howard Chu

author

2007-03-06 19:30 UTC #18 lore raw

Quoted 3 lines · Rick Jones

Quoted 1 line
This is probably not something that happens in real world deployments. I But it's not 60,000 concurrent connections, it's 60,000 within a 2 minute span.
Sounds like a case of Doctor! Doctor! It hurts when I do this.

I guess. In the cases where it matters, we use LDAP over Unix Domain Sockets instead of TCP. Smarter clients that do connection pooling would help too, but the fact that this even came to our attention is because not all clients out there are smart enough.

Since we have an alternative that works, I'm not really worried about it. I just thought it was worthwhile to raise the question.

Quoted 11 lines

Quoted 9 lines
I'm not saying this is a high priority problem, I only encountered it in a test scenario where I was deliberately trying to max out the server.
Quoted 7 lines
Quoted 7 lines
Ideally the 2MSL parameter would be dynamically adjusted based on the route to the destination and the weights associated with those routes. In the simplest case, connections between machines on the same subnet (i.e., no router hops involved) should have a much smaller default value than connections that traverse any routers. I'd settle for a two-level setting - with no router hops, use the small value; with any router hops use the large value.
With transparant bridging, nobody knows how long the datagram may be out there. Admittedly, the chances of a datagram living for a full two minutes these days is probably nil, but just being in the same IP subnet doesn't really mean anything when it comes to physical locality.

Bridging isn't necessarily a problem though. The 2MSL timeout is designed to prevent problems from delayed packets that got sent through multiple paths. In a bridging setup you don't allow multiple paths, that's what STP is designed to prevent. If you want to configure a network that allows multiple paths, you need to use a router, not a bridge.

Quoted 1 line

SPECweb benchmarking has had to deal with the issue of attempted TIME_WAIT reuse going back to 1997. It deals with it by not relying on the client's configured local/anonymous/ephemeral port number range and instead making explicit bind() calls in the (more or less) entire unpriv port range (actually it may just be from 5000 to 65535 but still)

That still doesn't solve the problem, it only ~doubles the available port range. That means it takes 0.6 seconds to trigger the problem instead of only 0.3 seconds...

Quoted 1 line

Now, if it weren't necessary to fully randomize the ISNs, the chances of a successful transition from TIME_WAIT to ESTABLISHED might be greater, but going back to the good old days of more or less purly clock driven ISN's isn't likely.

In an environment where connections are opened and closed very quickly with only a small amount of data carried per connection, it might make sense to remember the last sequence number used on a port and use that as the floor of the next randomly generated ISN. Monotonically increasing sequence numbers aren't a security risk if there's still a randomly determined gap from one connection to the next. But I don't think it's necessary to consider this at the moment.

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

Rick Jones

2007-03-06 20:41 UTC #19 lore raw

Quoted 4 lines

Quoted 1 line
With transparant bridging, nobody knows how long the datagram may be out there. Admittedly, the chances of a datagram living for a full two minutes these days is probably nil, but just being in the same IP subnet doesn't really mean anything when it comes to physical locality.
Bridging isn't necessarily a problem though. The 2MSL timeout is designed to prevent problems from delayed packets that got sent through multiple paths. In a bridging setup you don't allow multiple paths, that's what STP is designed to prevent. If you want to configure a network that allows multiple paths, you need to use a router, not a bridge.

Well, there is trunking at the data link layer, and in theory there could be an active-standby where the standby took a somewhat different path.

The timeout is also to cover datagrams which just got "stuck" somewhere too (IIRC) and may not necessarily require a multiple path situation.

Quoted 5 lines

Quoted 2 lines
SPECweb benchmarking has had to deal with the issue of attempted TIME_WAIT reuse going back to 1997. It deals with it by not relying on the client's configured local/anonymous/ephemeral port number range and instead making explicit bind() calls in the (more or less) entire unpriv port range (actually it may just be from 5000 to 65535 but still)
That still doesn't solve the problem, it only ~doubles the available port range. That means it takes 0.6 seconds to trigger the problem instead of only 0.3 seconds...

True. Thankfully, the web learned to use persistent connections so later versions of SPECweb benchmarking make use of persistent connections.

Quoted 1 line

In an environment where connections are opened and closed very quickly with only a small amount of data carried per connection, it might make sense to remember the last sequence number used on a port and use that as the floor of the next randomly generated ISN. Monotonically increasing sequence numbers aren't a security risk if there's still a randomly determined gap from one connection to the next. But I don't think it's necessary to consider this at the moment.

I thought that all the "security types" started squawking if the ISN wasn't completely random?

I've not tried this, but if a client does want to cycle through thousands of connections per second, and if it is the one to initiate connection close, would it be sufficient to only use something like:

socket()
bind()
loop:
connect()
request()
response()
shudtown(SHUT_RDWR)
goto loop

ie not call close on the FD so there is still a direct link to the connection in TIME_WAIT so one could in theory initiate a new connection from TIME_WAIT? Then in theory the randomness could be _almost_ the entire sequence space, less the previous connection's window (IIRC).

rick jones

Howard Chu

author

2007-03-07 03:42 UTC #20 lore raw

Quoted 1 line · Rick Jones

The timeout is also to cover datagrams which just got "stuck" somewhere too (IIRC) and may not necessarily require a multiple path situation.

I guess that's a fair point. Originally, the only possible place for a packet to get "stuck" was in a router but I suppose that may no longer be true.

Quoted 1 line

True. Thankfully, the web learned to use persistent connections so later versions of SPECweb benchmarking make use of persistent connections.

As a complete aside, I think it's about time for a SPECldap benchmark...

Signature · 4 lines

  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

David Miller

maintainer

2007-03-05 20:59 UTC #21 lore raw

In reply to Howard Chu

You might want to post networking questions to the networking developer mailing list, [email protected], instead of linux-kernel which most of the networking developers do not read.

Thank you.