Skip to contentSkip to conversation

NFSD: Remove NFSD_TCP kernel build option

PATCH In review Chuck Lever 6 participants 12 messages Feb 5, 2008 7 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 4/5 NFSD: Remove NFSD_TCP kernel build option

TCP support in the Linux NFS server is stable enough that we can leave it on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to "Y" anyway.

A run-time switch might be more appropriate if people feel they would like to disable NFSD's TCP support.

---

 fs/Kconfig       |   10 ----------
 fs/nfsd/nfssvc.c |    2 --
 2 files changed, 0 insertions(+), 12 deletions(-)
fs/Kconfig +0 -10
index e6d4034..5ccff9a 100644
@@ -1686,7 +1686,6 @@ config NFSD
 	select EXPORTFS
 	select NFSD_V2_ACL if NFSD_V3_ACL
 	select NFS_ACL_SUPPORT if NFSD_V2_ACL
-	select NFSD_TCP if NFSD_V4
 	select CRYPTO_MD5 if NFSD_V4
 	select CRYPTO if NFSD_V4
 	select FS_POSIX_ACL if NFSD_V4
@@ -1763,15 +1762,6 @@ config NFSD_V4
 
 	  If unsure, say N.
 
-config NFSD_TCP
-	bool "Provide NFS server over TCP support"
-	depends on NFSD
-	default y
-	help
-	  If you want your NFS server to support TCP connections, say Y here.
-	  TCP connections usually perform better than the default UDP when
-	  the network is lossy or congested.  If unsure, say Y.
-
 config LOCKD
 	tristate
 
fs/nfsd/nfssvc.c +0 -2
index 9647b0f..941041f 100644
@@ -244,7 +244,6 @@ static int nfsd_init_socks(int port)
 	if (error < 0)
 		return error;
 
-#ifdef CONFIG_NFSD_TCP
 	error = lockd_up(IPPROTO_TCP);
 	if (error >= 0) {
 		error = svc_create_xprt(nfsd_serv, "tcp", port,
@@ -254,7 +253,6 @@ static int nfsd_init_socks(int port)
 	}
 	if (error < 0)
 		return error;
-#endif
 	return 0;
 }
 

A message that is not in the archive (sent off-list or not yet received): 20080205000442.18602.29035.stgit-meopP2rzCrTwdl/1UfZZQIVfYA8g3rJ/@public.gmane.org

#2 lore raw
Quoted 8 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to "Y" anyway.

A run-time switch might be more appropriate if people feel they would like to disable NFSD's TCP support.

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

A message that is not in the archive (sent off-list or not yet received): 47A7AB89.7020709-cP1dWloDopni96+mSzHFpQC/[email protected]

#4 lore raw
Quoted 5 lines · Greg Banks
Quoted 4 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to "Y" anyway.

Quoted 8 lines
Quoted 4 lines

A run-time switch might be more appropriate if people feel they would like to disable NFSD's TCP support.

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

That will break older clients.

A message that is not in the archive (sent off-list or not yet received): 1202170754.28484.57.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#6 lore raw
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Greg Banks
Quoted 9 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to "Y" anyway.

A run-time switch might be more appropriate if people feel they would like to disable NFSD's TCP support.

Quoted 9 lines
Quoted 5 lines

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

That will break older clients.

Hence the default, rather than removing the code entirely.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

A message that is not in the archive (sent off-list or not yet received): 47A7AE03.10401-cP1dWloDopni96+mSzHFpQC/[email protected]

#8 lore raw
Quoted 16 lines · Greg Banks
Quoted 15 lines · Trond Myklebust
Quoted 13 lines · Greg Banks
Quoted 9 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to "Y" anyway.

A run-time switch might be more appropriate if people feel they would like to disable NFSD's TCP support.

Looks good.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Actually, I'd be inclined to go one step further and set UDP support off by default.

That will break older clients.

Hence the default, rather than removing the code entirely.

What might make sense is to remove NFSD_TCP, but add NFSD_UDP, defaulting to Y.

Then in a year or two we can change the default to N.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#9 lore raw
Quoted 27 lines · Chuck Lever
Quoted 26 lines · Greg Banks
Quoted 25 lines · Trond Myklebust
Quoted 14 lines · Greg Banks
Quoted 6 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it
on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
"Y" anyway.

Quoted 6 lines

A run-time switch might be more appropriate if people feel they would like
to disable NFSD's TCP support.

Quoted 5 lines

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

That will break older clients.

Quoted 8 lines
Quoted 1 line

Hence the default, rather than removing the code entirely.

What might make sense is to remove NFSD_TCP, but add NFSD_UDP, defaulting to Y.

Then in a year or two we can change the default to N.

Fine by me.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.
#10 lore raw
Quoted 7 lines · On Tuesday February 5, gnb-cP1dWloDopni96+mSzHFpQC/[email protected]…
Quoted 6 lines

What might make sense is to remove NFSD_TCP, but add NFSD_UDP, defaulting to Y.

Then in a year or two we can change the default to N.

Fine by me.

I, on the other hand, think we should leave support for UDP and TCP in the kernel indefinitely and rely on the 'rpc.nfsd' tool to choose what to listen to.

NeilBrown

A message that is not in the archive (sent off-list or not yet received): 47A7F8F3.3020907-cP1dWloDopni96+mSzHFpQC/[email protected]

#12 lore raw
Quoted 31 lines · Greg Banks
Quoted 30 lines · Chuck Lever
Quoted 27 lines · Greg Banks
Quoted 16 lines · Trond Myklebust
Quoted 14 lines · Greg Banks
Quoted 12 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it
on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
"Y" anyway.

A run-time switch might be more appropriate if people feel they would like
to disable NFSD's TCP support.

Quoted 9 lines
Quoted 5 lines

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

That will break older clients.

Hence the default, rather than removing the code entirely.

Quoted 6 lines
Quoted 5 lines

What might make sense is to remove NFSD_TCP, but add NFSD_UDP, defaulting to Y.

Then in a year or two we can change the default to N.

Fine by me.

Last time I checked (around 2.6.22) writing large files on NFSv3 over UDP was 20% faster compared to TCP (Gb LAN with one switch connecting all machines).

TCP and its timeout/retransmission behavior isn't always the best choice.

Signature · 1 line
Frank
#13 lore raw
Quoted 5 lines · Frank van Maarseveen

Last time I checked (around 2.6.22) writing large files on NFSv3 over UDP was 20% faster compared to TCP (Gb LAN with one switch connecting all machines).

TCP and its timeout/retransmission behavior isn't always the best choice.

If your environment has only 1 client working against a dedicated NFS server on a clean network, then that may indeed be the case, but as soon as you have more than 1 client, TCP almost always ends up outperforming UDP.

Cheers
Trond

A message that is not in the archive (sent off-list or not yet received): 1202233839.8452.31.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#15 lore raw
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen

Last time I checked (around 2.6.22) writing large files on NFSv3 over UDP was 20% faster compared to TCP (Gb LAN with one switch connecting all machines).

Quoted 6 lines
Quoted 1 line

TCP and its timeout/retransmission behavior isn't always the best choice.

If your environment has only 1 client working against a dedicated NFS server on a clean network, then that may indeed be the case, but as soon as you have more than 1 client, TCP almost always ends up outperforming UDP.

There are multiple clients (12 per physical server on average) but the clients generally don't write (or read) at the same time. Anyway, I'll keep this in mind.

Signature · 1 line
Frank
#16 lore raw
In reply to Frank van Maarseveen
Quoted 45 lines · Frank van Maarseveen
Quoted 43 lines · Greg Banks
Quoted 41 lines · Chuck Lever
Quoted 27 lines · Greg Banks
Quoted 25 lines · Trond Myklebust
Quoted 22 lines · Greg Banks
Quoted 6 lines · Chuck Lever

TCP support in the Linux NFS server is stable enough that we can leave it
on always. CONFIG_NFSD_TCP adds about 10 lines of code, and defaults to
"Y" anyway.

Quoted 7 lines

A run-time switch might be more appropriate if people feel they would like
to disable NFSD's TCP support.

Looks good.

Actually, I'd be inclined to go one step further and set UDP support off by default.

Quoted 6 lines
Quoted 4 lines

That will break older clients.

Hence the default, rather than removing the code entirely.

What might make sense is to remove NFSD_TCP, but add NFSD_UDP, defaulting to Y.

Then in a year or two we can change the default to N.

Quoted 7 lines
Quoted 2 lines

Fine by me.

Last time I checked (around 2.6.22) writing large files on NFSv3 over UDP was 20% faster compared to TCP (Gb LAN with one switch connecting all machines).

Did all of your file arrive at the server, and in the same order it left the client? NFS on UDP relies on IP fragmentation, which is known to introduce silent data corruption at high data rates (google for "IPID aliasing").

Also, last time I checked, UDP support in the server uses a single socket for all traffic, and processes need to serialise on the svc_sock lock to send, so aggregate UDP throughput is strictly limited compared to TCP. As in, 145 MB/s for UDP compared to filling 12 1gige pipes for TCP. I have a patch to fix this, but given the inherent data corruption issues of UDP I haven't bothered posting the most recent version.

Quoted 3 lines

TCP and its timeout/retransmission behavior isn't always the best choice.

The timeout & retrans that sunrpc implements on top of UDP is arguably worse, especially if you use the "soft" mount option.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.

A message that is not in the archive (sent off-list or not yet received): 47A8EBC3.7050900-cP1dWloDopni96+mSzHFpQC/[email protected]

#18 lore raw
Quoted 8 lines · Greg Banks
Quoted 4 lines · Frank van Maarseveen

Last time I checked (around 2.6.22) writing large files on NFSv3 over UDP was 20% faster compared to TCP (Gb LAN with one switch connecting all machines).

Did all of your file arrive at the server, and in the same order it left the client? NFS on UDP relies on IP fragmentation, which is known to introduce silent data corruption at high data rates (google for "IPID aliasing").

The right query appears to be "IPID aliasing NFS", which (at least for me) gets you a nice explanation from Olaf's 2006 OLS paper as the first hit....

--b.

Quoted 20 lines

Also, last time I checked, UDP support in the server uses a single socket for all traffic, and processes need to serialise on the svc_sock lock to send, so aggregate UDP throughput is strictly limited compared to TCP. As in, 145 MB/s for UDP compared to filling 12 1gige pipes for TCP. I have a patch to fix this, but given the inherent data corruption issues of UDP I haven't bothered posting the most recent version.

Quoted 3 lines

TCP and its timeout/retransmission behavior isn't always the best choice.

The timeout & retrans that sunrpc implements on top of UDP is arguably worse, especially if you use the "soft" mount option.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.