Skip to contentSkip to conversation

Side effects of having NFSv4 mounted over udp

DISCUSSION DENIEL Philippe 4 participants 6 messages Dec 13, 2010 – Dec 14, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?

Thanks in advance for your answer.

Regards

Philippe

#1 lore raw
Quoted 3 lines · DENIEL Philippe

Hi,

I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?

You might find that some implementations have UDP disabled and it might then make it hard to triage what is going on.

I believe both NetApp's OnTap and Oracle's Solaris have it turned off.

Quoted 10 lines

Thanks in advance for your answer.

Regards

Philippe
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 5 lines · Thomas Haynes
Quoted 2 lines · DENIEL Philippe

Hi,

Quoted 7 lines
Quoted 1 line

I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?

You might find that some implementations have UDP disabled and it might then make it hard to triage what is going on.

I believe both NetApp's OnTap and Oracle's Solaris have it turned off.

The Linux server allows it, but I've been considering that a (low-priority) bug, so it wouldn't be safe to assume it will continue working.

That aside, if you have a perfect network, NFSv4.0 at least will probably work. (Not 4.1 since backchannel setup will fail?)

Are you really sure that you can't make tcp scale to thousands of clients?

--b.

Quoted 17 lines
Quoted 5 lines

Thanks in advance for your answer.

Quoted 7 lines

Regards

Philippe
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 18 lines · J. Bruce Fields
Quoted 5 lines · Thomas Haynes
Quoted 2 lines · DENIEL Philippe

Hi,

Quoted 7 lines
Quoted 1 line

I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?

You might find that some implementations have UDP disabled and it might then make it hard to triage what is going on.

I believe both NetApp's OnTap and Oracle's Solaris have it turned off.

The Linux server allows it, but I've been considering that a (low-priority) bug, so it wouldn't be safe to assume it will continue working.

Quoted 5 lines

That aside, if you have a perfect network, NFSv4.0 at least will probably work. (Not 4.1 since backchannel setup will fail?)

Are you really sure that you can't make tcp scale to thousands of clients?

Maybe SCTP would be the way to go then? We've often talked about it over the years, but nobody has yet (to my knowledge) written an ONC RPC transport for it.

Cheers
Trond

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#4 lore raw
In reply to J. Bruce Fields

Hi

Quoted 4 lines

The Linux server allows it, but I've been considering that a (low-priority) bug, so it wouldn't be safe to assume it will continue working.

I am mostly thinking at the client side using as well udp and tcp. Would the "NFSv4 client over UDP" behaves differently then the same client over TCP ?

Quoted 1 line

That aside, if you have a perfect network,

I can trust my network. It's not a WAN, it's located into a very massive cluster (it's kind of "internal LAN"). It's a very high throughput network (IB based) so I believe there are less "hardware based reason" to loose packets.

Quoted 2 lines

NFSv4.0 at least will
probably work.

OK.

Quoted 2 lines

(Not 4.1 since backchannel setup will fail?)

What is erroneous in using UDP for NFSv4.1 backchannels ?

Quoted 3 lines

Are you really sure that you can't make tcp scale to thousands of clients?

I am a bit afraid of a "No more file descriptors" effect. If I have one TCP socket per client and thousands of clients, I have less remaining fds for other purposes. Another point : UDP is a "cheap" protocol. I can have bunches of clients without overloading the server (a new client will almost cost nothing to the server, just the cost of a new clientid negotiation) . I was wondering if it could be reliable to use it for NFSv4 inside a large cluster.

Philippe

#5 lore raw
Quoted 14 lines · DENIEL Philippe

Hi

Quoted 3 lines

The Linux server allows it, but I've been considering that a (low-priority) bug, so it wouldn't be safe to assume it will continue working.

I am mostly thinking at the client side using as well udp and tcp. Would the "NFSv4 client over UDP" behaves differently then the same client over TCP ?

Quoted 1 line

That aside, if you have a perfect network,

I can trust my network. It's not a WAN, it's located into a very massive cluster (it's kind of "internal LAN"). It's a very high throughput network (IB based) so I believe there are less "hardware based reason" to loose packets.

Quoted 5 lines
Quoted 2 lines

NFSv4.0 at least will
probably work.

OK.

Quoted 1 line

(Not 4.1 since backchannel setup will fail?)

What is erroneous in using UDP for NFSv4.1 backchannels ?

I assumed it would require some stronger notion of a "connection" than UDP could provide. But maybe it could be made to work somehow.

Quoted 8 lines
Quoted 2 lines

Are you really sure that you can't make tcp scale to thousands of clients?

I am a bit afraid of a "No more file descriptors" effect. If I have one TCP socket per client and thousands of clients, I have less remaining fds for other purposes. Another point : UDP is a "cheap" protocol. I can have bunches of clients without overloading the server (a new client will almost cost nothing to the server, just the cost of a new clientid negotiation) .

Is the tcp state all that much more?

Quoted 2 lines

I was wondering if it
could be reliable to use it for NFSv4 inside a large cluster.

Well, it'd require some spec changes in any case.

--b.