Skip to contentSkip to conversation

NFS over QUIC

DISCUSSION J. Bruce Fields 3 participants 5 messages Sep 3, 2020 – Sep 4, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

I've been thinking about what might be required for NFS to run over QUIC.

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans. For Linux, I don't even know if there's a kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/

For example, section 4.3, which explains how to authenticate on top of an already-encrypted session, should also apply to QUIC.

QUIC runs over UDP, so I think all that would be required to negotiate support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

	https://tools.ietf.org/html/rfc5661#section-2.9

requires transports support reliable and in-order transmission, forbids clients from retrying a request unless a connection is lost, and forbids servers from dropping a request without closing a connection. I'm still vague on how those requirements interact with QUIC's connection management and 0-RTT reconnection.

https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. Sounds like that's what they do for HTTP/QUIC to make server push notifications work:

https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5

HTTP clients are expected to use QUIC PING frames to keep connections open. Servers SHOULD NOT use PING frames to keep a connection open. A client SHOULD NOT use PING frames for this purpose unless there are responses outstanding for requests or server pushes.

QUIC allows multiple streams per connection--I wonder how we might use that. RFC 5661 justifies the requirement for an ordered transport with:

Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol.

So as long as we don't try to split a single RPC among streams, I think we're OK. Would a stream per session slot be reasonable? I'm not sure what the cost of a stream is.

Do we need to add a new universal address type so the protocol can specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP address and maybe a port, so maybe the existing UDP/TCP addresses are enough?

--b.

#1 lore raw

Hi Bruce-

Quoted 5 lines · J. Bruce Fields

I've been thinking about what might be required for NFS to run over QUIC.

Quoted 8 lines

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans. For Linux, I don't even know if there's a kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/

The link to the latest version of that document is

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

Quoted 2 lines

For example, section 4.3, which explains how to authenticate on top of an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining RPC-over-QUIC, for example the ALPN defined in Section 8.2. Lars Eggert, a chair of the QUIC WG, has been helping guide the RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over- QUIC as a new RPC transport type with a netid and uaddr along with a definition of the transport semantics (a la TI-RPC). The document would need to explain record marking, peer authentication, how to use multi-path and multi-stream support, and so on.

Making NFS work on that transport should then be straightforward enough that perhaps additional standards work wouldn't be necessary.

Quoted 35 lines

QUIC runs over UDP, so I think all that would be required to negotiate support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

	https://tools.ietf.org/html/rfc5661#section-2.9

requires transports support reliable and in-order transmission, forbids clients from retrying a request unless a connection is lost, and forbids servers from dropping a request without closing a connection. I'm still vague on how those requirements interact with QUIC's connection management and 0-RTT reconnection.

https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. Sounds like that's what they do for HTTP/QUIC to make server push notifications work:

https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5

HTTP clients are expected to use QUIC PING frames to keep connections open. Servers SHOULD NOT use PING frames to keep a connection open. A client SHOULD NOT use PING frames for this purpose unless there are responses outstanding for requests or server pushes.

QUIC allows multiple streams per connection--I wonder how we might use that. RFC 5661 justifies the requirement for an ordered transport with:

Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol.

Quoted 8 lines

So as long as we don't try to split a single RPC among streams, I think we're OK. Would a stream per session slot be reasonable? I'm not sure what the cost of a stream is.

Do we need to add a new universal address type so the protocol can specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP address and maybe a port, so maybe the existing UDP/TCP addresses are enough?

--
Chuck Lever
[email protected]

#2 lore raw
Quoted 36 lines · Chuck Lever

Hi Bruce-

Quoted 5 lines · J. Bruce Fields

I've been thinking about what might be required for NFS to run over QUIC.

Quoted 8 lines

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans. For Linux, I don't even know if there's a kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/

The link to the latest version of that document is

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

Quoted 2 lines

For example, section 4.3, which explains how to authenticate on top of an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining RPC-over-QUIC, for example the ALPN defined in Section 8.2. Lars Eggert, a chair of the QUIC WG, has been helping guide the RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over- QUIC as a new RPC transport type with a netid and uaddr along with a definition of the transport semantics (a la TI-RPC). The document would need to explain record marking, peer authentication, how to use multi-path and multi-stream support, and so on.

Quoted 3 lines

Making NFS work on that transport should then be straightforward enough that perhaps additional standards work wouldn't be necessary.

Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know there was a plan.

--b.

Quoted 49 lines
Quoted 35 lines

QUIC runs over UDP, so I think all that would be required to negotiate support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

	https://tools.ietf.org/html/rfc5661#section-2.9

requires transports support reliable and in-order transmission, forbids clients from retrying a request unless a connection is lost, and forbids servers from dropping a request without closing a connection. I'm still vague on how those requirements interact with QUIC's connection management and 0-RTT reconnection.

https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. Sounds like that's what they do for HTTP/QUIC to make server push notifications work:

https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5

HTTP clients are expected to use QUIC PING frames to keep connections open. Servers SHOULD NOT use PING frames to keep a connection open. A client SHOULD NOT use PING frames for this purpose unless there are responses outstanding for requests or server pushes.

QUIC allows multiple streams per connection--I wonder how we might use that. RFC 5661 justifies the requirement for an ordered transport with:

Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol.

Quoted 8 lines

So as long as we don't try to split a single RPC among streams, I think we're OK. Would a stream per session slot be reasonable? I'm not sure what the cost of a stream is.

Do we need to add a new universal address type so the protocol can specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP address and maybe a port, so maybe the existing UDP/TCP addresses are enough?

--
Chuck Lever
[email protected]

#3 lore raw
Quoted 41 lines · Bruce Fields
Quoted 36 lines · Chuck Lever

Hi Bruce-

Quoted 5 lines · J. Bruce Fields

I've been thinking about what might be required for NFS to run over QUIC.

Quoted 8 lines

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans. For Linux, I don't even know if there's a kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/

The link to the latest version of that document is

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

Quoted 2 lines

For example, section 4.3, which explains how to authenticate on top of an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining RPC-over-QUIC, for example the ALPN defined in Section 8.2. Lars Eggert, a chair of the QUIC WG, has been helping guide the RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over- QUIC as a new RPC transport type with a netid and uaddr along with a definition of the transport semantics (a la TI-RPC). The document would need to explain record marking, peer authentication, how to use multi-path and multi-stream support, and so on.

Quoted 6 lines
Quoted 3 lines

Making NFS work on that transport should then be straightforward enough that perhaps additional standards work wouldn't be necessary.

Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know there was a plan.

That's all there is for the moment! :-)

Quoted 49 lines

--b.

Quoted 47 lines
Quoted 35 lines

QUIC runs over UDP, so I think all that would be required to negotiate support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

	https://tools.ietf.org/html/rfc5661#section-2.9

requires transports support reliable and in-order transmission, forbids clients from retrying a request unless a connection is lost, and forbids servers from dropping a request without closing a connection. I'm still vague on how those requirements interact with QUIC's connection management and 0-RTT reconnection.

https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. Sounds like that's what they do for HTTP/QUIC to make server push notifications work:

https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5

HTTP clients are expected to use QUIC PING frames to keep connections open. Servers SHOULD NOT use PING frames to keep a connection open. A client SHOULD NOT use PING frames for this purpose unless there are responses outstanding for requests or server pushes.

QUIC allows multiple streams per connection--I wonder how we might use that. RFC 5661 justifies the requirement for an ordered transport with:

Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol.

Quoted 8 lines

So as long as we don't try to split a single RPC among streams, I think we're OK. Would a stream per session slot be reasonable? I'm not sure what the cost of a stream is.

Do we need to add a new universal address type so the protocol can specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP address and maybe a port, so maybe the existing UDP/TCP addresses are enough?

--
Chuck Lever
[email protected]

--
Chuck Lever
[email protected]

#4 lore raw

We (open source SMB3 developers) have been thinking a lot about SMB3.1.1 over QUIC especially after the interesting talks on this at the last few SDC conferences e.g.

https://www.snia.org/sites/default/files/SDC/2019/presentations/SMB/Li_Frank_Surfing_the_World_Wide_File_SMB3_Improvements_for_Safe_and_Efficient_Internet_Access.pdf (slides 8 and following).

And some of the broader coverage:
https://redmondmag.com/articles/2020/03/02/microsoft-smb-over-quic-to-windows.aspx

I have gotten a chance to talk with some of the developers of the recently open sourced quic libraries (https://github.com/microsoft/msquic) and there are various code style issues that would prevent it being merged into Linux kernel without significant changes (not just to fix camelCase) but ... it is a very very promising place to start.

One of the ideas being bounced around is to do this in stages - first try to cleanup the kernel style issues with the open sourced msquic project and merge it experimentally into the Linux client (cifs.ko) since there should be SMB3.1.1. servers to test it against reasonably soon (presumably we will know a lot more after the 2020 SNIA SDC talk on SMB3.1.1 over QUIC in a few weeks updating us on status https://www.snia.org/events/storage-developer/2020/abstracts#smb-Dantuluri) after we are convinced that the quic driver interoperates ok then move it into the Linux networking stack and let the networking experts clean it up more and remove the 'experimental' tag from it when satisfied.

More discussion is needed on this so we should add samba-technical and some of the network experts as well.

I am very excited about the possibilities opened up by a decent quic kernel driver, but the current cross platform open sourced driver is fairly large - ~90KLOC so will have to be trimmed down (and a lot of boring changes to convert to kernel style needed).

-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Thursday, September 3, 2020 7:53 PM
To: Bruce Fields <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; NFSv4 <[email protected]>; Steven French <[email protected]>
Subject: [EXTERNAL] Re: [nfsv4] NFS over QUIC

Quoted 49 lines · Bruce Fields
Quoted 46 lines · Chuck Lever

Hi Bruce-

Quoted 13 lines · J. Bruce Fields

I've been thinking about what might be required for NFS to run over QUIC.

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans. For Linux, I don't even know if there's a kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

Quoted 6 lines

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fda
tatracker.ietf.org%2Fdoc%2Fdraft-cel-nfsv4-rpc-tls%2F&amp;data=02%7C 01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee6 3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&am p;sdata=pOnlPJcTkyJiqQm%2BCbo7dbOAqHMKFIi01qrShVZTH%2Bk%3D&amp;reser ved=0

The link to the latest version of that document is

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdat
atracker.ietf.org%2Fdoc%2Fdraft-ietf-nfsv4-rpc-tls%2F&amp;data=02%7C0 1%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63% 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&amp;s data=YYBPryN5sFJApWm%2BkQvmjYzGXxjxI1fXRUWVXvVGE24%3D&amp;reserved=0

Quoted 2 lines

For example, section 4.3, which explains how to authenticate on top of an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining RPC-over-QUIC, for example the ALPN defined in Section 8.2. Lars Eggert, a chair of the QUIC WG, has been helping guide the RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over- QUIC as a new RPC transport type with a netid and uaddr along with a definition of the transport semantics (a la TI-RPC).
The document would need to explain record marking, peer authentication, how to use multi-path and multi-stream support, and so on.

Quoted 5 lines
Quoted 2 lines

Making NFS work on that transport should then be straightforward enough that perhaps additional standards work wouldn't be necessary.

Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know there was a plan.

That's all there is for the moment! :-)

Quoted 60 lines

--b.

Quoted 58 lines
Quoted 46 lines

QUIC runs over UDP, so I think all that would be required to negotiate support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
ols.ietf.org%2Fhtml%2Frfc5661%23section-2.9&amp;data=02%7C01%7CSteve n.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%7C72f988 bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&amp;sdata=ej gN6OuHrcDMzejeEKbaP5UiD1GAxHfQcPyLZZ45vbo%3D&amp;reserved=0

requires transports support reliable and in-order transmission, forbids clients from retrying a request unless a connection is lost, and forbids servers from dropping a request without closing a connection. I'm still vague on how those requirements interact with QUIC's connection management and 0-RTT reconnection.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
w.ietf.org%2Fid%2Fdraft-ietf-quic-applicability-07.txt&amp;data=02%7
C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee
63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&amp;sdata=TZeoR%2FT22hg%2BkG0LllDEttlgdzv%2FAsH8PeXbdEHQ9Mk%3D&amp;reserved=0 looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. Sounds like that's what they do for HTTP/QUIC to make server push
notifications work:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
ols.ietf.org%2Fhtml%2Fdraft-ietf-quic-http-09%23section-5&amp;data=0 2%7C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506 cee63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63734777607589360 4&amp;sdata=rF91Tq8OXHeN7W5iuSbtYotq1XBC4Zc8DGEGzwPUuTg%3D&amp;reser ved=0

HTTP clients are expected to use QUIC PING frames to keep connections open. Servers SHOULD NOT use PING frames to keep a connection open. A client SHOULD NOT use PING frames for this purpose unless there are responses outstanding for requests or server pushes.

QUIC allows multiple streams per connection--I wonder how we might use that. RFC 5661 justifies the requirement for an ordered transport with:

Ordered delivery simplifies detection of transmit errors, and simplifies the sending of arbitrary sized requests and responses via the record marking protocol.

Quoted 8 lines

So as long as we don't try to split a single RPC among streams, I think we're OK. Would a stream per session slot be reasonable? I'm not sure what the cost of a stream is.

Do we need to add a new universal address type so the protocol can specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP address and maybe a port, so maybe the existing UDP/TCP addresses are enough?

--
Chuck Lever
[email protected]

--
Chuck Lever
[email protected]