Skip to contentSkip to conversation

NFS over RMDA

DISCUSSION poma 2 participants 3 messages Nov 25, 2014 4 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

poma

author
#4 lore raw
In reply to a message not in the archive
Quoted 47 lines · Josh Boyer
Quoted 39 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman

Hi,

Is anyone successfully running NFS over RDMA on Fedora 20+?

I've edited /etc/sysconfig/nfs and set the the following config paramter.

RDMA_PORT=20049

Upon restarting the nfs server I get the following:

modprobe: FATAL: Module svcrdma not found /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supported

So it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.

Quoted 5 lines
Quoted 3 lines

Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?

[snip unedited copy from someone's terminal]

0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for NFSoRDMA client and server support").

1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit fd469c7db4e6 ("Linux v3.15.2"). It dropped CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.

Quoted 6 lines

2) A similar commit for Rawhide was 700baa35a69e ("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.

3) Perhaps Justin or Josh, authors of those commits, might recall why only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.

Quoted 9 lines

At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers. I believe this also matches what wound up in RHEL7.

The NFS developers haven't asked us to turn it back on, so it has stayed off since.

josh

Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER'?

poma

#5 lore raw
Quoted 45 lines · poma
Quoted 44 lines · Josh Boyer
Quoted 43 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman

Hi,

Is anyone successfully running NFS over RDMA on Fedora 20+?

I've edited /etc/sysconfig/nfs and set the the following config paramter.

RDMA_PORT=20049

Upon restarting the nfs server I get the following:

modprobe: FATAL: Module svcrdma not found /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supported

So it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.

Quoted 5 lines
Quoted 3 lines

Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?

[snip unedited copy from someone's terminal]

0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for NFSoRDMA client and server support").

1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit fd469c7db4e6 ("Linux v3.15.2"). It dropped CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.

2) A similar commit for Rawhide was 700baa35a69e ("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

3) Perhaps Justin or Josh, authors of those commits, might recall why only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.

At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers.

The NFS/RDMA server crashed often and there was no identified upstream resource to help with it, so it was surgically disabled rather than fixed.

Quoted 10 lines
Quoted 8 lines

I believe this
also matches what wound up in RHEL7.

The NFS developers haven't asked us to turn it back on, so it has stayed off since.

josh

Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?

The server in 3.17 and later kernels no longer crashes regularly.

As to whether you should re-enable it, here’s full disclosure:

As I understand it the NFS/RDMA server-side transport effort was defunded after a prototype was merged. It was hoped that a production-quality implementation would be funded but it never was.

The code as it stands needs some work, but is operational. I use it for NFS/RDMA client work every day, so it is getting some exercise.

Distributions can enable this if they have the resources (ie, test and support engineers and RDMA adapters) to test and support customer issues. I assume RH/Fedora do, as they support client side NFS/RDMA already. But perhaps consider it as “tech preview” for your enterprise kernels.

Or they may choose to wait until upstream has changed the default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.

Note that community support for NFS/RDMA server-side is still best-effort only. We don’t have any designated specialists, just a few folks who need the server to work for other reasons. We are actively looking for interested parties.

Upstream bugs can be filed here:

https://bugzilla.linux-nfs.org

HTH

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#6 lore raw
Quoted 102 lines · Chuck Lever
Quoted 45 lines · poma
Quoted 44 lines · Josh Boyer
Quoted 43 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman

Hi,

Is anyone successfully running NFS over RDMA on Fedora 20+?

I've edited /etc/sysconfig/nfs and set the the following config paramter.

RDMA_PORT=20049

Upon restarting the nfs server I get the following:

modprobe: FATAL: Module svcrdma not found /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supported

So it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.

Quoted 5 lines
Quoted 3 lines

Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?

[snip unedited copy from someone's terminal]

0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for NFSoRDMA client and server support").

1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit fd469c7db4e6 ("Linux v3.15.2"). It dropped CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.

2) A similar commit for Rawhide was 700baa35a69e ("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

3) Perhaps Justin or Josh, authors of those commits, might recall why only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.

At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers.

The NFS/RDMA server crashed often and there was no identified upstream resource to help with it, so it was surgically disabled rather than fixed.

Quoted 10 lines
Quoted 8 lines

I believe this
also matches what wound up in RHEL7.

The NFS developers haven't asked us to turn it back on, so it has stayed off since.

josh

Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?

The server in 3.17 and later kernels no longer crashes regularly.

As to whether you should re-enable it, here’s full disclosure:

As I understand it the NFS/RDMA server-side transport effort was defunded after a prototype was merged. It was hoped that a production-quality implementation would be funded but it never was.

The code as it stands needs some work, but is operational. I use it for NFS/RDMA client work every day, so it is getting some exercise.

Distributions can enable this if they have the resources (ie, test and support engineers and RDMA adapters) to test and support customer issues. I assume RH/Fedora do, as they support client side NFS/RDMA already. But perhaps consider it as “tech preview” for your enterprise kernels.

Or they may choose to wait until upstream has changed the default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.

Note that community support for NFS/RDMA server-side is still best-effort only. We don’t have any designated specialists, just a few folks who need the server to work for other reasons. We are actively looking for interested parties.

Upstream bugs can be filed here:

Quoted 10 lines

https://bugzilla.linux-nfs.org

HTH

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Thanks man.

poma