NFS over RMDA
Subsystems (2)
Quoted 47 lines · Josh Boyer
Quoted 39 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman
Hi,
Is anyone successfully running NFS over RDMA on Fedora 20+?
I've edited
/etc/sysconfig/nfsand set the the following config paramter.RDMA_PORT=20049
Upon restarting the nfs server I get the following:
modprobe: FATAL: Module svcrdma not found
/usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supportedSo it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.
Quoted 5 lines
Quoted 3 lines
Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?
[snip unedited copy from someone's terminal]
0) In mainline kernel v3.15 the config option
SUNRPC_XPRT_RDMAwas split in two options:SUNRPC_XPRT_RDMA_CLIENTandSUNRPC_XPRT_RDMA_SERVER. See commit2e8c12e1b765("xprtrdma: add separateKconfigoptions for NFSoRDMA client and server support").1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit
fd469c7db4e6("Linux v3.15.2"). It droppedCONFIG_SUNRPC_XPRT_RDMA=mfrom the config files (as it was useless). It setCONFIG_SUNRPC_XPRT_RDMA_CLIENTto 'm' but did not setCONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.Quoted 6 lines
2) A similar commit for Rawhide was
700baa35a69e("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.3) Perhaps Justin or Josh, authors of those commits, might recall why only
CONFIG_SUNRPC_XPRT_RDMA_CLIENTwas set.
Quoted 9 lines
At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers. I believe this also matches what wound up in RHEL7.
The NFS developers haven't asked us to turn it back on, so it has stayed off since.
josh
Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER'?
poma
Quoted 45 lines · poma
Quoted 44 lines · Josh Boyer
Quoted 43 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman
Hi,
Is anyone successfully running NFS over RDMA on Fedora 20+?
I've edited
/etc/sysconfig/nfsand set the the following config paramter.RDMA_PORT=20049
Upon restarting the nfs server I get the following:
modprobe: FATAL: Module svcrdma not found
/usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supportedSo it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.
Quoted 5 lines
Quoted 3 lines
Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?
[snip unedited copy from someone's terminal]
0) In mainline kernel v3.15 the config option
SUNRPC_XPRT_RDMAwas split in two options:SUNRPC_XPRT_RDMA_CLIENTandSUNRPC_XPRT_RDMA_SERVER. See commit2e8c12e1b765("xprtrdma: add separateKconfigoptions for NFSoRDMA client and server support").1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit
fd469c7db4e6("Linux v3.15.2"). It droppedCONFIG_SUNRPC_XPRT_RDMA=mfrom the config files (as it was useless). It setCONFIG_SUNRPC_XPRT_RDMA_CLIENTto 'm' but did not setCONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.2) A similar commit for Rawhide was
700baa35a69e("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
3) Perhaps Justin or Josh, authors of those commits, might recall why only
CONFIG_SUNRPC_XPRT_RDMA_CLIENTwas set.At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers.
The NFS/RDMA server crashed often and there was no identified upstream resource to help with it, so it was surgically disabled rather than fixed.
Quoted 10 lines
Quoted 8 lines
I believe this
also matches what wound up in RHEL7.The NFS developers haven't asked us to turn it back on, so it has stayed off since.
josh
Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
The server in 3.17 and later kernels no longer crashes regularly.
As to whether you should re-enable it, here’s full disclosure:
As I understand it the NFS/RDMA server-side transport effort was defunded after a prototype was merged. It was hoped that a production-quality implementation would be funded but it never was.
The code as it stands needs some work, but is operational. I use it for NFS/RDMA client work every day, so it is getting some exercise.
Distributions can enable this if they have the resources (ie, test and support engineers and RDMA adapters) to test and support customer issues. I assume RH/Fedora do, as they support client side NFS/RDMA already. But perhaps consider it as “tech preview” for your enterprise kernels.
Or they may choose to wait until upstream has changed the default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.
Note that community support for NFS/RDMA server-side is still best-effort only. We don’t have any designated specialists, just a few folks who need the server to work for other reasons. We are actively looking for interested parties.
Upstream bugs can be filed here:
https://bugzilla.linux-nfs.org
HTH
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 102 lines · Chuck Lever
Quoted 45 lines · poma
Quoted 44 lines · Josh Boyer
Quoted 43 lines · Paul Bolle
Quoted 19 lines · poma
Quoted 18 lines · Ian Chapman
Hi,
Is anyone successfully running NFS over RDMA on Fedora 20+?
I've edited
/etc/sysconfig/nfsand set the the following config paramter.RDMA_PORT=20049
Upon restarting the nfs server I get the following:
modprobe: FATAL: Module svcrdma not found
/usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supportedSo it looks like the kernel module svcrdma is missing and the last kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to nfs-utils.
Quoted 5 lines
Quoted 3 lines
Is svcrdma intentionally not built in the current kernels or has it been replaced by something else?
[snip unedited copy from someone's terminal]
0) In mainline kernel v3.15 the config option
SUNRPC_XPRT_RDMAwas split in two options:SUNRPC_XPRT_RDMA_CLIENTandSUNRPC_XPRT_RDMA_SERVER. See commit2e8c12e1b765("xprtrdma: add separateKconfigoptions for NFSoRDMA client and server support").1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20). Looking at the git history of the Fedora kernel package I found commit commit
fd469c7db4e6("Linux v3.15.2"). It droppedCONFIG_SUNRPC_XPRT_RDMA=mfrom the config files (as it was useless). It setCONFIG_SUNRPC_XPRT_RDMA_CLIENTto 'm' but did not setCONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this choice. Perhaps it was discusses somewhere else.2) A similar commit for Rawhide was
700baa35a69e("Linux v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice either.Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
3) Perhaps Justin or Josh, authors of those commits, might recall why only
CONFIG_SUNRPC_XPRT_RDMA_CLIENTwas set.At the time, server support for NFSoRDMA wasn't in the greatest shape and we disabled it at the request of the NFS developers.
The NFS/RDMA server crashed often and there was no identified upstream resource to help with it, so it was surgically disabled rather than fixed.
Quoted 10 lines
Quoted 8 lines
I believe this
also matches what wound up in RHEL7.The NFS developers haven't asked us to turn it back on, so it has stayed off since.
josh
Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
The server in 3.17 and later kernels no longer crashes regularly.
As to whether you should re-enable it, here’s full disclosure:
As I understand it the NFS/RDMA server-side transport effort was defunded after a prototype was merged. It was hoped that a production-quality implementation would be funded but it never was.
The code as it stands needs some work, but is operational. I use it for NFS/RDMA client work every day, so it is getting some exercise.
Distributions can enable this if they have the resources (ie, test and support engineers and RDMA adapters) to test and support customer issues. I assume RH/Fedora do, as they support client side NFS/RDMA already. But perhaps consider it as “tech preview” for your enterprise kernels.
Or they may choose to wait until upstream has changed the default setting of
CONFIG_SUNRPC_XPRT_RDMA_SERVER.Note that community support for NFS/RDMA server-side is still best-effort only. We don’t have any designated specialists, just a few folks who need the server to work for other reasons. We are actively looking for interested parties.
Upstream bugs can be filed here:
Quoted 10 lines
https://bugzilla.linux-nfs.org
HTH
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Thanks man.
poma