unable to run NFSD in container if "options sunrpc pool_mode=pernode"
Subsystems (2)
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Mike
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Jeff Layton
maintainerQuoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Mike
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Huh, strange. I've no idea why that would be. What kernel is this?
FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Quoted 6 lines
Quoted 4 lines
Mike
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Huh, strange. I've no idea why that would be. What kernel is this?
It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testing
Basically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).
But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16
Here is the NFS related config:
CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not setCONFIG_NFS_V3=mCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set CONFIG_NFS_V4_1=yCONFIG_NFS_V4_2=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=m CONFIG_PNFS_FLEXFILE_LAYOUT=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" # CONFIG_NFS_V4_1_MIGRATION is not set CONFIG_NFS_V4_SECURITY_LABEL=y CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFS_DEBUG=y CONFIG_NFS_DISABLE_UDP_SUPPORT=y
# CONFIG_NFS_V4_2_READ_PLUS is not set CONFIG_NFSD=m
# CONFIG_NFSD_V2 is not set CONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_NFSD_PNFS=y
# CONFIG_NFSD_BLOCKLAYOUT is not set CONFIG_NFSD_SCSILAYOUT=y
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_V4_2_INTER_SSC is not set CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_LEGACY_CLIENT_TRACKING is not set # CONFIG_NFSD_V4_DELEG_TIMESTAMPS is not set CONFIG_GRACE_PERIOD=mCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=y CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m CONFIG_NFS_LOCALIO=y CONFIG_NFS_V4_2_SSC_HELPER=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_RPCSEC_GSS_KRB5=m CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=y CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=y CONFIG_SUNRPC_DEBUG=y CONFIG_SUNRPC_XPRT_RDMA=m
Quoted 4 lines
FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.
OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Mike
Quoted 84 lines · Mike Snitzer
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Quoted 6 lines
Quoted 4 lines
Mike
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Huh, strange. I've no idea why that would be. What kernel is this?
It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testingBasically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).
But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16Here is the NFS related config:
CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=m
#CONFIG_NFS_V2is not setCONFIG_NFS_V3=mCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=m
#CONFIG_NFS_SWAPis not setCONFIG_NFS_V4_1=yCONFIG_NFS_V4_2=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=mCONFIG_PNFS_FLEXFILE_LAYOUT=mCONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" #CONFIG_NFS_V4_1_MIGRATIONis not setCONFIG_NFS_V4_SECURITY_LABEL=yCONFIG_NFS_FSCACHE=y
#CONFIG_NFS_USE_LEGACY_DNSis not setCONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFS_DEBUG=yCONFIG_NFS_DISABLE_UDP_SUPPORT=y
#CONFIG_NFS_V4_2_READ_PLUSis not setCONFIG_NFSD=m
#CONFIG_NFSD_V2is not setCONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_NFSD_PNFS=y
#CONFIG_NFSD_BLOCKLAYOUTis not setCONFIG_NFSD_SCSILAYOUT=y
#CONFIG_NFSD_FLEXFILELAYOUTis not set
#CONFIG_NFSD_V4_2_INTER_SSCis not setCONFIG_NFSD_V4_SECURITY_LABEL=y
#CONFIG_NFSD_LEGACY_CLIENT_TRACKINGis not set #CONFIG_NFSD_V4_DELEG_TIMESTAMPSis not setCONFIG_GRACE_PERIOD=mCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=yCONFIG_NFS_COMMON_LOCALIO_SUPPORT=mCONFIG_NFS_LOCALIO=yCONFIG_NFS_V4_2_SSC_HELPER=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=mCONFIG_SUNRPC_BACKCHANNEL=yCONFIG_RPCSEC_GSS_KRB5=mCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=yCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=yCONFIG_SUNRPC_DEBUG=yCONFIG_SUNRPC_XPRT_RDMA=mQuoted 4 lines
FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.
Quoted 6 lines
OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Actually, I do see nfsd threads running.. just that if I try to issue IO (using pNFS flexfiles to file on v3 DS) it hangs.
Mike
Jeff Layton
maintainerQuoted 10 lines · Mike Snitzer
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
Quoted 74 lines
Quoted 6 lines
Quoted 4 lines
Mike
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Huh, strange. I've no idea why that would be. What kernel is this?
It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testingBasically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).
But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16Here is the NFS related config:
CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=m
#CONFIG_NFS_V2is not setCONFIG_NFS_V3=mCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=m
#CONFIG_NFS_SWAPis not setCONFIG_NFS_V4_1=yCONFIG_NFS_V4_2=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=mCONFIG_PNFS_FLEXFILE_LAYOUT=mCONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" #CONFIG_NFS_V4_1_MIGRATIONis not setCONFIG_NFS_V4_SECURITY_LABEL=yCONFIG_NFS_FSCACHE=y
#CONFIG_NFS_USE_LEGACY_DNSis not setCONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFS_DEBUG=yCONFIG_NFS_DISABLE_UDP_SUPPORT=y
#CONFIG_NFS_V4_2_READ_PLUSis not setCONFIG_NFSD=m
#CONFIG_NFSD_V2is not setCONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_NFSD_PNFS=y
#CONFIG_NFSD_BLOCKLAYOUTis not setCONFIG_NFSD_SCSILAYOUT=y
#CONFIG_NFSD_FLEXFILELAYOUTis not set
#CONFIG_NFSD_V4_2_INTER_SSCis not setCONFIG_NFSD_V4_SECURITY_LABEL=y
#CONFIG_NFSD_LEGACY_CLIENT_TRACKINGis not set #CONFIG_NFSD_V4_DELEG_TIMESTAMPSis not setCONFIG_GRACE_PERIOD=mCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=yCONFIG_NFS_COMMON_LOCALIO_SUPPORT=mCONFIG_NFS_LOCALIO=yCONFIG_NFS_V4_2_SSC_HELPER=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=mCONFIG_SUNRPC_BACKCHANNEL=yCONFIG_RPCSEC_GSS_KRB5=mCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=yCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=yCONFIG_SUNRPC_DEBUG=yCONFIG_SUNRPC_XPRT_RDMA=mQuoted 4 lines
FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.
Quoted 7 lines
OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
I'll have a look soon, but if you figure it out in the meantime, let us know.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 76 lines
Quoted 76 lines
Quoted 4 lines
Quoted 2 lines
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Huh, strange. I've no idea why that would be. What kernel is this?
It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testingBasically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).
But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16Here is the NFS related config:
CONFIG_NETWORK_FILESYSTEMS=yCONFIG_NFS_FS=m
#CONFIG_NFS_V2is not setCONFIG_NFS_V3=mCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=m
#CONFIG_NFS_SWAPis not setCONFIG_NFS_V4_1=yCONFIG_NFS_V4_2=yCONFIG_PNFS_FILE_LAYOUT=mCONFIG_PNFS_BLOCK=mCONFIG_PNFS_FLEXFILE_LAYOUT=mCONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" #CONFIG_NFS_V4_1_MIGRATIONis not setCONFIG_NFS_V4_SECURITY_LABEL=yCONFIG_NFS_FSCACHE=y
#CONFIG_NFS_USE_LEGACY_DNSis not setCONFIG_NFS_USE_KERNEL_DNS=yCONFIG_NFS_DEBUG=yCONFIG_NFS_DISABLE_UDP_SUPPORT=y
#CONFIG_NFS_V4_2_READ_PLUSis not setCONFIG_NFSD=m
#CONFIG_NFSD_V2is not setCONFIG_NFSD_V3_ACL=yCONFIG_NFSD_V4=yCONFIG_NFSD_PNFS=y
#CONFIG_NFSD_BLOCKLAYOUTis not setCONFIG_NFSD_SCSILAYOUT=y
#CONFIG_NFSD_FLEXFILELAYOUTis not set
#CONFIG_NFSD_V4_2_INTER_SSCis not setCONFIG_NFSD_V4_SECURITY_LABEL=y
#CONFIG_NFSD_LEGACY_CLIENT_TRACKINGis not set #CONFIG_NFSD_V4_DELEG_TIMESTAMPSis not setCONFIG_GRACE_PERIOD=mCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_NFS_ACL_SUPPORT=mCONFIG_NFS_COMMON=yCONFIG_NFS_COMMON_LOCALIO_SUPPORT=mCONFIG_NFS_LOCALIO=yCONFIG_NFS_V4_2_SSC_HELPER=yCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=mCONFIG_SUNRPC_BACKCHANNEL=yCONFIG_RPCSEC_GSS_KRB5=mCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=yCONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=yCONFIG_SUNRPC_DEBUG=yCONFIG_SUNRPC_XPRT_RDMA=mQuoted 4 lines
FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.
OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?
Quoted 6 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
# cat /proc/8087/stack [<0>] rpc_wait_bit_killable+0x25/0x80 [sunrpc] [<0>] __rpc_execute+0x151/0x480 [sunrpc] [<0>] rpc_execute+0xca/0xf0 [sunrpc] [<0>] rpc_run_task+0x110/0x180 [sunrpc] [<0>] nfs4_call_sync_custom+0xb/0x30 [nfsv4] [<0>] nfs4_do_call_sync+0x69/0x90 [nfsv4] [<0>] _nfs4_proc_getattr+0x128/0x160 [nfsv4] [<0>] nfs4_proc_getattr+0x73/0x100 [nfsv4] [<0>] nfs4_do_open+0x775/0x9d0 [nfsv4] [<0>] nfs4_atomic_open+0xf7/0x100 [nfsv4] [<0>] nfs_atomic_open+0x1e7/0x6c0 [nfs] [<0>] path_openat+0xd38/0x11f0 [<0>] do_filp_open+0xae/0x120 [<0>] do_sys_openat2+0x24d/0x2a0 [<0>] do_sys_open+0x4f/0x90 [<0>] do_syscall_64+0x7b/0x160 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
And if I just try to mount using v3 it fails with:
# mount -vvvvvvv -o vers=3,nolock 10.200.80.89:/cvol_12_0 /mnt/test mount.nfs: timeout set for Fri May 23 22:52:04 2025 mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049 mount.nfs: portmap query retrying: RPC: Timed out mount.nfs: prog 100003, trying vers=3, prot=17 mount.nfs: portmap query failed: RPC: Program not registered mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049 mount.nfs: portmap query retrying: RPC: Timed out mount.nfs: prog 100003, trying vers=3, prot=17 mount.nfs: portmap query failed: RPC: Program not registered mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049 mount.nfs: portmap query retrying: RPC: Timed out mount.nfs: prog 100003, trying vers=3, prot=17 mount.nfs: portmap query failed: RPC: Program not registered mount.nfs: requested NFS version or transport protocol is not supported for /mnt/test
# rpcinfo -p 10.200.80.89
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100024 1 udp 45252 status
100024 1 tcp 60557 status
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100021 1 udp 40987 nlockmgr
100021 3 udp 40987 nlockmgr
100021 4 udp 40987 nlockmgr
100021 1 tcp 36527 nlockmgr
100021 3 tcp 36527 nlockmgr
100021 4 tcp 36527 nlockmgr(Not sure what's up with portmap issues and it not progressing to trying program 100005.. which as you can see below it does)
But if I just use sunrpc's default pool_mode=global:
# mount -vvvvvvv -o vers=3,nolock 10.200.80.89:/cvol_12_0 /mnt/test mount.nfs: timeout set for Fri May 23 22:55:43 2025 mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 10.200.80.89 prog 100005 vers 3 prot UDP port 20048
# rpcinfo -p 10.200.80.89
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 54037 status
100024 1 tcp 46339 status
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100021 1 udp 36268 nlockmgr
100021 3 udp 36268 nlockmgr
100021 4 udp 36268 nlockmgr
100021 1 tcp 44195 nlockmgr
100021 3 tcp 44195 nlockmgr
100021 4 tcp 44195 nlockmgrQuoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
Mike
Jeff Layton
maintainerQuoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
This is probably a kernel problem.
Newer nfs-utils uses nfsdctl to start the server, whereas the older nfs-utils would use rpc.nfsd. nfsdctl uses the netlink interfaces to start the server instead of /proc/fs/nfsd which rpc.nfsd uses.
If you're getting different results with the two different nfs-utils versions then the problem is likely there. I'll take a look.
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerQuoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Quoted 5 lines
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode
(from EL9.5 container with nfs-utils 2.8.2)
# nfsdctl pool-mode pool-mode: pernode npools: 2
(on host)
# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 11665 MB node 0 free: 9892 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 6042 MB node 1 free: 5127 MB node distances: node 0 1 0: 10 20 1: 20 10
(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)
Jeff Layton
maintainerQuoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Quoted 5 lines
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode(from EL9.5 container with nfs-utils 2.8.2)
# nfsdctl pool-mode pool-mode: pernode npools: 2(on host)
# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 11665 MB node 0 free: 9892 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 6042 MB node 1 free: 5127 MB node distances: node 0 1 0: 10 20 1: 20 10
Quoted 3 lines
(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)
rpc.nfsd creates sockets in userland and then passes them to the kernel. With the netlink interface, we now create the socket in the kernel. If I had to guess, the problem is related to that difference.
I can check out a NUMA box at work this week and try to reproduce this, if you don't figure it out over the weekend.
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerQuoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Quoted 5 lines
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode(from EL9.5 container with nfs-utils 2.8.2)
# nfsdctl pool-mode pool-mode: pernode npools: 2(on host)
# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 11665 MB node 0 free: 9892 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 6042 MB node 1 free: 5127 MB node distances: node 0 1 0: 10 20 1: 20 10
Quoted 3 lines
(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)
Ok, I can reproduce this on a true NUMA machine. The first thing that's interesting is that it seems to be intermittent. Occasionally I can mount and operate on the socket, but socket requests hang most of the time.
I turned up all of the nfsd and sunrpc tracepoints. After attempting a mount that hung, I see only a single tracepoint fire:
<idle>-0 [038] ..s.. 5942.572721: svc_xprt_enqueue: server=[::]:2049 client=(einval) flags=CONN|CHNGBUF|LISTENER|CACHE_AUTH|CONG_CTRL
Based on the flags, svc_xprt_ready should have returned true. That should make the xprt be enqueued and an idle thread be awoken. It looks like that last bit may not be happening for some reason.
At this point, I'll probably have to add some debugging. I'll keep poking at it -- stay tuned.
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerQuoted 92 lines · Jeff Layton
Quoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Quoted 5 lines
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode(from EL9.5 container with nfs-utils 2.8.2)
# nfsdctl pool-mode pool-mode: pernode npools: 2(on host)
# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 11665 MB node 0 free: 9892 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 6042 MB node 1 free: 5127 MB node distances: node 0 1 0: 10 20 1: 20 10Quoted 3 lines
(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)
Ok, I can reproduce this on a true NUMA machine. The first thing that's interesting is that it seems to be intermittent. Occasionally I can mount and operate on the socket, but socket requests hang most of the time.
I turned up all of the nfsd and sunrpc tracepoints. After attempting a mount that hung, I see only a single tracepoint fire:
Quoted 8 lines
<idle>-0 [038] ..s.. 5942.572721: svc_xprt_enqueue: server=[::]:2049 client=(einval) flags=CONN|CHNGBUF|LISTENER|CACHE_AUTH|CONG_CTRLBased on the flags,
svc_xprt_readyshould have returned true. That should make the xprt be enqueued and an idle thread be awoken. It looks like that last bit may not be happening for some reason.At this point, I'll probably have to add some debugging. I'll keep poking at it -- stay tuned.
Ok, I figured it out. The problem is that the netlink interfaces don't correctly handle the special case of a non-global pool_mode and userland passing down a single-element array.
The old interface interpreted that as "distribute all of these threads amongst the available nodes", but the netlink interfaces take it literally and start all of the threads in the first node. If a socket request comes in on the other node, then you're out of luck and the socket never gets serviced.
I have a patch I'm testing now that should do the right thing. In the meantime, if you explicitly set the number of threads in each pool it should work around the problem. E.g., if you want 128 threads total, distributed evenly over both numa nodes, try a setting like this for now:
[nfsd]
threads=64,64
I'll send a proper fix for the kernel once I've tested it a bit more.
Thanks for the bug report!
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerQuoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer
I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.
If pool_mode=pernode then mounting the container's NFSv3 export fails.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.
Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.
OK.
Quoted 4 lines
Quoted 3 lines
I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".
Quoted 7 lines
Quoted 2 lines
I'll have a look soon, but if you figure it out in the meantime, let us know.
Will do.
Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:
Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).
If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.
Quoted 8 lines
And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).
Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"
I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?
Quoted 5 lines
Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:
# nfsdctl pool-mode(from EL9.5 container with nfs-utils 2.8.2)
# nfsdctl pool-mode pool-mode: pernode npools: 2(on host)
# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 11665 MB node 0 free: 9892 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 6042 MB node 1 free: 5127 MB node distances: node 0 1 0: 10 20 1: 20 10
Quoted 3 lines
(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)
Hi Mike,
I submitted a patch for this a couple of weeks ago:
https://lore.kernel.org/linux-nfs/[email protected]/
Were you able to test it, and did it fix your issue?
Thanks,
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 2 lines · Mike Snitzer
ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)
Easier said than done. During bake-a-thon, I mentioned there are some historical discussions about this. Two I found:
https://lore.kernel.org/linux-nfs/[email protected]/
And the review comments on this patch:
https://lore.kernel.org/linux-nfs/[email protected]/
Signature · 1 line
Chuck Lever