Skip to contentSkip to conversation

unable to run NFSD in container if "options sunrpc pool_mode=pernode"

DISCUSSION Mike Snitzer 3 participants 15 messages May 23, 2025 – Jun 13, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Mike

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

maintainer
#1 lore raw
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Mike

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

Huh, strange. I've no idea why that would be. What kernel is this?

FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.

Signature · 1 line
Jeff Layton <[email protected]>
#2 lore raw
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Quoted 6 lines
Quoted 4 lines

Mike

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

Huh, strange. I've no idea why that would be. What kernel is this?

It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testing

Basically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).

But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16

Here is the NFS related config:

CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_PNFS_FLEXFILE_LAYOUT=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" # CONFIG_NFS_V4_1_MIGRATION is not set CONFIG_NFS_V4_SECURITY_LABEL=y CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFS_DEBUG=y CONFIG_NFS_DISABLE_UDP_SUPPORT=y
# CONFIG_NFS_V4_2_READ_PLUS is not set CONFIG_NFSD=m
# CONFIG_NFSD_V2 is not set CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_PNFS=y
# CONFIG_NFSD_BLOCKLAYOUT is not set CONFIG_NFSD_SCSILAYOUT=y
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_V4_2_INTER_SSC is not set CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_LEGACY_CLIENT_TRACKING is not set # CONFIG_NFSD_V4_DELEG_TIMESTAMPS is not set CONFIG_GRACE_PERIOD=m
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m CONFIG_NFS_LOCALIO=y CONFIG_NFS_V4_2_SSC_HELPER=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_RPCSEC_GSS_KRB5=m CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=y CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=y CONFIG_SUNRPC_DEBUG=y CONFIG_SUNRPC_XPRT_RDMA=m

Quoted 4 lines

FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.

OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Mike

#3 lore raw
Quoted 84 lines · Mike Snitzer
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Quoted 6 lines
Quoted 4 lines

Mike

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

Huh, strange. I've no idea why that would be. What kernel is this?

It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testing

Basically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).

But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16

Here is the NFS related config:

CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_PNFS_FLEXFILE_LAYOUT=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" # CONFIG_NFS_V4_1_MIGRATION is not set CONFIG_NFS_V4_SECURITY_LABEL=y CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFS_DEBUG=y CONFIG_NFS_DISABLE_UDP_SUPPORT=y
# CONFIG_NFS_V4_2_READ_PLUS is not set CONFIG_NFSD=m
# CONFIG_NFSD_V2 is not set CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_PNFS=y
# CONFIG_NFSD_BLOCKLAYOUT is not set CONFIG_NFSD_SCSILAYOUT=y
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_V4_2_INTER_SSC is not set CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_LEGACY_CLIENT_TRACKING is not set # CONFIG_NFSD_V4_DELEG_TIMESTAMPS is not set CONFIG_GRACE_PERIOD=m
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m CONFIG_NFS_LOCALIO=y CONFIG_NFS_V4_2_SSC_HELPER=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_RPCSEC_GSS_KRB5=m CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=y CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=y CONFIG_SUNRPC_DEBUG=y CONFIG_SUNRPC_XPRT_RDMA=m

Quoted 4 lines

FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.

Quoted 6 lines

OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Actually, I do see nfsd threads running.. just that if I try to issue IO (using pNFS flexfiles to file on v3 DS) it hangs.

Mike

maintainer
#4 lore raw
In reply to Mike Snitzer
Quoted 10 lines · Mike Snitzer
Quoted 9 lines · Jeff Layton
Quoted 8 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

Quoted 74 lines
Quoted 6 lines
Quoted 4 lines

Mike

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

Huh, strange. I've no idea why that would be. What kernel is this?

It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testing

Basically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).

But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16

Here is the NFS related config:

CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_PNFS_FLEXFILE_LAYOUT=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" # CONFIG_NFS_V4_1_MIGRATION is not set CONFIG_NFS_V4_SECURITY_LABEL=y CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFS_DEBUG=y CONFIG_NFS_DISABLE_UDP_SUPPORT=y
# CONFIG_NFS_V4_2_READ_PLUS is not set CONFIG_NFSD=m
# CONFIG_NFSD_V2 is not set CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_PNFS=y
# CONFIG_NFSD_BLOCKLAYOUT is not set CONFIG_NFSD_SCSILAYOUT=y
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_V4_2_INTER_SSC is not set CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_LEGACY_CLIENT_TRACKING is not set # CONFIG_NFSD_V4_DELEG_TIMESTAMPS is not set CONFIG_GRACE_PERIOD=m
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m CONFIG_NFS_LOCALIO=y CONFIG_NFS_V4_2_SSC_HELPER=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_RPCSEC_GSS_KRB5=m CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=y CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=y CONFIG_SUNRPC_DEBUG=y CONFIG_SUNRPC_XPRT_RDMA=m

Quoted 4 lines

FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.

Quoted 7 lines

OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

I'll have a look soon, but if you figure it out in the meantime, let us know.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 76 lines
Quoted 76 lines
Quoted 4 lines
Quoted 2 lines

ps. yet another reason why pool_mode=pernode should be the default if more than 1 NUMA node ;)

Huh, strange. I've no idea why that would be. What kernel is this?

It is this 6.12.24 based frankenbeast-ish kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/main-testing

Basically just 6.12.24 + NFS and NFSD sync'd through nfs-testing and nfsd-testing (so 6.15 NFS and NFSD going on 6.16).

But I also just verified that this kernel built on Chuck's nfsd-testing branch (with 2 extra patches) has the same issue:
https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=cel-nfsd-testing-6.16

Here is the NFS related config:

CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m CONFIG_PNFS_FLEXFILE_LAYOUT=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" # CONFIG_NFS_V4_1_MIGRATION is not set CONFIG_NFS_V4_SECURITY_LABEL=y CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFS_DEBUG=y CONFIG_NFS_DISABLE_UDP_SUPPORT=y
# CONFIG_NFS_V4_2_READ_PLUS is not set CONFIG_NFSD=m
# CONFIG_NFSD_V2 is not set CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_PNFS=y
# CONFIG_NFSD_BLOCKLAYOUT is not set CONFIG_NFSD_SCSILAYOUT=y
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_V4_2_INTER_SSC is not set CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_LEGACY_CLIENT_TRACKING is not set # CONFIG_NFSD_V4_DELEG_TIMESTAMPS is not set CONFIG_GRACE_PERIOD=m
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m CONFIG_NFS_LOCALIO=y CONFIG_NFS_V4_2_SSC_HELPER=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_RPCSEC_GSS_KRB5=m CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA1=y CONFIG_RPCSEC_GSS_KRB5_ENCTYPES_AES_SHA2=y CONFIG_SUNRPC_DEBUG=y CONFIG_SUNRPC_XPRT_RDMA=m

Quoted 4 lines

FWIW, I just built a localio-enabled on a v6.12-uek kernel for our own purposes yesterday and it's running pool_mode=pernode. It seemed to work fine as a v3 DS, but I didn't test mounting the container's export directly.

OK, but you were able to access the v3 DS just fine (assuming pNFS flexfiles layouts that point to your DS that is running NFSD in a container) ?

Quoted 6 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

# cat /proc/8087/stack
[<0>] rpc_wait_bit_killable+0x25/0x80 [sunrpc]
[<0>] __rpc_execute+0x151/0x480 [sunrpc]
[<0>] rpc_execute+0xca/0xf0 [sunrpc]
[<0>] rpc_run_task+0x110/0x180 [sunrpc]
[<0>] nfs4_call_sync_custom+0xb/0x30 [nfsv4]
[<0>] nfs4_do_call_sync+0x69/0x90 [nfsv4]
[<0>] _nfs4_proc_getattr+0x128/0x160 [nfsv4]
[<0>] nfs4_proc_getattr+0x73/0x100 [nfsv4]
[<0>] nfs4_do_open+0x775/0x9d0 [nfsv4]
[<0>] nfs4_atomic_open+0xf7/0x100 [nfsv4]
[<0>] nfs_atomic_open+0x1e7/0x6c0 [nfs]
[<0>] path_openat+0xd38/0x11f0
[<0>] do_filp_open+0xae/0x120
[<0>] do_sys_openat2+0x24d/0x2a0
[<0>] do_sys_open+0x4f/0x90
[<0>] do_syscall_64+0x7b/0x160
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

And if I just try to mount using v3 it fails with:

# mount -vvvvvvv -o vers=3,nolock 10.200.80.89:/cvol_12_0 /mnt/test
mount.nfs: timeout set for Fri May 23 22:52:04 2025
mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported for /mnt/test
# rpcinfo -p 10.200.80.89
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100024    1   udp  45252  status
    100024    1   tcp  60557  status
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100021    1   udp  40987  nlockmgr
    100021    3   udp  40987  nlockmgr
    100021    4   udp  40987  nlockmgr
    100021    1   tcp  36527  nlockmgr
    100021    3   tcp  36527  nlockmgr
    100021    4   tcp  36527  nlockmgr

(Not sure what's up with portmap issues and it not progressing to trying program 100005.. which as you can see below it does)

But if I just use sunrpc's default pool_mode=global:

# mount -vvvvvvv -o vers=3,nolock 10.200.80.89:/cvol_12_0 /mnt/test
mount.nfs: timeout set for Fri May 23 22:55:43 2025
mount.nfs: trying text-based options 'vers=3,nolock,addr=10.200.80.89'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.200.80.89 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.200.80.89 prog 100005 vers 3 prot UDP port 20048
# rpcinfo -p 10.200.80.89
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  54037  status
    100024    1   tcp  46339  status
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100021    1   udp  36268  nlockmgr
    100021    3   udp  36268  nlockmgr
    100021    4   udp  36268  nlockmgr
    100021    1   tcp  44195  nlockmgr
    100021    3   tcp  44195  nlockmgr
    100021    4   tcp  44195  nlockmgr
#6 lore raw
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

Mike

maintainer
#7 lore raw
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

This is probably a kernel problem.

Newer nfs-utils uses nfsdctl to start the server, whereas the older nfs-utils would use rpc.nfsd. nfsdctl uses the netlink interfaces to start the server instead of /proc/fs/nfsd which rpc.nfsd uses.

If you're getting different results with the two different nfs-utils versions then the problem is likely there. I'll take a look.

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#8 lore raw
In reply to Mike Snitzer
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode
Signature · 1 line
Jeff Layton <[email protected]>
#9 lore raw
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Quoted 5 lines

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode

(from EL9.5 container with nfs-utils 2.8.2)

# nfsdctl pool-mode
pool-mode: pernode
npools: 2

(on host)

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 11665 MB
node 0 free: 9892 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 6042 MB
node 1 free: 5127 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)

maintainer
#10 lore raw
Quoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Quoted 5 lines

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode

(from EL9.5 container with nfs-utils 2.8.2)

# nfsdctl pool-mode
pool-mode: pernode
npools: 2

(on host)

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 11665 MB
node 0 free: 9892 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 6042 MB
node 1 free: 5127 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10
Quoted 3 lines

(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)

rpc.nfsd creates sockets in userland and then passes them to the kernel. With the netlink interface, we now create the socket in the kernel. If I had to guess, the problem is related to that difference.

I can check out a NUMA box at work this week and try to reproduce this, if you don't figure it out over the weekend.

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#11 lore raw
In reply to Mike Snitzer
Quoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Quoted 5 lines

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode

(from EL9.5 container with nfs-utils 2.8.2)

# nfsdctl pool-mode
pool-mode: pernode
npools: 2

(on host)

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 11665 MB
node 0 free: 9892 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 6042 MB
node 1 free: 5127 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10
Quoted 3 lines

(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)

Ok, I can reproduce this on a true NUMA machine. The first thing that's interesting is that it seems to be intermittent. Occasionally I can mount and operate on the socket, but socket requests hang most of the time.

I turned up all of the nfsd and sunrpc tracepoints. After attempting a mount that hung, I see only a single tracepoint fire:

          <idle>-0       [038] ..s..  5942.572721: svc_xprt_enqueue: server=[::]:2049 client=(einval) flags=CONN|CHNGBUF|LISTENER|CACHE_AUTH|CONG_CTRL

Based on the flags, svc_xprt_ready should have returned true. That should make the xprt be enqueued and an idle thread be awoken. It looks like that last bit may not be happening for some reason.

At this point, I'll probably have to add some debugging. I'll keep poking at it -- stay tuned.

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#12 lore raw
Quoted 92 lines · Jeff Layton
Quoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Quoted 5 lines

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode

(from EL9.5 container with nfs-utils 2.8.2)

# nfsdctl pool-mode
pool-mode: pernode
npools: 2

(on host)

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 11665 MB
node 0 free: 9892 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 6042 MB
node 1 free: 5127 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10
Quoted 3 lines

(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)

Ok, I can reproduce this on a true NUMA machine. The first thing that's interesting is that it seems to be intermittent. Occasionally I can mount and operate on the socket, but socket requests hang most of the time.

I turned up all of the nfsd and sunrpc tracepoints. After attempting a mount that hung, I see only a single tracepoint fire:

Quoted 8 lines
          <idle>-0       [038] ..s..  5942.572721: svc_xprt_enqueue: server=[::]:2049 client=(einval) flags=CONN|CHNGBUF|LISTENER|CACHE_AUTH|CONG_CTRL

Based on the flags, svc_xprt_ready should have returned true. That should make the xprt be enqueued and an idle thread be awoken. It looks like that last bit may not be happening for some reason.

At this point, I'll probably have to add some debugging. I'll keep poking at it -- stay tuned.

Ok, I figured it out. The problem is that the netlink interfaces don't correctly handle the special case of a non-global pool_mode and userland passing down a single-element array.

The old interface interpreted that as "distribute all of these threads amongst the available nodes", but the netlink interfaces take it literally and start all of the threads in the first node. If a socket request comes in on the other node, then you're out of luck and the socket never gets serviced.

I have a patch I'm testing now that should do the right thing. In the meantime, if you explicitly set the number of threads in each pool it should work around the problem. E.g., if you want 128 threads total, distributed evenly over both numa nodes, try a setting like this for now:

[nfsd]
threads=64,64

I'll send a proper fix for the kernel once I've tested it a bit more.

Thanks for the bug report!

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#13 lore raw
In reply to Mike Snitzer
Quoted 79 lines · Mike Snitzer
Quoted 53 lines · Jeff Layton
Quoted 39 lines · Mike Snitzer
Quoted 23 lines · Mike Snitzer
Quoted 8 lines · Jeff Layton
Quoted 7 lines · Mike Snitzer
Quoted 6 lines · Jeff Layton
Quoted 5 lines · Mike Snitzer

I don't know if $SUBJECT ever worked... but with latest 6.15 or nfsd-testing if I just use pool_mode=global then all is fine.

If pool_mode=pernode then mounting the container's NFSv3 export fails.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I haven't started to dig into code yet but pool_mode=pernode works perfectly fine if NFSD isn't running in a container.

Oops, I went and looked and nfsd isn't running in a container on these boxes. There are some other containerized apps running on the box, but nfsd isn't running in a container.

OK.

Quoted 4 lines
Quoted 3 lines

I'm using nfs-utils-2.8.2. I don't see any nfsd threads running if I use "options sunrpc pool_mode=pernode".

Quoted 7 lines
Quoted 2 lines

I'll have a look soon, but if you figure it out in the meantime, let us know.

Will do.

Just the latest info I have, with sunrpc's pool_mode=pernode dd hangs with this stack trace:

Turns out this pool_mode=pernode issue is a regression caused by the very recent nfs-utils 2.8.2 (I rebuilt EL10's nfs-utils package, because why not upgrade to the latest!?).

If I use EL9.5's latest nfs-utils-2.5.4-37.el8.x86_64 then sunrpc's pool_mode=pernode works fine.

Quoted 8 lines

And this issue doesn't have anything to do with running in a container (it seemed to be container related purely because I happened to be seeing the issue with an EL9.5 container that had the EL10-based nfs-utils 2.8.2 installed).

Steved, unfortunately I'm not sure what the problem is with the newer nfs-utils and setting "options sunrpc pool_mode=pernode"

I tried to reproduce this using fedora-41 VMs (no f42 available for virt-builder yet), but everything worked. I don't have any actual NUMA hw here though, so maybe that matters?

Quoted 5 lines

Can you run this on the nfs server and send back the output? I'm wondering if this setting might not track the module option properly on that host for some reason:

    # nfsdctl pool-mode

(from EL9.5 container with nfs-utils 2.8.2)

# nfsdctl pool-mode
pool-mode: pernode
npools: 2

(on host)

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 11665 MB
node 0 free: 9892 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 6042 MB
node 1 free: 5127 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10
Quoted 3 lines

(and yeahh I was aware the newer nfs-utils uses the netlink interface, will be interesting to pin down what the issue is with pool-mode=pernode)

Hi Mike,

I submitted a patch for this a couple of weeks ago:

    https://lore.kernel.org/linux-nfs/[email protected]/

Were you able to test it, and did it fix your issue?

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>