Skip to contentSkip to conversation

leaked pNFS DS nfs_client references

DISCUSSION Jeff Layton 1 participant 2 messages Apr 21, 2025 – Apr 23, 2025

Subsystems (2)
lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

Hi Trond/Anna:

We (at Meta) have been hunting a number of problems surrounding leaked network namespaces with containerized workloads. We recently deployed a v6.9 based kernel on the clients that has all the known containerized NFS fixes from upstream.

Usually, when we've found problems with leaked netns's it has been because there were still outstanding RPCs associated with the rpc_clnt. Today, we found a host that seems to have some leaked nfs_client structures, but there is no associated RPC activity.

In this case, we had 2 leaked net namespaces. We discovered them by looking under /sys/kernel/debugfs/rpc_xprt for xprts associated with netns's that no longer have any userland tasks attached.

Some drgn (pardon my terrible Python):

Quoted 1 line
Quoted 1 line
Quoted 1 line

for net in for_each_net():

... if (net.ns.inum == 4026558887 or net.ns.inum == 4026558805):

...         print("netns:", net.ns.inum)                                                               
...         nfs_net = cast("struct nfs_net *", net.gen.ptr[prog["nfs_net_id"]])
...         print("Volume list empty:", list_empty(nfs_net.nfs_volume_list.address_of_()))
...         for clnt in list_for_each_entry("struct nfs_client", nfs_net.nfs_client_list.address_of_(), "cl_share_link"):

... rpcclnt = clnt.cl_rpcclient
... print(clnt.cl_count.refs.counter, clnt.cl_hostname, rpcclnt.cl_vers, "tasks: ", list_count_nodes(rpcclnt.cl_tasks.address_of_()))
...
netns: (unsigned int)4026558805
Volume list empty: True
(int)1 (char *)0xffff8a12e988a500 = "f00::3117:a4f1:a940:94af" (u32)3 tasks: 0 (int)1 (char *)0xffff8881a0f694c0 = "f00::bfaa:cec2:8ee2:295" (u32)3 tasks: 0 (int)1 (char *)0xffff889e81a74e40 = "f00::8f23:f52d:9d79:a7b0" (u32)3 tasks: 0 (int)1 (char *)0xffff8a027d8e0780 = "f00::d209:97ba:1c6:3282" (u32)3 tasks: 0
netns: (unsigned int)4026558887
Volume list empty: True
(int)1 (char *)0xffff8a14d5b0e2c0 = "f00::3f52:fea6:4ccb:96dd" (u32)3 tasks: 0 (int)1 (char *)0xffff8881e6626cc0 = "f00::705:c924:ddc1:51e4" (u32)3 tasks: 0

(int)1 (char *)0xffff8a149cdb6680 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
(int)1 (char *)0xffff8896ada2f800 = "f00::d56c:cd93:1f0c:99c7" (u32)3 tasks:  0
(int)1 (char *)0xffff8a159251f240 = "f00::614d:87c1:a73f:1f09" (u32)3 tasks:  0
(int)1 (char *)0xffff888e699f4940 = "f00::1285:b785:f114:d38b" (u32)3 tasks:  0
(int)1 (char *)0xffff88812ae41500 = "f00::fb1c:bc4a:3d9a:c2a6" (u32)3 tasks:  0
(int)1 (char *)0xffff8a137dbc4e00 = "f00::bd2f:5851:b552:5bce" (u32)3 tasks:  0

There are 12 leaked nfs_clients in 2 netns's. There are no longer any struct nfs_servers associated with either netns. Each leaked client has a single outstanding reference. They're all connections to different DS's (except for one between the two netns's, but I suspect that's just coincidence). They're all NFSv3, which indicates that they are pNFS DS clients. None of them have any running RPCs.

I took a look at the nfs_client refcount handling in the pNFS code but didn't see any obvious bugs.

One thing we could consider is adding a refcount tracker for these objects. That would tell us pretty quickly what took the leftover references in the first place, assuming this is reproducible.

This kernel is based on v6.9, so it's possible we missed a fix that we need. I didn't see anything obvious in recent git fixes though.

Any thoughts?

Signature · 1 line
Jeff Layton <[email protected]>
author maintainer
#1 lore raw
Quoted 54 lines · Jeff Layton

Hi Trond/Anna:

We (at Meta) have been hunting a number of problems surrounding leaked network namespaces with containerized workloads. We recently deployed a v6.9 based kernel on the clients that has all the known containerized NFS fixes from upstream.

Usually, when we've found problems with leaked netns's it has been because there were still outstanding RPCs associated with the rpc_clnt. Today, we found a host that seems to have some leaked nfs_client structures, but there is no associated RPC activity.

In this case, we had 2 leaked net namespaces. We discovered them by looking under /sys/kernel/debugfs/rpc_xprt for xprts associated with netns's that no longer have any userland tasks attached.

Some drgn (pardon my terrible Python):

Quoted 1 line
Quoted 1 line
Quoted 1 line

for net in for_each_net():

... if (net.ns.inum == 4026558887 or net.ns.inum == 4026558805):

...         print("netns:", net.ns.inum)                                                               
...         nfs_net = cast("struct nfs_net *", net.gen.ptr[prog["nfs_net_id"]])
...         print("Volume list empty:", list_empty(nfs_net.nfs_volume_list.address_of_()))
...         for clnt in list_for_each_entry("struct nfs_client", nfs_net.nfs_client_list.address_of_(), "cl_share_link"):

... rpcclnt = clnt.cl_rpcclient
... print(clnt.cl_count.refs.counter, clnt.cl_hostname, rpcclnt.cl_vers, "tasks: ", list_count_nodes(rpcclnt.cl_tasks.address_of_()))
...
netns: (unsigned int)4026558805
Volume list empty: True
(int)1 (char *)0xffff8a12e988a500 = "f00::3117:a4f1:a940:94af" (u32)3 tasks: 0 (int)1 (char *)0xffff8881a0f694c0 = "f00::bfaa:cec2:8ee2:295" (u32)3 tasks: 0 (int)1 (char *)0xffff889e81a74e40 = "f00::8f23:f52d:9d79:a7b0" (u32)3 tasks: 0 (int)1 (char *)0xffff8a027d8e0780 = "f00::d209:97ba:1c6:3282" (u32)3 tasks: 0
netns: (unsigned int)4026558887
Volume list empty: True
(int)1 (char *)0xffff8a14d5b0e2c0 = "f00::3f52:fea6:4ccb:96dd" (u32)3 tasks: 0 (int)1 (char *)0xffff8881e6626cc0 = "f00::705:c924:ddc1:51e4" (u32)3 tasks: 0

(int)1 (char *)0xffff8a149cdb6680 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
(int)1 (char *)0xffff8896ada2f800 = "f00::d56c:cd93:1f0c:99c7" (u32)3 tasks:  0
(int)1 (char *)0xffff8a159251f240 = "f00::614d:87c1:a73f:1f09" (u32)3 tasks:  0
(int)1 (char *)0xffff888e699f4940 = "f00::1285:b785:f114:d38b" (u32)3 tasks:  0
(int)1 (char *)0xffff88812ae41500 = "f00::fb1c:bc4a:3d9a:c2a6" (u32)3 tasks:  0
(int)1 (char *)0xffff8a137dbc4e00 = "f00::bd2f:5851:b552:5bce" (u32)3 tasks:  0

There are 12 leaked nfs_clients in 2 netns's. There are no longer any struct nfs_servers associated with either netns. Each leaked client has a single outstanding reference. They're all connections to different DS's (except for one between the two netns's, but I suspect that's just coincidence). They're all NFSv3, which indicates that they are pNFS DS clients. None of them have any running RPCs.

I took a look at the nfs_client refcount handling in the pNFS code but didn't see any obvious bugs.

Quoted 8 lines

One thing we could consider is adding a refcount tracker for these objects. That would tell us pretty quickly what took the leftover references in the first place, assuming this is reproducible.

This kernel is based on v6.9, so it's possible we missed a fix that we need. I didn't see anything obvious in recent git fixes though.

Any thoughts?

An update:

Omar and I worked together yesterday, and confirmed that this is the same problem that he reported a week or two ago. This kernel has the two patches you sent on April 6th:

[PATCH v2 1/2] NFSv4: Handle fatal ENETDOWN and ENETUNREACH errors [PATCH v2 2/2] NFSv4/pnfs: Layoutreturn on close must handle fatal networking errors

...so that's evidently not enough to fix it.

Note that this is a potential memory corruptor. The leaked layout segs were all sitting on the layouts plh_return_segs list. If they get freed later, then that will likely scribble over the list_head in the freed layout.

Looking over the code, it appears that when the inode is evicted, the plh_return_segs list should get cleaned out via:

nfs4_evict_inode
pnfs_destroy_layout_final
__pnfs_destroy_layout
pnfs_mark_layout_stateid_invalid
pnfs_free_returned_lsegs

pnfs_free_returned_lsegs() should put them all on the tmp_list and then __pnfs_destroy_layout() should free the contents of the list via pnfs_free_lseg_list().

It's not clear to me why that didn't happen here.

Signature · 1 line
Jeff Layton <[email protected]>