Skip to contentSkip to conversation

Make nfs stats visible in network NS

PATCH Superseded Josef Bacik 2 participants 5 messages Jan 26, 2024 – Jan 29, 2024

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover Make nfs stats visible in network NS

v1: https://lore.kernel.org/linux-nfs/[email protected]/
v2: https://lore.kernel.org/linux-nfs/[email protected]/
v3: https://lore.kernel.org/linux-nfs/[email protected]/

v3->v4:
- Fix a weird formatting thing that snuck into 1/3.

v2->v3:
- Split out the nfs and nfsd related changes into their own patches.
- Dropped the change adding sv_stats throuch svc_create()
- Changed the th_cnt to be global, re-arranged it's location.

v1->v2:
- rework the sunprc service creation to take a pointer to the sv_stats.
- dropped ->pg_stats from the svc_program.
- converted all of the nfsd global stats to per-network namespace.
- added the ability to point at a specific rpc_stat for rpc program creation.
- converted the rpc stats for nfs to per-network namespace.

-- Original email --
Hello,

We're currently deploying NFS internally and have run into some oddities with our usage of containers. All of the services that mount and export NFS volumes run inside of containers, specifically all the namespaces including network namespaces. Our monitoring is done on a per-container basis, so we need access to the nfs and nfsd stats that are under /proc/net/sunrpc. However these are only tied to the init_net, which makes them invisible to containers in a different network namespace.

Fix this so that these files are tied to the network namespace. This allows us to avoid the hack of bind mounting the hosts /proc into the container in order to do proper monitoring. Thanks,

Josef

Josef Bacik (3):
sunrpc: add a struct rpc_stats arg to rpc_create_args
nfs: expose /proc/net/sunrpc/nfs in net namespaces
nfs: make the rpc_stat per net namespace

 fs/nfs/client.c             | 5 ++++-
 fs/nfs/inode.c              | 8 ++++----
 fs/nfs/internal.h           | 2 --
 fs/nfs/netns.h              | 2 ++
 include/linux/sunrpc/clnt.h | 1 +
 net/sunrpc/clnt.c           | 2 +-
 6 files changed, 12 insertions(+), 8 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/3 sunrpc: add a struct rpc_stats arg to rpc_create_args

We want to be able to have our rpc stats handled in a per network namespace manner, so add an option to rpc_create_args to specify a different rpc_stats struct instead of using the one on the rpc_program.

---
 include/linux/sunrpc/clnt.h | 1 +
 net/sunrpc/clnt.c           | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
include/linux/sunrpc/clnt.h +1 -0
index 5e9d1469c6fa..5321585c778f 100644
@@ -139,6 +139,7 @@ struct rpc_create_args {
 	const char		*servername;
 	const char		*nodename;
 	const struct rpc_program *program;
+	struct rpc_stat		*stats;
 	u32			prognumber;	/* overrides program->number */
 	u32			version;
 	rpc_authflavor_t	authflavor;
net/sunrpc/clnt.c +1 -1
index cda0935a68c9..bc8c209fc0c7 100644
@@ -405,7 +405,7 @@ static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args,
 	clnt->cl_maxproc  = version->nrprocs;
 	clnt->cl_prog     = args->prognumber ? : program->number;
 	clnt->cl_vers     = version->number;
-	clnt->cl_stats    = program->stats;
+	clnt->cl_stats    = args->stats ? : program->stats;
 	clnt->cl_metrics  = rpc_alloc_iostats(clnt);
 	rpc_init_pipe_dir_head(&clnt->cl_pipedir_objects);
 	err = -ENOMEM;
Signature · 1 line
2.43.0
#2 lore raw

PATCH v4 2/3 nfs: expose /proc/net/sunrpc/nfs in net namespaces

In reply to Josef Bacik

We're using nfs mounts inside of containers in production and noticed that the nfs stats are not exposed in /proc. This is a problem for us as we use these stats for monitoring, and have to do this awkward bind mount from the main host into the container in order to get to these states.

Add the rpc_proc_register call to the pernet operations entry and exit points so these stats can be exposed inside of network namespaces.

---
 fs/nfs/inode.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
fs/nfs/inode.c +2 -4
index ebb8d60e1152..e11e9c34aa56 100644
@@ -2427,11 +2427,13 @@ EXPORT_SYMBOL_GPL(nfs_net_id);
 static int nfs_net_init(struct net *net)
 {
 	nfs_clients_init(net);
+	rpc_proc_register(net, &nfs_rpcstat);
 	return nfs_fs_proc_net_init(net);
 }
 
 static void nfs_net_exit(struct net *net)
 {
+	rpc_proc_unregister(net, "nfs");
 	nfs_fs_proc_net_exit(net);
 	nfs_clients_exit(net);
 }
@@ -2486,15 +2488,12 @@ static int __init init_nfs_fs(void)
 	if (err)
 		goto out1;
 
-	rpc_proc_register(&init_net, &nfs_rpcstat);
-
 	err = register_nfs_fs();
 	if (err)
 		goto out0;
 
 	return 0;
 out0:
-	rpc_proc_unregister(&init_net, "nfs");
 	nfs_destroy_directcache();
 out1:
 	nfs_destroy_writepagecache();
@@ -2524,7 +2523,6 @@ static void __exit exit_nfs_fs(void)
 	nfs_destroy_inodecache();
 	nfs_destroy_nfspagecache();
 	unregister_pernet_subsys(&nfs_net_ops);
-	rpc_proc_unregister(&init_net, "nfs");
 	unregister_nfs_fs();
 	nfs_fs_proc_exit();
 	nfsiod_stop();
Signature · 1 line
2.43.0
#3 lore raw

PATCH v4 3/3 nfs: make the rpc_stat per net namespace

In reply to Josef Bacik

Now that we're exposing the rpc stats on a per-network namespace basis, move this struct into struct nfs_net and use that to make sure only the per-network namespace stats are exposed.

---
 fs/nfs/client.c   | 5 ++++-
 fs/nfs/inode.c    | 4 +++-
 fs/nfs/internal.h | 2 --
 fs/nfs/netns.h    | 2 ++
 4 files changed, 9 insertions(+), 4 deletions(-)
fs/nfs/client.c +4 -1
index 44eca51b2808..4d9249c99989 100644
@@ -73,7 +73,6 @@ const struct rpc_program nfs_program = {
 	.number			= NFS_PROGRAM,
 	.nrvers			= ARRAY_SIZE(nfs_version),
 	.version		= nfs_version,
-	.stats			= &nfs_rpcstat,
 	.pipe_dir_name		= NFS_PIPE_DIRNAME,
 };
 
@@ -502,6 +501,7 @@ int nfs_create_rpc_client(struct nfs_client *clp,
 			  const struct nfs_client_initdata *cl_init,
 			  rpc_authflavor_t flavor)
 {
+	struct nfs_net		*nn = net_generic(clp->cl_net, nfs_net_id);
 	struct rpc_clnt		*clnt = NULL;
 	struct rpc_create_args args = {
 		.net		= clp->cl_net,
@@ -513,6 +513,7 @@ int nfs_create_rpc_client(struct nfs_client *clp,
 		.servername	= clp->cl_hostname,
 		.nodename	= cl_init->nodename,
 		.program	= &nfs_program,
+		.stats		= &nn->rpcstats,
 		.version	= clp->rpc_ops->version,
 		.authflavor	= flavor,
 		.cred		= cl_init->cred,
@@ -1175,6 +1176,8 @@ void nfs_clients_init(struct net *net)
 #endif
 	spin_lock_init(&nn->nfs_client_lock);
 	nn->boot_time = ktime_get_real();
+	memset(&nn->rpcstats, 0, sizeof(nn->rpcstats));
+	nn->rpcstats.program = &nfs_program;
 
 	nfs_netns_sysfs_setup(nn, net);
 }
fs/nfs/inode.c +3 -1
index e11e9c34aa56..91b4d811958a 100644
@@ -2426,8 +2426,10 @@ EXPORT_SYMBOL_GPL(nfs_net_id);
 
 static int nfs_net_init(struct net *net)
 {
+	struct nfs_net *nn = net_generic(net, nfs_net_id);
+
 	nfs_clients_init(net);
-	rpc_proc_register(net, &nfs_rpcstat);
+	rpc_proc_register(net, &nn->rpcstats);
 	return nfs_fs_proc_net_init(net);
 }
 
fs/nfs/internal.h +0 -2
index e3722ce6722e..06253695fe53 100644
@@ -449,8 +449,6 @@ int nfs_try_get_tree(struct fs_context *);
 int nfs_get_tree_common(struct fs_context *);
 void nfs_kill_super(struct super_block *);
 
-extern struct rpc_stat nfs_rpcstat;
-
 extern int __init register_nfs_fs(void);
 extern void __exit unregister_nfs_fs(void);
 extern bool nfs_sb_active(struct super_block *sb);
fs/nfs/netns.h +2 -0
index c8374f74dce1..a68b21603ea9 100644
@@ -9,6 +9,7 @@
 #include <linux/nfs4.h>
 #include <net/net_namespace.h>
 #include <net/netns/generic.h>
+#include <linux/sunrpc/stats.h>
 
 struct bl_dev_msg {
 	int32_t status;
@@ -34,6 +35,7 @@ struct nfs_net {
 	struct nfs_netns_client *nfs_client;
 	spinlock_t nfs_client_lock;
 	ktime_t boot_time;
+	struct rpc_stat rpcstats;
 #ifdef CONFIG_PROC_FS
 	struct proc_dir_entry *proc_nfsfs;
 #endif
Signature · 1 line
2.43.0
maintainer Reviewed-by on the v4 series
#4 lore raw
In reply to Josef Bacik
Quoted 41 lines · Josef Bacik

v1: https://lore.kernel.org/linux-nfs/[email protected]/
v2: https://lore.kernel.org/linux-nfs/[email protected]/
v3: https://lore.kernel.org/linux-nfs/[email protected]/

v3->v4:
- Fix a weird formatting thing that snuck into 1/3.

v2->v3:
- Split out the nfs and nfsd related changes into their own patches.
- Dropped the change adding sv_stats throuch svc_create()
- Changed the th_cnt to be global, re-arranged it's location.

v1->v2:
- rework the sunprc service creation to take a pointer to the sv_stats.
- dropped ->pg_stats from the svc_program.
- converted all of the nfsd global stats to per-network namespace.
- added the ability to point at a specific rpc_stat for rpc program creation.
- converted the rpc stats for nfs to per-network namespace.

-- Original email --
Hello,

We're currently deploying NFS internally and have run into some oddities with our usage of containers. All of the services that mount and export NFS volumes run inside of containers, specifically all the namespaces including network namespaces. Our monitoring is done on a per-container basis, so we need access to the nfs and nfsd stats that are under /proc/net/sunrpc. However these are only tied to the init_net, which makes them invisible to containers in a different network namespace.

Fix this so that these files are tied to the network namespace. This allows us to avoid the hack of bind mounting the hosts /proc into the container in order to do proper monitoring. Thanks,

Josef

Josef Bacik (3):
sunrpc: add a struct rpc_stats arg to rpc_create_args
nfs: expose /proc/net/sunrpc/nfs in net namespaces
nfs: make the rpc_stat per net namespace

Quoted 8 lines
 fs/nfs/client.c             | 5 ++++-
 fs/nfs/inode.c              | 8 ++++----
 fs/nfs/internal.h           | 2 --
 fs/nfs/netns.h              | 2 ++
 include/linux/sunrpc/clnt.h | 1 +
 net/sunrpc/clnt.c           | 2 +-
 6 files changed, 12 insertions(+), 8 deletions(-)

This all looks good to me. You can add: