Skip to contentSkip to conversation

provide sunrpc pool_mode module option

PATCH In review Greg Banks 2 participants 3 messages Feb 21, 2007 – Feb 22, 2007

Subsystems (3)
lore
Share & tools
lore

Greg Banks

author
#0 lore raw

PATCH 1/1 provide sunrpc pool_mode module option

Provide a module option "pool_mode" for sunrpc.ko which allows a sysadmin to choose the mode for mapping svc_pools to CPUs. Values are:

auto choose a mapping mode heuristically
global (default, same as the pre-2.6.19 code) a single global pool
percpu one pool per CPU
pernode one pool per NUMA node

Note that since 2.6.19 the hardcoded behaviour has been "auto", this patch makes the default "global".

---

 net/sunrpc/svc.c |   76 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 60 insertions(+), 16 deletions(-)
net/sunrpc/svc.c +60 -16
Index: linux/net/sunrpc/svc.c
===================================================================
@@ -31,7 +31,7 @@
  * Mode for mapping cpus to pools.
  */
 enum {
-	SVC_POOL_NONE = -1,	/* uninitialised, choose one of the others */
+	SVC_POOL_AUTO = -1,	/* choose one of the others */
 	SVC_POOL_GLOBAL,	/* no mapping, just a single global pool
 				 * (legacy & UP mode) */
 	SVC_POOL_PERCPU,	/* one pool per cpu */
@@ -46,13 +46,55 @@ static struct svc_pool_map {
 	int mode;			/* Note: int not enum to avoid
 					 * warnings about "enumeration value
 					 * not handled in switch" */
+	int initialised;
 	unsigned int npools;
 	unsigned int *pool_to;		/* maps pool id to cpu or node */
 	unsigned int *to_pool;		/* maps cpu or node to pool id */
 } svc_pool_map = {
-	.mode = SVC_POOL_NONE
+	.mode = SVC_POOL_GLOBAL,
+	.initialised = 0
 };
 
+static int
+param_set_pool_mode(const char *val, struct kernel_param *kp)
+{
+	int *ip = (int *)kp->arg;
+
+	if (!strcmp(val, "auto"))
+		*ip = SVC_POOL_AUTO;
+	else if (!strcmp(val, "global"))
+		*ip = SVC_POOL_GLOBAL;
+	else if (!strcmp(val, "percpu"))
+		*ip = SVC_POOL_PERCPU;
+	else if (!strcmp(val, "pernode"))
+		*ip = SVC_POOL_PERNODE;
+	else
+		return -EINVAL;
+	return 0;
+}
+
+static int
+param_get_pool_mode(char *buf, struct kernel_param *kp)
+{
+	int *ip = (int *)kp->arg;
+
+	switch (*ip)
+	{
+	case SVC_POOL_AUTO:
+		return strlcpy(buf, "auto", 20);
+	case SVC_POOL_GLOBAL:
+		return strlcpy(buf, "global", 20);
+	case SVC_POOL_PERCPU:
+		return strlcpy(buf, "percpu", 20);
+	case SVC_POOL_PERNODE:
+		return strlcpy(buf, "pernode", 20);
+	default:
+		return sprintf(buf, "%d", *ip);
+	}
+}
+
+module_param_call(pool_mode, param_set_pool_mode, param_get_pool_mode,
+		 &svc_pool_map.mode, 0644);
 
 /*
  * Detect best pool mapping mode heuristically,
@@ -174,10 +216,12 @@ svc_pool_map_init(void)
 	struct svc_pool_map *m = &svc_pool_map;
 	int npools = -1;
 
-	if (m->mode != SVC_POOL_NONE)
+	if (m->initialised)
 		return m->npools;
+	m->initialised = 1;
 
-	m->mode = svc_pool_map_choose_mode();
+	if (m->mode == SVC_POOL_AUTO)
+		m->mode = svc_pool_map_choose_mode();
 
 	switch (m->mode) {
 	case SVC_POOL_PERCPU:
@@ -212,10 +256,9 @@ svc_pool_map_set_cpumask(unsigned int pi
 
 	/*
 	 * The caller checks for sv_nrpools > 1, which
-	 * implies that we've been initialized and the
-	 * map mode is not NONE.
+	 * implies that we've been initialized.
 	 */
-	BUG_ON(m->mode == SVC_POOL_NONE);
+	BUG_ON(!m->initialised);
 
 	switch (m->mode)
 	{
@@ -246,18 +289,19 @@ svc_pool_for_cpu(struct svc_serv *serv, 
 	unsigned int pidx = 0;
 
 	/*
-	 * SVC_POOL_NONE happens in a pure client when
+	 * !m->initialised happens in a pure client when
 	 * lockd is brought up, so silently treat it the
 	 * same as SVC_POOL_GLOBAL.
 	 */
-
-	switch (m->mode) {
-	case SVC_POOL_PERCPU:
-		pidx = m->to_pool[cpu];
-		break;
-	case SVC_POOL_PERNODE:
-		pidx = m->to_pool[cpu_to_node(cpu)];
-		break;
+	if (m->initialised) {
+		switch (m->mode) {
+		case SVC_POOL_PERCPU:
+			pidx = m->to_pool[cpu];
+			break;
+		case SVC_POOL_PERNODE:
+			pidx = m->to_pool[cpu_to_node(cpu)];
+			break;
+		}
 	}
 	return &serv->sv_pools[pidx % serv->sv_nrpools];
 }
Signature · 11 lines
Greg (not Dave) Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 11 lines · On Thursday February 22, [email protected] wrote:

Provide a module option "pool_mode" for sunrpc.ko which allows a sysadmin to choose the mode for mapping svc_pools to CPUs. Values are:

auto choose a mapping mode heuristically
global (default, same as the pre-2.6.19 code) a single global pool
percpu one pool per CPU
pernode one pool per NUMA node

Note that since 2.6.19 the hardcoded behaviour has been "auto", this patch makes the default "global".

-ENODOC .... Documentation/kernel-parameters.txt should have this
documented as sunrpc.pool_mode

It bugs me slightly that you can only set this once. If you want to try a different option you have to reboot unless sunrpc is a module in which case you can rmmod and insmod.

What would you think of making 'initialised' a count of active services that gets decremented from svc_destroy? Then you could just stop all nfsd (and lockd) threads, write a new value to
/sys/module/sunrpc/parameters/pool_mode
and restart the threads.

I'm happy to hear arguments against this - it is just an idea.

Otherwise, patch looks ok,

Thanks,
NeilBrown

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#2 lore raw
Quoted 10 lines · Neil Brown
Quoted 9 lines · On Thursday February 22, [email protected] wrote:

Provide a module option "pool_mode" for sunrpc.ko which allows a sysadmin to choose the mode for mapping svc_pools to CPUs. Values are:

auto choose a mapping mode heuristically
global (default, same as the pre-2.6.19 code) a single global pool
percpu one pool per CPU
pernode one pool per NUMA node

Quoted 5 lines
Quoted 2 lines

Note that since 2.6.19 the hardcoded behaviour has been "auto", this patch makes the default "global".

-ENODOC .... Documentation/kernel-parameters.txt should have this
documented as sunrpc.pool_mode

My bad, I'll fix it.

Quoted 3 lines

It bugs me slightly that you can only set this once. If you want to try a different option you have to reboot unless sunrpc is a module in which case you can rmmod and insmod.

Indeed.

Quoted 5 lines

What would you think of making 'initialised' a count of active services that gets decremented from svc_destroy? Then you could just stop all nfsd (and lockd) threads, write a new value to
/sys/module/sunrpc/parameters/pool_mode
and restart the threads.

Sounds useful, I'll look into it. V2 patch coming soon.

Greg.

Signature · 11 lines
Greg (not Dave) Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs