Skip to contentSkip to conversation

Change default MSGMNI tunable to scale with lowmem

RFC PATCH Superseded [email protected] 3 participants 11 messages Jan 14, 2008 – Jan 15, 2008

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Change default MSGMNI tunable to scale with lowmem

Resending the set of patches after Andrew's remark about the fact that we should take the lowmem into account, rather than the total ram.

Also, these patches have been enhanced with memory hotplug management.

Note: they now have to applied to 2.6.24-rc7.

-------------------

On large systems we'd like to allow a larger number of message queues. In some cases up to 32K. However simply setting MSGMNI to a larger value may cause problems for smaller systems.

The first patch of this series introduces a default maximum number of message queue ids that scales with the amount of lowmem.

Since msgmni is per namespace and there is no amount of memory dedicated to each namespace so far, the second patch of this series scales msgmni to the number of ipc namespaces.

In the last patch, a notifier block is added to the ipc namespace structure to manage memory hotplug. The callback routine is activated upon memory add/remove and it recomputes msgmni. One callback routine is added to the memory notifier chain each time an ipc namespace is allocated. It is removed when the coresponding ipc namespace is freed.

I still have 1 issue that I'll try to solve next:
. use the notification mechanism to recompute all the msg_ctlmni each time
an ipc namespace is created / removed.

These patches should be applied to 2.6.24-rc7, in the following order:

[PATCH 1/4]: ipc_scale_msgmni_with_lowmem.patch [PATCH 2/4]: ipc_scale_msgmni_with_namespaces.patch [PATCH 3/4]: ipc_slab_memory_callback_prio_to_const.patch [PATCH 4/4]: ipc_recompute_msgmni_on_memory_hotplug.patch

Regards,
Nadia

--

#1 lore raw

RFC PATCH 1/4 Scaling msgmni to the amount of lowmem

[PATCH 01/04]

This patch computes msg_ctlmni to make it scale with the amount of lowmem. msg_ctlmni is now set to make the message queues occupy 1/32 of the available lowmem.

Some cleaning has also been done in the MSGXXX constants:
. MSGPOOL: the msgctl man page says it's not used, but it also defines it as
a size in bytes (the code expresses it in Kbytes).
. MSGSEG definition has been removed since it used only once in msgctl().

---
 include/linux/msg.h |   14 +++++++++++---
 ipc/msg.c           |   44 ++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 53 insertions(+), 5 deletions(-)
include/linux/msg.h +11 -3
Index: linux-2.6.24-rc7/include/linux/msg.h
===================================================================
@@ -49,17 +49,25 @@ struct msginfo {
 	unsigned short  msgseg; 
 };
 
+/*
+ * Scaling factor to compute msgmni:
+ * the memory dedicated to msg queues (msgmni * msgmnb) should occupy
+ * at most 1/MSG_MEM_SCALE of the lowmem (see the formula in ipc/msg.c):
+ * up to 8MB       : msgmni = 16 (MSGMNI)
+ * 4 GB            : msgmni = 8K
+ * more than 16 GB : msgmni = 32K (IPCMNI)
+ */
+#define MSG_MEM_SCALE 32
+
 #define MSGMNI    16   /* <= IPCMNI */     /* max # of msg queue identifiers */
 #define MSGMAX  8192   /* <= INT_MAX */   /* max size of message (bytes) */
 #define MSGMNB 16384   /* <= INT_MAX */   /* default max size of a message queue */
 
 /* unused */
-#define MSGPOOL (MSGMNI*MSGMNB/1024)  /* size in kilobytes of message pool */
+#define MSGPOOL (MSGMNI * MSGMNB) /* size in bytes of message pool */
 #define MSGTQL  MSGMNB            /* number of system message headers */
 #define MSGMAP  MSGMNB            /* number of entries in message map */
 #define MSGSSZ  16                /* message segment size */
-#define __MSGSEG ((MSGPOOL*1024)/ MSGSSZ) /* max no. of segments */
-#define MSGSEG (__MSGSEG <= 0xffff ? __MSGSEG : 0xffff)
 
 #ifdef __KERNEL__
 #include <linux/list.h>
ipc/msg.c +42 -2

67 lines not on this page · raw message

--

#2 lore raw

RFC PATCH 2/4 Scaling msgmni to the number of ipc namespaces

In reply to [email protected]

[PATCH 02/04]

Since all the namespaces see the same amount of memory (the total one) this patch introduces a new variable that counts the ipc namespaces and divides msg_ctlmni by this counter.

---
 include/linux/ipc.h |    1 +
 ipc/msg.c           |    8 ++++++--
 ipc/util.c          |    7 +++++++
 3 files changed, 14 insertions(+), 2 deletions(-)
include/linux/ipc.h +1 -0
Index: linux-2.6.24-rc7/include/linux/ipc.h
===================================================================
@@ -121,6 +121,7 @@ struct ipc_namespace {
 };
 
 extern struct ipc_namespace init_ipc_ns;
+extern atomic_t nr_ipc_ns;
 
 #ifdef CONFIG_SYSVIPC
 #define INIT_IPC_NS(ns)		.ns		= &init_ipc_ns,
ipc/util.c +7 -0
Index: linux-2.6.24-rc7/ipc/util.c
===================================================================
@@ -51,6 +51,9 @@ struct ipc_namespace init_ipc_ns = {
 	},
 };
 
+atomic_t nr_ipc_ns = ATOMIC_INIT(1);
+
+
 static struct ipc_namespace *clone_ipc_ns(struct ipc_namespace *old_ns)
 {
 	int err;
@@ -61,6 +64,8 @@ static struct ipc_namespace *clone_ipc_n
 	if (ns == NULL)
 		goto err_mem;
 
+	atomic_inc(&nr_ipc_ns);
+
 	err = sem_init_ns(ns);
 	if (err)
 		goto err_sem;
@@ -80,6 +85,7 @@ err_msg:
 	sem_exit_ns(ns);
 err_sem:
 	kfree(ns);
+	atomic_dec(&nr_ipc_ns);
 err_mem:
 	return ERR_PTR(err);
 }
@@ -109,6 +115,7 @@ void free_ipc_ns(struct kref *kref)
 	msg_exit_ns(ns);
 	shm_exit_ns(ns);
 	kfree(ns);
+	atomic_dec(&nr_ipc_ns);
 }
 
 /**
ipc/msg.c +6 -2
Index: linux-2.6.24-rc7/ipc/msg.c
===================================================================
@@ -83,15 +83,19 @@ static int sysvipc_msg_proc_show(struct 
 /*
  * Scale msgmni with the available lowmem size: the memory dedicated to msg
  * queues should occupy at most 1/MSG_MEM_SCALE of lowmem.
+ * Also take into account the number of nsproxies created so far.
  */
 static void recompute_msgmni(struct ipc_namespace *ns)
 {
 	struct sysinfo i;
 	unsigned long allowed;
+	int nb_ns;
 
 	si_meminfo(&i);
 	allowed = (((i.totalram - i.totalhigh) / MSG_MEM_SCALE) * i.mem_unit)
 		/ MSGMNB;
+	nb_ns = atomic_read(&nr_ipc_ns);
+	allowed /= nb_ns;
 
 	/*
 	 * Never fall under the current minimum value
@@ -104,8 +108,8 @@ static void recompute_msgmni(struct ipc_
 	/*
 	 * Never go over the current max value
 	 */
-	if (allowed > IPCMNI) {
-		ns->msg_ctlmni = IPCMNI;
+	if (allowed > IPCMNI / nb_ns) {
+		ns->msg_ctlmni = IPCMNI / nb_ns;
 		goto out_callback;
 	}
 

--

#3 lore raw

RFC PATCH 3/4 Defining the slab_memory_callback priority as a constant

In reply to [email protected]

[PATCH 03/04]

This is a trivial patch that defines the priority of slab_memory_callback in the callback chain as a constant.
This is to prepare for next patch in the series.

---
 include/linux/memory.h |    6 ++++++
 mm/slub.c              |    2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/memory.h +6 -0
Index: linux-2.6.24-rc7/include/linux/memory.h
===================================================================
@@ -54,6 +54,12 @@ struct memory_notify {
 struct notifier_block;
 struct mem_section;
 
+/*
+ * Priorities for the hotplug memory callback routines (stored in decreasing
+ * order in the callback chain)
+ */
+#define SLAB_CALLBACK_PRI       1
+
 #ifndef CONFIG_MEMORY_HOTPLUG_SPARSE
 static inline int memory_dev_init(void)
 {
mm/slub.c +1 -1
Index: linux-2.6.24-rc7/mm/slub.c
===================================================================
@@ -2816,7 +2816,7 @@ void __init kmem_cache_init(void)
 	kmalloc_caches[0].refcount = -1;
 	caches++;
 
-	hotplug_memory_notifier(slab_memory_callback, 1);
+	hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI);
 #endif
 
 	/* Able to allocate the per node structures */

--

#4 lore raw

RFC PATCH 4/4 Recomputing msgmni on memory add / remove

In reply to [email protected]

[PATCH 04/04]

This patch introduces the registration of a callback routine that recomputes msg_ctlmni upon memory add / remove.

A notifier block structure has been added to the ipc namespace structure.

Each time a new ipc namespace is allocated, this notifier block callback routine is registered into the memory hotplug notifier chain (it is unregistered when the ipc namespace is freed). That routine is then activated when memory is added or removed, in order to recompute msgmni for that namespace.

---
 include/linux/ipc.h    |   25 +++++++++++++++++++++
 include/linux/memory.h |    1 
 ipc/msg.c              |    2 -
 ipc/util.c             |   57 +++++++++++++++++++++++++++++++++++++++++++++++++
 ipc/util.h             |    2 +
 5 files changed, 86 insertions(+), 1 deletion(-)
include/linux/ipc.h +25 -0
Index: linux-2.6.24-rc7/include/linux/ipc.h
===================================================================
@@ -81,6 +81,10 @@ struct ipc_kludge {
 
 #include <linux/kref.h>
 #include <linux/spinlock.h>
+#ifdef CONFIG_MEMORY_HOTPLUG
+#include <linux/notifier.h>
+#include <linux/memory.h>
+#endif /* CONFIG_MEMORY_HOTPLUG */
 
 #define IPCMNI 32768  /* <= MAX_INT limit for ipc arrays (including sysctl changes) */
 
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };
 
 extern struct ipc_namespace init_ipc_ns;
@@ -153,6 +161,23 @@ static inline void put_ipc_ns(struct ipc
 #endif
 }
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+
+extern void register_ipc_hotplug_notifier(struct ipc_namespace *);
+
+static inline void unregister_ipc_hotplug_notifier(struct ipc_namespace *ns)
+{
+	unregister_memory_notifier(&ns->ipc_memory_hotplug);
+}
+
+#else /* CONFIG_MEMORY_HOTPLUG */
+
+#define register_ipc_hotplug_notifier(ns)   do { } while (0)
+#define unregister_ipc_hotplug_notifier(ns)  do { } while (0)
+
+#endif /* CONFIG_MEMORY_HOTPLUG */
+
+
 #endif /* __KERNEL__ */
 
 #endif /* _LINUX_IPC_H */
include/linux/memory.h +1 -0
Index: linux-2.6.24-rc7/include/linux/memory.h
===================================================================
@@ -59,6 +59,7 @@ struct mem_section;
  * order in the callback chain)
  */
 #define SLAB_CALLBACK_PRI       1
+#define IPC_CALLBACK_PRI        10
 
 #ifndef CONFIG_MEMORY_HOTPLUG_SPARSE
 static inline int memory_dev_init(void)
ipc/util.h +2 -0
Index: linux-2.6.24-rc7/ipc/util.h
===================================================================
@@ -134,6 +134,8 @@ extern int ipcget_new(struct ipc_namespa
 extern int ipcget_public(struct ipc_namespace *, struct ipc_ids *,
 			struct ipc_ops *, struct ipc_params *);
 
+extern void recompute_msgmni(struct ipc_namespace *);
+
 static inline int ipc_buildid(int id, int seq)
 {
 	return SEQ_MULTIPLIER * seq + id;
ipc/util.c +57 -0

87 lines not on this page · raw message

ipc/msg.c +1 -1
Index: linux-2.6.24-rc7/ipc/msg.c
===================================================================
@@ -85,7 +85,7 @@ static int sysvipc_msg_proc_show(struct 
  * queues should occupy at most 1/MSG_MEM_SCALE of lowmem.
  * Also take into account the number of nsproxies created so far.
  */
-static void recompute_msgmni(struct ipc_namespace *ns)
+void recompute_msgmni(struct ipc_namespace *ns)
 {
 	struct sysinfo i;
 	unsigned long allowed;

--

#5 lore raw

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

Hello Nadia-san.

include/linux/ipc.h struct ipc_namespace { · line 129

Quoted 9 lines
include/linux/ipc.h +4 -0
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };

I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, the recompute_msgmni() calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?

BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)

Bye.

Signature · 1 line
Yasunori Goto 
#6 lore raw

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

Quoted 14 lines · Yasunori Goto

Hello Nadia-san.

include/linux/ipc.h struct ipc_namespace { · line 129

Quoted 9 lines
include/linux/ipc.h +4 -0
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };
Quoted 7 lines

I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, the recompute_msgmni() calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?

Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces. I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.

Quoted 4 lines

BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)

Well, I tested it but not in "real configuration": what I did is that I changed the status by hand under sysfs to offline. I also changed remove_memory() in mm/memory_hotplug.c in the following way (instead of returninf EINVAL):
1) decrease the total_ram pages
2) call memory_notify(MEM_OFFLINE, NULL)

and checked that the msgmni was recomputed.

But sure, if you are candidate to test it, that would be great!

Regards,
Nadia

#7 lore raw

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

Quoted 23 lines
Quoted 14 lines · Yasunori Goto

Hello Nadia-san.

include/linux/ipc.h struct ipc_namespace { · line 129

Quoted 9 lines
include/linux/ipc.h +4 -0
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };
Quoted 7 lines

I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, the recompute_msgmni() calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?

Quoted 6 lines

Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces.
I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.

The recompute_msg() must be called when new ipc_namespace is created/removed as you mentioned. I think namespaces should be linked each other for it in the end....

Quoted 4 lines
Quoted 3 lines

BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)

Quoted 8 lines

Well, I tested it but not in "real configuration": what I did is that I changed the status by hand under sysfs to offline. I also changed remove_memory() in mm/memory_hotplug.c in the following way (instead of returninf EINVAL):
1) decrease the total_ram pages
2) call memory_notify(MEM_OFFLINE, NULL)

and checked that the msgmni was recomputed.

You can also online again after offline by writing sysfs.

Quoted 1 line

But sure, if you are candidate to test it, that would be great!

Ok. I'll check it too.
Bye.

Signature · 1 line
Yasunori Goto 
#8 lore raw

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

Quoted 25 lines · Yasunori Goto
Quoted 25 lines
Quoted 22 lines · Yasunori Goto

Hello Nadia-san.

include/linux/ipc.h struct ipc_namespace { · line 129

Quoted 9 lines
include/linux/ipc.h +4 -0
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };

I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, the recompute_msgmni() calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?

Quoted 7 lines
Quoted 2 lines

Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces. I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.

The recompute_msg() must be called when new ipc_namespace is created/removed as you mentioned. I think namespaces should be linked each other for it in the end....

Not if I do it the same way as for memory hotplug:
1) definee a "namespace event notifier"
2) insert another notifier block into the ipc namespace.
3) The callback routines in the notifier chain would be activated upon namespace creation / removal.

But I'm still thinking about it: Matt Helsley suggested that we might just copy the old namespace's value when creating a new namespace.

There's also the issue of an msgmni that has been changed via procfs: may be we should not unconditionally recompute it upon namespace creation / removal, so unregister the callback for the owning namespace as soon as msgmni has been changed from userspace.

Regards,
Nadia

#9 lore raw
In reply to [email protected]

I'm realizing that I wrote Andrew's address wrong. So please if ever you're answering in this thread correct his address by hand. Sorry for that!
Now if you find it more convenient that I start a new thread with the right address, please tell me and I'll do it.

Regards,
Nadia

#10 lore raw
Quoted 8 lines · Nadia Derbey

I'm realizing that I wrote Andrew's address wrong. So please if ever you're answering in this thread correct his address by hand. Sorry for that!
Now if you find it more convenient that I start a new thread with the right address, please tell me and I'll do it.

Regards,
Nadia

Not a big deal. If I do reply please double check that I haven't forgotten. It's easy to miss -- I only just noticed this :)

Cheers,
-Matt