Change default MSGMNI tunable to scale with lowmem
Subsystems (3)
cover Change default MSGMNI tunable to scale with lowmem
Resending the set of patches after Andrew's remark about the fact that we should take the lowmem into account, rather than the total ram.
Also, these patches have been enhanced with memory hotplug management.
Note: they now have to applied to 2.6.24-rc7.
-------------------
On large systems we'd like to allow a larger number of message queues. In some cases up to 32K. However simply setting MSGMNI to a larger value may cause problems for smaller systems.
The first patch of this series introduces a default maximum number of message queue ids that scales with the amount of lowmem.
Since msgmni is per namespace and there is no amount of memory dedicated to each namespace so far, the second patch of this series scales msgmni to the number of ipc namespaces.
In the last patch, a notifier block is added to the ipc namespace structure to manage memory hotplug. The callback routine is activated upon memory add/remove and it recomputes msgmni. One callback routine is added to the memory notifier chain each time an ipc namespace is allocated. It is removed when the coresponding ipc namespace is freed.
I still have 1 issue that I'll try to solve next:
. use the notification mechanism to recompute all the msg_ctlmni each time
an ipc namespace is created / removed.
These patches should be applied to 2.6.24-rc7, in the following order:
[PATCH 1/4]: ipc_scale_msgmni_with_lowmem.patch [PATCH 2/4]: ipc_scale_msgmni_with_namespaces.patch [PATCH 3/4]: ipc_slab_memory_callback_prio_to_const.patch [PATCH 4/4]: ipc_recompute_msgmni_on_memory_hotplug.patch
Regards,
Nadia
--
RFC PATCH 1/4 Scaling msgmni to the amount of lowmem
[PATCH 01/04]
This patch computes msg_ctlmni to make it scale with the amount of lowmem. msg_ctlmni is now set to make the message queues occupy 1/32 of the available lowmem.
Some cleaning has also been done in the MSGXXX constants:
. MSGPOOL: the msgctl man page says it's not used, but it also defines it as
a size in bytes (the code expresses it in Kbytes).
. MSGSEG definition has been removed since it used only once in msgctl().
--- include/linux/msg.h | 14 +++++++++++--- ipc/msg.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 53 insertions(+), 5 deletions(-)
include/linux/msg.h +11 -3
@@ -49,17 +49,25 @@ struct msginfo {unsigned short msgseg; }; +/* +* Scaling factor to compute msgmni: +* the memory dedicated to msg queues (msgmni * msgmnb) should occupy +* at most 1/MSG_MEM_SCALE of the lowmem (see the formula in ipc/msg.c): +* up to 8MB : msgmni = 16 (MSGMNI) +* 4 GB : msgmni = 8K +* more than 16 GB : msgmni = 32K (IPCMNI) +*/ +#define MSG_MEM_SCALE 32 +#define MSGMNI 16/* <= IPCMNI */ /* max # of msg queue identifiers */ #define MSGMAX 8192/* <= INT_MAX */ /* max size of message (bytes) */ #define MSGMNB 16384/* <= INT_MAX */ /* default max size of a message queue */ /* unused */ -#define MSGPOOL (MSGMNI*MSGMNB/1024)/* size in kilobytes of message pool */ +#define MSGPOOL (MSGMNI * MSGMNB)/* size in bytes of message pool */ #define MSGTQL MSGMNB/* number of system message headers */ #define MSGMAP MSGMNB/* number of entries in message map */ #define MSGSSZ 16/* message segment size */ -#define __MSGSEG ((MSGPOOL*1024)/ MSGSSZ)/* max no. of segments */ -#define MSGSEG (__MSGSEG <= 0xffff ? __MSGSEG : 0xffff)#ifdef __KERNEL__#include <linux/list.h>
ipc/msg.c +42 -2
67 lines not on this page · raw message
--
RFC PATCH 2/4 Scaling msgmni to the number of ipc namespaces
In reply to [email protected][PATCH 02/04]
Since all the namespaces see the same amount of memory (the total one) this patch introduces a new variable that counts the ipc namespaces and divides msg_ctlmni by this counter.
--- include/linux/ipc.h | 1 + ipc/msg.c | 8 ++++++-- ipc/util.c | 7 +++++++ 3 files changed, 14 insertions(+), 2 deletions(-)
include/linux/ipc.h +1 -0
@@ -121,6 +121,7 @@ struct ipc_namespace { };extern struct ipc_namespace init_ipc_ns; +extern atomic_t nr_ipc_ns;#ifdef CONFIG_SYSVIPC #define INIT_IPC_NS (ns) .ns = &init_ipc_ns,
ipc/util.c +7 -0
@@ -51,6 +51,9 @@ struct ipc_namespace init_ipc_ns = { }, }; +atomic_t nr_ipc_ns =ATOMIC_INIT (1 ); + +static struct ipc_namespace *clone_ipc_ns (struct ipc_namespace *old_ns) {int err;
@@ -61,6 +64,8 @@ static struct ipc_namespace *clone_ipc_nif (ns ==NULL ) goto err_mem; +atomic_inc (&nr_ipc_ns); + err =sem_init_ns (ns);if (err) goto err_sem;
@@ -80,6 +85,7 @@ err_msg:sem_exit_ns (ns); err_sem:kfree (ns); +atomic_dec (&nr_ipc_ns); err_mem:return ERR_PTR (err); }
@@ -109,6 +115,7 @@ void free_ipc_ns(struct kref *kref)msg_exit_ns (ns);shm_exit_ns (ns);kfree (ns); +atomic_dec (&nr_ipc_ns); }/**
ipc/msg.c +6 -2
@@ -83,15 +83,19 @@ static int sysvipc_msg_proc_show(struct/* * Scale msgmni with the available lowmem size: the memory dedicated to msg * queues should occupy at most 1/MSG_MEM_SCALE of lowmem. +* Also take into account the number of nsproxies created so far. */ static void recompute_msgmni (struct ipc_namespace *ns) {struct sysinfo i;unsigned long allowed; +int nb_ns;si_meminfo (&i); allowed = (((i.totalram - i.totalhigh) /MSG_MEM_SCALE ) * i.mem_unit) /MSGMNB ; + nb_ns =atomic_read (&nr_ipc_ns); + allowed /= nb_ns;/* * Never fall under the current minimum value
@@ -104,8 +108,8 @@ static void recompute_msgmni(struct ipc_/* * Never go over the current max value */ -if (allowed >IPCMNI ) { - ns->msg_ctlmni =IPCMNI ; +if (allowed >IPCMNI / nb_ns) { + ns->msg_ctlmni =IPCMNI / nb_ns; goto out_callback; }
--
RFC PATCH 3/4 Defining the slab_memory_callback priority as a constant
In reply to [email protected][PATCH 03/04]
This is a trivial patch that defines the priority of slab_memory_callback in the callback chain as a constant.
This is to prepare for next patch in the series.
--- include/linux/memory.h | 6 ++++++ mm/slub.c | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/memory.h +6 -0
@@ -54,6 +54,12 @@ struct memory_notify {struct notifier_block ;struct mem_section ; +/* +* Priorities for the hotplug memory callback routines (stored in decreasing +* order in the callback chain) +*/ +#define SLAB_CALLBACK_PRI 1 +#ifndef CONFIG_MEMORY_HOTPLUG_SPARSE static inline int memory_dev_init (void ) {
mm/slub.c +1 -1
@@ -2816,7 +2816,7 @@ void __init kmem_cache_init(void) kmalloc_caches[0 ].refcount =-1 ; caches++; -hotplug_memory_notifier (slab_memory_callback,1 ); +hotplug_memory_notifier (slab_memory_callback,SLAB_CALLBACK_PRI );#endif /* Able to allocate the per node structures */
--
RFC PATCH 4/4 Recomputing msgmni on memory add / remove
In reply to [email protected][PATCH 04/04]
This patch introduces the registration of a callback routine that recomputes msg_ctlmni upon memory add / remove.
A notifier block structure has been added to the ipc namespace structure.
Each time a new ipc namespace is allocated, this notifier block callback routine is registered into the memory hotplug notifier chain (it is unregistered when the ipc namespace is freed). That routine is then activated when memory is added or removed, in order to recompute msgmni for that namespace.
--- include/linux/ipc.h | 25 +++++++++++++++++++++ include/linux/memory.h | 1 ipc/msg.c | 2 - ipc/util.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++ ipc/util.h | 2 + 5 files changed, 86 insertions(+), 1 deletion(-)
include/linux/ipc.h +25 -0
@@ -81,6 +81,10 @@ struct ipc_kludge {#include <linux/kref.h> #include <linux/spinlock.h> +#ifdef CONFIG_MEMORY_HOTPLUG +#include <linux/notifier.h> +#include <linux/memory.h> +#endif /* CONFIG_MEMORY_HOTPLUG */ #define IPCMNI 32768/* <= MAX_INT limit for ipc arrays (including sysctl changes) */
@@ -118,6 +122,10 @@ struct ipc_namespace {size_t shm_ctlall;int shm_ctlmni;int shm_tot; + +#ifdef CONFIG_MEMORY_HOTPLUG +struct notifier_block ipc_memory_hotplug; +#endif };extern struct ipc_namespace init_ipc_ns;
@@ -153,6 +161,23 @@ static inline void put_ipc_ns(struct ipc#endif } +#ifdef CONFIG_MEMORY_HOTPLUG + +extern void register_ipc_hotplug_notifier (struct ipc_namespace *); + +static inline void unregister_ipc_hotplug_notifier (struct ipc_namespace *ns) +{ +unregister_memory_notifier (&ns->ipc_memory_hotplug); +} + +#else /* CONFIG_MEMORY_HOTPLUG */ + +#define register_ipc_hotplug_notifier (ns) do { } while (0) +#define unregister_ipc_hotplug_notifier (ns) do { } while (0) + +#endif /* CONFIG_MEMORY_HOTPLUG */ + +#endif /* __KERNEL__ */ #endif /* _LINUX_IPC_H */
include/linux/memory.h +1 -0
@@ -59,6 +59,7 @@ struct mem_section;* order in the callback chain) */ #define SLAB_CALLBACK_PRI 1 +#define IPC_CALLBACK_PRI 10#ifndef CONFIG_MEMORY_HOTPLUG_SPARSE static inline int memory_dev_init (void )
ipc/util.h +2 -0
@@ -134,6 +134,8 @@ extern int ipcget_new(struct ipc_namespaextern int ipcget_public (struct ipc_namespace *,struct ipc_ids *,struct ipc_ops *,struct ipc_params *); +extern void recompute_msgmni (struct ipc_namespace *); +static inline int ipc_buildid (int id,int seq) {return SEQ_MULTIPLIER * seq + id;
ipc/util.c +57 -0
87 lines not on this page · raw message
ipc/msg.c +1 -1
@@ -85,7 +85,7 @@ static int sysvipc_msg_proc_show(struct* queues should occupy at most 1/MSG_MEM_SCALE of lowmem. * Also take into account the number of nsproxies created so far. */ -static void recompute_msgmni (struct ipc_namespace *ns) +void recompute_msgmni (struct ipc_namespace *ns) {struct sysinfo i;unsigned long allowed;
--
Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove
Hello Nadia-san.
include/linux/ipc.h struct ipc_namespace { · line 129
Quoted 9 lines
include/linux/ipc.h +4
-0@@ -118,6 +122,10 @@ struct ipc_namespace {size_t shm_ctlall;int shm_ctlmni;int shm_tot; + +#ifdef CONFIG_MEMORY_HOTPLUG +struct notifier_block ipc_memory_hotplug; +#endif };
I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, the recompute_msgmni() calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?
BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)
Bye.
Signature · 1 line
Yasunori Goto
Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove
Quoted 14 lines · Yasunori Goto
Hello Nadia-san.
include/linux/ipc.h struct ipc_namespace { · line 129
Quoted 9 lines
include/linux/ipc.h +4
-0@@ -118,6 +122,10 @@ struct ipc_namespace {size_t shm_ctlall;int shm_ctlmni;int shm_tot; + +#ifdef CONFIG_MEMORY_HOTPLUG +struct notifier_block ipc_memory_hotplug; +#endif };
Quoted 7 lines
I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, therecompute_msgmni()calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?
Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces. I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.
Quoted 4 lines
BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)
Well, I tested it but not in "real configuration": what I did is that I changed the status by hand under sysfs to offline. I also changed remove_memory() in mm/memory_hotplug.c in the following way (instead of returninf EINVAL):
1) decrease the total_ram pages
2) call memory_notify(MEM_OFFLINE, NULL)
and checked that the msgmni was recomputed.
But sure, if you are candidate to test it, that would be great!
Regards,
Nadia
Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove
Quoted 23 lines
Quoted 14 lines · Yasunori Goto
Hello Nadia-san.
include/linux/ipc.h struct ipc_namespace { · line 129
Quoted 9 lines
include/linux/ipc.h +4
-0@@ -118,6 +122,10 @@ struct ipc_namespace {size_t shm_ctlall;int shm_ctlmni;int shm_tot; + +#ifdef CONFIG_MEMORY_HOTPLUG +struct notifier_block ipc_memory_hotplug; +#endif };Quoted 7 lines
I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, therecompute_msgmni()calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?
Quoted 6 lines
Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces.
I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.
The recompute_msg() must be called when new ipc_namespace is created/removed as you mentioned. I think namespaces should be linked each other for it in the end....
Quoted 4 lines
Quoted 3 lines
BTW, have you ever tested this patch? If you don't have any test environment for memory hotplug code, then I'll check it. :-)
Quoted 8 lines
Well, I tested it but not in "real configuration": what I did is that I changed the status by hand under sysfs to offline. I also changed
remove_memory()inmm/memory_hotplug.cin the following way (instead of returninf EINVAL):
1) decrease thetotal_rampages
2) callmemory_notify(MEM_OFFLINE, NULL)and checked that the msgmni was recomputed.
You can also online again after offline by writing sysfs.
Quoted 1 line
But sure, if you are candidate to test it, that would be great!
Ok. I'll check it too.
Bye.
Signature · 1 line
Yasunori Goto
Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove
Quoted 25 lines · Yasunori Goto
Quoted 25 lines
Quoted 22 lines · Yasunori Goto
Hello Nadia-san.
include/linux/ipc.h struct ipc_namespace { · line 129
Quoted 9 lines
include/linux/ipc.h +4
-0@@ -118,6 +122,10 @@ struct ipc_namespace {size_t shm_ctlall;int shm_ctlmni;int shm_tot; + +#ifdef CONFIG_MEMORY_HOTPLUG +struct notifier_block ipc_memory_hotplug; +#endif };I'm sorry, but I don't see why each ipc namespace must have each callbacks of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace. In addition, therecompute_msgmni()calculation looks very similar for all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?
Quoted 7 lines
Quoted 2 lines
Actually, this is what I wanted to do at the very beginning: have a single callback that would recompute the msgmni for each ipc namespace. But the issue here is that the namespaces are not linked to each other, so I had no simple way to go through all the namespaces. I solved the issue by having a callback for any single ipc namespace and make it recompute the msgmni value for itslef.
The
recompute_msg()must be called when newipc_namespaceis created/removed as you mentioned. I think namespaces should be linked each other for it in the end....
Not if I do it the same way as for memory hotplug:
1) definee a "namespace event notifier"
2) insert another notifier block into the ipc namespace.
3) The callback routines in the notifier chain would be activated upon namespace creation / removal.
But I'm still thinking about it: Matt Helsley suggested that we might just copy the old namespace's value when creating a new namespace.
There's also the issue of an msgmni that has been changed via procfs: may be we should not unconditionally recompute it upon namespace creation / removal, so unregister the callback for the owning namespace as soon as msgmni has been changed from userspace.
Regards,
Nadia
I'm realizing that I wrote Andrew's address wrong. So please if ever you're answering in this thread correct his address by hand. Sorry for that!
Now if you find it more convenient that I start a new thread with the right address, please tell me and I'll do it.
Regards,
Nadia
Quoted 8 lines · Nadia Derbey
I'm realizing that I wrote Andrew's address wrong. So please if ever you're answering in this thread correct his address by hand. Sorry for that!
Now if you find it more convenient that I start a new thread with the right address, please tell me and I'll do it.Regards,
Nadia
Not a big deal. If I do reply please double check that I haven't forgotten. It's easy to miss -- I only just noticed this :)
Cheers,
-Matt