sysv ipc: scale msgmnb with the number of cpus
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · [email protected] wrote:
The size in bytes of a SysV IPC message queue, msgmnb, is too small for large machines, but we don't want to bloat small machines
Several methods are used already to modify (mainly increase) msgmnb:
. distribution specific patch
. system wide sysctl.conf
. application specific tuning via/proc/sys/kernel/msgmnb
Which distributions use a patch?
The whole configuration can be done from user space, thus I assumed that a sysctl.conf value (or in the worst case: a dbus/hal daemon that updates /proc/sys/kernel/msgnmb) could do the job.
--
Manfred
Quoted 10 lines · Manfred Spraul
Quoted 9 lines
The size in bytes of a SysV IPC message queue, msgmnb, is too small for large machines, but we don't want to bloat small machines
Several methods are used already to modify (mainly increase) msgmnb:
. distribution specific patch
. system wide sysctl.conf
. application specific tuning via/proc/sys/kernel/msgmnbWhich distributions use a patch?
opensuse has this:
"The defaults are too small for most users."
[...]
#define MSGMNI 16 /* <= IPCMNI */ /* max # of msg queue identifiers */ -#define MSGMAX 8192 /* <= INT_MAX */ /* max size of message (bytes) */ -#define MSGMNB 16384 /* <= INT_MAX */ /* default max size of a message queue */
+#define MSGMAX 65536 /* <= INT_MAX */ /* max size of message (bytes) */
+#define MSGMNB 65536 /* <= INT_MAX */ /* default max size of a message queue */
[...]
Signature · 1 line
solofo
Hi Manfred,
This part is more difficult to answer than the other one:
Quoted 2 lines
The whole configuration can be done from user space, thus I assumed that a sysctl.conf value [...] could do the job
Yes, while this is (still) possible, this can become cumbersome with namespaces, hotplug, ...
Quoted 2 lines
(or in the worst case: a dbus/hal daemon that
updates/proc/sys/kernel/msgnmb) [...]
This would probably mean one daemon per ipc namespace. The patches seems lighter.
There has been related discussions regarding kernel space vs. userspace approach in the threads:
. "Change in default vm_dirty_ratio"
http://lkml.org/lkml/2007/6/18/471
. "[RFC][PATCH 0/6] Automatic kernel tunables (AKT)"
http://lkml.org/lkml/2007/1/16/16
(and probably others)
So it seems there is no consensus in doing it either way: kernel or user space.
Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?
Signature · 1 line
solofo
Quoted 5 lines · [email protected] wrote:
Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?
I was afraid that it might break user space applications that queue a few kb of messages. That's also the reason for
Quoted 4 lines
if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
1 + msq->q_qnum <= msq->q_qbytes) {
break;
}It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].
--
Manfred
Quoted 7 lines · Manfred Spraul
Quoted 5 lines
Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?I was afraid that it might break user space applications that queue a few kb of messages.
Ok, the choice of a maximum value of 65536 which is already is use for several months/years was made partly for the same concern. Beside, as the values are not enforced, we should be relatively safe.
Searching the archives, I also found usage of a value "around a MB".
Quoted 7 lines
That's also the reason for
Quoted 4 lines
if (msgsz + msq->q_cbytes <= msq->q_qbytes && 1 + msq->q_qnum <= msq->q_qbytes) { break; }It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].
Thanks for this information. I should add that I checked that no regression was introduced with ltp-full-20080531, but I did not look more closely (e.g. coverage of this part of the code).
Signature · 1 line
solofo
Quoted 11 lines · Manfred Spraul
Quoted 5 lines · [email protected] wrote:
Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?I was afraid that it might break user space applications that queue a few kb of messages. That's also the reason for
Quoted 7 lines
Quoted 4 lines
if (msgsz + msq->q_cbytes <= msq->q_qbytes && 1 + msq->q_qnum <= msq->q_qbytes) { break; }It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].
Manfred,
If I'm not missign anything when reading the code, sending up to MSGMNB 0-bytes messages would make us enqueue MSGMNB msg_msg structures (this is in the worst case where no receiver is waiting for those messages). ==> MSGMNB * 24 bytes (or 48 bytes in 64-bit mode))
==> 384 KB with current MSGMNB value (16K).
But 1,5 MB with a MSGMNB=64K
Even if it is a worst case, it should be considered and may be we should refine the formula Solofo has proposed if you think this is not a reasonable value. May be add a dependency on the memory size?
Regards,
Nadia