Skip to contentSkip to conversation

sysv ipc: scale msgmnb with the number of cpus

RFC Manfred Spraul 3 participants 6 messages Jun 7, 2008 – Jun 25, 2008 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 9 lines · [email protected] wrote:

The size in bytes of a SysV IPC message queue, msgmnb, is too small for large machines, but we don't want to bloat small machines

Several methods are used already to modify (mainly increase) msgmnb:
. distribution specific patch
. system wide sysctl.conf
. application specific tuning via /proc/sys/kernel/msgmnb

Which distributions use a patch?

The whole configuration can be done from user space, thus I assumed that a sysctl.conf value (or in the worst case: a dbus/hal daemon that updates /proc/sys/kernel/msgnmb) could do the job.

--
Manfred

#2 lore raw
Quoted 10 lines · Manfred Spraul
Quoted 9 lines

The size in bytes of a SysV IPC message queue, msgmnb, is too small for large machines, but we don't want to bloat small machines

Several methods are used already to modify (mainly increase) msgmnb:
. distribution specific patch
. system wide sysctl.conf
. application specific tuning via /proc/sys/kernel/msgmnb

Which distributions use a patch?

opensuse has this:

"The defaults are too small for most users."

[...]

 #define MSGMNI    16   /* <= IPCMNI */     /* max # of msg queue identifiers */
-#define MSGMAX  8192   /* <= INT_MAX */   /* max size of message (bytes) */
-#define MSGMNB 16384   /* <= INT_MAX */   /* default max size of a message queue */

+#define MSGMAX 65536 /* <= INT_MAX */ /* max size of message (bytes) */
+#define MSGMNB 65536 /* <= INT_MAX */ /* default max size of a message queue */
[...]

Signature · 1 line
solofo
#3 lore raw
In reply to Manfred Spraul

Hi Manfred,

This part is more difficult to answer than the other one:

Quoted 2 lines

The whole configuration can be done from user space, thus I assumed that a sysctl.conf value [...] could do the job

Yes, while this is (still) possible, this can become cumbersome with namespaces, hotplug, ...

Quoted 2 lines

(or in the worst case: a dbus/hal daemon that
updates /proc/sys/kernel/msgnmb) [...]

This would probably mean one daemon per ipc namespace. The patches seems lighter.

There has been related discussions regarding kernel space vs. userspace approach in the threads:
. "Change in default vm_dirty_ratio"
http://lkml.org/lkml/2007/6/18/471
. "[RFC][PATCH 0/6] Automatic kernel tunables (AKT)"
http://lkml.org/lkml/2007/1/16/16
(and probably others)

So it seems there is no consensus in doing it either way: kernel or user space.

Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?

Signature · 1 line
solofo
#4 lore raw
Quoted 5 lines · [email protected] wrote:

Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?

I was afraid that it might break user space applications that queue a few kb of messages. That's also the reason for

Quoted 4 lines
                if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
                                1 + msq->q_qnum <= msq->q_qbytes) {
                        break;
                }

It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].

--
Manfred

#5 lore raw
Quoted 7 lines · Manfred Spraul
Quoted 5 lines

Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?

I was afraid that it might break user space applications that queue a few kb of messages.

Ok, the choice of a maximum value of 65536 which is already is use for several months/years was made partly for the same concern. Beside, as the values are not enforced, we should be relatively safe.

Searching the archives, I also found usage of a value "around a MB".

Quoted 7 lines

That's also the reason for

Quoted 4 lines
                if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
                                1 + msq->q_qnum <= msq->q_qbytes) {
                        break;
                }

It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].

Thanks for this information. I should add that I checked that no regression was introduced with ltp-full-20080531, but I did not look more closely (e.g. coverage of this part of the code).

Signature · 1 line
solofo
#6 lore raw
In reply to Manfred Spraul
Quoted 11 lines · Manfred Spraul
Quoted 5 lines · [email protected] wrote:

Humm... now this make me think that you did not change the MSGMNB value when you changed MSGMNI and MSGMAX.
Maybe that was on purpose?

I was afraid that it might break user space applications that queue a few kb of messages. That's also the reason for

Quoted 7 lines
Quoted 4 lines
                if (msgsz + msq->q_cbytes <= msq->q_qbytes &&
                                1 + msq->q_qnum <= msq->q_qbytes) {
                        break;
                }

It's possible to send 0-byte messages even if the message queue is full [except that you can't send more than MSGMNB messages].

Manfred,

If I'm not missign anything when reading the code, sending up to MSGMNB 0-bytes messages would make us enqueue MSGMNB msg_msg structures (this is in the worst case where no receiver is waiting for those messages). ==> MSGMNB * 24 bytes (or 48 bytes in 64-bit mode))
==> 384 KB with current MSGMNB value (16K).

But 1,5 MB with a MSGMNB=64K

Even if it is a worst case, it should be considered and may be we should refine the formula Solofo has proposed if you think this is not a reasonable value. May be add a dependency on the memory size?

Regards,
Nadia