New: default shmmax
A message that is not in the archive (sent off-list or not yet received): [email protected]/
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 48 lines · On Wed, 20 Aug 2008 05:58:57 -0700 (PDT) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=11381
Summary: default shmmax Product: Other Version: 2.5 KernelVersion: 2.6.26.2 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: enhancement Priority: P1 Component: Other AssignedTo: [email protected] ReportedBy: [email protected]I would like to request that the default shmmax setting be increased or the downsides of that be documented. Allow me to explain.
I am with the PostgreSQL development team. PostgreSQL is probably one of the few users of large amounts of SysV shared memory. Users would usually want to configure anywhere between 10% and 50% of their physical RAM to be used as shared memory, which would translate to something on the order of gigabytes nowadays. One of the uniformly annoying things about setting this up is that you need to reconfigure the Linux kernel to allow that. sysctl is nice and all, but it still requires users to learn about operating system and kernel details, requires root access, and distros don't handle sysctl uniformly either. Maybe there is even a good reason for that, but I couldn't find it, and at least I would like to learn it, so that we can pass that information on to our users.
I did some kernel version archeology and found out that up until kernels 2.2 the shmmax setting appears to have been restricted by CPU-specific constraints, as indicated by the default setting being different across CPUs and being defined in an asm header. The default setting on i386 was increased from 16 MB to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB in the latest kernel on all architectures.
Now one question is whether there is a space or time overhead involved with setting a high shmmax limit that isn't actually used. If so, it would be interesting to know what that overhead is. The feeling I get from browsing the kernel source code over time is that there was some management overhead and/or some restrictions about this in old kernels, but that nowadays it doesn't really seem to matter much anymore. I suspect instead that this whole thing was just forgotten, because few applications use large amounts of shared memory.
Quoted 3 lines
So, if you want to do us a favor, could you please see about increasing the default shmmax setting to whatever the theoretical maximum is?
I don't think anybody has even thought about the shmmax default in years. Sure, it might be time to reexamine that.
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Also, from a quick read it looks to me that shmmax is busted in the non-init namespace.
clone_ipc_ns() calls shm_init_ns() which does
ns->shm_ctlmax = SHMMAX;
which a) fails to inherit the parent's setting and b) cannot be altered from SHMMAX via the sysctl?
Quoted 2 lines
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Red Hat provide a sysctl tuning config file and I believe things like the Oracle install docs cover this.
There is btw no earthly reason why a postgres package can't include a tool to do this or postgres can't check and update it as part of its own set up and config file options
Alan
Quoted 9 lines · Alan Cox
Quoted 2 lines
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Red Hat provide a sysctl tuning config file and I believe things like the Oracle install docs cover this.
There is btw no earthly reason why a postgres package can't include a tool to do this or postgres can't check and update it as part of its own set up and config file options
I have explored this, it would be possible, but it would be a pain in the neck. You'd need to add an option to postgres to print out how much shared memory you would need for a given configuration (because there are multiple factors in this), run this across all postgres instances on the machine, sum it up, then arrange for this to be fed back to sysctl or pasted back into sysctl.conf, being careful not to override or lower an existing setting that might have wanted to reserve space for other things. Note that to make this robust you would have to rerun this after every configuration change + restart cycle across all postgres instances on the machine, race conditions included. And you'd probably violate a few packaging guidelines on the way. Plus, this would have to be distro-specific (not Linux-specific, but distro-specific), doesn't help managed hosting with no root access, and doesn't help people building from source. So altogether it is very complicated.
In the meantime I am trying to explore why the shmmax default is what it is. Or why it has to exist at all. If we can figure that out, maybe we can solve a lot of users a lot of trouble with much less effort (including Oracle users, why not).
Quoted 66 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 48 lines · On Wed, 20 Aug 2008 05:58:57 -0700 (PDT) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=11381
Summary: default shmmax Product: Other Version: 2.5 KernelVersion: 2.6.26.2 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: enhancement Priority: P1 Component: Other AssignedTo: [email protected] ReportedBy: [email protected]I would like to request that the default shmmax setting be increased or the downsides of that be documented. Allow me to explain.
I am with the PostgreSQL development team. PostgreSQL is probably one of the few users of large amounts of SysV shared memory. Users would usually want to configure anywhere between 10% and 50% of their physical RAM to be used as shared memory, which would translate to something on the order of gigabytes nowadays. One of the uniformly annoying things about setting this up is that you need to reconfigure the Linux kernel to allow that. sysctl is nice and all, but it still requires users to learn about operating system and kernel details, requires root access, and distros don't handle sysctl uniformly either. Maybe there is even a good reason for that, but I couldn't find it, and at least I would like to learn it, so that we can pass that information on to our users.
I did some kernel version archeology and found out that up until kernels 2.2 the shmmax setting appears to have been restricted by CPU-specific constraints, as indicated by the default setting being different across CPUs and being defined in an asm header. The default setting on i386 was increased from 16 MB to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB in the latest kernel on all architectures.
Now one question is whether there is a space or time overhead involved with setting a high shmmax limit that isn't actually used. If so, it would be interesting to know what that overhead is. The feeling I get from browsing the kernel source code over time is that there was some management overhead and/or some restrictions about this in old kernels, but that nowadays it doesn't really seem to matter much anymore. I suspect instead that this whole thing was just forgotten, because few applications use large amounts of shared memory.
Quoted 3 lines
So, if you want to do us a favor, could you please see about increasing the default shmmax setting to whatever the theoretical maximum is?
I don't think anybody has even thought about the shmmax default in years. Sure, it might be time to reexamine that.
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Quoted 8 lines
Also, from a quick read it looks to me that shmmax is busted in the non-init namespace.
clone_ipc_ns()callsshm_init_ns()which doesns->shm_ctlmax = SHMMAX;which a) fails to inherit the parent's setting and
This is debatable if such behaviour should be default, this makes one ipc_ns more ipc_ns than others.
Quoted 1 line
b) cannot be altered from SHMMAX via the sysctl?
It can be altered. See kludge called get_ipc().
(well, I haven't checked personally, but it's a bug if IPC sysctls aren't independently controllable after CLONE_NEWIPC, complain loudly).
Quoted 70 lines · On Wed, Aug 20, 2008 at 11:12:57PM +0400, wrote:
Quoted 69 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 48 lines · On Wed, 20 Aug 2008 05:58:57 -0700 (PDT) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=11381
Summary: default shmmax Product: Other Version: 2.5 KernelVersion: 2.6.26.2 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: enhancement Priority: P1 Component: Other AssignedTo: [email protected] ReportedBy: [email protected]I would like to request that the default shmmax setting be increased or the downsides of that be documented. Allow me to explain.
I am with the PostgreSQL development team. PostgreSQL is probably one of the few users of large amounts of SysV shared memory. Users would usually want to configure anywhere between 10% and 50% of their physical RAM to be used as shared memory, which would translate to something on the order of gigabytes nowadays. One of the uniformly annoying things about setting this up is that you need to reconfigure the Linux kernel to allow that. sysctl is nice and all, but it still requires users to learn about operating system and kernel details, requires root access, and distros don't handle sysctl uniformly either. Maybe there is even a good reason for that, but I couldn't find it, and at least I would like to learn it, so that we can pass that information on to our users.
I did some kernel version archeology and found out that up until kernels 2.2 the shmmax setting appears to have been restricted by CPU-specific constraints, as indicated by the default setting being different across CPUs and being defined in an asm header. The default setting on i386 was increased from 16 MB to 32 MB somewhere around 1998 in the kernel 2.0 line, and it remains at 32 MB in the latest kernel on all architectures.
Now one question is whether there is a space or time overhead involved with setting a high shmmax limit that isn't actually used. If so, it would be interesting to know what that overhead is. The feeling I get from browsing the kernel source code over time is that there was some management overhead and/or some restrictions about this in old kernels, but that nowadays it doesn't really seem to matter much anymore. I suspect instead that this whole thing was just forgotten, because few applications use large amounts of shared memory.
Quoted 3 lines
So, if you want to do us a favor, could you please see about increasing the default shmmax setting to whatever the theoretical maximum is?
I don't think anybody has even thought about the shmmax default in years. Sure, it might be time to reexamine that.
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Also, from a quick read it looks to me that shmmax is busted in the non-init namespace.
Quoted 8 lines
Quoted 5 lines
clone_ipc_ns()callsshm_init_ns()which doesns->shm_ctlmax = SHMMAX;which a) fails to inherit the parent's setting and
This is debatable if such behaviour should be default, this makes one
ipc_nsmoreipc_nsthan others.
Oh, I forgot for a moment, that mainline has hierarchical ipc_ns.
A message that is not in the archive (sent off-list or not yet received): 20080820120043.e6e7e436.akpm-de/[email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andrew Morton
Andrew>I don't think anybody has even thought about the shmmax default in Andrew> years.
IIRC, Matt (explicit CC: added) did a review of IPC tunables and how they are changed in the distributions.
Andrew> It would be useful to get distro input on this. Do they Andrew> override the kernel default at boot time? If so, what do Andrew> they do?
OpenSuse (11.0 at least) has this:
-#define SHMMAX 0x2000000 /* max shared seg size (bytes) */
+#define SHMMAX ULONG_MAX /* max shared seg size (bytes) */
Review
Quoted 3 lines · [email protected] wrote:
OpenSuse (11.0 at least) has this:
(quoted diff) +1
-1-#define SHMMAX 0x2000000 /* max shared seg size (bytes) */ +#define SHMMAX ULONG_MAX /* max shared seg size (bytes) */
Interesting, I don't see that in SLES 10, so it must be quite new. If no problems are found with this, this would be great for the upstream kernel.
Quoted 2 lines · Andrew Morton
It would be useful to get distro input on this. Do they override the kernel default at boot time? If so, what do they do?
Debian and Ubuntu have a sysctl.conf that is processed at boot time, but it does not contain anything related to shm* by default.
Suse has a sysctl.conf, but it is not processed at boot by default. You have to activate a special boot service in the runlevel system to make it actually run at boot. (Try explaining that to a PostgreSQL user.) There is nothing related to shm* in the default configuration.