cannot fork w/ 1000s of procs (but still mem avail.)
hopefully a simple question to answer: I get "cannot fork" messages on my machine running some 20000 processes and threads (1 master proc, 3 threads) where each (master) process opens a socket and does IP traffic over it. Although there is plenty of memory left (4GB box, 2GB used, 0 swap), I get "cannot fork - out of memory" when trying to increase the number of procs. (If none of the procs does IP, I can start more [of course?!].) Anything I can do to increase the number of active processes using IP? Any kernel paramter, limit, sizing?
You run out of process numbers, I suppose.
pid is 15 bits only - look at getpid() in fork.c.
It is very easy to make it 31 bits instead, and one of my machines has been running a system with 31 bit pids for a long time.
(Note, I am not talking about sizeof(pid_t) but just about code like
if((++last_pid) & 0xffff8000) {
last_pid = 300; /* Skip daemons etc. */
goto inside;
}that must be removed.)
I sent Linus a patch once or twice, will include it below if I can find it.
Andries
Yes, found it [hand-edited to remove unrelated stuff]. I have not checked whether it still applies and still is correct.
==================================================================
Quoted 1 line
From aeb Sun Jan 23 22:47:48 2000
To: [email protected], [email protected]
Subject: [PATCH] large pid and a few other fixes
Below a patch with 31-bit pids and various compilation fixes. The fork patch makes a fork measurably faster on a system with more than 5000 processes.
I started making CONFIG_15BIT_PID a config option,
but decided against it as long as no need had been established. (Yesterday or so I did a grep on "all Linux sources" for msqid_ds and shmid_ds, more in particular msg_lspid, msg_lrpid, shm_cpid, shm_lpid, and these are used in only a handful of places, but I have not yet studied these places. Maybe it will turn out to be necessary to have a config option after all, say for ibcs users, we'll see. In case you like the opposite default, change the #ifdef CONFIG_15BIT_PID
into
#if 1
or so, below in threads.h. (But as long as 2.4 has not been released some testing is good, to see the effects of a larger pid. Maybe someone will adapt ps.)
Andries
linux/fs/proc/base.c renamed from linux-2.3.40/linux/fs/proc/base.c +10 -6
@@ -29,9 +29,12 @@* inumbers of the rest of procfs (currently those are in 0x0000--0xffff). * As soon as we'll get a separate superblock we will be able to forget * about magical ranges too. +* +* For example, the define below may be replaced by +* #define fake_ino(pid,ino) 0x10000 */ -#define fake_ino (pid,ino) (((pid)<<16)|(ino)) +#define fake_ino (pid,ino) (((1)<<16)|(ino))ssize_t proc_pid_read_maps (struct task_struct *,struct file *,char *,size_t ,loff_t *);int proc_pid_stat (struct task_struct *,char *);
@@ -610,7 +613,9 @@return 1 ; p = base_stuff + i;while (p->name) { -if (filldir (dirent, p->name, p->len, filp->f_pos,fake_ino (pid, p->type)) <0 ) +if (filldir (dirent, p->name, p->len, + filp->f_pos, +fake_ino (pid, p->type)) <0 )return 0 ; filp->f_pos++; p++;
@@ -621,7 +626,8 @@/* building an inode */ -static struct inode *proc_pid_make_inode (struct super_block * sb,struct task_struct *task,int ino) +static struct inode *proc_pid_make_inode (struct super_block * sb, +struct task_struct *task,int ino) {struct inode * inode;
@@ -923,9 +929,7 @@ goto out; pid *=10 ; pid += c; -if (!pid) - goto out; -if (pid &0xffff0000 ) +if (pid <=0 || pid >=PID_MAX ) goto out; }
linux/include/linux/threads.h renamed from linux-2.3.40/linux/include/linux/threads.h +11 -1
@@ -15,8 +15,18 @@#define MIN_THREADS_LEFT_FOR_ROOT 4/* -* This controls the maximum pid allocated to a process +* One more than the maximum pid allocated to a process +* (should be positive when assigned to an int) */ +#ifdef CONFIG_15BIT_PID #define PID_MAX 0x8000 +#else +#define PID_MAX 0x7fffffff +#endif + +/* +* Place where we start again after a full cycle +*/ +#define PID_MIN 300#endif
linux/kernel/fork.c renamed from linux-2.3.40/linux/kernel/fork.c +8 -15
@@ -194,26 +194,19 @@return current->pid;spin_lock (&lastpid_lock); -if ((++last_pid) &0xffff8000 ) { - last_pid =300 ;/* Skip daemons etc. */ - goto inside; - } -if (last_pid >= next_safe) { -inside: - next_safe =PID_MAX ; +if (++last_pid >= next_safe) {read_lock (&tasklist_lock); repeat: +if (last_pid >=PID_MAX ) + last_pid =PID_MIN ; + next_safe =PID_MAX ; +for_each_task (p) {if (p->pid == last_pid || p->pgrp == last_pid || - p->session == last_pid) { -if (++last_pid >= next_safe) { -if (last_pid &0xffff8000 ) - last_pid =300 ; - next_safe =PID_MAX ; - } - goto repeat; - } + p->session == last_pid) +if (++last_pid >= next_safe) + goto repeat;if (p->pid > last_pid && next_safe > p->pid) next_safe = p->pid;if (p->pgrp > last_pid && next_safe > p->pgrp)
It would be nice to do as follows:
1. have a process limit, with a default based on memory size
2. have a pid limit, 2x the above + 300, rounded up to 9999, 99999, etc.
So if we require 32 kB per process, then...
0 to 11 MB --> 3-digit 12 MB to 155 MB --> 4-digit 156 MB to 1.6 GB --> 5-digit (max kernel memory on x86) 1.7 GB to 16 GB --> 6-digit
The 15-bit limit isn't so bad though, as you can see from the above. On x86, you get 1 PID for every 28 kB of kernel memory. You will have some other uses for kernel memory too, if we may assume that this box has some real work to do. So more likely, you can't really have anywhere near that many processes anyway.
Thus there isn't any good reason to break the old user apps.