RLIMIT_NPROC enforcement during execve() calls

Enforces the RLIMIT_NPROC limit by adding an additional check for execve(), as
such limit is checked only during fork() calls.

The patch is also available at:
http://pearls.tuxedo-es.org/patches/security/rlimit_nproc-enforcing-execve.patch

---

 linux-2.6.11-lorenzo/fs/compat.c |    8 ++++++++
 linux-2.6.11-lorenzo/fs/exec.c   |    9 +++++++++
 2 files changed, 17 insertions(+)

diff -puN fs/exec.c~rlimit_nproc-enforcing-execve fs/exec.c
--- linux-2.6.11/fs/exec.c~rlimit_nproc-enforcing-execve 2005-04-16
16:28:56.000000000 +0200
+++ linux-2.6.11-lorenzo/fs/exec.c 2005-04-16 19:26:47.000000000 +0200
@@ -1140,6 +1140,15 @@ int do_execve(char * filename,
if (IS_ERR(file))
goto out_kfree;

+ /* RLIMIT_NPROC enforcement */
+ if (current->user && (atomic_read(&current->user->processes) >
+ current->signal->rlim[RLIMIT_NPROC].rlim_cur) &&
+ !capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE)) {
+ allow_write_access(file);
+ fput(file);
+ return -EAGAIN;
+ }
+
sched_exec();

bprm->p = PAGE_SIZE*MAX_ARG_PAGES-sizeof(void *);
diff -puN fs/compat.c~rlimit_nproc-enforcing-execve fs/compat.c
--- linux-2.6.11/fs/compat.c~rlimit_nproc-enforcing-execve 2005-04-16
16:28:56.000000000 +0200
+++ linux-2.6.11-lorenzo/fs/compat.c 2005-04-16 19:26:58.000000000 +0200
@@ -1450,6 +1450,14 @@ int compat_do_execve(char * filename,
if (!bprm->mm)
goto out_file;

+ /* RLIMIT_NPROC enforcement */
+ retval = -EAGAIN;
+ if (current->user && (atomic_read(&current->user->processes) >
+ current->signal->rlim[RLIMIT_NPROC].rlim_cur) &&
+ !capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE)) {
+ goto out_file;
+ }
+
retval = init_new_context(current, bprm->mm); if (retval < 0)
goto out_mm;
_

Cheers,

Lorenzo Hernández García-Hierro <[email protected]> 
[1024D/6F2B2DEC] & [2048g/9AE91A22][http://tuxedo-es.org]

What's the point? exec doesn't create new process and exec() shouldn't start to fail just because someone lowered the rlimit a short while ago.

The limit is only checked when process is created on a fork() call, but during execution it's uid can change, thus, the limit for the new uid could be exceed.

It comes from the Openwall kernel patch, as well implemented in grSecurity and vSecurity.

Cheers,

Lorenzo Hernández García-Hierro <[email protected]> 
[1024D/6F2B2DEC] & [2048g/9AE91A22][http://tuxedo-es.org]

The only two ways I can see this happening are (1) if the process is running as uid 0 (or capability-equivalent) at fork() time and have called set*uid() before execve(), or (2) we just exec'ed a set-UID binary.

In both cases the "obvious" thing to do is to re-check the target UID's rlimit, but there may be some squirrelly corner case where this isn't true...