Skip to contentSkip to conversation

Containerise nproc count

RFC PATCH In review Nikolay Borisov 3 participants 5 messages Sep 8, 2015 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Containerise nproc count

From: Nikolay Borisov <n.borisov-/[email protected]>

Hello,

This is an initial try to have nproc count apply per-userns, rather than per the global user struct. The implementation is really simple - a hashtable holding uid->nproc mapping for each id inside the respective namespace. In its current form I have also left the debugging code so that people who want to have a play with it can easily see what's happening.

Now, this is only an RFC and I'd like to gather your thoughts about the semantics. Currently as it stands I have tested the patchset by invoking multiple LXC containers, with identical uid mappings and users with the same uid inside the containers and it was working correctly.

There is an issue however, when using the unshare syscall and then doing the mappings e.g. using "unshare -r" util from util-linux the initial process (the one which have done the unsharing) is accounted to the overflowuid but then again when exiting from the resulting shell the UID for user 0 is being freed which causes the BUG_ON in nsuser_nproc_dec to trigger. My initial idea for fixing this was to add code which upon writing to /proc/[pid]/uid_map would map all current processes from overflowuid to the 'ns->uid_map.extent[0].first'. This was working correctly but it was breaking the use case of lxc, since lxc is changing the uids after creating the uid_mapping (maybe this is a deficiency in the unshare util implementation?)

Another thing that needs improving is the locking occuring on the nsuser_nproc_hash, since in its current coarse-grained form it is serialisign process/thread creation on a per-usernamespace basis.

I'm happy to discuss any concerns and improvements that people might have regarding this patchset.

Nikolay Borisov (2):
userns: Implement per-userns nproc infrastructure userns/nproc: Add hooks for userns nproc management

 include/linux/user_namespace.h |  15 +++++-
 kernel/cred.c                  |  36 +++++++++++++-
 kernel/exit.c                  |   9 ++++
 kernel/fork.c                  |  33 ++++++++++---
 kernel/user.c                  |   3 ++
 kernel/user_namespace.c        | 105 +++++++++++++++++++++++++++++++++++++++++
 6 files changed, 192 insertions(+), 9 deletions(-)
Signature · 1 line
2.5.0
#1 lore raw

RFC PATCH 1/2 userns: Implement per-userns nproc infrastructure

From: Nikolay Borisov <n.borisov-/[email protected]>

This patch add a simple hashtable to the user_namespace structure and the necessary functions to work with it. The idea is to keep a uid->nproc counts per-namespace.

---
 include/linux/user_namespace.h |  15 +++++-
 kernel/user.c                  |   3 ++
 kernel/user_namespace.c        | 105 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 122 insertions(+), 1 deletion(-)
include/linux/user_namespace.h +14 -1
index 8297e5b..6eb9414 100644
@@ -5,6 +5,9 @@
 #include <linux/nsproxy.h>
 #include <linux/ns_common.h>
 #include <linux/sched.h>
+#include <linux/hashtable.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
 #include <linux/err.h>
 
 #define UID_GID_MAP_MAX_EXTENTS 5
@@ -21,7 +24,7 @@ struct uid_gid_map {	/* 64 bytes -- 1 cache line */
 #define USERNS_SETGROUPS_ALLOWED 1UL
 
 #define USERNS_INIT_FLAGS USERNS_SETGROUPS_ALLOWED
-
+#define NPROC_HASH_ORDER 7
 struct user_namespace {
 	struct uid_gid_map	uid_map;
 	struct uid_gid_map	gid_map;
@@ -33,6 +36,8 @@ struct user_namespace {
 	kgid_t			group;
 	struct ns_common	ns;
 	unsigned long		flags;
+	struct spinlock		nproc_hash_lock;
+	DECLARE_HASHTABLE(nproc_hash, NPROC_HASH_ORDER);
 
 	/* Register of per-UID persistent keyrings for this namespace */
 #ifdef CONFIG_PERSISTENT_KEYRINGS
@@ -72,6 +77,9 @@ extern ssize_t proc_projid_map_write(struct file *, const char __user *, size_t,
 extern ssize_t proc_setgroups_write(struct file *, const char __user *, size_t, loff_t *);
 extern int proc_setgroups_show(struct seq_file *m, void *v);
 extern bool userns_may_setgroups(const struct user_namespace *ns);
+extern void userns_nproc_inc(struct user_namespace *ns, uid_t uid);
+extern void userns_nproc_dec(struct user_namespace *ns, uid_t uid);
+extern uint32_t get_userns_nproc(struct user_namespace *ns, uid_t uid);
 #else
 
 static inline struct user_namespace *get_user_ns(struct user_namespace *ns)
@@ -100,6 +108,11 @@ static inline bool userns_may_setgroups(const struct user_namespace *ns)
 {
 	return true;
 }
+
+
+static void userns_nproc_inc(ustruct user_namespace *ns, id_t uid) {}
+static void userns_nproc_dec(ustruct user_namespace *ns, id_t uid) {}
+static uint32_t get_userns_nproc(ustruct user_namespace *ns, id_t uid) { return 0; }
 #endif
 
 #endif /* _LINUX_USER_H */
kernel/user.c +3 -0
index b2a552f..3a0e36e 100644
@@ -54,6 +54,9 @@ struct user_namespace init_user_ns = {
 #ifdef CONFIG_USER_NS
 	.ns.ops = &userns_operations,
 #endif
+	//Intentially leave the nproc_hash_lock uninitialised
+	//as it shouldn't be used in the init namespace
+
 	.flags = USERNS_INIT_FLAGS,
 #ifdef CONFIG_PERSISTENT_KEYRINGS
 	.persistent_keyring_register_sem =
kernel/user_namespace.c +105 -0

135 lines not on this page · raw message

Signature · 1 line
2.5.0
#2 lore raw

RFC PATCH 2/2 userns/nproc: Add hooks for userns nproc management

In reply to Nikolay Borisov

From: Nikolay Borisov <n.borisov-/[email protected]>

This patch introduce the usage of the userns_nproc_* functions where necessary to have correct accounting of the processes.

---
 kernel/cred.c | 36 ++++++++++++++++++++++++++++++++++--
 kernel/exit.c |  9 +++++++++
 kernel/fork.c | 33 +++++++++++++++++++++++++++------
 3 files changed, 70 insertions(+), 8 deletions(-)
kernel/cred.c +34 -2
index b7581dc..79565b8 100644
@@ -320,6 +320,7 @@ struct cred *prepare_exec_creds(void)
 int copy_creds(struct task_struct *p, unsigned long clone_flags)
 {
 	struct cred *new;
+	struct user_namespace *ns;
 	int ret;
 
 	if (
@@ -331,10 +332,15 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
 		p->real_cred = get_cred(p->cred);
 		get_cred(p->cred);
 		alter_cred_subscribers(p->cred, 2);
+		ns = p->real_cred->user_ns;
 		kdebug("share_creds(%p{%d,%d})",
 		       p->cred, atomic_read(&p->cred->usage),
 		       read_cred_subscribers(p->cred));
 		atomic_inc(&p->cred->user->processes);
+		if (ns != &init_user_ns) {
+			pr_info ("%s: incrementing nproc from due copy_process (CLONE_THREAD)\n", __func__);
+			userns_nproc_inc(ns, from_kuid_munged(ns, p->real_cred->uid));
+		}
 		return 0;
 	}
 
@@ -343,6 +349,7 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
 		return -ENOMEM;
 
 	if (clone_flags & CLONE_NEWUSER) {
+		pr_debug("%s: Creating new usernamespace\n", __func__);
 		ret = create_user_ns(new);
 		if (ret < 0)
 			goto error_put;
@@ -369,6 +376,11 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
 
 	atomic_inc(&new->user->processes);
 	p->cred = p->real_cred = get_cred(new);
+	ns = p->real_cred->user_ns;
+	if (ns != &init_user_ns) {
+		pr_info("%s: Incrementing due to not-being a thread\n",	__func__);
+		userns_nproc_inc(ns, from_kuid_munged(ns, p->real_cred->uid));
+	}
 	alter_cred_subscribers(new, 2);
 	validate_creds(new);
 	return 0;
@@ -454,17 +466,37 @@ int commit_creds(struct cred *new)
 	if (!gid_eq(new->fsgid, old->fsgid))
 		key_fsgid_changed(task);
 
+	/* Handle cases when a process is moving from one userns to another */
+	if (old->user_ns != new->user_ns) {
+		if (new->user_ns != &init_user_ns) {
+			pr_info ("\t%s: incrementing user count in %p\n", __func__, new->user_ns);
+			userns_nproc_inc(new->user_ns, from_kuid_munged(new->user_ns, new->uid));
+		}
+		if (old->user_ns != &init_user_ns) {
+			pr_info ("\t%s: decrementing user_count in %p\n", __func__, old->user_ns);
+			userns_nproc_dec(old->user_ns, from_kuid_munged(old->user_ns, old->uid));
+		}
+	}
+
 	/* do it
 	 * RLIMIT_NPROC limits on user->processes have already been checked
 	 * in set_user().
 	 */
 	alter_cred_subscribers(new, 2);
-	if (new->user != old->user)
+	if (new->user != old->user) {
 		atomic_inc(&new->user->processes);
+		if (new->user_ns != &init_user_ns)
+			userns_nproc_inc(new->user_ns,
+				 from_kuid_munged(new->user_ns, new->uid));
+	}
 	rcu_assign_pointer(task->real_cred, new);
 	rcu_assign_pointer(task->cred, new);
-	if (new->user != old->user)
+	if (new->user != old->user) {
 		atomic_dec(&old->user->processes);
+		if (old->user_ns != &init_user_ns)
+			userns_nproc_dec(old->user_ns,
+				 from_kuid_munged(old->user_ns, old->uid));
+	}
 	alter_cred_subscribers(old, -2);
 
 	/* send notifications */
kernel/exit.c +9 -0
index 22fcc05..dde172b 100644
@@ -170,13 +170,22 @@ void release_task(struct task_struct *p)
 {
 	struct task_struct *leader;
 	int zap_leader;
+	struct user_namespace *ns;
+	kuid_t uid;
 repeat:
 	/* don't need to get the RCU readlock here - the process is dead and
 	 * can't be modifying its own credentials. But shut RCU-lockdep up */
 	rcu_read_lock();
 	atomic_dec(&__task_cred(p)->user->processes);
+	ns = get_user_ns(__task_cred(p)->user_ns);
+	uid = __task_cred(p)->uid;
 	rcu_read_unlock();
 
+	if (ns != &init_user_ns)
+		userns_nproc_dec(ns, from_kuid_munged(ns, uid));
+
+	put_user_ns(ns);
+
 	proc_flush_task(p);
 
 	write_lock_irq(&tasklist_lock);
kernel/fork.c +27 -6
index f9826a3..c537b6a 100644
@@ -1308,18 +1308,34 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 	DEBUG_LOCKS_WARN_ON(!p->softirqs_enabled);
 #endif
 	retval = -EAGAIN;
-	if (atomic_read(&p->real_cred->user->processes) >=
-			task_rlimit(p, RLIMIT_NPROC)) {
-		if (p->real_cred->user != INIT_USER &&
-		    !capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
-			goto bad_fork_free;
-	}
+	//If we are in the root namespace use this check
+	if (p->real_cred->user_ns == &init_user_ns) {
+		if (atomic_read(&p->real_cred->user->processes) >=
+				task_rlimit(p, RLIMIT_NPROC)) {
+			if (p->real_cred->user != INIT_USER &&
+			    !capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+				goto bad_fork_free;
+		}
 	current->flags &= ~PF_NPROC_EXCEEDED;
+	}
 
 	retval = copy_creds(p, clone_flags);
 	if (retval < 0)
 		goto bad_fork_free;
 
+	//Otherwise perform the non-root userns check here
+	//since we want the stuff in copy_cred to have already happened
+	if (p->real_cred->user_ns != &init_user_ns) {
+		struct user_namespace *ns = p->real_cred->user_ns;
+		int32_t processes = get_userns_nproc(ns, from_kuid_munged(ns, p->real_cred->uid));
+		retval = -EAGAIN;
+
+		if (processes >= task_rlimit(p, RLIMIT_NPROC))
+			goto bad_fork_cleanup_userns_count;
+		else
+			current->flags &= ~PF_NPROC_EXCEEDED;
+	}
+
 	/*
 	 * If multiple threads are within copy_process(), then this check
 	 * triggers too late. This doesn't hurt, the check is only there
@@ -1652,6 +1668,9 @@ bad_fork_cleanup_threadgroup_lock:
 	delayacct_tsk_free(p);
 bad_fork_cleanup_count:
 	atomic_dec(&p->cred->user->processes);
+bad_fork_cleanup_userns_count:
+	if (p->cred->user_ns != &init_user_ns)
+		userns_nproc_dec(p->cred->user_ns, from_kuid_munged(p->cred->user_ns, p->cred->uid));
 	exit_creds(p);
 bad_fork_free:
 	free_task(p);
@@ -1936,6 +1955,7 @@ SYSCALL_DEFINE1(unshare, unsigned long, unshare_flags)
 	int do_sysvsem = 0;
 	int err;
 
+	pr_info("%s begin\n", __func__);
 	/*
 	 * If unsharing a user namespace must also unshare the thread.
 	 */
@@ -2037,6 +2057,7 @@ bad_unshare_cleanup_fs:
 		free_fs_struct(new_fs);
 
 bad_unshare_out:
+	pr_info("%s end\n", __func__);
 	return err;
 }
 
Signature · 1 line
2.5.0

A message that is not in the archive (sent off-list or not yet received): 1441699873-26653-1-git-send-email-kernel-6AxghH7DbtA@public.gmane.org

#4 lore raw
Quoted 28 lines · Nikolay Borisov

From: Nikolay Borisov <n.borisov-/[email protected]>

Hello,

This is an initial try to have nproc count apply per-userns, rather than per the global user struct. The implementation is really simple - a hashtable holding uid->nproc mapping for each id inside the respective namespace. In its current form I have also left the debugging code so that people who want to have a play with it can easily see what's happening.

Now, this is only an RFC and I'd like to gather your thoughts about the semantics. Currently as it stands I have tested the patchset by invoking multiple LXC containers, with identical uid mappings and users with the same uid inside the containers and it was working correctly.

There is an issue however, when using the unshare syscall and then doing the mappings e.g. using "unshare -r" util from util-linux the initial process (the one which have done the unsharing) is accounted to the overflowuid but then again when exiting from the resulting shell the UID for user 0 is being freed which causes the BUG_ON in nsuser_nproc_dec to trigger. My initial idea for fixing this was to add code which upon writing to /proc/[pid]/uid_map would map all current processes from overflowuid to the 'ns->uid_map.extent[0].first'. This was working correctly but it was breaking the use case of lxc, since lxc is changing the uids after creating the uid_mapping (maybe this is a deficiency in the unshare util implementation?)

Quoted 6 lines

Another thing that needs improving is the locking occuring on the nsuser_nproc_hash, since in its current coarse-grained form it is serialisign process/thread creation on a per-usernamespace basis.

I'm happy to discuss any concerns and improvements that people might have regarding this patchset.

So. no.

Changing rlimit nproc this way breaks per user process accounting. Effectively this allows any user to escape their NPROC limit by creating a new user namespace. Which means that to even consider anything like this we need hierarchical limits.

I am not particularly convinced that reusing uids between containers is all that smart. Certainly it is bad to assume that there is no leakage and containers can never interact. So something like this also needs a description of why this new set of semantics is a good direction to go in.

Eric

#5 lore raw
In reply to Nikolay Borisov
Quoted 28 lines · Nikolay Borisov

From: Nikolay Borisov <[email protected]>

Hello,

This is an initial try to have nproc count apply per-userns, rather than per the global user struct. The implementation is really simple - a hashtable holding uid->nproc mapping for each id inside the respective namespace. In its current form I have also left the debugging code so that people who want to have a play with it can easily see what's happening.

Now, this is only an RFC and I'd like to gather your thoughts about the semantics. Currently as it stands I have tested the patchset by invoking multiple LXC containers, with identical uid mappings and users with the same uid inside the containers and it was working correctly.

There is an issue however, when using the unshare syscall and then doing the mappings e.g. using "unshare -r" util from util-linux the initial process (the one which have done the unsharing) is accounted to the overflowuid but then again when exiting from the resulting shell the UID for user 0 is being freed which causes the BUG_ON in nsuser_nproc_dec to trigger. My initial idea for fixing this was to add code which upon writing to /proc/[pid]/uid_map would map all current processes from overflowuid to the 'ns->uid_map.extent[0].first'. This was working correctly but it was breaking the use case of lxc, since lxc is changing the uids after creating the uid_mapping (maybe this is a deficiency in the unshare util implementation?)

Quoted 6 lines

Another thing that needs improving is the locking occuring on the nsuser_nproc_hash, since in its current coarse-grained form it is serialisign process/thread creation on a per-usernamespace basis.

I'm happy to discuss any concerns and improvements that people might have regarding this patchset.

So. no.

Changing rlimit nproc this way breaks per user process accounting. Effectively this allows any user to escape their NPROC limit by creating a new user namespace. Which means that to even consider anything like this we need hierarchical limits.

I am not particularly convinced that reusing uids between containers is all that smart. Certainly it is bad to assume that there is no leakage and containers can never interact. So something like this also needs a description of why this new set of semantics is a good direction to go in.

Eric