Skip to contentSkip to conversation

Containers (V8): Generic Process Containers

PATCH In review [email protected] 5 participants 24 messages Apr 7, 2007 – Apr 25, 2007 1 message not in the archive

Subsystems (9)
Documentation VFS MM core Scheduler +5 more subsystems
lore
Share & tools
lore
#0 lore raw

cover Containers (V8): Generic Process Containers

--

This is an update to my multi-hierarchy multi-subsystem generic process containers patch. Changes since V7 (12th Feb) include:

- Removed the config-time choice of the number of supported hierarchies - this is now completely dynamic; new hierarchies are allocated on demand, and freed when no longer in use.

- Subsystems are now registered at compile-time in linux/container_subsys.h. This allows for faster access to subsystem state since the id is a compile-time constant, so there's only a single extra pointer dereference compared to having a pointer directly in the task_struct. It also avoids wasting space with unused subsystem pointers.

- Removed the container pointers from container_group - this results in a structure very similar to Srivatsa Vaddagiri's rcfs approach. (RCFS uses the nsproxy object rather than the container_group object; merging container_group and nsproxy would be pretty straightforward if desired).

- Removed callback_mutex from container subsystem to be purely back in the cpuset subsystem. Renamed manage_mutex to container_mutex.

- Condensed post_attach_task() into attach_task() now that callback_mutex is purely within cpuset.c

- Simplified the container_subsys_state reference counting - stricter rules on liveness make adding reference counts cheaper.

Still TODO:

- decide whether "Containers" is an acceptable name for the system given its usage by some other development groups, or whether something else (ProcessSets? ResourceGroups?) would be better

- decide whether merging container_group and nsproxy is desirable

- add a hash-table based lookup for container_group objects.

- use seq_file properly in container tasks files (and also in cpuset_attach_task) to avoid having to allocate a big array for all the container's task pointers.

- add back support for the "release agent" functionality

- lots more testing

- define standards for container file names

Generic Process Containers
--------------------------

There have recently been various proposals floating around for resource management/accounting and other task grouping subsystems in the kernel, including ResGroups, User BeanCounters, NSProxy containers, and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, or control other behaviour of the processes, and all implement this grouping in different ways.

Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards to synchronization rules).

This patchset extracts the process grouping code from cpusets into a generic container system, and makes the cpusets code a client of the container system.

It also provides several example clients of the container system, including ResGroups, BeanCounters and namespace proxy.

The change is implemented in three implementation patches, plus four example subsystems that aren't necessarily intended to be merged as part of this patch set, but demonstrate the applicability of the framework.

1) extract the process grouping code from cpusets into a standalone system

2) remove the process grouping code from cpusets and hook into the container system

3) convert the container system to present a generic multi-hierarchy API, and make cpusets a client of that API

4) example of a simple CPU accounting container subsystem. Useful as a boilerplate for people implementing their own subsystems.

5) example of implementing ResGroups and its numtasks controller over generic containers

6) example of implementing BeanCounters and its numfiles counter over generic containers

7) example of integrating the namespace isolation code (sys_unshare() or various clone flags) with generic containers, allowing virtual servers to take advantage of other resource control efforts.

The intention is that the various resource management and virtualization efforts can also become container clients, with the result that:

- the userspace APIs are (somewhat) normalised

- it's easier to test out e.g. the ResGroups CPU controller in conjunction with the BeanCounters memory controller, or use either of
them as the resource-control portion of a virtual server system.

- the additional kernel footprint of any of the competing resource management systems is substantially reduced, since it doesn't need to provide process grouping/containment, hence improving their chances of getting into the kernel

#1 lore raw

PATCH 1/7 Containers (V8): Generic container system abstracted from cpusets code

This patch creates a generic process container system based on (and parallel top) the cpusets code. At a coarse level it was created by copying kernel/cpuset.c, doing s/cpuset/container/g, and stripping out any code that was cpuset-specific rather than applicable to any process container subsystem.

---
 Documentation/containers.txt |  229 +++++++
 fs/proc/base.c               |    7 
 include/linux/container.h    |   96 +++
 include/linux/sched.h        |    5 
 init/Kconfig                 |    9 
 init/main.c                  |    3 
 kernel/Makefile              |    1 
 kernel/container.c           | 1260 +++++++++++++++++++++++++++++++++++++++++++
 kernel/exit.c                |    2 
 kernel/fork.c                |    3 
 10 files changed, 1614 insertions(+), 1 deletion(-)
fs/proc/base.c +7 -0
Index: container-2.6.20-new/fs/proc/base.c
===================================================================
@@ -68,6 +68,7 @@
 #include <linux/security.h>
 #include <linux/ptrace.h>
 #include <linux/seccomp.h>
+#include <linux/container.h>
 #include <linux/cpuset.h>
 #include <linux/audit.h>
 #include <linux/poll.h>
@@ -1870,6 +1871,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_CPUSETS
 	REG("cpuset",     S_IRUGO, cpuset),
 #endif
+#ifdef CONFIG_CONTAINERS
+	REG("container",  S_IRUGO, container),
+#endif
 	INF("oom_score",  S_IRUGO, oom_score),
 	REG("oom_adj",    S_IRUGO|S_IWUSR, oom_adjust),
 #ifdef CONFIG_AUDITSYSCALL
@@ -2151,6 +2155,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_CPUSETS
 	REG("cpuset",    S_IRUGO, cpuset),
 #endif
+#ifdef CONFIG_CONTAINERS
+	REG("container",  S_IRUGO, container),
+#endif
 	INF("oom_score", S_IRUGO, oom_score),
 	REG("oom_adj",   S_IRUGO|S_IWUSR, oom_adjust),
 #ifdef CONFIG_AUDITSYSCALL
include/linux/container.h new file +96 -0

96 lines not on this page · raw message

include/linux/sched.h +4 -1
Index: container-2.6.20-new/include/linux/sched.h
===================================================================
@@ -743,8 +743,8 @@ extern unsigned int max_cache_size;
 
 
 struct io_context;			/* See blkdev.h */
+struct container;
 struct cpuset;
-
 #define NGROUPS_SMALL		32
 #define NGROUPS_PER_BLOCK	((int)(PAGE_SIZE / sizeof(gid_t)))
 struct group_info {
@@ -1031,6 +1031,9 @@ struct task_struct {
 	int cpuset_mems_generation;
 	int cpuset_mem_spread_rotor;
 #endif
+#ifdef CONFIG_CONTAINERS
+	struct container *container;
+#endif
 	struct robust_list_head __user *robust_list;
 #ifdef CONFIG_COMPAT
 	struct compat_robust_list_head __user *compat_robust_list;
init/Kconfig +9 -0
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -238,6 +238,15 @@ config IKCONFIG_PROC
 	  This option enables access to the kernel configuration file
 	  through /proc/config.gz.
 
+config CONTAINERS
+	bool "Container support"
+	help
+	  This option will let you create and manage process containers,
+	  which can be used to aggregate multiple processes, e.g. for
+	  the purposes of resource tracking.
+
+	  Say N if unsure
+
 config CPUSETS
 	bool "Cpuset support"
 	depends on SMP
init/main.c +3 -0
Index: container-2.6.20-new/init/main.c
===================================================================
@@ -39,6 +39,7 @@
 #include <linux/writeback.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/efi.h>
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
@@ -485,6 +486,7 @@ asmlinkage void __init start_kernel(void
 	char * command_line;
 	extern struct kernel_param __start___param[], __stop___param[];
 
+	container_init_early();
 	smp_setup_processor_id();
 
 	/*
@@ -608,6 +610,7 @@ asmlinkage void __init start_kernel(void
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
 #endif
+	container_init();
 	cpuset_init();
 	taskstats_init_early();
 	delayacct_init();
kernel/container.c new file +1260 -0

1,260 lines not on this page · raw message

kernel/exit.c +2 -0
Index: container-2.6.20-new/kernel/exit.c
===================================================================
@@ -31,6 +31,7 @@
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/syscalls.h>
 #include <linux/signal.h>
 #include <linux/posix-timers.h>
@@ -927,6 +928,7 @@ fastcall NORET_TYPE void do_exit(long co
 	__exit_fs(tsk);
 	exit_thread();
 	cpuset_exit(tsk);
+	container_exit(tsk);
 	exit_keys(tsk);
 
 	if (group_dead && tsk->signal->leader)
kernel/fork.c +3 -0
Index: container-2.6.20-new/kernel/fork.c
===================================================================
@@ -31,6 +31,7 @@
 #include <linux/capability.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/security.h>
 #include <linux/swap.h>
 #include <linux/syscalls.h>
@@ -1058,6 +1059,7 @@ static struct task_struct *copy_process(
 	p->io_context = NULL;
 	p->io_wait = NULL;
 	p->audit_context = NULL;
+	container_fork(p);
 	cpuset_fork(p);
 #ifdef CONFIG_NUMA
  	p->mempolicy = mpol_copy(p->mempolicy);
@@ -1291,6 +1293,7 @@ bad_fork_cleanup_policy:
 bad_fork_cleanup_cpuset:
 #endif
 	cpuset_exit(p);
+	container_exit(p);
 bad_fork_cleanup_delays_binfmt:
 	delayacct_tsk_free(p);
 	if (p->binfmt)
kernel/Makefile +1 -0
Index: container-2.6.20-new/kernel/Makefile
===================================================================
@@ -35,6 +35,7 @@ obj-$(CONFIG_PM) += power/
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_COMPAT) += compat.o
+obj-$(CONFIG_CONTAINERS) += container.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
Documentation/containers.txt new file +229 -0

229 lines not on this page · raw message

--

#2 lore raw

PATCH 2/7 Containers (V8): Cpusets hooked into containers

In reply to [email protected]

This patch removes the process grouping code from the cpusets code, instead hooking it into the generic container system. This temporarily adds cpuset-specific code in kernel/container.c, which is removed by the next patch in the series.

---
 Documentation/cpusets.txt |   81 +--
 fs/proc/base.c            |    4 
 fs/super.c                |    5 
 include/linux/container.h |   10 
 include/linux/cpuset.h    |   16 
 include/linux/fs.h        |    2 
 include/linux/mempolicy.h |    2 
 include/linux/sched.h     |    4 
 init/Kconfig              |   14 
 kernel/container.c        |  117 ++++
 kernel/cpuset.c           | 1100 +++++-----------------------------------------
 kernel/exit.c             |    2 
 kernel/fork.c             |    7 
 13 files changed, 286 insertions(+), 1078 deletions(-)
include/linux/container.h +7 -3
Index: container-2.6.20-new/include/linux/container.h
===================================================================
@@ -27,9 +27,6 @@ extern struct file_operations proc_conta
 extern void container_lock(void);
 extern void container_unlock(void);
 
-extern void container_manage_lock(void);
-extern void container_manage_unlock(void);
-
 struct container {
 	unsigned long flags;		/* "unsigned long" so bitops work */
 
@@ -47,6 +44,10 @@ struct container {
 
 	struct container *parent;	/* my parent */
 	struct dentry *dentry;		/* container fs entry */
+
+#ifdef CONFIG_CPUSETS
+	struct cpuset *cpuset;
+#endif
 };
 
 /* struct cftype:
@@ -79,6 +80,9 @@ struct cftype {
 int container_add_file(struct container *cont, const struct cftype *cft);
 
 int container_is_removed(const struct container *cont);
+void container_set_release_agent_path(const char *path);
+
+int container_path(const struct container *cont, char *buf, int buflen);
 
 #else /* !CONFIG_CONTAINERS */
 
include/linux/cpuset.h +11 -5
Index: container-2.6.20-new/include/linux/cpuset.h
===================================================================
@@ -11,16 +11,15 @@
 #include <linux/sched.h>
 #include <linux/cpumask.h>
 #include <linux/nodemask.h>
+#include <linux/container.h>
 
 #ifdef CONFIG_CPUSETS
 
-extern int number_of_cpusets;	/* How many cpusets are defined in system? */
+extern int number_of_cpusets;  /* How many cpusets are defined in system? */
 
 extern int cpuset_init_early(void);
 extern int cpuset_init(void);
 extern void cpuset_init_smp(void);
-extern void cpuset_fork(struct task_struct *p);
-extern void cpuset_exit(struct task_struct *p);
 extern cpumask_t cpuset_cpus_allowed(struct task_struct *p);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
@@ -75,13 +74,20 @@ static inline int cpuset_do_slab_mem_spr
 
 extern void cpuset_track_online_nodes(void);
 
+extern int cpuset_can_attach_task(struct container *cont,
+				  struct task_struct *tsk);
+extern void cpuset_attach_task(struct container *cont,
+			       struct container *oldcont,
+			       struct task_struct *tsk);
+extern int cpuset_populate_dir(struct container *cont);
+extern int cpuset_create(struct container *cont);
+extern void cpuset_destroy(struct container *cont);
+
 #else /* !CONFIG_CPUSETS */
 
 static inline int cpuset_init_early(void) { return 0; }
 static inline int cpuset_init(void) { return 0; }
 static inline void cpuset_init_smp(void) {}
-static inline void cpuset_fork(struct task_struct *p) {}
-static inline void cpuset_exit(struct task_struct *p) {}
 
 static inline cpumask_t cpuset_cpus_allowed(struct task_struct *p)
 {
kernel/exit.c +0 -2
Index: container-2.6.20-new/kernel/exit.c
===================================================================
@@ -30,7 +30,6 @@
 #include <linux/mempolicy.h>
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
-#include <linux/cpuset.h>
 #include <linux/container.h>
 #include <linux/syscalls.h>
 #include <linux/signal.h>
@@ -927,7 +926,6 @@ fastcall NORET_TYPE void do_exit(long co
 	__exit_files(tsk);
 	__exit_fs(tsk);
 	exit_thread();
-	cpuset_exit(tsk);
 	container_exit(tsk);
 	exit_keys(tsk);
 
kernel/fork.c +2 -5
Index: container-2.6.20-new/kernel/fork.c
===================================================================
@@ -30,7 +30,6 @@
 #include <linux/nsproxy.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
-#include <linux/cpuset.h>
 #include <linux/container.h>
 #include <linux/security.h>
 #include <linux/swap.h>
@@ -1060,13 +1059,12 @@ static struct task_struct *copy_process(
 	p->io_wait = NULL;
 	p->audit_context = NULL;
 	container_fork(p);
-	cpuset_fork(p);
 #ifdef CONFIG_NUMA
  	p->mempolicy = mpol_copy(p->mempolicy);
  	if (IS_ERR(p->mempolicy)) {
  		retval = PTR_ERR(p->mempolicy);
  		p->mempolicy = NULL;
- 		goto bad_fork_cleanup_cpuset;
+ 		goto bad_fork_cleanup_container;
  	}
 	mpol_fix_fork_child_flag(p);
 #endif
@@ -1290,9 +1288,8 @@ bad_fork_cleanup_security:
 bad_fork_cleanup_policy:
 #ifdef CONFIG_NUMA
 	mpol_free(p->mempolicy);
-bad_fork_cleanup_cpuset:
+bad_fork_cleanup_container:
 #endif
-	cpuset_exit(p);
 	container_exit(p);
 bad_fork_cleanup_delays_binfmt:
 	delayacct_tsk_free(p);
kernel/container.c +102 -15

246 lines not on this page · raw message

kernel/cpuset.c +120 -980

1,537 lines not on this page · raw message

init/Kconfig +7 -7
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -239,17 +239,12 @@ config IKCONFIG_PROC
 	  through /proc/config.gz.
 
 config CONTAINERS
-	bool "Container support"
-	help
-	  This option will let you create and manage process containers,
-	  which can be used to aggregate multiple processes, e.g. for
-	  the purposes of resource tracking.
-
-	  Say N if unsure
+	bool
 
 config CPUSETS
 	bool "Cpuset support"
 	depends on SMP
+	select CONTAINERS
 	help
 	  This option will let you create and manage CPUSETs which
 	  allow dynamically partitioning a system into sets of CPUs and
@@ -278,6 +273,11 @@ config SYSFS_DEPRECATED
 	  If you are using a distro that was released in 2006 or later,
 	  it should be safe to say N here.
 
+config PROC_PID_CPUSET
+	bool "Include legacy /proc/<pid>/cpuset file"
+	depends on CPUSETS
+	default y
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help
include/linux/sched.h +1 -3
Index: container-2.6.20-new/include/linux/sched.h
===================================================================
@@ -744,7 +744,6 @@ extern unsigned int max_cache_size;
 
 struct io_context;			/* See blkdev.h */
 struct container;
-struct cpuset;
 #define NGROUPS_SMALL		32
 #define NGROUPS_PER_BLOCK	((int)(PAGE_SIZE / sizeof(gid_t)))
 struct group_info {
@@ -1026,7 +1025,6 @@ struct task_struct {
 	short il_next;
 #endif
 #ifdef CONFIG_CPUSETS
-	struct cpuset *cpuset;
 	nodemask_t mems_allowed;
 	int cpuset_mems_generation;
 	int cpuset_mem_spread_rotor;
@@ -1471,7 +1469,7 @@ static inline int thread_group_empty(str
 /*
  * Protects ->fs, ->files, ->mm, ->group_info, ->comm, keyring
  * subscriptions and synchronises with wait4().  Also used in procfs.  Also
- * pins the final release of task.io_context.  Also protects ->cpuset.
+ * pins the final release of task.io_context.  Also protects ->container.
  *
  * Nests both inside and outside of read_lock(&tasklist_lock).
  * It must not be nested with write_lock_irq(&tasklist_lock),
Documentation/cpusets.txt +31 -50

166 lines not on this page · raw message

fs/super.c +0 -5
Index: container-2.6.20-new/fs/super.c
===================================================================
@@ -39,11 +39,6 @@
 #include <linux/mutex.h>
 #include <asm/uaccess.h>
 
-
-void get_filesystem(struct file_system_type *fs);
-void put_filesystem(struct file_system_type *fs);
-struct file_system_type *get_fs_type(const char *name);
-
 LIST_HEAD(super_blocks);
 DEFINE_SPINLOCK(sb_lock);
 
include/linux/fs.h +2 -0
Index: container-2.6.20-new/include/linux/fs.h
===================================================================
@@ -1841,6 +1841,8 @@ extern int vfs_fstat(unsigned int, struc
 
 extern int vfs_ioctl(struct file *, unsigned int, unsigned int, unsigned long);
 
+extern void get_filesystem(struct file_system_type *fs);
+extern void put_filesystem(struct file_system_type *fs);
 extern struct file_system_type *get_fs_type(const char *name);
 extern struct super_block *get_super(struct block_device *);
 extern struct super_block *user_get_super(dev_t);
include/linux/mempolicy.h +1 -1
Index: container-2.6.20-new/include/linux/mempolicy.h
===================================================================
@@ -152,7 +152,7 @@ extern void mpol_fix_fork_child_flag(str
 
 #ifdef CONFIG_CPUSETS
 #define current_cpuset_is_being_rebound() \
-				(cpuset_being_rebound == current->cpuset)
+ (cpuset_being_rebound == current->container->cpuset)
 #else
 #define current_cpuset_is_being_rebound() 0
 #endif
fs/proc/base.c +2 -2
Index: container-2.6.20-new/fs/proc/base.c
===================================================================
@@ -1868,7 +1868,7 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_SCHEDSTATS
 	INF("schedstat",  S_IRUGO, pid_schedstat),
 #endif
-#ifdef CONFIG_CPUSETS
+#ifdef CONFIG_PROC_PID_CPUSET
 	REG("cpuset",     S_IRUGO, cpuset),
 #endif
 #ifdef CONFIG_CONTAINERS
@@ -2152,7 +2152,7 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_SCHEDSTATS
 	INF("schedstat", S_IRUGO, pid_schedstat),
 #endif
-#ifdef CONFIG_CPUSETS
+#ifdef CONFIG_PROC_PID_CPUSET
 	REG("cpuset",    S_IRUGO, cpuset),
 #endif
 #ifdef CONFIG_CONTAINERS

--

#3 lore raw

Re: [PATCH 2/7] Containers (V8): Cpusets hooked into containers

Quoted 7 lines · [email protected] wrote:

This patch removes the process grouping code from the cpusets code, instead hooking it into the generic container system. This temporarily adds cpuset-specific code in kernel/container.c, which is removed by the next patch in the series.

[snip]

Quoted 6 lines

Index: container-2.6.20-new/init/Kconfig
===================================================================

init/Kconfig +0 -0
@@ -239,17 +239,12 @@ config IKCONFIG_PROC
 	  through /proc/config.gz.

init/Kconfig config IKCONFIG_PROC · line 242

Quoted 10 lines
init/Kconfig +1 -7
 
 config CONTAINERS
-	bool "Container support"
-	help
-	  This option will let you create and manage process containers,
-	  which can be used to aggregate multiple processes, e.g. for
-	  the purposes of resource tracking.
-
-	  Say N if unsure
+	bool

Hi Paul,

This looks like some patch generation error. Description for containers should not be removed after applying this patch.

--Vaidy

Quoted 20 lines
init/Kconfig +6 -0
 config CPUSETS
 	bool "Cpuset support"
 	depends on SMP
+	select CONTAINERS
 	help
 	  This option will let you create and manage CPUSETs which
 	  allow dynamically partitioning a system into sets of CPUs and
@@ -278,6 +273,11 @@ config SYSFS_DEPRECATED
 	  If you are using a distro that was released in 2006 or later,
 	  it should be safe to say N here.
 
+config PROC_PID_CPUSET
+	bool "Include legacy /proc/<pid>/cpuset file"
+	depends on CPUSETS
+	default y
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help

[snip]

#4 lore raw

Re: [PATCH 2/7] Containers (V8): Cpusets hooked into containers

Quoted 11 lines · Vaidyanathan Srinivasan

init/Kconfig config IKCONFIG_PROC · line 242

Quoted 10 lines

config CONTAINERS

init/Kconfig +1 -7
-     bool "Container support"
-     help
-       This option will let you create and manage process containers,
-       which can be used to aggregate multiple processes, e.g. for
-       the purposes of resource tracking.
-
-       Say N if unsure
+     bool
Quoted 4 lines

Hi Paul,

This looks like some patch generation error. Description for containers should not be removed after applying this patch.

No, this is intentional - in the first patch in the series, CONFIG_CONTAINER was a user-selectable option so it had a description; in the second it becomes an option that's only selected if other selected systems (e.g. cpusets) depend on it. So it no longer needs help text.

Cheers,

Paul

#5 lore raw

PATCH 4/7 Containers (V8): Simple CPU accounting container subsystem

In reply to [email protected]

This example demonstrates how to use the generic container subsystem for a simple resource tracker that counts, for the processes in a container, the total CPU time used and the %CPU used in the last complete 10 second interval.

Portions contributed by Balbir Singh <[email protected]>

---
 include/linux/container_subsys.h |    6 +
 include/linux/cpu_acct.h         |   14 ++
 init/Kconfig                     |    7 +
 kernel/Makefile                  |    1 
 kernel/cpu_acct.c                |  204 +++++++++++++++++++++++++++++++++++++++
 kernel/sched.c                   |   14 ++
 6 files changed, 243 insertions(+), 3 deletions(-)
include/linux/cpu_acct.h new file +14 -0
Index: container-2.6.20-new/include/linux/cpu_acct.h
===================================================================
@@ -0,0 +1,14 @@
+
+#ifndef _LINUX_CPU_ACCT_H
+#define _LINUX_CPU_ACCT_H
+
+#include <linux/container.h>
+#include <asm/cputime.h>
+
+#ifdef CONFIG_CONTAINER_CPUACCT
+extern void cpuacct_charge(struct task_struct *, cputime_t cputime);
+#else
+static void inline cpuacct_charge(struct task_struct *p, cputime_t cputime) {}
+#endif
+
+#endif
init/Kconfig +7 -0
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -278,6 +278,13 @@ config PROC_PID_CPUSET
 	depends on CPUSETS
 	default y
 
+config CONTAINER_CPUACCT
+	bool "Simple CPU accounting container subsystem"
+	select CONTAINERS
+	help
+	  Provides a simple Resource Controller for monitoring the
+	  total CPU consumed by the tasks in a container
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help
kernel/cpu_acct.c new file +204 -0

204 lines not on this page · raw message

kernel/Makefile +1 -0
Index: container-2.6.20-new/kernel/Makefile
===================================================================
@@ -37,6 +37,7 @@ obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_COMPAT) += compat.o
 obj-$(CONFIG_CONTAINERS) += container.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
+obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
kernel/sched.c +11 -3
Index: container-2.6.20-new/kernel/sched.c
===================================================================
@@ -52,6 +52,7 @@
 #include <linux/tsacct_kern.h>
 #include <linux/kprobes.h>
 #include <linux/delayacct.h>
+#include <linux/cpu_acct.h>
 #include <asm/tlb.h>
 
 #include <asm/unistd.h>
@@ -3066,9 +3067,13 @@ void account_user_time(struct task_struc
 {
 	struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
 	cputime64_t tmp;
+	struct rq *rq = this_rq();
 
 	p->utime = cputime_add(p->utime, cputime);
 
+	if (p != rq->idle)
+		cpuacct_charge(p, cputime);
+
 	/* Add user time to cpustat. */
 	tmp = cputime_to_cputime64(cputime);
 	if (TASK_NICE(p) > 0)
@@ -3098,9 +3103,10 @@ void account_system_time(struct task_str
 		cpustat->irq = cputime64_add(cpustat->irq, tmp);
 	else if (softirq_count())
 		cpustat->softirq = cputime64_add(cpustat->softirq, tmp);
-	else if (p != rq->idle)
+	else if (p != rq->idle) {
 		cpustat->system = cputime64_add(cpustat->system, tmp);
-	else if (atomic_read(&rq->nr_iowait) > 0)
+		cpuacct_charge(p, cputime);
+	} else if (atomic_read(&rq->nr_iowait) > 0)
 		cpustat->iowait = cputime64_add(cpustat->iowait, tmp);
 	else
 		cpustat->idle = cputime64_add(cpustat->idle, tmp);
@@ -3125,8 +3131,10 @@ void account_steal_time(struct task_stru
 			cpustat->iowait = cputime64_add(cpustat->iowait, tmp);
 		else
 			cpustat->idle = cputime64_add(cpustat->idle, tmp);
-	} else
+	} else {
 		cpustat->steal = cputime64_add(cpustat->steal, tmp);
+		cpuacct_charge(p, -tmp);
+	}
 }
 
 static void task_running_tick(struct rq *rq, struct task_struct *p)
include/linux/container_subsys.h +6 -0
Index: container-2.6.20-new/include/linux/container_subsys.h
===================================================================
@@ -11,4 +11,10 @@ SUBSYS(cpuset)
 
 /* */
 
+#ifdef CONFIG_CONTAINER_CPUACCT
+SUBSYS(cpuacct)
+#endif
+
+/* */
+
 /* */

--

#6 lore raw

Re: [PATCH 4/7] Containers (V8): Simple CPU accounting container subsystem

kernel/cpu_acct.c line 204

Quoted 7 lines · On Fri, Apr 06, 2007 at 04:32:25PM -0700, [email protected] wrote:
kernel/cpu_acct.c new file +7 -0
+struct container_subsys cpuacct_subsys = {
+	.name = "cpuacct",
+	.create = cpuacct_create,
+	.destroy = cpuacct_destroy,
+	.populate = cpuacct_populate,
+	.subsys_id = cpuacct_subsys_id,
+};

container_register_subsys(&cpuacct_subsys) is missing ..

Signature · 2 lines
Regards,
vatsa
#7 lore raw

PATCH 5/7 Containers (V8): Resource Groups over generic containers

In reply to [email protected]

This patch provides the RG core and numtasks controller as container subsystems, intended as an example of how to implement a more complex resource control system over generic process containers. The changes to the core involve primarily removing the group management, task membership and configfs support and adding interface layers to talk to the generic container layer instead.

Each resource controller becomes an independent container subsystem; the RG core is essentially a library that the resource controllers can use to provide the RG API to userspace. Rather than a single shares and stats file in each group, there's a <controller>_shares and a <controller>_stats file, each linked to the appropriate resource controller.

 include/linux/container_subsys.h |    6 
 include/linux/moduleparam.h      |   12 -
 include/linux/numtasks.h         |   28 ++
 include/linux/res_group.h        |   86 +++++++
 include/linux/res_group_rc.h     |  125 ++++++++++
 init/Kconfig                     |   22 +
 kernel/Makefile                  |    1 
 kernel/fork.c                    |    7 
 kernel/res_group/Makefile        |    2 
 kernel/res_group/local.h         |   38 +++
 kernel/res_group/numtasks.c      |  451 +++++++++++++++++++++++++++++++++++++++
 kernel/res_group/res_group.c     |  135 +++++++++++
 kernel/res_group/rgcs.c          |  302 ++++++++++++++++++++++++++
 kernel/res_group/shares.c        |  228 +++++++++++++++++++
 14 files changed, 1439 insertions(+), 4 deletions(-)
include/linux/moduleparam.h +9 -3
Index: container-2.6.20-new/include/linux/moduleparam.h
===================================================================
@@ -78,11 +78,17 @@ struct kparam_array
 /* Helper functions: type is byte, short, ushort, int, uint, long,
    ulong, charp, bool or invbool, or XXX if you define param_get_XXX,
    param_set_XXX and param_check_XXX. */
-#define module_param_named(name, value, type, perm)			   \
-	param_check_##type(name, &(value));				   \
-	module_param_call(name, param_set_##type, param_get_##type, &value, perm); \
+#define module_param_named_call(name, value, type, set, perm)		\
+	param_check_##type(name, &(value));				\
+	module_param_call(name, set, param_get_##type, &(value), perm); \
 	__MODULE_PARM_TYPE(name, #type)
 
+#define module_param_named(name, value, type, perm)			   \
+	module_param_named_call(name, value, type, param_set_##type, perm)
+
+#define module_param_set_call(name, type, setfn, perm) \
+	module_param_named_call(name, name, type, setfn, perm)
+
 #define module_param(name, type, perm)				\
 	module_param_named(name, name, type, perm)
 
include/linux/numtasks.h new file +28 -0
Index: container-2.6.20-new/include/linux/numtasks.h
===================================================================
@@ -0,0 +1,28 @@
+/* numtasks.h - No. of tasks resource controller for Resource Groups
+ *
+ * Copyright (C) Chandra Seetharaman, IBM Corp. 2003, 2004, 2005
+ *
+ * Provides No. of tasks resource controller for Resource Groups
+ *
+ * Latest version, more details at http://ckrm.sf.net
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ */
+#ifndef _LINUX_NUMTASKS_H
+#define _LINUX_NUMTASKS_H
+
+#ifdef CONFIG_RES_GROUPS_NUMTASKS
+#include <linux/res_group_rc.h>
+
+extern int numtasks_allow_fork(struct task_struct *);
+
+#else /* CONFIG_RES_GROUPS_NUMTASKS */
+
+#define numtasks_allow_fork(task) (0)
+
+#endif /* CONFIG_RES_GROUPS_NUMTASKS */
+#endif /* _LINUX_NUMTASKS_H */
include/linux/res_group.h new file +86 -0

86 lines not on this page · raw message

include/linux/res_group_rc.h new file +125 -0

125 lines not on this page · raw message

init/Kconfig +22 -0
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -329,6 +329,28 @@ config TASK_IO_ACCOUNTING
 
 	  Say N if unsure.
 
+menu "Resource Groups"
+
+config RES_GROUPS
+	bool "Resource Groups"
+	depends on EXPERIMENTAL
+	select CONTAINERS
+	help
+	  Resource Groups is a framework for controlling and monitoring
+	  resource allocation of user-defined groups of tasks. For more
+	  information, please visit http://ckrm.sf.net.
+
+config RES_GROUPS_NUMTASKS
+	bool "Number of Tasks Resource Controller"
+	depends on RES_GROUPS
+	default y
+	help
+	  Provides a Resource Controller for Resource Groups that allows
+	  limiting number of tasks a resource group can have.
+
+	  Say N if unsure, Y to use the feature.
+
+endmenu
 config SYSCTL
 	bool
 
kernel/Makefile +1 -0
Index: container-2.6.20-new/kernel/Makefile
===================================================================
@@ -52,6 +52,7 @@ obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
+obj-$(CONFIG_RES_GROUPS) += res_group/
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/fork.c +6 -1
Index: container-2.6.20-new/kernel/fork.c
===================================================================
@@ -49,6 +49,7 @@
 #include <linux/delayacct.h>
 #include <linux/taskstats_kern.h>
 #include <linux/random.h>
+#include <linux/numtasks.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>
@@ -1362,7 +1363,7 @@ long do_fork(unsigned long clone_flags,
 	      int __user *child_tidptr)
 {
 	struct task_struct *p;
-	int trace = 0;
+	int trace = 0, rc;
 	struct pid *pid = alloc_pid();
 	long nr;
 
@@ -1375,6 +1376,10 @@ long do_fork(unsigned long clone_flags,
 			clone_flags |= CLONE_PTRACE;
 	}
 
+	rc = numtasks_allow_fork(current);
+	if (rc)
+		return rc;
+
 	p = copy_process(clone_flags, stack_start, regs, stack_size, parent_tidptr, child_tidptr, nr);
 	/*
 	 * Do this prior waking up the new thread - the thread pointer
kernel/res_group/Makefile new file +2 -0
Index: container-2.6.20-new/kernel/res_group/Makefile
===================================================================
@@ -0,0 +1,2 @@
+obj-y = res_group.o shares.o rgcs.o
+obj-$(CONFIG_RES_GROUPS_NUMTASKS) += numtasks.o
kernel/res_group/local.h new file +38 -0
Index: container-2.6.20-new/kernel/res_group/local.h
===================================================================
@@ -0,0 +1,38 @@
+/*
+ * Contains function definitions that are local to the Resource Groups.
+ * NOT to be included by controllers.
+ */
+
+#include <linux/res_group_rc.h>
+
+extern struct res_controller *get_controller_by_name(const char *);
+extern struct res_controller *get_controller_by_id(unsigned int);
+extern void put_controller(struct res_controller *);
+extern struct resource_group *alloc_res_group(struct resource_group *,
+							const char *);
+extern int free_res_group(struct resource_group *);
+extern void release_res_group(struct kref *);
+extern int set_controller_shares(struct resource_group *,
+			struct res_controller *, const struct res_shares *);
+/* Set shares for the given resource group and resource to default values */
+extern void set_shares_to_default(struct resource_group *,
+						struct res_controller *);
+extern void res_group_teardown(void);
+extern int set_res_group(pid_t, struct resource_group *);
+extern void move_tasks_to_parent(struct resource_group *);
+
+ssize_t res_group_file_read(struct container *cont,
+			    struct cftype *cft,
+			    struct file *file,
+			    char __user *buf,
+			    size_t nbytes, loff_t *ppos);
+ssize_t res_group_file_write(struct container *cont,
+			     struct cftype *cft,
+			     struct file *file,
+			     const char __user *userbuf,
+			     size_t nbytes, loff_t *ppos);
+
+enum {
+	RG_FILE_SHARES,
+	RG_FILE_STATS,
+};
kernel/res_group/numtasks.c new file +451 -0

451 lines not on this page · raw message

kernel/res_group/res_group.c new file +135 -0

135 lines not on this page · raw message

kernel/res_group/rgcs.c new file +302 -0

302 lines not on this page · raw message

kernel/res_group/shares.c new file +228 -0

228 lines not on this page · raw message

include/linux/container_subsys.h +6 -0
Index: container-2.6.20-new/include/linux/container_subsys.h
===================================================================
@@ -17,4 +17,10 @@ SUBSYS(cpuacct)
 
 /* */
 
+#ifdef CONFIG_RES_GROUPS_NUMTASKS
+SUBSYS(numtasks)
+#endif
+
+/* */
+
 /* */

--

#8 lore raw

PATCH 6/7 Containers (V8): BeanCounters over generic process containers

In reply to [email protected]

This patch implements the BeanCounter resource control abstraction over generic process containers. It contains the beancounter core code, plus the numfiles resource counter. It doesn't currently contain any of the memory tracking code or the code for switching beancounter context in interrupts.

Currently all the beancounters resource counters are lumped into a single hierarchy; ideally it would be possible for each resource counter to be a separate container subsystem, allowing them to be connected to different hierarchies.

---
 fs/file_table.c                  |   11 +
 include/bc/beancounter.h         |  194 ++++++++++++++++++++
 include/bc/misc.h                |   27 ++
 include/bc/task.h                |   64 ++++++
 include/linux/container_subsys.h |    6 
 include/linux/fs.h               |    3 
 include/linux/sched.h            |    3 
 init/Kconfig                     |    4 
 init/main.c                      |    3 
 kernel/Makefile                  |    1 
 kernel/bc/Kconfig                |   17 +
 kernel/bc/Makefile               |    7 
 kernel/bc/beancounter.c          |  368 +++++++++++++++++++++++++++++++++++++++
 kernel/bc/misc.c                 |   57 ++++++
 kernel/softirq.c                 |    7 
 15 files changed, 771 insertions(+), 1 deletion(-)
init/Kconfig +4 -0
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -607,6 +607,10 @@ config STOP_MACHINE
 	  Need stop_machine() primitive.
 endmenu
 
+menu "Beancounters"
+source "kernel/bc/Kconfig"
+endmenu
+
 menu "Block layer"
 source "block/Kconfig"
 endmenu
kernel/Makefile +1 -0
Index: container-2.6.20-new/kernel/Makefile
===================================================================
@@ -12,6 +12,7 @@ obj-y     = sched.o fork.o exec_domain.o
 
 obj-$(CONFIG_STACKTRACE) += stacktrace.o
 obj-y += time/
+obj-$(CONFIG_BEANCOUNTERS) += bc/
 obj-$(CONFIG_DEBUG_MUTEXES) += mutex-debug.o
 obj-$(CONFIG_LOCKDEP) += lockdep.o
 ifeq ($(CONFIG_PROC_FS),y)
kernel/bc/Kconfig new file +17 -0
Index: container-2.6.20-new/kernel/bc/Kconfig
===================================================================
@@ -0,0 +1,17 @@
+config BEANCOUNTERS
+	bool "Enable resource accounting/control"
+	default n
+	select CONTAINERS
+	help
+	  When Y this option provides accounting and allows configuring
+	  limits for user's consumption of exhaustible system resources.
+	  The most important resource controlled by this patch is unswappable
+	  memory (either mlock'ed or used by internal kernel structures and
+	  buffers). The main goal of this patch is to protect processes
+	  from running short of important resources because of accidental
+	  misbehavior of processes or malicious activity aiming to ``kill''
+	  the system. It's worth mentioning that resource limits configured
+	  by setrlimit(2) do not give an acceptable level of protection
+	  because they cover only a small fraction of resources and work on a
+	  per-process basis.  Per-process accounting doesn't prevent malicious
+	  users from spawning a lot of resource-consuming processes.
kernel/bc/Makefile new file +7 -0
Index: container-2.6.20-new/kernel/bc/Makefile
===================================================================
@@ -0,0 +1,7 @@
+#
+# kernel/bc/Makefile
+#
+# Copyright (C) 2006 OpenVZ SWsoft Inc.
+#
+
+obj-y = beancounter.o misc.o
include/bc/beancounter.h new file +194 -0

194 lines not on this page · raw message

init/main.c +3 -0
Index: container-2.6.20-new/init/main.c
===================================================================
@@ -54,6 +54,8 @@
 #include <linux/pid_namespace.h>
 #include <linux/device.h>
 
+#include <bc/beancounter.h>
+
 #include <asm/io.h>
 #include <asm/bugs.h>
 #include <asm/setup.h>
@@ -487,6 +489,7 @@ asmlinkage void __init start_kernel(void
 	extern struct kernel_param __start___param[], __stop___param[];
 
 	container_init_early();
+	bc_init_early();
 	smp_setup_processor_id();
 
 	/*
kernel/bc/beancounter.c new file +368 -0

368 lines not on this page · raw message

include/bc/misc.h new file +27 -0
Index: container-2.6.20-new/include/bc/misc.h
===================================================================
@@ -0,0 +1,27 @@
+/*
+ * include/bc/misc.h
+ *
+ * Copyright (C) 2006 OpenVZ SWsoft Inc
+ *
+ */
+
+#ifndef __BC_MISC_H__
+#define __BC_MISC_H__
+
+struct file;
+
+#ifdef CONFIG_BEANCOUNTERS
+int __must_check bc_file_charge(struct file *);
+void bc_file_uncharge(struct file *);
+#else
+static inline int __must_check bc_file_charge(struct file *f)
+{
+	return 0;
+}
+
+static inline void bc_file_uncharge(struct file *f)
+{
+}
+#endif
+
+#endif
kernel/bc/misc.c new file +57 -0

57 lines not on this page · raw message

fs/file_table.c +10 -1
Index: container-2.6.20-new/fs/file_table.c
===================================================================
@@ -22,6 +22,8 @@
 #include <linux/sysctl.h>
 #include <linux/percpu_counter.h>
 
+#include <bc/misc.h>
+
 #include <asm/atomic.h>
 
 /* sysctl tunables... */
@@ -43,6 +45,7 @@ static inline void file_free_rcu(struct 
 static inline void file_free(struct file *f)
 {
 	percpu_counter_dec(&nr_files);
+	bc_file_uncharge(f);
 	call_rcu(&f->f_u.fu_rcuhead, file_free_rcu);
 }
 
@@ -107,8 +110,10 @@ struct file *get_empty_filp(void)
 	if (f == NULL)
 		goto fail;
 
-	percpu_counter_inc(&nr_files);
 	memset(f, 0, sizeof(*f));
+	if (bc_file_charge(f))
+		goto fail_charge;
+	percpu_counter_inc(&nr_files);
 	if (security_file_alloc(f))
 		goto fail_sec;
 
@@ -135,6 +140,10 @@ fail_sec:
 	file_free(f);
 fail:
 	return NULL;
+
+ fail_charge:
+	kmem_cache_free(filp_cachep, f);
+	return NULL;
 }
 
 EXPORT_SYMBOL(get_empty_filp);
include/linux/fs.h +3 -0
Index: container-2.6.20-new/include/linux/fs.h
===================================================================
@@ -739,6 +739,9 @@ struct file {
 	spinlock_t		f_ep_lock;
 #endif /* #ifdef CONFIG_EPOLL */
 	struct address_space	*f_mapping;
+#ifdef CONFIG_BEANCOUNTERS
+	struct beancounter      *f_bc;
+#endif
 };
 extern spinlock_t files_lock;
 #define file_list_lock() spin_lock(&files_lock);
include/bc/task.h new file +64 -0

64 lines not on this page · raw message

include/linux/sched.h +3 -0
Index: container-2.6.20-new/include/linux/sched.h
===================================================================
@@ -1052,6 +1052,9 @@ struct task_struct {
 #ifdef CONFIG_FAULT_INJECTION
 	int make_it_fail;
 #endif
+#ifdef CONFIG_BEANCOUNTERS
+	struct beancounter *tmp_exec_bc;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
kernel/softirq.c +7 -0
Index: container-2.6.20-new/kernel/softirq.c
===================================================================
@@ -18,6 +18,8 @@
 #include <linux/rcupdate.h>
 #include <linux/smp.h>
 
+#include <bc/task.h>
+
 #include <asm/irq.h>
 /*
    - No shared variables, all the data are CPU local.
@@ -209,6 +211,7 @@ asmlinkage void __do_softirq(void)
 	__u32 pending;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	int cpu;
+	struct beancounter *bc;
 
 	pending = local_softirq_pending();
 	account_system_vtime(current);
@@ -225,6 +228,8 @@ restart:
 
 	h = softirq_vec;
 
+	bc = set_exec_bc(&init_bc);
+
 	do {
 		if (pending & 1) {
 			h->action(h);
@@ -234,6 +239,8 @@ restart:
 		pending >>= 1;
 	} while (pending);
 
+	reset_exec_bc(bc, &init_bc);
+
 	local_irq_disable();
 
 	pending = local_softirq_pending();
include/linux/container_subsys.h +6 -0
Index: container-2.6.20-new/include/linux/container_subsys.h
===================================================================
@@ -23,4 +23,10 @@ SUBSYS(numtasks)
 
 /* */
 
+#ifdef CONFIG_BEANCOUNTERS
+SUBSYS(bc)
+#endif
+
+/* */
+
 /* */

--

#9 lore raw

Re: [PATCH 6/7] Containers (V8): BeanCounters over generic process containers

Quoted 10 lines · [email protected] wrote:

This patch implements the BeanCounter resource control abstraction over generic process containers. It contains the beancounter core code, plus the numfiles resource counter. It doesn't currently contain any of the memory tracking code or the code for switching beancounter context in interrupts.

Currently all the beancounters resource counters are lumped into a single hierarchy; ideally it would be possible for each resource counter to be a separate container subsystem, allowing them to be connected to different hierarchies.

Thanks for your attention, but since we have decided to build a "new" resource controllers above your containers infrastructure, I think there is no need in further BC code porting.

Thanks,
Pavel.

#10 lore raw

Re: [PATCH 6/7] Containers (V8): BeanCounters over generic process containers

In reply to [email protected]
Quoted 8 lines · On Fri, Apr 06, 2007 at 04:32:27PM -0700, [email protected] wrote:

This patch implements the BeanCounter resource control abstraction over generic process containers. It contains the beancounter core code, plus the numfiles resource counter. It doesn't currently contain any of the memory tracking code or the code for switching beancounter context in interrupts.
Currently all the beancounters resource counters are lumped into a single hierarchy; ideally it would be possible for each resource counter to be a separate container subsystem, allowing them to be

I'm very happy to see the BeanCounter work revived. The utility of this work should extend far beyond feature work like containers and into stability and reliability areas as well.

-- wli

#11 lore raw

PATCH 7/7 Containers (V8): Container interface to nsproxy subsystem

In reply to [email protected]

This is intended as a simple illustration of how a virtual server system could be integrated with generic containers, and hence take advantage of other resource-control efforts. A real implementation would probably allow parameters such as configuring what kinds of namespace creations triggered new containers, etc.

When a task enters a new namespace via a clone() or unshare(), a new container is created and the task moves into it. Developed by Serge Hallyn <[email protected]>, adapted by Paul Menage <[email protected]>

---
 include/linux/container_subsys.h |    6 ++
 include/linux/nsproxy.h          |    6 ++
 init/Kconfig                     |    9 +++
 kernel/Makefile                  |    1 
 kernel/fork.c                    |    4 +
 kernel/ns_container.c            |   99 +++++++++++++++++++++++++++++++++++++++
 kernel/nsproxy.c                 |    6 ++
 7 files changed, 131 insertions(+)
include/linux/nsproxy.h +6 -0
Index: container-2.6.20-new/include/linux/nsproxy.h
===================================================================
@@ -53,4 +53,10 @@ static inline void exit_task_namespaces(
 		put_nsproxy(ns);
 	}
 }
+#ifdef CONFIG_CONTAINER_NS
+int ns_container_clone(struct task_struct *tsk);
+#else
+static inline int ns_container_clone(struct task_struct *tsk) { return 0; }
+#endif
+
 #endif
kernel/Makefile +1 -0
Index: container-2.6.20-new/kernel/Makefile
===================================================================
@@ -39,6 +39,7 @@ obj-$(CONFIG_COMPAT) += compat.o
 obj-$(CONFIG_CONTAINERS) += container.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
 obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o
+obj-$(CONFIG_CONTAINER_NS) += ns_container.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
kernel/fork.c +4 -0
Index: container-2.6.20-new/kernel/fork.c
===================================================================
@@ -1668,6 +1668,9 @@ asmlinkage long sys_unshare(unsigned lon
 			err = -ENOMEM;
 			goto bad_unshare_cleanup_ipc;
 		}
+		err = ns_container_clone(current);
+		if (err)
+			goto bad_unshare_cleanup_dupns;
 	}
 
 	if (new_fs || new_ns || new_mm || new_fd || new_ulist ||
@@ -1722,6 +1725,7 @@ asmlinkage long sys_unshare(unsigned lon
 		task_unlock(current);
 	}
 
+ bad_unshare_cleanup_dupns:
 	if (new_nsproxy)
 		put_nsproxy(new_nsproxy);
 
kernel/ns_container.c new file +99 -0

99 lines not on this page · raw message

kernel/nsproxy.c +6 -0
Index: container-2.6.20-new/kernel/nsproxy.c
===================================================================
@@ -116,10 +116,16 @@ int copy_namespaces(int flags, struct ta
 	if (err)
 		goto out_pid;
 
+	err = ns_container_clone(tsk);
+	if (err)
+		goto out_container;
 out:
 	put_nsproxy(old_ns);
 	return err;
 
+ out_container:
+	if (new_ns->pid_ns)
+		put_pid_ns(new_ns->pid_ns);
 out_pid:
 	if (new_ns->ipc_ns)
 		put_ipc_ns(new_ns->ipc_ns);
include/linux/container_subsys.h +6 -0
Index: container-2.6.20-new/include/linux/container_subsys.h
===================================================================
@@ -29,4 +29,10 @@ SUBSYS(bc)
 
 /* */
 
+#ifdef CONFIG_CONTAINER_NS
+SUBSYS(ns)
+#endif
+
+/* */
+
 /* */
init/Kconfig +9 -0
Index: container-2.6.20-new/init/Kconfig
===================================================================
@@ -285,6 +285,15 @@ config CONTAINER_CPUACCT
 	  Provides a simple Resource Controller for monitoring the
 	  total CPU consumed by the tasks in a container
 
+config CONTAINER_NS
+        bool "Namespace container subsystem"
+        select CONTAINERS
+        help
+          Provides a simple namespace container subsystem to
+          provide hierarchical naming of sets of namespaces,
+          for instance virtual servers and checkpoint/restart
+          jobs.
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help

--

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw

Re: [ckrm-tech] [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

Quoted 5 lines · On 4/6/07, [email protected] <[email protected]> wrote:

This patch removes all cpuset-specific knowlege from the container system, replacing it with a generic API that can be used by multiple subsystems. Cpusets is adapted to be a container subsystem.

(quoted diff) +1 -0
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+       /* Set of subsystem states, one for each subsystem. NULL for
+        * subsystems that aren't part of this hierarchy. These
+        * pointers reduce the number of dereferences required to get
+        * from a task to its state for a given container, but result
+        * in increased space usage if tasks are in wildly different
+        * groupings across different hierarchies. This array is
+        * mostly immutable after creation - a newly registered
+        * subsystem can result in a pointer in this array
+        * transitioning from NULL to non-NULL */
+       struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT];

Oops, this comment is rather out of date and should have been trimmed for this patch. It should be disregarded apart from the first sentence.

Paul

#14 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive
Quoted 2 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:

+static int attach_task(struct container *cont, struct task_struct *tsk)
{

[snip]

Review

Quoted 1 line
(quoted diff) +1 -0
+	task_lock(tsk);

You need to check here if task state is PF_EXITING and fail with -ESRCH if so? Otherwise we risk breaking refcount on init_container_group.

Quoted 2 lines
(quoted diff) +2 -0
+	rcu_assign_pointer(tsk->containers, newcg);
+	task_unlock(tsk);
Signature · 2 lines
Regards,
vatsa
#15 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

Quoted 4 lines · Srivatsa Vaddagiri
Quoted 2 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:

+static int attach_task(struct container *cont, struct task_struct *tsk)
{

Quoted 8 lines

[snip]

Quoted 1 line

+ task_lock(tsk);

You need to check here if task state is PF_EXITING and fail with -ESRCH if so? Otherwise we risk breaking refcount on init_container_group.

Yes, I think you're right; I've now changed it to this in my tree:

       task_lock(tsk);
       if (tsk->flags & PF_EXITING) {
               task_unlock(tsk);
               put_container_group(newcg);
               return -ESRCH;
       }
       rcu_assign_pointer(tsk->containers, newcg);
       task_unlock(tsk);

Thanks,

Paul

#16 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive

Review

Quoted 10 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:
(quoted diff) +2 -3
-static int container_create_dir(struct container *cont, const char *name, int mode)
+static int container_create_dir(struct container *cont, struct dentry *dentry,
+				int mode)
 {
-	struct dentry *dentry = NULL;
 	struct dentry *parent;
 	int error = 0;
 
 	parent = cont->parent->dentry;
-	dentry = container_get_dentry(parent, name);

If you removed this line, then:

Review

Quoted 11 lines
(quoted diff) +2 -2
 	if (IS_ERR(dentry))
 		return PTR_ERR(dentry);
-	error = container_create_file(dentry, S_IFDIR | mode);
+	error = container_create_file(dentry, S_IFDIR | mode, cont->root->sb);
 	if (!error) {
 		dentry->d_fsdata = cont;
-		parent->d_inode->i_nlink++;
+		inc_nlink(parent->d_inode);
 		cont->dentry = dentry;
 	}
 	dput(dentry);

We should remove this dput as well, otherwise we will be short of required reference count on dentry by 1?

Basically, when user does:

        # cd /dev/container
        # mkdir a

and mkdir returns, what should be the (min) ref count of a's dentry? From my understanding it should be total number of files under a + 1. The dput above will cause this reqd count to be short by 1 ..

Signature · 2 lines
Regards,
vatsa
#17 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive

Review

Quoted 13 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:
(quoted diff) +13 -0
+struct container_subsys {
+	int (*create)(struct container_subsys *ss,
+		      struct container *cont);
+	void (*destroy)(struct container_subsys *ss, struct container *cont);
+	int (*can_attach)(struct container_subsys *ss,
+			  struct container *cont, struct task_struct *tsk);
+	void (*attach)(struct container_subsys *ss, struct container *cont,
+			struct container *old_cont, struct task_struct *tsk);
+	void (*fork)(struct container_subsys *ss, struct task_struct *task);
+	void (*exit)(struct container_subsys *ss, struct task_struct *task);
+	int (*populate)(struct container_subsys *ss,
+			struct container *cont);
+	void (*bind)(struct container_subsys *ss, struct container *root);

Is the first argument into all the callbacks, struct container_subsys *ss, necessary?

Review

Quoted 10 lines
(quoted diff) +10 -0
+struct container_subsys cpuset_subsys = {
+	.name = "cpuset",
+	.create = cpuset_create,
+	.destroy  = cpuset_destroy,
+	.can_attach = cpuset_can_attach,
+	.attach = cpuset_attach,
+	.populate = cpuset_populate,
+	.subsys_id = cpuset_subsys_id,
+	.early_init = 1,
+};

We seem to be missing a register call here as well ..

Signature · 2 lines
Regards,
vatsa
#18 lore raw

Re: [ckrm-tech] [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

Quoted 3 lines · Srivatsa Vaddagiri

Is the first argument into all the callbacks, struct container_subsys *ss, necessary?

I added it to support library-like abstractions - where one subsystem can have its container callbacks and file accesses all handled by a library which converts to a higher-level abstraction. See the res_group example patch to see how it can be used.

Quoted 13 lines

Review

Quoted 11 lines
(quoted diff) +10 -0
+struct container_subsys cpuset_subsys = {
+     .name = "cpuset",
+     .create = cpuset_create,
+     .destroy  = cpuset_destroy,
+     .can_attach = cpuset_can_attach,
+     .attach = cpuset_attach,
+     .populate = cpuset_populate,
+     .subsys_id = cpuset_subsys_id,
+     .early_init = 1,
+};

We seem to be missing a register call here as well ..

Registration is done at compile time in the new patch set, thanks to the static declaration of subsystems and subsystem ids.

#19 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive

Review

Quoted 2 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:
(quoted diff) +1 -1
-int cpuset_create(struct container *cont)
+int cpuset_create(struct container_subsys *ss, struct container *cont)

Minor nit: The static declaration for cpuset_create (and friends) can be re-introduced, since they won't be accessed directly from other files?

Signature · 2 lines
Regards,
vatsa
#20 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive

[ Sorry abt piece meal reviews, I am sending comments as and when I spot something ]

Quoted 3 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:
(quoted diff) +1 -1
-void container_exit(struct task_struct *tsk)
+void container_exit(struct task_struct *tsk, int run_callbacks)
 {

[snip]

Quoted 2 lines
(quoted diff) +1 -0
+	/* Reassign the task to the init_container_group. */
 	task_lock(tsk);

[snip]

Review

Quoted 1 line
(quoted diff) +1 -0
+	if (tsk->containers != &init_container_group) {

Is this check needed? If we have the check, then:

Review

Quoted 3 lines
(quoted diff) +3 -0
+	task_unlock(tsk);
+	if (cg)
+		put_container_group(cg);

init_container_group refcount is leaky (fork increments it, but exit doesnt). Not a big prob perhaps, but ..would be nice to avoid?

Signature · 2 lines
Regards,
vatsa
#21 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

Quoted 2 lines · Srivatsa Vaddagiri

[ Sorry abt piece meal reviews, I am sending comments as and when I spot something ]

That's no problem.

Quoted 15 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:
Quoted 3 lines
(quoted diff) +1 -1
-void container_exit(struct task_struct *tsk)
+void container_exit(struct task_struct *tsk, int run_callbacks)
 {

[snip]

Quoted 2 lines

+ /* Reassign the task to the init_container_group. */

      task_lock(tsk);

[snip]

Quoted 1 line

+ if (tsk->containers != &init_container_group) {

Quoted 9 lines

Is this check needed? If we have the check, then:

Review

Quoted 3 lines
(quoted diff) +3 -0
+     task_unlock(tsk);
+     if (cg)
+             put_container_group(cg);

init_container_group refcount is leaky (fork increments it, but exit doesnt). Not a big prob perhaps, but ..would be nice to avoid?

Yes, I think you're right - over a long uptime that would just cause problems ...

Paul

#22 lore raw

Re: [PATCH 3/7] Containers (V8): Add generic multi-subsystem API to containers

In reply to a message not in the archive

Coming from a "simplify things" pov:

Quoted 16 lines · On Fri, Apr 06, 2007 at 04:32:24PM -0700, [email protected] wrote:

struct container {

(quoted diff) +0 -5
 	unsigned long flags;		/* "unsigned long" so bitops work */
 
 	/*
-	 * Count is atomic so can incr (fork) or decr (exit) without a lock.
-	 */
-	atomic_t count;			/* count tasks using this container */
-
-	/*
 	 * We link our 'sibling' struct into our parent's 'children'.
 	 * Our children link their 'sibling' into our 'children'.
 	 */
@@ -43,11 +106,13 @@ struct container {
 	struct list_head children;	/* my children */
 
 	struct container *parent;	/* my parent */

Review struct container { · line 114

Quoted 10 lines
(quoted diff) +5 -4
-	struct dentry *dentry;		/* container fs entry */
+	struct dentry *dentry;	  	/* container fs entry */
 
-#ifdef CONFIG_CPUSETS
-	struct cpuset *cpuset;
-#endif
+	/* Private pointers for each registered subsystem */
+	struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT];
+
+	struct containerfs_root *root;

Could this root pointer derived from dentry pointer (cont->dentry->d_sb->s_fs_info)?

Review struct container { · line 115

Quoted 1 line
(quoted diff) +1 -0
+	struct container *top_container;

and this as well?

	cont->dentry->d_sb->s_fs_info->top_container

Review struct container { · line 116

Quoted 1 line
(quoted diff) +0 -0
 };

So we have the foward subsys pointer array being stored in both 'struct container' and 'struct container_group' and reverse container pointer stored in struct container_subsys_state. Can we reduce this pointer maze by:

	struct container {
		/* All shared stuff like flags, parent/child pointers etc */
		..

		struct container_group *my_group;
		
	}

The forward mapping from 'struct container' to subsys objects is made via 'my_group'. This also lets 'struct container' be a placeholder strictly for shared state.

On further thoughts, perhaps even my_group can be avoided by having:

	dentry->d_fsdata point to my_group 

and cont->dentry->d_fsdata will point to my_group which we wanted to store above.

I don't see distinct adv of doing this, but I suspect it will simplify the structure relationship (and code) a bit.

Signature · 2 lines
Regards,
vatsa
#23 lore raw
In reply to [email protected]

Hi Paul,

In [patch 3/7] Containers (V8): Add generic multi-subsystem API to containers, you have forcefully enabled interrupt in container_init_subsys() with spin_unlock_irq() which breaks on PPC64.

Quoted 23 lines
(quoted diff) +4 -0
+static void container_init_subsys(struct container_subsys *ss) {
+	int retval;
+	struct list_head *l;
+	printk(KERN_ERR "Initializing container subsys %s\n",

ss->name);

(quoted diff) +17 -0
+
+	/* Create the top container state for this subsystem */
+	ss->root = &rootnode;
+	retval = ss->create(ss, dummytop);
+	BUG_ON(retval);
+	init_container_css(ss, dummytop);
+
+	/* Update all container groups to contain a subsys
+	 * pointer to this state - since the subsystem is
+	 * newly registered, all tasks and hence all container
+	 * groups are in the subsystem's top container. */
+	spin_lock_irq(&container_group_lock);
+	l = &init_container_group.list;
+	do {
+		struct container_group *cg =
+			list_entry(l, struct container_group, list);
+		cg->subsys[ss->subsys_id] =

dummytop->subsys[ss->subsys_id];

Review

Quoted 3 lines
(quoted diff) +3 -0
+		l = l->next;
+	} while (l != &init_container_group.list);
+	spin_unlock_irq(&container_group_lock);

Interrupt gets enabled here and on PPC64, the kernel takes a pending decrementer and crashes because it is too early to handle them.

Use of irqsave and restore routines would fix the problem. I have included the fix along with minor Kconfig correction.

Also your 3/7 patch did not showup on LKML.

--Vaidy

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+	need_forkexit_callback |= ss->fork || ss->exit;
kernel/container.c +3 -2
Index: linux-2.6.20/kernel/container.c
===================================================================
@@ -1638,6 +1638,7 @@ int container_is_descendant(const struct
 
 static void container_init_subsys(struct container_subsys *ss) {
 	int retval;
+	unsigned long flags;
 	struct list_head *l;
 	printk(KERN_ERR "Initializing container subsys %s\n", ss->name);
 
@@ -1651,7 +1652,7 @@ static void container_init_subsys(struct
 	 * pointer to this state - since the subsystem is
 	 * newly registered, all tasks and hence all container
 	 * groups are in the subsystem's top container. */
-	spin_lock_irq(&container_group_lock);
+	spin_lock_irqsave(&container_group_lock, flags);
 	l = &init_container_group.list;
 	do {
 		struct container_group *cg =
@@ -1659,7 +1660,7 @@ static void container_init_subsys(struct
 		cg->subsys[ss->subsys_id] = dummytop->subsys[ss->subsys_id];
 		l = l->next;
 	} while (l != &init_container_group.list);
-	spin_unlock_irq(&container_group_lock);
+	spin_unlock_irqrestore(&container_group_lock, flags);
 
 	need_forkexit_callback |= ss->fork || ss->exit;
 
init/Kconfig +7 -2
Index: linux-2.6.20/init/Kconfig
===================================================================
@@ -239,8 +239,13 @@ config IKCONFIG_PROC
 	  through /proc/config.gz.
 
 config CONTAINERS
-	bool
-	
+	bool "Container support"
+	help
+	  This option will let you create and manage process containers,
+	  which can be used to aggregate multiple processes, e.g. for
+	  the purposes of resource tracking.
+
+	  Say N if unsure
 
 config CPUSETS
 	bool "Cpuset support"
#24 lore raw
Quoted 37 lines · Vaidyanathan Srinivasan

Hi Paul,

In [patch 3/7] Containers (V8): Add generic multi-subsystem API to containers, you have forcefully enabled interrupt in container_init_subsys() with spin_unlock_irq() which breaks on PPC64.

Quoted 23 lines
(quoted diff) +4 -0
+static void container_init_subsys(struct container_subsys *ss) {
+     int retval;
+     struct list_head *l;
+     printk(KERN_ERR "Initializing container subsys %s\n",

ss->name);

(quoted diff) +17 -0
+
+     /* Create the top container state for this subsystem */
+     ss->root = &rootnode;
+     retval = ss->create(ss, dummytop);
+     BUG_ON(retval);
+     init_container_css(ss, dummytop);
+
+     /* Update all container groups to contain a subsys
+      * pointer to this state - since the subsystem is
+      * newly registered, all tasks and hence all container
+      * groups are in the subsystem's top container. */
+     spin_lock_irq(&container_group_lock);
+     l = &init_container_group.list;
+     do {
+             struct container_group *cg =
+                     list_entry(l, struct container_group, list);
+             cg->subsys[ss->subsys_id] =

dummytop->subsys[ss->subsys_id];

Review

Quoted 3 lines
(quoted diff) +3 -0
+             l = l->next;
+     } while (l != &init_container_group.list);
+     spin_unlock_irq(&container_group_lock);

Interrupt gets enabled here and on PPC64, the kernel takes a pending decrementer and crashes because it is too early to handle them.

Quoted 1 line

Use of irqsave and restore routines would fix the problem.

OK, thanks. I'll add that change.

Paul