Skip to contentSkip to conversation

Generic container system

RFC PATCH In review Paul Menage 4 participants 14 messages Oct 2, 2006 – Oct 4, 2006

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover Generic container system

This is essentially the same as the patch set that I posted last week, with the following fixes/changes:

- CONFIG_CONTAINERS is no longer a user-selectable option - subsystems such as cpusets that require it should select it in Kconfig.

- Each container subsystem type now has a name, and a <name>_enabled file in the top container directory. This file contains 0 or 1 to indicate whether the container subsystem is enabled, and can only be modified when there are no subcontainers; disabled container subsystems don't get new instances created when a subcontainer is created; the subsystem-specific state is simply inherited from the parent container.

- include a config option to default to enabled, for backwards compatibility

- Documentation tweaks

- builds properly without CONFIG_CONTAINER_CPUACCT configured on

- should build properly with newer gccs. (I've not actually had a chance to try building it with anything newer than gcc 3.2.2, but I've fixed all the potential warnings/errors that PaulJ pointed out when compiling with some unspecified newer gcc).

I've also looked at converting ResGroups to be a client of the container system. This isn't yet complete; my thoughts so far include:

- each resource controller can be implemented as an independent container subsystem; rather than a single "shares" and "stats" file in each directory there will be e.g. "numtasks_shares", "cpurc_shares", etc

- the ResGroups core will basically end up as a library that provides the common parsing/displaying for the shares and stats file for each controller, and the logic for propagating resources up and down the parent/child tree.

- for some of the resource controllers we will probably require a few extra callbacks from the container system, e.g. at fork/exit time. I might make these a config option that the controller must "select" in Kconfig, to avoid extra locking/overhead for subsystems such as cpusets that don't require such callbacks.

-------------------------------------

There have recently been various proposals floating around for resource management/accounting subsystems in the kernel, including Res Groups, User BeanCounters and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, and all implement this grouping in different ways.

Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards to synchronization rules).

This patchset extracts the process grouping code from cpusets into a generic container system, and makes the cpusets code a client of the container system.

It also provides a very simple additional container subsystem to do per-container CPU usage accounting; this is primarily to demonstrate use of the container subsystem API, but is useful in its own right.

The change is implemented in four stages:

1) extract the process grouping code from cpusets into a standalone system

2) remove the process grouping code from cpusets and hook into the container system

3) convert the container system to present a generic API, and make cpusets a client of that API

4) add a simple CPU accounting container subsystem as an example

The intention is that the various resource management efforts can also become container clients, with the result that:

- the userspace APIs are (somewhat) normalised

- it's easier to test out e.g. the ResGroups CPU controller in conjunction with the UBC memory controller

- the additional kernel footprint of any of the competing resource management systems is substantially reduced, since it doesn't need to provide process grouping/containment, hence improving their chances of getting into the kernel

Possible TODOs include:

- define a convention for populating the per-container directories so that different subsystems don't clash with one another

- provide higher-level primitives (e.g. an easy interface to seq_file) for files registered by subsystems.

- support subsystem deregistering

---

#1 lore raw

RFC PATCH 1/4 Generic container system abstracted from cpusets code

This patch creates a generic process container system based on (and parallel top) the cpusets code. At a coarse level it was created by copying kernel/cpuset.c, doing s/cpuset/container/g, and stripping out any code that was cpuset-specific rather than applicable to any process container subsystem.

---
 Documentation/containers.txt |  229 +++++++
 fs/proc/base.c               |   19 
 include/linux/container.h    |   96 +++
 include/linux/sched.h        |    5 
 include/linux/utsrelease.h   |    1 
 init/Kconfig                 |    9 
 init/main.c                  |    3 
 kernel/Makefile              |    1 
 kernel/container.c           | 1344 +++++++++++++++++++++++++++++++++++++++++++
 kernel/exit.c                |    2 
 kernel/fork.c                |    3 
 11 files changed, 1711 insertions(+), 1 deletion(-)
fs/proc/base.c +19 -0
Index: linux-2.6.18/fs/proc/base.c
===================================================================
@@ -69,6 +69,7 @@
 #include <linux/ptrace.h>
 #include <linux/seccomp.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/audit.h>
 #include <linux/poll.h>
 #include "internal.h"
@@ -124,6 +125,9 @@ enum pid_directory_inos {
 #ifdef CONFIG_CPUSETS
 	PROC_TGID_CPUSET,
 #endif
+#ifdef CONFIG_CONTAINERS
+	PROC_TGID_CONTAINER,
+#endif
 #ifdef CONFIG_SECURITY
 	PROC_TGID_ATTR,
 	PROC_TGID_ATTR_CURRENT,
@@ -167,6 +171,9 @@ enum pid_directory_inos {
 #ifdef CONFIG_CPUSETS
 	PROC_TID_CPUSET,
 #endif
+#ifdef CONFIG_CONTAINERS
+	PROC_TID_CONTAINER,
+#endif
 #ifdef CONFIG_SECURITY
 	PROC_TID_ATTR,
 	PROC_TID_ATTR_CURRENT,
@@ -235,6 +242,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_CPUSETS
 	E(PROC_TGID_CPUSET,    "cpuset",  S_IFREG|S_IRUGO),
 #endif
+#ifdef CONFIG_CONTAINERS
+	E(PROC_TGID_CONTAINER,    "container",  S_IFREG|S_IRUGO),
+#endif
 	E(PROC_TGID_OOM_SCORE, "oom_score",S_IFREG|S_IRUGO),
 	E(PROC_TGID_OOM_ADJUST,"oom_adj", S_IFREG|S_IRUGO|S_IWUSR),
 #ifdef CONFIG_AUDITSYSCALL
@@ -277,6 +287,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_CPUSETS
 	E(PROC_TID_CPUSET,     "cpuset",  S_IFREG|S_IRUGO),
 #endif
+#ifdef CONFIG_CONTAINERS
+	E(PROC_TID_CONTAINER,    "container",  S_IFREG|S_IRUGO),
+#endif
 	E(PROC_TID_OOM_SCORE,  "oom_score",S_IFREG|S_IRUGO),
 	E(PROC_TID_OOM_ADJUST, "oom_adj", S_IFREG|S_IRUGO|S_IWUSR),
 #ifdef CONFIG_AUDITSYSCALL
@@ -1819,6 +1832,12 @@ static struct dentry *proc_pident_lookup
 			inode->i_fop = &proc_cpuset_operations;
 			break;
 #endif
+#ifdef CONFIG_CONTAINERS
+		case PROC_TID_CONTAINER:
+		case PROC_TGID_CONTAINER:
+			inode->i_fop = &proc_container_operations;
+			break;
+#endif
 		case PROC_TID_OOM_SCORE:
 		case PROC_TGID_OOM_SCORE:
 			inode->i_fop = &proc_info_file_operations;
include/linux/container.h new file +96 -0

96 lines not on this page · raw message

include/linux/sched.h +4 -1
Index: linux-2.6.18/include/linux/sched.h
===================================================================
@@ -710,8 +710,8 @@ extern unsigned int max_cache_size;
 
 struct io_context;			/* See blkdev.h */
 void exit_io_context(void);
+struct container;
 struct cpuset;
-
 #define NGROUPS_SMALL		32
 #define NGROUPS_PER_BLOCK	((int)(PAGE_SIZE / sizeof(gid_t)))
 struct group_info {
@@ -979,6 +979,9 @@ struct task_struct {
 	int cpuset_mems_generation;
 	int cpuset_mem_spread_rotor;
 #endif
+#ifdef CONFIG_CONTAINERS
+	struct container *container;
+#endif
 	struct robust_list_head __user *robust_list;
 #ifdef CONFIG_COMPAT
 	struct compat_robust_list_head __user *compat_robust_list;
include/linux/utsrelease.h new file +1 -0
Index: linux-2.6.18/include/linux/utsrelease.h
===================================================================
@@ -0,0 +1 @@
+#define UTS_RELEASE "2.6.18"
init/Kconfig +9 -0
Index: linux-2.6.18/init/Kconfig
===================================================================
@@ -220,6 +220,15 @@ config IKCONFIG_PROC
 	  This option enables access to the kernel configuration file
 	  through /proc/config.gz.
 
+config CONTAINERS
+	bool "Container support"
+	help
+	  This option will let you create and manage process containers,
+	  which can be used to aggregate multiple processes, e.g. for
+	  the purposes of resource tracking.
+
+	  Say N if unsure
+
 config CPUSETS
 	bool "Cpuset support"
 	depends on SMP
init/main.c +3 -0
Index: linux-2.6.18/init/main.c
===================================================================
@@ -40,6 +40,7 @@
 #include <linux/writeback.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/efi.h>
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
@@ -545,6 +546,7 @@ asmlinkage void __init start_kernel(void
 	}
 #endif
 	vfs_caches_init_early();
+	container_init_early();
 	cpuset_init_early();
 	mem_init();
 	kmem_cache_init();
@@ -575,6 +577,7 @@ asmlinkage void __init start_kernel(void
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
 #endif
+	container_init();
 	cpuset_init();
 	taskstats_init_early();
 	delayacct_init();
kernel/container.c new file +1344 -0

1,344 lines not on this page · raw message

kernel/exit.c +2 -0
Index: linux-2.6.18/kernel/exit.c
===================================================================
@@ -28,6 +28,7 @@
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/syscalls.h>
 #include <linux/signal.h>
 #include <linux/posix-timers.h>
@@ -919,6 +920,7 @@ fastcall NORET_TYPE void do_exit(long co
 	exit_namespace(tsk);
 	exit_thread();
 	cpuset_exit(tsk);
+	container_exit(tsk);
 	exit_keys(tsk);
 
 	if (group_dead && tsk->signal->leader)
kernel/fork.c +3 -0
Index: linux-2.6.18/kernel/fork.c
===================================================================
@@ -30,6 +30,7 @@
 #include <linux/capability.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/container.h>
 #include <linux/security.h>
 #include <linux/swap.h>
 #include <linux/syscalls.h>
@@ -1044,6 +1045,7 @@ static struct task_struct *copy_process(
 	p->io_context = NULL;
 	p->io_wait = NULL;
 	p->audit_context = NULL;
+	container_fork(p);
 	cpuset_fork(p);
 #ifdef CONFIG_NUMA
  	p->mempolicy = mpol_copy(p->mempolicy);
@@ -1278,6 +1280,7 @@ bad_fork_cleanup_policy:
 bad_fork_cleanup_cpuset:
 #endif
 	cpuset_exit(p);
+	container_exit(p);
 bad_fork_cleanup_delays_binfmt:
 	delayacct_tsk_free(p);
 	if (p->binfmt)
kernel/Makefile +1 -0
Index: linux-2.6.18/kernel/Makefile
===================================================================
@@ -36,6 +36,7 @@ obj-$(CONFIG_PM) += power/
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_COMPAT) += compat.o
+obj-$(CONFIG_CONTAINERS) += container.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
Documentation/containers.txt new file +229 -0

229 lines not on this page · raw message

--

#2 lore raw

RFC PATCH 2/4 Cpusets hooked into containers

In reply to Paul Menage

This patch removes the process grouping code from the cpusets code, instead hooking it into the generic container system. This temporarily adds cpuset-specific code in kernel/container.c, which is removed by the next patch in the series.

---
 Documentation/cpusets.txt |   80 +-
 fs/proc/base.c            |   18 
 include/linux/container.h |    4 
 include/linux/cpuset.h    |   25 
 include/linux/sched.h     |    4 
 init/Kconfig              |   18 
 kernel/container.c        |   51 +
 kernel/cpuset.c           | 1307 ++++------------------------------------------
 kernel/exit.c             |    2 
 kernel/fork.c             |    7 
 mm/oom_kill.c             |    6 
 11 files changed, 239 insertions(+), 1283 deletions(-)
include/linux/container.h +4 -0
Index: linux-2.6.18/include/linux/container.h
===================================================================
@@ -47,6 +47,10 @@ struct container {
 
 	struct container *parent;	/* my parent */
 	struct dentry *dentry;		/* container fs entry */
+
+#ifdef CONFIG_CPUSETS
+	struct cpuset *cpuset;
+#endif
 };
 
 /* struct cftype:
include/linux/cpuset.h +13 -12
Index: linux-2.6.18/include/linux/cpuset.h
===================================================================
@@ -11,16 +11,15 @@
 #include <linux/sched.h>
 #include <linux/cpumask.h>
 #include <linux/nodemask.h>
+#include <linux/container.h>
 
 #ifdef CONFIG_CPUSETS
 
-extern int number_of_cpusets;	/* How many cpusets are defined in system? */
+extern int number_of_cpusets;  /* How many cpusets are defined in system? */
 
 extern int cpuset_init_early(void);
 extern int cpuset_init(void);
 extern void cpuset_init_smp(void);
-extern void cpuset_fork(struct task_struct *p);
-extern void cpuset_exit(struct task_struct *p);
 extern cpumask_t cpuset_cpus_allowed(struct task_struct *p);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 void cpuset_init_current_mems_allowed(void);
@@ -47,10 +46,6 @@ extern void __cpuset_memory_pressure_bum
 
 extern struct file_operations proc_cpuset_operations;
 extern char *cpuset_task_status_allowed(struct task_struct *task, char *buffer);
-
-extern void cpuset_lock(void);
-extern void cpuset_unlock(void);
-
 extern int cpuset_mem_spread_node(void);
 
 static inline int cpuset_do_page_mem_spread(void)
@@ -63,13 +58,22 @@ static inline int cpuset_do_slab_mem_spr
 	return current->flags & PF_SPREAD_SLAB;
 }
 
+extern int cpuset_can_attach_task(struct container *cont,
+				  struct task_struct *tsk);
+extern void cpuset_attach_task(struct container *cont,
+				struct task_struct *tsk);
+extern void cpuset_post_attach_task(struct container *cont,
+				    struct container *oldcont,
+				    struct task_struct *tsk);
+extern int cpuset_populate_dir(struct container *cont);
+extern int cpuset_create(struct container *cont);
+extern void cpuset_destroy(struct container *cont);
+
 #else /* !CONFIG_CPUSETS */
 
 static inline int cpuset_init_early(void) { return 0; }
 static inline int cpuset_init(void) { return 0; }
 static inline void cpuset_init_smp(void) {}
-static inline void cpuset_fork(struct task_struct *p) {}
-static inline void cpuset_exit(struct task_struct *p) {}
 
 static inline cpumask_t cpuset_cpus_allowed(struct task_struct *p)
 {
@@ -108,9 +112,6 @@ static inline char *cpuset_task_status_a
 	return buffer;
 }
 
-static inline void cpuset_lock(void) {}
-static inline void cpuset_unlock(void) {}
-
 static inline int cpuset_mem_spread_node(void)
 {
 	return 0;
kernel/exit.c +0 -2
Index: linux-2.6.18/kernel/exit.c
===================================================================
@@ -27,7 +27,6 @@
 #include <linux/mempolicy.h>
 #include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
-#include <linux/cpuset.h>
 #include <linux/container.h>
 #include <linux/syscalls.h>
 #include <linux/signal.h>
@@ -919,7 +918,6 @@ fastcall NORET_TYPE void do_exit(long co
 	__exit_fs(tsk);
 	exit_namespace(tsk);
 	exit_thread();
-	cpuset_exit(tsk);
 	container_exit(tsk);
 	exit_keys(tsk);
 
kernel/fork.c +2 -5
Index: linux-2.6.18/kernel/fork.c
===================================================================
@@ -29,7 +29,6 @@
 #include <linux/fs.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
-#include <linux/cpuset.h>
 #include <linux/container.h>
 #include <linux/security.h>
 #include <linux/swap.h>
@@ -1046,13 +1045,12 @@ static struct task_struct *copy_process(
 	p->io_wait = NULL;
 	p->audit_context = NULL;
 	container_fork(p);
-	cpuset_fork(p);
 #ifdef CONFIG_NUMA
  	p->mempolicy = mpol_copy(p->mempolicy);
  	if (IS_ERR(p->mempolicy)) {
  		retval = PTR_ERR(p->mempolicy);
  		p->mempolicy = NULL;
- 		goto bad_fork_cleanup_cpuset;
+ 		goto bad_fork_cleanup_container;
  	}
 	mpol_fix_fork_child_flag(p);
 #endif
@@ -1277,9 +1275,8 @@ bad_fork_cleanup_security:
 bad_fork_cleanup_policy:
 #ifdef CONFIG_NUMA
 	mpol_free(p->mempolicy);
-bad_fork_cleanup_cpuset:
+bad_fork_cleanup_container:
 #endif
-	cpuset_exit(p);
 	container_exit(p);
 bad_fork_cleanup_delays_binfmt:
 	delayacct_tsk_free(p);
kernel/container.c +48 -3

111 lines not on this page · raw message

kernel/cpuset.c +126 -1181

1,747 lines not on this page · raw message

init/Kconfig +11 -7
Index: linux-2.6.18/init/Kconfig
===================================================================
@@ -221,17 +221,12 @@ config IKCONFIG_PROC
 	  through /proc/config.gz.
 
 config CONTAINERS
-	bool "Container support"
-	help
-	  This option will let you create and manage process containers,
-	  which can be used to aggregate multiple processes, e.g. for
-	  the purposes of resource tracking.
-
-	  Say N if unsure
+	bool
 
 config CPUSETS
 	bool "Cpuset support"
 	depends on SMP
+	select CONTAINERS
 	help
 	  This option will let you create and manage CPUSETs which
 	  allow dynamically partitioning a system into sets of CPUs and
@@ -240,6 +235,15 @@ config CPUSETS
 
 	  Say N if unsure.
 
+config CPUSETS_DEFAULT_ENABLED
+	bool "Default the cpuset container type to be enabled"
+	depends on CPUSETS
+	default y
+	help
+	  This option makes the cpuset container type enabled by
+	  default at boot, for compatibility with legacy cpuset
+	  semantics
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help
mm/oom_kill.c +3 -3
Index: linux-2.6.18/mm/oom_kill.c
===================================================================
@@ -326,7 +326,7 @@ void out_of_memory(struct zonelist *zone
 		show_mem();
 	}
 
-	cpuset_lock();
+	container_lock();
 	read_lock(&tasklist_lock);
 
 	/*
@@ -360,7 +360,7 @@ retry:
 		/* Found nothing?!?! Either we hang forever, or we panic. */
 		if (!p) {
 			read_unlock(&tasklist_lock);
-			cpuset_unlock();
+			container_unlock();
 			panic("Out of memory and no killable processes...\n");
 		}
 
@@ -372,7 +372,7 @@ retry:
 
 out:
 	read_unlock(&tasklist_lock);
-	cpuset_unlock();
+	container_unlock();
 
 	/*
 	 * Give "p" a good chance of killing itself before we
fs/proc/base.c +0 -18
Index: linux-2.6.18/fs/proc/base.c
===================================================================
@@ -122,9 +122,6 @@ enum pid_directory_inos {
 #ifdef CONFIG_SCHEDSTATS
 	PROC_TGID_SCHEDSTAT,
 #endif
-#ifdef CONFIG_CPUSETS
-	PROC_TGID_CPUSET,
-#endif
 #ifdef CONFIG_CONTAINERS
 	PROC_TGID_CONTAINER,
 #endif
@@ -168,9 +165,6 @@ enum pid_directory_inos {
 #ifdef CONFIG_SCHEDSTATS
 	PROC_TID_SCHEDSTAT,
 #endif
-#ifdef CONFIG_CPUSETS
-	PROC_TID_CPUSET,
-#endif
 #ifdef CONFIG_CONTAINERS
 	PROC_TID_CONTAINER,
 #endif
@@ -239,9 +233,6 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_SCHEDSTATS
 	E(PROC_TGID_SCHEDSTAT, "schedstat", S_IFREG|S_IRUGO),
 #endif
-#ifdef CONFIG_CPUSETS
-	E(PROC_TGID_CPUSET,    "cpuset",  S_IFREG|S_IRUGO),
-#endif
 #ifdef CONFIG_CONTAINERS
 	E(PROC_TGID_CONTAINER,    "container",  S_IFREG|S_IRUGO),
 #endif
@@ -284,9 +275,6 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_SCHEDSTATS
 	E(PROC_TID_SCHEDSTAT, "schedstat",S_IFREG|S_IRUGO),
 #endif
-#ifdef CONFIG_CPUSETS
-	E(PROC_TID_CPUSET,     "cpuset",  S_IFREG|S_IRUGO),
-#endif
 #ifdef CONFIG_CONTAINERS
 	E(PROC_TID_CONTAINER,    "container",  S_IFREG|S_IRUGO),
 #endif
@@ -1826,12 +1814,6 @@ static struct dentry *proc_pident_lookup
 			ei->op.proc_read = proc_pid_schedstat;
 			break;
 #endif
-#ifdef CONFIG_CPUSETS
-		case PROC_TID_CPUSET:
-		case PROC_TGID_CPUSET:
-			inode->i_fop = &proc_cpuset_operations;
-			break;
-#endif
 #ifdef CONFIG_CONTAINERS
 		case PROC_TID_CONTAINER:
 		case PROC_TGID_CONTAINER:
include/linux/sched.h +1 -3
Index: linux-2.6.18/include/linux/sched.h
===================================================================
@@ -711,7 +711,6 @@ extern unsigned int max_cache_size;
 struct io_context;			/* See blkdev.h */
 void exit_io_context(void);
 struct container;
-struct cpuset;
 #define NGROUPS_SMALL		32
 #define NGROUPS_PER_BLOCK	((int)(PAGE_SIZE / sizeof(gid_t)))
 struct group_info {
@@ -974,7 +973,6 @@ struct task_struct {
 	short il_next;
 #endif
 #ifdef CONFIG_CPUSETS
-	struct cpuset *cpuset;
 	nodemask_t mems_allowed;
 	int cpuset_mems_generation;
 	int cpuset_mem_spread_rotor;
@@ -1352,7 +1350,7 @@ static inline int thread_group_empty(str
 /*
  * Protects ->fs, ->files, ->mm, ->group_info, ->comm, keyring
  * subscriptions and synchronises with wait4().  Also used in procfs.  Also
- * pins the final release of task.io_context.  Also protects ->cpuset.
+ * pins the final release of task.io_context.  Also protects ->container.
  *
  * Nests both inside and outside of read_lock(&tasklist_lock).
  * It must not be nested with write_lock_irq(&tasklist_lock),
Documentation/cpusets.txt +31 -50

166 lines not on this page · raw message

--

#3 lore raw

RFC PATCH 3/4 Add generic multi-subsystem API to containers

In reply to Paul Menage

This patch removes all cpuset-specific knowlege from the container system, replacing it with a generic API that can be used by multiple subsystems. Cpusets is adapted to be a container subsystem.

---
 Documentation/containers.txt |  127 +++++++++++++++++++++++++++++++
 include/linux/container.h    |   34 +++++++-
 include/linux/cpuset.h       |   11 --
 kernel/container.c           |  172 ++++++++++++++++++++++++++++++++++++-------
 kernel/cpuset.c              |  119 +++++++++++++++++++----------
 5 files changed, 378 insertions(+), 85 deletions(-)
include/linux/container.h +31 -3
Index: linux-2.6.18/include/linux/container.h
===================================================================
@@ -30,6 +30,8 @@ extern void container_unlock(void);
 extern void container_manage_lock(void);
 extern void container_manage_unlock(void);
 
+#define MAX_CONTAINER_SUBSYS 8
+
 struct container {
 	unsigned long flags;		/* "unsigned long" so bitops work */
 
@@ -48,9 +50,8 @@ struct container {
 	struct container *parent;	/* my parent */
 	struct dentry *dentry;		/* container fs entry */
 
-#ifdef CONFIG_CPUSETS
-	struct cpuset *cpuset;
-#endif
+	/* Private pointers for each registered subsystem */
+	void *subsys[MAX_CONTAINER_SUBSYS];
 };
 
 /* struct cftype:
@@ -84,6 +85,33 @@ int container_add_file(struct container 
 
 int container_is_removed(const struct container *cont);
 
+/* Container subsystem type. See Documentation/containers.txt for details */
+
+struct container_subsys {
+	int (*create)(struct container *cont);
+	void (*destroy)(struct container *cont);
+	int (*can_attach)(struct container *cont, struct task_struct *tsk);
+	void (*attach)(struct container *cont, struct task_struct *tsk);
+	void (*post_attach)(struct container *cont,
+			   struct container *old_cont,
+			   struct task_struct *tsk);
+	int (*populate)(struct container *cont);
+
+	int subsys_id;
+#define MAX_CONTAINER_TYPE_NAMELEN 32
+	const char *name;
+
+	/* file handler for enable/disable - managed by container.c */
+	struct cftype enable_cft;
+	char enable_cft_filename[MAX_CONTAINER_TYPE_NAMELEN + 10];
+
+	/* enabled/disabled flag - managed by container.c, but can be
+	 * set to a default value before registration */
+	int enabled;
+};
+
+int container_register_subsys(struct container_subsys *subsys);
+
 #else /* !CONFIG_CONTAINERS */
 
 static inline int container_init_early(void) { return 0; }
include/linux/cpuset.h +0 -11
Index: linux-2.6.18/include/linux/cpuset.h
===================================================================
@@ -58,17 +58,6 @@ static inline int cpuset_do_slab_mem_spr
 	return current->flags & PF_SPREAD_SLAB;
 }
 
-extern int cpuset_can_attach_task(struct container *cont,
-				  struct task_struct *tsk);
-extern void cpuset_attach_task(struct container *cont,
-				struct task_struct *tsk);
-extern void cpuset_post_attach_task(struct container *cont,
-				    struct container *oldcont,
-				    struct task_struct *tsk);
-extern int cpuset_populate_dir(struct container *cont);
-extern int cpuset_create(struct container *cont);
-extern void cpuset_destroy(struct container *cont);
-
 #else /* !CONFIG_CPUSETS */
 
 static inline int cpuset_init_early(void) { return 0; }
kernel/container.c +144 -28

273 lines not on this page · raw message

kernel/cpuset.c +79 -40

296 lines not on this page · raw message

Documentation/containers.txt +124 -3

140 lines not on this page · raw message

--

#4 lore raw

RFC PATCH 4/4 Simple CPU accounting container subsystem

In reply to Paul Menage

This demonstrates how to use the generic container subsystem for a simple resource tracker that counts the total CPU time used by all processes in a container, during the time that they're members of the container.

---
 include/linux/cpu_acct.h |   14 ++++++
 init/Kconfig             |    7 +++
 kernel/Makefile          |    1 
 kernel/cpu_acct.c        |  108 +++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched.c           |    6 ++
 5 files changed, 136 insertions(+)
include/linux/cpu_acct.h new file +14 -0
Index: linux-2.6.18/include/linux/cpu_acct.h
===================================================================
@@ -0,0 +1,14 @@
+
+#ifndef _LINUX_CPU_ACCT_H
+#define _LINUX_CPU_ACCT_H
+
+#include <linux/container.h>
+#include <asm/cputime.h>
+
+#ifdef CONFIG_CONTAINER_CPUACCT
+extern void cpuacct_charge(struct task_struct *, cputime_t cputime);
+#else
+static void inline cpuacct_charge(struct task_struct *p, cputime_t cputime) {}
+#endif
+
+#endif
init/Kconfig +7 -0
Index: linux-2.6.18/init/Kconfig
===================================================================
@@ -244,6 +244,13 @@ config CPUSETS_DEFAULT_ENABLED
 	  default at boot, for compatibility with legacy cpuset
 	  semantics
 
+config CONTAINER_CPUACCT
+	bool "Simple CPU accounting container subsystem"
+	select CONTAINERS
+	help
+	  Provides a simple Resource Controller for monitoring the
+	  total CPU consumed by the tasks in a container
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help
kernel/cpu_acct.c new file +108 -0

108 lines not on this page · raw message

kernel/Makefile +1 -0
Index: linux-2.6.18/kernel/Makefile
===================================================================
@@ -38,6 +38,7 @@ obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_COMPAT) += compat.o
 obj-$(CONFIG_CONTAINERS) += container.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
+obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
kernel/sched.c +6 -0
Index: linux-2.6.18/kernel/sched.c
===================================================================
@@ -52,6 +52,7 @@
 #include <linux/acct.h>
 #include <linux/kprobes.h>
 #include <linux/delayacct.h>
+#include <linux/cpu_acct.h>
 #include <asm/tlb.h>
 
 #include <asm/unistd.h>
@@ -2918,6 +2919,8 @@ void account_user_time(struct task_struc
 
 	p->utime = cputime_add(p->utime, cputime);
 
+	cpuacct_charge(p, cputime);
+
 	/* Add user time to cpustat. */
 	tmp = cputime_to_cputime64(cputime);
 	if (TASK_NICE(p) > 0)
@@ -2941,6 +2944,9 @@ void account_system_time(struct task_str
 
 	p->stime = cputime_add(p->stime, cputime);
 
+	if (p != rq->idle)
+		cpuacct_charge(p, cputime);
+
 	/* Add system time to cpustat. */
 	tmp = cputime_to_cputime64(cputime);
 	if (hardirq_count() - hardirq_offset)

--

#5 lore raw
In reply to Paul Menage

Hi Paul,

Thanks for doing the exercise of removing the container part of cpuset to provide some process aggregation.

With this model, I think I agree with you that RG can be split into individual controllers (need to look at it closely).

I have few questions/concerns w.r.t this implementation:

- Since we are re-implementing anyways, why not use configfs instead of having our own filesystem ?
- I am little nervous about notify_on_release, as RG would want classes/RGs to be available even when there are no tasks or sub- classes. (Documentation says that the user level program can rmdir the container, which would be a problem). Can the user level program be _not_ called when there are other subsystems registered ? Also, shouldn't it be cpuset specific, instead of global ?
- Export of the locks: These locks protect container data structures. But, most of the usages in cpuset.c are to protect the cpuset data structure itself. Shouldn't the cpuset subsystem have its own locks ? IMO, these locks should be used by subsystem only when they want data integrity in the container data structure itself (like walking thru the sibling list).
- Tight coupling of subsystems: I like your idea (you mentioned in a reply to the previous thread) of having an array of containers in task structure than the current implementation.

regards,

chandra

Quoted 104 lines · Paul Menage

This is essentially the same as the patch set that I posted last week, with the following fixes/changes:

- CONFIG_CONTAINERS is no longer a user-selectable option - subsystems such as cpusets that require it should select it in Kconfig.

- Each container subsystem type now has a name, and a <name>_enabled file in the top container directory. This file contains 0 or 1 to indicate whether the container subsystem is enabled, and can only be modified when there are no subcontainers; disabled container subsystems don't get new instances created when a subcontainer is created; the subsystem-specific state is simply inherited from the parent container.

- include a config option to default to enabled, for backwards compatibility

- Documentation tweaks

- builds properly without CONFIG_CONTAINER_CPUACCT configured on

- should build properly with newer gccs. (I've not actually had a chance to try building it with anything newer than gcc 3.2.2, but I've fixed all the potential warnings/errors that PaulJ pointed out when compiling with some unspecified newer gcc).

I've also looked at converting ResGroups to be a client of the container system. This isn't yet complete; my thoughts so far include:

- each resource controller can be implemented as an independent container subsystem; rather than a single "shares" and "stats" file in each directory there will be e.g. "numtasks_shares", "cpurc_shares", etc

- the ResGroups core will basically end up as a library that provides the common parsing/displaying for the shares and stats file for each controller, and the logic for propagating resources up and down the parent/child tree.

- for some of the resource controllers we will probably require a few extra callbacks from the container system, e.g. at fork/exit time. I might make these a config option that the controller must "select" in Kconfig, to avoid extra locking/overhead for subsystems such as cpusets that don't require such callbacks.

-------------------------------------

There have recently been various proposals floating around for resource management/accounting subsystems in the kernel, including Res Groups, User BeanCounters and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, and all implement this grouping in different ways.

Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards to synchronization rules).

This patchset extracts the process grouping code from cpusets into a generic container system, and makes the cpusets code a client of the container system.

It also provides a very simple additional container subsystem to do per-container CPU usage accounting; this is primarily to demonstrate use of the container subsystem API, but is useful in its own right.

The change is implemented in four stages:

1) extract the process grouping code from cpusets into a standalone system

2) remove the process grouping code from cpusets and hook into the container system

3) convert the container system to present a generic API, and make cpusets a client of that API

4) add a simple CPU accounting container subsystem as an example

The intention is that the various resource management efforts can also become container clients, with the result that:

- the userspace APIs are (somewhat) normalised

- it's easier to test out e.g. the ResGroups CPU controller in conjunction with the UBC memory controller

- the additional kernel footprint of any of the competing resource management systems is substantially reduced, since it doesn't need to provide process grouping/containment, hence improving their chances of getting into the kernel

Possible TODOs include:

- define a convention for populating the per-container directories so that different subsystems don't clash with one another

- provide higher-level primitives (e.g. an easy interface to seq_file) for files registered by subsystems.

- support subsystem deregistering

---

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------
#6 lore raw
Quoted 6 lines · Chandra Seetharaman

Hi Paul,

Thanks for doing the exercise of removing the container part of cpuset to provide some process aggregation.

Quoted 7 lines

With this model, I think I agree with you that RG can be split into individual controllers (need to look at it closely).

I have few questions/concerns w.r.t this implementation:

- Since we are re-implementing anyways, why not use configfs instead of having our own filesystem ?

The filesystem was lifted straight from cpuset.c, and hence isn't a reimplementation, it's a migration of code already in the tree. Wasn't there also a problem with the maximum output size of a configfs file, which would cause problems e.g. listing the task members in a container?

Quoted 6 lines

- I am little nervous about notify_on_release, as RG would want classes/RGs to be available even when there are no tasks or sub- classes. (Documentation says that the user level program can rmdir the container, which would be a problem). Can the user level program be _not_ called when there are other subsystems registered ? Also, shouldn't it be cpuset specific, instead of global ?

This again is taken straight from cpusets. The idea is that if you don't have some kind of middleware polling the container/cpuset/res_group directories to see if they're empty, you can instead ask the kernel to call you back (via "container_release_agent") at a point when a container is empty and hence removable. I don't think there's any guarantee that the container will still be empty by the time the userspace agent runs.

Quoted 6 lines

- Export of the locks: These locks protect container data structures. But, most of the usages in cpuset.c are to protect the cpuset data structure itself. Shouldn't the cpuset subsystem have its own locks ? IMO, these locks should be used by subsystem only when they want data integrity in the container data structure itself (like walking thru the sibling list).

It would certainly be possible to have finer-grained locking. But the cpuset code seems pretty happy with coarse-grained locking (only one writer at any one time) and having just the two global locks does make the whole synchronization an awful lot simpler. There's nothing to stop you having additional analogues of the callback_mutex to protect specific data in a particular resource controller's private data.

My inclination would be to find a situation where generic fine-grained locking is really required before forcing it on all container subsystems. The locking model in RG is certainly finer-grained than in cpusets, but don't a lot of the operations end up taking the root_group->group_lock anyway as their first action?

Quoted 3 lines

- Tight coupling of subsystems: I like your idea (you mentioned in a reply to the previous thread) of having an array of containers in task structure than the current implementation.

Can you suggest some scenarios that require this?

Paul

#7 lore raw
Quoted 1 line · Paul M

The filesystem was lifted straight from cpuset.c,

The primary author of the cpuset file system code was Simon Derr.

I'd encourage you to include him on the cc list of future posts of this patch set.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#8 lore raw
In reply to Paul Menage
Quoted 10 lines · Paul Menage
Quoted 9 lines · Chandra Seetharaman

Hi Paul,

Thanks for doing the exercise of removing the container part of cpuset to provide some process aggregation.

With this model, I think I agree with you that RG can be split into individual controllers (need to look at it closely).

Quoted 7 lines
Quoted 4 lines

I have few questions/concerns w.r.t this implementation:

- Since we are re-implementing anyways, why not use configfs instead of having our own filesystem ?

The filesystem was lifted straight from cpuset.c, and hence isn't a reimplementation, it's a migration of code already in the tree. Wasn't

Ok. I can't call it re-implementing :). But, I guess you get the point. This is an oppurtunity to remove the filesystem implementation and use existing infrastructure, configfs. configfs didn't exist when cpuset went in, otherwise they might have chosen to use it instead of writing their own.

Quoted 3 lines

there also a problem with the maximum output size of a configfs file, which would cause problems e.g. listing the task members in a container?

Yes, Joel is aware of it and is open to make that change.
http://marc.theaimsgroup.com/?l=ckrm-tech&m=115619222129067&w=2. Having a in-tree user (this infrastructure + cpuset) for that feature will increase the need for it.

Quoted 8 lines
Quoted 7 lines

- I am little nervous about notify_on_release, as RG would want classes/RGs to be available even when there are no tasks or sub- classes. (Documentation says that the user level program can rmdir the container, which would be a problem). Can the user level program be _not_ called when there are other subsystems registered ? Also, shouldn't it be cpuset specific, instead of global ?

Quoted 5 lines

This again is taken straight from cpusets. The idea is that if you don't have some kind of middleware polling the container/cpuset/res_group directories to see if they're empty, you can instead ask the kernel to call you back (via "container_release_agent") at a point when a container is empty and

I understand the purpose and usage.

Quoted 2 lines

hence removable. I don't think there's any guarantee that the container will still be empty by the time the userspace agent runs.

My concern is that the container _will_ be considered empty if there is no task attached with the container _and_ there is no sub-container.

CKRM/RG would want a empty container to exist.

We can hack it around by artificially incrementing the counter, but it will beat the original purpose of this feature.

Quoted 10 lines
Quoted 7 lines

- Export of the locks: These locks protect container data structures. But, most of the usages in cpuset.c are to protect the cpuset data structure itself. Shouldn't the cpuset subsystem have its own locks ? IMO, these locks should be used by subsystem only when they want data integrity in the container data structure itself (like walking thru the sibling list).

It would certainly be possible to have finer-grained locking. But the cpuset code seems pretty happy with coarse-grained locking (only one

cpuset may be happy today. But, It will not be happy when there are tens of other container subsystems use the same locks to protect their own data structures. Using such coarse locking will certainly affect the scalability.

Quoted 2 lines

writer at any one time) and having just the two global locks does make the whole synchronization an awful lot simpler. There's nothing to

No questions about that. But, do recall BKL and how much effort has gone in to break it to add scalability ( I am not saying that these locks are same as that). When we are starting afresh, why not start with scalability in mind.

Quoted 5 lines

stop you having additional analogues of the callback_mutex to protect specific data in a particular resource controller's private data.

My inclination would be to find a situation where generic fine-grained locking is really required before forcing it on all container

My thinking was like this: cpuset was the first user of this interface, any future container subsystem writers will certainly use cpuset as an example to write their subsystems. In effect, use the container-global locks to protect their data structures, which is not good in the long run.

Quoted 4 lines

subsystems. The locking model in RG is certainly finer-grained than in cpusets, but don't a lot of the operations end up taking the root_group->group_lock anyway as their first action?

Only if they are going to depend on the core data structure being intact (like list traversal).

Quoted 5 lines
Quoted 3 lines

- Tight coupling of subsystems: I like your idea (you mentioned in a reply to the previous thread) of having an array of containers in task structure than the current implementation.

Can you suggest some scenarios that require this?

Consider a scenario where you have only the system level cpuset and have multiple RGs. With this model you would be forced to create multiple cpusets (with the same set of cpus) so as to allow multiple RG's. Now, consider you want to create a cpuset that is a subset of the high level cpuset, where in the hierarchy you would create this cpuset (at top level or one level below) ?

Extend this scenario to multiple subsystems and see how complicated the interface would become to the user.

If we have it this way, then the notify_on_release issue (above) will disappear too.

Quoted 2 lines

Paul

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------
#9 lore raw
Quoted 9 lines
Quoted 3 lines

It would certainly be possible to have finer-grained locking. But the cpuset code seems pretty happy with coarse-grained locking (only one

cpuset may be happy today. But, It will not be happy when there are tens of other container subsystems use the same locks to protect their own data structures. Using such coarse locking will certainly affect the scalability.

All of this (and the rest of the snipped email with suggested improvements) makes pretty good sense. But would it not be better to do this in stages?

1) Split the code out from cpusets
2) Move to configfs
3) Work on locking scalability, etc ...

Else it'd seem that we'll never get anywhere, and it'll all be impossible to review anyway. Incremental improvement would seem to be a much easier way to fix this stuff, to me.

M.

#10 lore raw

On Wed, 2006-10-04 at 12:36 -0700, Martin Bligh wrote:

I agree with you, Martin.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

It would certainly be possible to have finer-grained locking. But the cpuset code seems pretty happy with coarse-grained locking (only one

cpuset may be happy today. But, It will not be happy when there are tens of other container subsystems use the same locks to protect their own data structures. Using such coarse locking will certainly affect the scalability.

Quoted 5 lines

All of this (and the rest of the snipped email with suggested improvements) makes pretty good sense. But would it not be better to do this in stages?

1) Split the code out from cpusets

Paul (Menage) is already work on this.

We will work out the rest.

Quoted 8 lines

2) Move to configfs
3) Work on locking scalability, etc ...

Else it'd seem that we'll never get anywhere, and it'll all be impossible to review anyway. Incremental improvement would seem to be a much easier way to fix this stuff, to me.

M.

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------
#11 lore raw
Quoted 7 lines · Chandra Seetharaman
Quoted 5 lines

All of this (and the rest of the snipped email with suggested improvements) makes pretty good sense. But would it not be better to do this in stages?

1) Split the code out from cpusets

Paul (Menage) is already work on this.

The split out is done - right now I'm tidying up an example of RG moved over the container API - basically dumping the group membership code (since container.c supplies that) and migrating the shares/stats file model to containerfs rather than configfs. I'll try to send it out today.

Paul

#12 lore raw
In reply to Chandra Seetharaman
Quoted 8 lines · Chandra Seetharaman
Quoted 2 lines

The filesystem was lifted straight from cpuset.c, and hence isn't a reimplementation, it's a migration of code already in the tree. Wasn't

Ok. I can't call it re-implementing :). But, I guess you get the point. This is an oppurtunity to remove the filesystem implementation and use existing infrastructure, configfs. configfs didn't exist when cpuset went in, otherwise they might have chosen to use it instead of writing their own.

I guess I'm mostly agnostic about this issue. But looking at the configfs interfacing code in rgcs.c versus the filesystem detail code in container.c, it's about 200 lines vs 300 lines, so not exactly a huge complexity saving.

Quoted 4 lines

Yes, Joel is aware of it and is open to make that change.
http://marc.theaimsgroup.com/?l=ckrm-tech&m=115619222129067&w=2. Having a in-tree user (this infrastructure + cpuset) for that feature will increase the need for it.

Great - when that happens I'd be much happier to move over configfs.

Quoted 3 lines

My concern is that the container _will_ be considered empty if there is no task attached with the container _and_ there is no sub-container.

Right, that's the definition of an empty container.

Quoted 2 lines

CKRM/RG would want a empty container to exist.

Even if the user wants to be told when it's OK to clean up their RG containers?

I don't see why this is a problem - all this does is allows the user (*if* they want to) to get a callback when there's no longer anything alive in the container. It doesn't cause the container to be removed, without additional explicit action from userspace.

I don't see it as a feature that I'd make much use of myself, but it preserves the existing cpusets API.

Quoted 5 lines

cpuset may be happy today. But, It will not be happy when there are tens of other container subsystems use the same locks to protect their own data structures. Using such coarse locking will certainly affect the scalability.

The locks exported by the container system simply guarantee:

- if you hold container_manage_lock(), then no-one else will make any changes to the core container groupings, and you won't block readers

- if you hold container_lock(), then no changes will be made to the container groupings.

While there's nothing to stop a container subsystem from also using them to protect its own data, there's equally nothing to stop a container subsystem from using its own finer-grained locking, if it feels that it needs to. E.g. the simple cpu_acct subsystem that I posted as an example has a per-container spinlock that it uses to protect the cpu stats for that container. So the callback doesn't need to take either container_lock() or container_manage_lock(), it just has to ensure it's in an rcu_read_lock() section and takes its own spinlock.

Quoted 4 lines

No questions about that. But, do recall BKL and how much effort has gone in to break it to add scalability ( I am not saying that these locks are same as that). When we are starting afresh, why not start with scalability in mind.

Looking at the group_lock in RG (including memrc and cpurc), the only places that it appears to get taken are when creating and destroying groups, or when changing shares. These are things that are going to be done by the middleware - how many concurrent middleware systems are you expecting to need to scale to? :-)

Incidentally, I have a few questions about the locking in the example numtasks controller:

- recalc_and_propagate() does:

lock rgroup->group_lock
for each child:
lock child->group_lock
recalc_and_propagate(child)

So the first thing that the recursive call does is to take the lock that the previous level of recursion has already taken. How's that supposed to work?

E.g.

cd /mnt/configfs/res_groups
mkdir foo
echo -n res=numtasks,min_shares=10,max_shares=10 > shares

causes a lockup.

- in dec_usage_count() when a task exits, am I right that the only locking done is on the group from which the task exited? The calls can cascade up to the parent if cnt_borrowed is non-zero. In that case, in the situation where the parent and two child groups each have a cnt_borrowed of 1, what's to stop the dec_usage_count() calls racing with one another in the parent, hence leaving parent->cnt_borrowed at -1, and both calling dec_usage_count() on the grandparent?

I realise that these issues can be fixed, but they do serve to illustrate the pitfalls (or at least, reduced clarity) associated with fine-grained locking ...

Quoted 6 lines

My thinking was like this: cpuset was the first user of this interface, any future container subsystem writers will certainly use cpuset as an example to write their subsystems. In effect, use the container-global locks to protect their data structures, which is not good in the long run.

Fair point. I'll see about reducing some of the coarse-grained locking in cpuset.c. But a lot of the time, the cpuset code needs to be able to prevent both changes to current->cpuset, and changes to its reference - so it has to take container_lock() anyway, and might as well use that as the only lock, rather than having to take two locks.

One optimization would be to have callback_mutex be an rwsem, so we could export container_lock()/container_unlock() as well as container_read_lock() and container_read_unlock(). (PaulJ, any thoughts on that?)

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

- Tight coupling of subsystems: I like your idea (you mentioned in a reply to the previous thread) of having an array of containers in task structure than the current implementation.

Can you suggest some scenarios that require this?

Consider a scenario where you have only the system level cpuset and have multiple RGs. With this model you would be forced to create multiple cpusets (with the same set of cpus) so as to allow multiple RG's.

No - see the intro to this patch set. I added a "<subsys>_enabled" file for each subsystem, managed by the container.c code; if this is set to 0, then child containers inherit the same per-subsystem state for that subsystem as the root container.

So if you do echo 0 > /mnt/container/cpuset_enabled, and create child containers, you still only have one cpuset.

Quoted 4 lines

Now,
consider you want to create a cpuset that is a subset of the high level cpuset, where in the hierarchy you would create this cpuset (at top level or one level below) ?

For a distinct set of tasks, or for the same set of tasks? If the former, then it can be a new hierarchy under the top-level, no problem. If it's for the same set of tasks then you'd need to give more details about how/why you're trying to split up resources. I would imagine that almost all situations like this, there would be a natural way to split things up hierarchically. E.g. you want to pin each customer's tasks on to a particular set of nodes (so use cpuset parameters at the top level) and within those you want to let the customer have different groups of tasks with e.g. different memory limits (so use RG mem controller parameters at the next level)

Paul

#13 lore raw
Quoted 5 lines · Paul Menage
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

- Tight coupling of subsystems: I like your idea (you mentioned in a reply to the previous thread) of having an array of containers in task structure than the current implementation.

...

BTW, that's not to say that having parallel hierarchies of containers is necessarily a bad thing - I can imagine just mounting multiple instances of containerfs, each managing one of the container pointers in task_struct - but I think that could be added on afterwards. Even if we did have the parallel support, we'd still need to support multiple subsystems/controllers on the same hierarchy, since I think that's going to be the much more common case.

Paul