Containers(V10): Generic Process Containers
Subsystems (8)
cover Containers(V10): Generic Process Containers
This is an update to my multi-hierarchy multi-subsystem generic process containers patch. Changes since V9 (April 27th) include:
- The patchset has been rebased over 2.6.22-rc2-mm1
- A lattice of lists linking tasks to their css_groups and css_groups to their containers has been added to support more efficient iteration across the member tasks of a container.
- Support for the cpusets "release agent" functionality has been added back in; this is based on a workqueue concept similar to the changes that Cliff Wickman has been pushing for supporting CPU hot-unplug.
- Several uses of tasklist_lock replaced by reliance on RCU
- Misc cleanups
- Tested with a tweaked version of PaulJ's cpuset_test script
Still TODO:
- decide whether "Containers" is an acceptable name for the system given its usage by some other development groups, or whether something else (ProcessSets? ResourceGroups? TaskGroups?) would be better. I'm inclined to leave this political decision to Andrew/Linus once they're happy with the technical aspects of the patches.
- add a hash-table based lookup for css_group objects.
- use seq_file properly in container tasks files to avoid having to allocate a big array for all the container's task pointers.
- lots more testing
- define standards for container file names
--
Generic Process Containers
--------------------------
There have recently been various proposals floating around for resource management/accounting and other task grouping subsystems in the kernel, including ResGroups, User BeanCounters, NSProxy containers, and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, or control other behaviour of the processes, and all implement this grouping in different ways.
Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards to synchronization rules).
This patchset extracts the process grouping code from cpusets into a generic container system, and makes the cpusets code a client of the container system, along with a couple of simple example subsystems.
The patch set is structured as follows:
1) Basic container framework - filesystem and tracking structures
2) Simple CPU Accounting example subsystem
3) Support for the "tasks" control file
4) Hooks for fork() and exit()
5) Support for the container_clone() operation
6) Add /proc reporting interface
7) Make cpusets a container subsystem
8) Share container subsystem pointer arrays between tasks with the same assignments
9) Simple container debugging subsystem
10) Support for a userspace "release agent", similar to the cpusets release agent functionality
The intention is that the various resource management and virtualization efforts can also become container clients, with the result that:
- the userspace APIs are (somewhat) normalised
- it's easier to test out e.g. the ResGroups CPU controller in conjunction with the BeanCounters memory controller, or use either of
them as the resource-control portion of a virtual server system.
- the additional kernel footprint of any of the competing resource management systems is substantially reduced, since it doesn't need to provide process grouping/containment, hence improving their chances of getting into the kernel
PATCH 1/10 Containers(V10): Basic container framework
This patch adds the main containers framework - the container filesystem, and the basic structures for tracking membership and associating subsystem state objects to tasks.
--- Documentation/containers.txt | 524 +++++++++++++++++ include/linux/container.h | 198 ++++++ include/linux/container_subsys.h | 10 include/linux/sched.h | 34 + init/Kconfig | 3 init/main.c | 3 kernel/Makefile | 1 kernel/container.c | 1155 +++++++++++++++++++++++++++++++++++++++ 8 files changed, 1927 insertions(+), 1 deletion(-)
Documentation/containers.txt new file +524 -0
@@ -0,0 +1,524 @@ + CONTAINERS + ------- + +Written by Paul Menage <[email protected]> based on Documentation/cpusets.txt + +Original copyright statements from cpusets.txt: +Portions Copyright (C) 2004 BULL SA. +Portions Copyright (c) 2004-2006 Silicon Graphics, Inc. +Modified by Paul Jackson <[email protected]> +Modified by Christoph Lameter <[email protected]> + +CONTENTS: +========= + +1. Containers + 1.1 What are containers ? + 1.2 Why are containers needed ? + 1.3 How are containers implemented ? + 1.4 What does notify_on_release do ? + 1.5 How do I use containers ? +2. Usage Examples and Syntax + 2.1 Basic Usage + 2.2 Attaching processes +3. Kernel API + 3.1 Overview + 3.2 Synchronization + 3.3 Subsystem API +4. Questions + +1. Containers +========== + +1.1 What are containers ? +---------------------- + +Containers provide a mechanism for aggregating/partitioning sets of +tasks, and all their future children, into hierarchical groups with +specialized behaviour. + +Definitions: + +A *container* associates a set of tasks with a set of parameters for one +or more subsystems. + +A *subsystem* is a module that makes use of the task grouping +facilities provided by containers to treat groups of tasks in +particular ways. A subsystem is typically a "resource controller" that +schedules a resource or applies per-container limits, but it may be +anything that wants to act on a group of processes, e.g. a +virtualization subsystem. + +A *hierarchy* is a set of containers arranged in a tree, such that +every task in the system is in exactly one of the containers in the +hierarchy, and a set of subsystems; each subsystem has system-specific +state attached to each container in the hierarchy. Each hierarchy has +an instance of the container virtual filesystem associated with it. + +At any one time there may be multiple active hierachies of task +containers. Each hierarchy is a partition of all tasks in the system. + +User level code may create and destroy containers by name in an +instance of the container virtual file system, specify and query to +which container a task is assigned, and list the task pids assigned to +a container. Those creations and assignments only affect the hierarchy +associated with that instance of the container file system. + +On their own, the only use for containers is for simple job +tracking. The intention is that other subsystems hook into the generic +container support to provide new attributes for containers, such as +accounting/limiting the resources which processes in a container can +access. For example, cpusets (see Documentation/cpusets.txt) allows +you to associate a set of CPUs and a set of memory nodes with the +tasks in each container. + +1.2 Why are containers needed ? +---------------------------- + +There are multiple efforts to provide process aggregations in the +Linux kernel, mainly for resource tracking purposes. Such efforts +include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server +namespaces. These all require the basic notion of a +grouping/partitioning of processes, with newly forked processes ending +in the same group (container) as their parent process. + +The kernel container patch provides the minimum essential kernel +mechanisms required to efficiently implement such groups. It has +minimal impact on the system fast paths, and provides hooks for +specific subsystems such as cpusets to provide additional behaviour as +desired. + +Multiple hierarchy support is provided to allow for situations where +the division of tasks into containers is distinctly different for +different subsystems - having parallel hierarchies allows each +hierarchy to be a natural division of tasks, without having to handle +complex combinations of tasks that would be present if several +unrelated subsystems needed to be forced into the same tree of +containers. + +At one extreme, each resource controller or subsystem could be in a +separate hierarchy; at the other extreme, all subsystems +would be attached to the same hierarchy. + +As an example of a scenario (originally proposed by [email protected]) +that can benefit from multiple hierarchies, consider a large +university server with various users - students, professors, system +tasks etc. The resource planning for this server could be along the +following lines: + + CPU : Top cpuset + / \ + CPUSet1 CPUSet2 + | | + (Profs) (Students) + + In addition (system tasks) are attached to topcpuset (so + that they can run anywhere) with a limit of 20% + + Memory : Professors (50%), students (30%), system (20%) + + Disk : Prof (50%), students (30%), system (20%) + + Network : WWW browsing (20%), Network File System (60%), others (20%) + / \ + Prof (15%) students (5%) + +Browsers like firefox/lynx go into the WWW network class, while (k)nfsd go +into NFS network class. + +At the same time firefox/lynx will share an appropriate CPU/Memory class +depending on who launched it (prof/student). + +With the ability to classify tasks differently for different resources +(by putting those resource subsystems in different hierarchies) then +the admin can easily set up a script which receives exec notifications +and depending on who is launching the browser he can + + # echo browser_pid > /mnt/<restype>/<userclass>/tasks + +With only a single hierarchy, he now would potentially have to create +a separate container for every browser launched and associate it with +approp network and other resource class. This may lead to +proliferation of such containers. + +Also lets say that the administrator would like to give enhanced network +access temporarily to a student's browser (since it is night and the user +wants to do online gaming :) OR give one of the students simulation +apps enhanced CPU power, + +With ability to write pids directly to resource classes, its just a +matter of : + + # echo pid > /mnt/network/<new_class>/tasks + (after some time) + # echo pid > /mnt/network/<orig_class>/tasks + +Without this ability, he would have to split the container into +multiple separate ones and then associate the new containers with the +new resource classes. + + + +1.3 How are containers implemented ? +--------------------------------- + +Containers extends the kernel as follows: + + - Each task in the system has a reference-counted pointer to a + css_group. + + - A css_group contains a set of reference-counted pointers to + container_subsys_state objects, one for each container subsystem + registered in the system. There is no direct link from a task to + the container of which it's a member in each hierarchy, but this + can be determined by following pointers through the + container_subsys_state objects. This is because accessing the + subsystem state is something that's expected to happen frequently + and in performance-critical code, whereas operations that require a + task's actual container assignments (in particular, moving between + containers) are less common. + + - A container hierarchy filesystem can be mounted for browsing and + manipulation from user space. + + - You can list all the tasks (by pid) attached to any container. + +The implementation of containers requires a few, simple hooks +into the rest of the kernel, none in performance critical paths: + + - in init/main.c, to initialize the root containers and initial + css_group at system boot. + + - in fork and exit, to attach and detach a task from its css_group. + +In addition a new file system, of type "container" may be mounted, to +enable browsing and modifying the containers presently known to the +kernel. When mounting a container hierarchy, you may specify a +comma-separated list of subsystems to mount as the filesystem mount +options. By default, mounting the container filesystem attempts to +mount a hierarchy containing all registered subsystems. + +If an active hierarchy with exactly the same set of subsystems already +exists, it will be reused for the new mount. If no existing hierarchy +matches, and any of the requested subsystems are in use in an existing +hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy +is activated, associated with the requested subsystems. + +It's not currently possible to bind a new subsystem to an active +container hierarchy, or to unbind a subsystem from an active container +hierarchy. This may be possible in future, but is fraught with nasty +error-recovery issues. + +When a container filesystem is unmounted, if there are any +subcontainers created below the top-level container, that hierarchy +will remain active even though unmounted; if there are no +subcontainers then the hierarchy will be deactivated. + +No new system calls are added for containers - all support for +querying and modifying containers is via this container file system. + +Each task under /proc has an added file named 'container' displaying, +for each active hierarchy, the subsystem names and the container name +as the path relative to the root of the container file system. + +Each container is represented by a directory in the container file system +containing the following files describing that container: + + - tasks: list of tasks (by pid) attached to that container + - notify_on_release flag: run /sbin/container_release_agent on exit? + +Other subsystems such as cpusets may add additional files in each +container dir + +New containers are created using the mkdir system call or shell +command. The properties of a container, such as its flags, are +modified by writing to the appropriate file in that containers +directory, as listed above. + +The named hierarchical structure of nested containers allows partitioning +a large system into nested, dynamically changeable, "soft-partitions". + +The attachment of each task, automatically inherited at fork by any +children of that task, to a container allows organizing the work load +on a system into related sets of tasks. A task may be re-attached to +any other container, if allowed by the permissions on the necessary +container file system directories. + +When a task is moved from one container to another, it gets a new +css_group pointer - if there's an already existing css_group with the +desired collection of containers then that group is reused, else a new +css_group is allocated. Note that the current implementation uses a +linear search to locate an appropriate existing css_group, so isn't +very efficient. A future version will use a hash table for better +performance. + +The use of a Linux virtual file system (vfs) to represent the +container hierarchy provides for a familiar permission and name space +for containers, with a minimum of additional kernel code. + +1.4 What does notify_on_release do ? +------------------------------------ + +*** notify_on_release is disabled in the current patch set. It will be +*** reactivated in a future patch in a less-intrusive manner + +If the notify_on_release flag is enabled (1) in a container, then +whenever the last task in the container leaves (exits or attaches to +some other container) and the last child container of that container +is removed, then the kernel runs the command specified by the contents +of the "release_agent" file in that hierarchy's root directory, +supplying the pathname (relative to the mount point of the container +file system) of the abandoned container. This enables automatic +removal of abandoned containers. The default value of +notify_on_release in the root container at system boot is disabled +(0). The default value of other containers at creation is the current +value of their parents notify_on_release setting. The default value of +a container hierarchy's release_agent path is empty. + +1.5 How do I use containers ? +-------------------------- + +To start a new job that is to be contained within a container, using +the "cpuset" container subsystem, the steps are something like: + + 1) mkdir /dev/container + 2) mount -t container -ocpuset cpuset /dev/container + 3) Create the new container by doing mkdir's and write's (or echo's) in + the /dev/container virtual file system. + 4) Start a task that will be the "founding father" of the new job. + 5) Attach that task to the new container by writing its pid to the + /dev/container tasks file for that container. + 6) fork, exec or clone the job tasks from this founding father task. + +For example, the following sequence of commands will setup a container +named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, +and then start a subshell 'sh' in that container: + + mount -t container cpuset -ocpuset /dev/container + cd /dev/container + mkdir Charlie + cd Charlie + /bin/echo $$ > tasks + sh + # The subshell 'sh' is now running in container Charlie + # The next line should display '/Charlie' + cat /proc/self/container + +2. Usage Examples and Syntax +============================ + +2.1 Basic Usage +--------------- + +Creating, modifying, using the containers can be done through the container +virtual filesystem. + +To mount a container hierarchy will all available subsystems, type: +# mount -t container xxx /dev/container + +The "xxx" is not interpreted by the container code, but will appear in +/proc/mounts so may be any useful identifying string that you like. + +To mount a container hierarchy with just the cpuset and numtasks +subsystems, type: +# mount -t container -o cpuset,numtasks hier1 /dev/container + +To change the set of subsystems bound to a mounted hierarchy, just +remount with different options: + +# mount -o remount,cpuset,ns /dev/container + +Note that changing the set of subsystems is currently only supported +when the hierarchy consists of a single (root) container. Supporting +the ability to arbitrarily bind/unbind subsystems from an existing +container hierarchy is intended to be implemented in the future. + +Then under /dev/container you can find a tree that corresponds to the +tree of the containers in the system. For instance, /dev/container +is the container that holds the whole system. + +If you want to create a new container under /dev/container: +# cd /dev/container +# mkdir my_container + +Now you want to do something with this container. +# cd my_container + +In this directory you can find several files: +# ls +notify_on_release release_agent tasks +(plus whatever files are added by the attached subsystems) + +Now attach your shell to this container: +# /bin/echo $$ > tasks + +You can also create containers inside your container by using mkdir in this +directory. +# mkdir my_sub_cs + +To remove a container, just use rmdir: +# rmdir my_sub_cs + +This will fail if the container is in use (has containers inside, or +has processes attached, or is held alive by other subsystem-specific +reference). + +2.2 Attaching processes +----------------------- + +# /bin/echo PID > tasks + +Note that it is PID, not PIDs. You can only attach ONE task at a time. +If you have several tasks to attach, you have to do it one after another: + +# /bin/echo PID1 > tasks +# /bin/echo PID2 > tasks + ... +# /bin/echo PIDn > tasks + +3. Kernel API +============= + +3.1 Overview +------------ + +Each kernel subsystem that wants to hook into the generic container +system needs to create a container_subsys object. This contains +various methods, which are callbacks from the container system, along +with a subsystem id which will be assigned by the container system. + +Other fields in the container_subsys object include: + +- subsys_id: a unique array index for the subsystem, indicating which + entry in container->subsys[] this subsystem should be + managing. Initialized by container_register_subsys(); prior to this + it should be initialized to -1 + +- hierarchy: an index indicating which hierarchy, if any, this + subsystem is currently attached to. If this is -1, then the + subsystem is not attached to any hierarchy, and all tasks should be + considered to be members of the subsystem's top_container. It should + be initialized to -1. + +- name: should be initialized to a unique subsystem name prior to + calling container_register_subsystem. Should be no longer than + MAX_CONTAINER_TYPE_NAMELEN + +Each container object created by the system has an array of pointers, +indexed by subsystem id; this pointer is entirely managed by the +subsystem; the generic container code will never touch this pointer. + +3.2 Synchronization +------------------- + +There is a global mutex, container_mutex, used by the container +system. This should be taken by anything that wants to modify a +container. It may also be taken to prevent containers from being +modified, but more specific locks may be more appropriate in that +situation. + +See kernel/container.c for more details. + +Subsystems can take/release the container_mutex via the functions +container_lock()/container_unlock(), and can +take/release the callback_mutex via the functions +container_lock()/container_unlock(). + +Accessing a task's container pointer may be done in the following ways: +- while holding container_mutex +- while holding the task's alloc_lock (via task_lock()) +- inside an rcu_read_lock() section via rcu_dereference() + +3.3 Subsystem API +-------------------------- + +Each subsystem should: + +- add an entry in linux/container_subsys.h +- define a container_subsys object called <name>_subsys + +Each subsystem may export the following methods. The only mandatory +methods are create/destroy. Any others that are null are presumed to +be successful no-ops. + +int create(struct container *cont) +LL=container_mutex + +Called to create a subsystem state object for a container. The +subsystem should set its subsystem pointer for the passed container, +returning 0 on success or a negative error code. On success, the +subsystem pointer should point to a structure of type +container_subsys_state (typically embedded in a larger +subsystem-specific object), which will be initialized by the container +system. Note that this will be called at initialization to create the +root subsystem state for this subsystem; this case can be identified +by the passed container object having a NULL parent (since it's the +root of the hierarchy) and may be an appropriate place for +initialization code. + +void destroy(struct container *cont) +LL=container_mutex + +The container system is about to destroy the passed container; the +subsystem should do any necessary cleanup + +int can_attach(struct container_subsys *ss, struct container *cont, + struct task_struct *task) +LL=container_mutex + +Called prior to moving a task into a container; if the subsystem +returns an error, this will abort the attach operation. If a NULL +task is passed, then a successful result indicates that *any* +unspecified task can be moved into the container. Note that this isn't +called on a fork. If this method returns 0 (success) then this should +remain valid while the caller holds container_mutex. + +void attach(struct container_subsys *ss, struct container *cont, + struct container *old_cont, struct task_struct *task) +LL=container_mutex + + +Called after the task has been attached to the container, to allow any +post-attachment activity that requires memory allocations or blocking. + +void fork(struct container_subsy *ss, struct task_struct *task) +LL=callback_mutex, maybe read_lock(tasklist_lock) + +Called when a task is forked into a container. Also called during +registration for all existing tasks. + +void exit(struct container_subsys *ss, struct task_struct *task) +LL=callback_mutex + +Called during task exit + +int populate(struct container_subsys *ss, struct container *cont) +LL=none + +Called after creation of a container to allow a subsystem to populate +the container directory with file entries. The subsystem should make +calls to container_add_file() with objects of type cftype (see +include/linux/container.h for details). Note that although this +method can return an error code, the error code is currently not +always handled well. + +void bind(struct container_subsys *ss, struct container *root) +LL=callback_mutex + +Called when a container subsystem is rebound to a different hierarchy +and root container. Currently this will only involve movement between +the default hierarchy (which never has sub-containers) and a hierarchy +that is being created/destroyed (and hence has no sub-containers). + +4. Questions +============ + +Q: what's up with this '/bin/echo' ? +A: bash's builtin 'echo' command does not check calls to write() against + errors. If you use it in the container file system, you won't be + able to tell whether a command succeeded or failed. + +Q: When I attach processes, only the first of the line gets really attached ! +A: We can only return one error code per call to write(). So you should also + put only ONE pid. +
include/linux/container.h new file +198 -0
198 lines not on this page · raw message
include/linux/container_subsys.h new file +10 -0
@@ -0,0 +1,10 @@ +/* Add subsystem definitions of the form SUBSYS(<name>) in this +* file. Surround each one by a line of comment markers so that +* patches don't collide +*/ + +/* */ + +/* */ + +/* */
include/linux/sched.h +33 -1
@@ -851,6 +851,34 @@ struct sched_class {void (*task_new) (struct rq *rq,struct task_struct *p); }; +#ifdef CONFIG_CONTAINERS + +#define SUBSYS (_x) _x ## _subsys_id, +enum container_subsys_id { +#include <linux/container_subsys.h> + CONTAINER_SUBSYS_COUNT +}; +#undef SUBSYS + +/* A css_group is a structure holding pointers to a set of +* container_subsys_state objects. +*/ + +struct css_group { + +/* Set of subsystem states, one for each subsystem. NULL for +* subsystems that aren't part of this hierarchy. These +* pointers reduce the number of dereferences required to get +* from a task to its state for a given container, but result +* in increased space usage if tasks are in wildly different +* groupings across different hierarchies. This array is +* immutable after creation */ +struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT ]; + +}; + +#endif /* CONFIG_CONTAINERS */ +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;
@@ -1107,6 +1135,9 @@ struct task_struct {int cpuset_mems_generation;int cpuset_mem_spread_rotor;#endif +#ifdef CONFIG_CONTAINERS +struct css_group containers; +#endif struct robust_list_head __user *robust_list;#ifdef CONFIG_COMPAT struct compat_robust_list_head __user *compat_robust_list;
@@ -1553,7 +1584,8 @@ static inline int thread_group_empty(str/* * Protects ->fs, ->files, ->mm, ->group_info, ->comm, keyring * subscriptions and synchronises with wait4(). Also used in procfs. Also -* pins the final release of task.io_context. Also protects ->cpuset. +* pins the final release of task.io_context. Also protects ->cpuset and +* ->container.subsys[]. * * Nests both inside and outside of read_lock(&tasklist_lock). * It must not be nested with write_lock_irq(&tasklist_lock),
init/Kconfig +3 -0
@@ -303,6 +303,9 @@ config LOG_BUF_SHIFT 13 => 8 KB 12 => 4 KB +config CONTAINERS + bool + config CPUSETS bool "Cpuset support" depends on SMP
init/main.c +3 -0
@@ -39,6 +39,7 @@#include <linux/writeback.h> #include <linux/cpu.h> #include <linux/cpuset.h> +#include <linux/container.h> #include <linux/efi.h> #include <linux/tick.h> #include <linux/interrupt.h>
@@ -499,6 +500,7 @@ asmlinkage void __init start_kernel(voidchar * command_line;extern struct kernel_param __start___param[], __stop___param[]; +container_init_early ();smp_setup_processor_id ();/*
@@ -624,6 +626,7 @@ asmlinkage void __init start_kernel(void#ifdef CONFIG_PROC_FS proc_root_init ();#endif +container_init ();cpuset_init ();taskstats_init_early ();delayacct_init ();
kernel/Makefile +1 -0
@@ -36,6 +36,7 @@ obj-$(CONFIG_PM) += power/ obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_COMPAT) += compat.o +obj-$(CONFIG_CONTAINERS) += container.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_IKCONFIG) += configs.o obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
kernel/container.c new file +1155 -0
1,155 lines not on this page · raw message
--
Andrew Morton
maintainerRe: [PATCH 01/10] Containers(V10): Basic container framework
Documentation/containers.txt line 305
Quoted 13 lines · On Tue, 29 May 2007 06:01:05 -0700 [email protected] wrote:
Documentation/containers.txt new file +13
-0+For example, the following sequence of commands will setup a container +named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, +and then start a subshell 'sh' in that container: + + mount -t container cpuset -ocpuset /dev/container + cd /dev/container + mkdir Charlie + cd Charlie + /bin/echo $$ > tasks + sh + # The subshell 'sh' is now running in container Charlie + # The next line should display '/Charlie' + cat /proc/self/container
Once this has been done, can tasks inside `Charlie' escape from it?
And what permissions are needed to expand the various allotments (if that's the approved term) for `Charlie'?
Andrew Morton
maintainerRe: [PATCH 01/10] Containers(V10): Basic container framework
In reply to [email protected]Quoted 42 lines · On Tue, 29 May 2007 06:01:05 -0700 [email protected] wrote:
This patch adds the main containers framework - the container filesystem, and the basic structures for tracking membership and associating subsystem state objects to tasks.
...
include/linux/container_subsys.h new file +10
-0@@ -0,0 +1,10 @@ +/* Add subsystem definitions of the form SUBSYS(<name>) in this +* file. Surround each one by a line of comment markers so that +* patches don't collide +*/ + +/* */ + +/* */ + +/* */ Index:
container-2.6.22-rc2-mm1/include/linux/sched.h
===================================================================include/linux/sched.h +15
-0@@ -851,6 +851,34 @@ struct sched_class {void (*task_new) (struct rq *rq,struct task_struct *p); }; +#ifdef CONFIG_CONTAINERS + +#define SUBSYS (_x) _x ## _subsys_id, +enum container_subsys_id { +#include <linux/container_subsys.h> + CONTAINER_SUBSYS_COUNT +}; +#undef SUBSYS + +/* A css_group is a structure holding pointers to a set of +* container_subsys_state objects. +*/ + +struct css_group { +
include/linux/sched.h struct sched_class { · line 878
Quoted 10 lines
include/linux/sched.h +10
-0+/* Set of subsystem states, one for each subsystem. NULL for +* subsystems that aren't part of this hierarchy. These +* pointers reduce the number of dereferences required to get +* from a task to its state for a given container, but result +* in increased space usage if tasks are in wildly different +* groupings across different hierarchies. This array is +* immutable after creation */ +struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT ]; + +};
hm, missing forward declaration of struct container_subsys_state, but it all seems to work out (via nested include) once the patches are applied and the config option is enableable.
Quoted 33 lines
...
kernel/container.c new file +25
-0+#include <linux/cpu.h> +#include <linux/cpumask.h> +#include <linux/container.h> +#include <linux/err.h> +#include <linux/errno.h> +#include <linux/file.h> +#include <linux/fs.h> +#include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/kernel.h> +#include <linux/kmod.h> +#include <linux/list.h> +#include <linux/mempolicy.h> +#include <linux/mm.h> +#include <linux/module.h> +#include <linux/mount.h> +#include <linux/namei.h> +#include <linux/pagemap.h> +#include <linux/proc_fs.h> +#include <linux/rcupdate.h> +#include <linux/sched.h> +#include <linux/seq_file.h> +#include <linux/security.h> +#include <linux/slab.h> +#include <linux/smp_lock.h>
kernel/container.c line 61
Quoted 10 lines
kernel/container.c new file +10
-0+#include <linux/spinlock.h> +#include <linux/stat.h> +#include <linux/string.h> +#include <linux/time.h> +#include <linux/backing-dev.h> +#include <linux/sort.h> + +#include <asm/uaccess.h> +#include <asm/atomic.h> +#include <linux/mutex.h>
Holy cow, do we need all those?
kernel/container.c line 118
Quoted 3 lines
kernel/container.c new file +3
-0+typedef enum { +CONT_REMOVED , +}container_flagbits_t ;
typedefs are verboten. Fortunately this one is never referred to - only the values are used, so we can delete it.
Quoted 30 lines
...
kernel/container.c new file +27
-0+static void container_clear_directory (struct dentry *dentry) +{ +struct list_head *node; +BUG_ON (!mutex_is_locked (&dentry->d_inode->i_mutex)); +spin_lock (&dcache_lock); + node = dentry->d_subdirs.next; +while (node != &dentry->d_subdirs) { + struct dentry *d = list_entry(node, struct dentry, d_u.d_child); +list_del_init (node); +if (d->d_inode) { +/* This should never be called on a container +* directory with child containers */ +BUG_ON (d->d_inode->i_mode &S_IFDIR ); + d =dget_locked (d); +spin_unlock (&dcache_lock); +d_delete (d); +simple_unlink (dentry->d_inode, d); +dput (d); +spin_lock (&dcache_lock); + } + node = dentry->d_subdirs.next; + } +spin_unlock (&dcache_lock); +} + +/* +* NOTE : the dentry must have been dget()'ed
kernel/container.c line 312
Quoted 10 lines
kernel/container.c new file +10
-0+*/ +static void container_d_remove_dir (struct dentry *dentry) +{ +container_clear_directory (dentry); + +spin_lock (&dcache_lock); +list_del_init (&dentry->d_u.d_child); +spin_unlock (&dcache_lock); +remove_dir (dentry); +}
Taking dcache_lock in here is unfortunate. A filesystem really shouldn't be playing with that lock.
But about 20 filesystems do so. Ho hum.
kernel/container.c line 315
Quoted 2 lines
kernel/container.c new file +2
-0+static int rebind_subsystems (struct containerfs_root *root, +unsigned long final_bits)
The code's a bit short on comments.
Quoted 68 lines
kernel/container.c new file +66
-0+{ +unsigned long added_bits, removed_bits; +struct container *cont = &root->top_container; +int i; + + removed_bits = root->subsys_bits & ~final_bits; + added_bits = final_bits & ~root->subsys_bits; +/* Check that any added subsystems are currently free */ +for (i =0 ; i <CONTAINER_SUBSYS_COUNT ; i++) { +unsigned long long bit =1ull << i; +struct container_subsys *ss = subsys[i]; +if (!(bit & added_bits)) +continue ; +if (ss->root != &rootnode) { +/* Subsystem isn't free */ +return -EBUSY ; + } + } + +/* Currently we don't handle adding/removing subsystems when +* any subcontainers exist. This is theoretically supportable +* but involves complex erro r handling, so it's being left until +* later */ +if (!list_empty (&cont->children)) { +return -EBUSY ; + } + +/* Process each subsystem */ +for (i =0 ; i <CONTAINER_SUBSYS_COUNT ; i++) { +struct container_subsys *ss = subsys[i]; +unsigned long bit =1UL << i; +if (bit & added_bits) { +/* We're binding this subsystem to this hierarchy */ +BUG_ON (cont->subsys[i]); +BUG_ON (!dummytop->subsys[i]); +BUG_ON (dummytop->subsys[i]->container != dummytop); + cont->subsys[i] = dummytop->subsys[i]; + cont->subsys[i]->container = cont; +list_add (&ss->sibling, &root->subsys_list); +rcu_assign_pointer (ss->root, root); +if (ss->bind) + ss->bind (ss, cont); + + }else if (bit & removed_bits) { +/* We're removing this subsystem */ +BUG_ON (cont->subsys[i] != dummytop->subsys[i]); +BUG_ON (cont->subsys[i]->container != cont); +if (ss->bind) + ss->bind (ss, dummytop); + dummytop->subsys[i]->container = dummytop; + cont->subsys[i] =NULL ; +rcu_assign_pointer (subsys[i]->root, &rootnode); +list_del (&ss->sibling); + }else if (bit & final_bits) { +/* Subsystem state should already exist */ +BUG_ON (!cont->subsys[i]); + }else { +/* Subsystem state shouldn't exist */ +BUG_ON (cont->subsys[i]); + } + } + root->subsys_bits = final_bits; +synchronize_rcu (); + +return 0 ; +}
kernel/container.c line 399
Quoted 10 lines
kernel/container.c new file +9
-0+static int container_remount (struct super_block *sb,int *flags,char *data) +{ +int ret =0 ; +unsigned long subsys_bits; +struct containerfs_root *root = sb->s_fs_info; +struct container *cont = &root->top_container; + +mutex_lock (&cont->dentry->d_inode->i_mutex); +mutex_lock (&container_mutex);
So container_mutex nests inside i_mutex. That mean that we'll get lockdep moaning if anyone does a __GFP_FS allocation inside container_mutex (some filesystems can take i_mutex on the ->writepage path, iirc).
Probably a false positive, we can cross that bridege if/when we come to it.
Quoted 37 lines
kernel/container.c new file +34
-0+/* See what subsystems are wanted */ + ret =parse_containerfs_options (data, &subsys_bits); +if (ret) + goto out_unlock; + + ret =rebind_subsystems (root, subsys_bits); + +/* (re)populate subsystem files */ +if (!ret) +container_populate_dir (cont); + + out_unlock: +mutex_unlock (&container_mutex); +mutex_unlock (&cont->dentry->d_inode->i_mutex); +return ret; +} + + +static int container_fill_super (struct super_block *sb,void *options, +int unused_silent) +{ +struct inode *inode; +struct dentry *root; +struct containerfs_root *hroot = options; + + sb->s_blocksize =PAGE_CACHE_SIZE ; + sb->s_blocksize_bits =PAGE_CACHE_SHIFT ; + sb->s_magic =CONTAINER_SUPER_MAGIC ; + sb->s_op = &container_ops; + + inode =container_new_inode (S_IFDIR |S_IRUGO |S_IXUGO |S_IWUSR , sb); +if (!inode) +return -ENOMEM ; +
kernel/container.c line 519
Quoted 10 lines
kernel/container.c new file +10
-0+ inode->i_op = &simple_dir_inode_operations; + inode->i_fop = &simple_dir_operations; + inode->i_op = &container_dir_inode_operations; +/* directories start off with i_nlink == 2 (for "." entry) */ +inc_nlink (inode); + + root =d_alloc_root (inode); +if (!root) { +iput (inode); +return -ENOMEM ;
I bet that iput() hasn't been tested ;)
People have hit unpleasant problems before now running iput() against partially-constructed inodes.
Quoted 54 lines
kernel/container.c new file +51
-0+ } + sb->s_root = root; + root->d_fsdata = &hroot->top_container; + hroot->top_container.dentry = root; + + sb->s_fs_info = hroot; + hroot->sb = sb; + +return 0 ; +} + +static int container_get_sb (struct file_system_type *fs_type, +int flags,const char *unused_dev_name, +void *data,struct vfsmount *mnt) +{ +unsigned long subsys_bits =0 ; +int ret =0 ; +struct containerfs_root *root =NULL ; +int use_existing =0 ; + +mutex_lock (&container_mutex); + +/* First find the desired set of resource controllers */ + ret =parse_containerfs_options (data, &subsys_bits); +if (ret) + goto out_unlock; + +/* See if we already have a hierarchy containing this set */ + +for_each_root (root) { +/* We match - use this hieracrchy */ +if (root->subsys_bits == subsys_bits) { + use_existing =1 ; +break ; + } +/* We clash - fail */ +if (root->subsys_bits & subsys_bits) { + ret = -EBUSY ; + goto out_unlock; + } + } + +if (!use_existing) { +/* We need a new root */ + root =kzalloc (sizeof (*root),GFP_KERNEL ); +if (!root) { + ret = -ENOMEM ; + goto out_unlock; + } +init_container_root (root); + }
kernel/container.c line 592
Quoted 10 lines
kernel/container.c new file +10
-0+ +if (!root->sb) { +/* We need a new superblock for this container combination */ +struct container *cont = &root->top_container; + +BUG_ON (root->subsys_bits); + ret =get_sb_nodev (fs_type, flags, root, + container_fill_super, mnt); +if (ret) + goto out_unlock;
Did we just leak *root?
Quoted 33 lines
kernel/container.c new file +30
-0+BUG_ON (!list_empty (&cont->sibling)); +BUG_ON (!list_empty (&cont->children)); +BUG_ON (root->number_of_containers !=1 ); + + ret =rebind_subsystems (root, subsys_bits); + +/* It's safe to nest i_mutex inside container_mutex in +* this case, since no-one else can be accessing this +* directory yet */ +mutex_lock (&cont->dentry->d_inode->i_mutex); +container_populate_dir (cont); +mutex_unlock (&cont->dentry->d_inode->i_mutex); +BUG_ON (ret); + + }else { +/* Reuse the existing superblock */ + ret =simple_set_mnt (mnt, root->sb); +if (!ret) +atomic_inc (&root->sb->s_active); + } + + out_unlock: +mutex_unlock (&container_mutex); +return ret; +} +static inline void get_first_subsys (const struct container *cont, +struct container_subsys_state **css, +int *subsys_id) { +const struct containerfs_root *root = cont->root; +const struct container_subsys *test_ss;
kernel/container.c line 680
Quoted 10 lines
kernel/container.c new file +10
-0+BUG_ON (list_empty (&root->subsys_list)); + test_ss =list_entry (root->subsys_list.next, + struct container_subsys, sibling); +if (css) { + *css = cont->subsys[test_ss->subsys_id]; +BUG_ON (!*css); + } +if (subsys_id) + *subsys_id = test_ss->subsys_id; +}
This ends up having several callers and its too large to inline.
kernel/container.c line 688
Quoted 7 lines
kernel/container.c new file +7
-0+/* The various types of files and directories in a container file system */ + +typedef enum { +FILE_ROOT , +FILE_DIR , +FILE_TASKLIST , +}container_filetype_t ;
Another typedef!
kernel/container.c line 778
Quoted 7 lines
kernel/container.c new file +7
-0+static struct file_operations container_file_operations = { + .read = container_file_read, + .write = container_file_write, + .llseek = generic_file_llseek, + .open = container_file_open, + .release = container_file_release, +};
Do we actually want to support lseek on these things?
If not we can leave this null and use nonseekable_open() in ->open.
Quoted 17 lines
kernel/container.c new file +17
-0+static int container_create_file (struct dentry *dentry,int mode,struct super_block *sb) +{ +struct inode *inode; + +if (!dentry) +return -ENOENT ; +if (dentry->d_inode) +return -EEXIST ; + + inode =container_new_inode (mode, sb); +if (!inode) +return -ENOMEM ; + +if (S_ISDIR (mode)) { + inode->i_op = &container_dir_inode_operations; + inode->i_fop = &simple_dir_operations; +
kernel/container.c line 813
Quoted 10 lines
kernel/container.c new file +10
-0+/* start off with i_nlink == 2 (for "." entry) */ +inc_nlink (inode); + +/* start with the directory inode held, so that we can +* populate it without racing with another mkdir */ +mutex_lock (&inode->i_mutex); + }else if (S_ISREG (mode)) { + inode->i_size =0 ; + inode->i_fop = &container_file_operations; + }
The S_ISREG files have no ->i_ops?
kernel/container.c line 827
Quoted 13 lines
kernel/container.c new file +13
-0+d_instantiate (dentry, inode); +dget (dentry);/* Extra count - pin the dentry in core */ +return 0 ; +} + +/* +* container_create_dir - create a directory for an object. +* cont: the container we create the directory for. +* It must have a valid ->parent field +* And we are going to fill its ->dentry field. +* name: The name to give to the container directory. Will be copied. +* mode: mode to set on new directory. +*/
This comment is almost kernel-doc-like, but not quite.
There doesn't seem much point in converting it, as it'd be pretty much the only kernel-doc think in there.
Quoted 20 lines
kernel/container.c new file +19
-0+static int container_create_dir (struct container *cont,struct dentry *dentry, +int mode) +{ +struct dentry *parent; +int error =0 ; + + parent = cont->parent->dentry; +if (IS_ERR (dentry)) +return PTR_ERR (dentry); + error =container_create_file (dentry,S_IFDIR | mode, cont->root->sb); +if (!error) { + dentry->d_fsdata = cont; +inc_nlink (parent->d_inode); + cont->dentry = dentry; + } +dput (dentry); + +return error; +}
kernel/container.c line 909
Quoted 10 lines
kernel/container.c new file +8
-0+/* +* container_create - create a container +* parent: container that will be parent of the new container. +* name: name of the new container. Will be strcpy'ed. +* mode: mode to set on new inode +* +* Must be called with the mutex on the parent inode held +*/
OK, there's another.
Re: [PATCH 01/10] Containers(V10): Basic container framework
Quoted 2 lines · Andrew Morton
Holy cow, do we need all those?
I'll experiment to see which ones we can get rid of.
Quoted 7 lines
kernel/container.c line 118
Quoted 4 lines
kernel/container.c new file +3
-0+typedef enum { +CONT_REMOVED , +}container_flagbits_t ;typedefs are verboten. Fortunately this one is never referred to - only the values are used, so we can delete it.
OK.
Quoted 3 lines
Taking
dcache_lockin here is unfortunate. A filesystem really shouldn't be playing with that lock.
Is there a recommended way to do what I want to do, i.e. clear out all the dentries from a virtual fs directory and rebuild them whilst holding the directory's i_sem so no one can see the transiently empty directory?
Quoted 2 lines
The code's a bit short on comments.
I'll add some.
Quoted 6 lines
kernel/container.c line 519
Quoted 4 lines
kernel/container.c new file +4
-0+ root =d_alloc_root (inode); +if (!root) { +iput (inode); +return -ENOMEM ;I bet that
iput()hasn't been tested ;)
Correct.
Quoted 3 lines
People have hit unpleasant problems before now running
iput()against partially-constructed inodes.
What kinds of problems? Are there bits of state that I should fully construct even if I'm going to iput() it, or is there a better function to call? fs/ext3/super.c seems to do the same thing.
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (ret) + goto out_unlock;Did we just leak *root?
I believe we did. I'll fix that.
Quoted 10 lines
Quoted 10 lines
kernel/container.c new file +9
-0+static inline void get_first_subsys (const struct container *cont, +struct container_subsys_state **css, +int *subsys_id) { +const struct containerfs_root *root = cont->root; +const struct container_subsys *test_ss; +BUG_ON (list_empty (&root->subsys_list)); + test_ss =list_entry (root->subsys_list.next, + struct container_subsys, sibling); +if (css) {
Quoted 8 lines
kernel/container.c line 680
Quoted 6 lines
kernel/container.c new file +6
-0+ *css = cont->subsys[test_ss->subsys_id]; +BUG_ON (!*css); + } +if (subsys_id) + *subsys_id = test_ss->subsys_id; +}This ends up having several callers and its too large to inline.
Two large from a compiler PoV or from a style PoV? It's basically just six dereferences and two comparisons, plus the BUG_ON()s.
Quoted 4 lines
Do we actually want to support lseek on these things?
If not we can leave this null and use
nonseekable_open()in ->open.
I inherited that from cpusets without thinking about it too much. I guess that we don't really need seekability.
Quoted 6 lines
kernel/container.c line 813
Quoted 4 lines
kernel/container.c new file +4
-0+ }else if (S_ISREG (mode)) { + inode->i_size =0 ; + inode->i_fop = &container_file_operations; + }The
S_ISREGfiles have no ->i_ops?
Not currently. I don't see anything in inode_operations that we want to be able to do on non-directories.
Paul
Andrew Morton
maintainerRe: [PATCH 01/10] Containers(V10): Basic container framework
Quoted 8 lines · On Wed, 30 May 2007 07:02:00 -0700 "Paul Menage" <[email protected]> wrote:
Quoted 4 lines
People have hit unpleasant problems before now running
iput()against partially-constructed inodes.What kinds of problems? Are there bits of state that I should fully construct even if I'm going to
iput()it, or is there a better function to call?fs/ext3/super.cseems to do the same thing.
I don't recall, actually. But it crashed.
I guess the fault-injection code could be used to trigger errors here.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
kernel/container.c new file +9
-0+static inline void get_first_subsys (const struct container *cont, +struct container_subsys_state **css, +int *subsys_id) { +const struct containerfs_root *root = cont->root; +const struct container_subsys *test_ss; +BUG_ON (list_empty (&root->subsys_list)); + test_ss =list_entry (root->subsys_list.next, + struct container_subsys, sibling); +if (css) {Quoted 8 lines
kernel/container.c line 680
Quoted 6 lines
kernel/container.c new file +6
-0+ *css = cont->subsys[test_ss->subsys_id]; +BUG_ON (!*css); + } +if (subsys_id) + *subsys_id = test_ss->subsys_id; +}This ends up having several callers and its too large to inline.
Quoted 2 lines
Two large from a compiler PoV or from a style PoV? It's basically just six dereferences and two comparisons, plus the BUG_ON()s.
It will end up generating more .text this way. We figure that this makes it slower, due to increased icache footprint.
Re: [PATCH 01/10] Containers(V10): Basic container framework
In reply to [email protected]Hi,
Quoted 13 lines · On Tue, May 29, 2007 at 06:01:05AM -0700, [email protected] wrote:
+1.5 How do I use containers ?
+--------------------------(quoted diff) +11
-0+ +To start a new job that is to be contained within a container, using +the "cpuset" container subsystem, the steps are something like: + + 1) mkdir /dev/container + 2) mount -t container -ocpuset cpuset /dev/container + 3) Create the new container by doing mkdir's and write's (or echo's) in + the /dev/container virtual file system. + 4) Start a task that will be the "founding father" of the new job. + 5) Attach that task to the new container by writing its pid to the + /dev/container tasks file for that container.
Review
Quoted 10 lines
(quoted diff) +10
-0+ 6) fork, exec or clone the job tasks from this founding father task. + +For example, the following sequence of commands will setup a container +named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, +and then start a subshell 'sh' in that container: + + mount -t container cpuset -ocpuset /dev/container + cd /dev/container + mkdir Charlie + cd Charlie
This example does not work. To do so we need to do
/bin/echo 2-3 > cpus
/bin/echo 1 > mems
Review
Quoted 5 lines
(quoted diff) +5
-0+ /bin/echo $$ > tasks + sh + # The subshell 'sh' is now running in container Charlie + # The next line should display '/Charlie' + cat /proc/self/container
The following patch does that.
thanks and regards
Dhaval
----------------------
Documentation/containers.txt +2 -0
@@ -310,6 +310,8 @@ and then start a subshell 'sh' in that c cd /dev/container mkdir Charlie cd Charlie + /bin/echo 2-3 > cpus + /bin/echo 1 > mems /bin/echo $$ > tasks sh # The subshell 'sh' is now running in container Charlie
PATCH 2/10 Containers(V10): Example CPU accounting subsystem
In reply to [email protected]This example demonstrates how to use the generic container subsystem for a simple resource tracker that counts, for the processes in a container, the total CPU time used and the %CPU used in the last complete 10 second interval.
Portions contributed by Balbir Singh <[email protected]>
--- include/linux/container_subsys.h | 6 + include/linux/cpu_acct.h | 14 ++ init/Kconfig | 7 + kernel/Makefile | 1 kernel/cpu_acct.c | 185 +++++++++++++++++++++++++++++++++++++++ kernel/sched.c | 14 ++ 6 files changed, 224 insertions(+), 3 deletions(-)
include/linux/container_subsys.h +6 -0
@@ -7,4 +7,10 @@/* */ +#ifdef CONFIG_CONTAINER_CPUACCT +SUBSYS(cpuacct) +#endif + +/* */ +/* */
include/linux/cpu_acct.h new file +14 -0
@@ -0,0 +1,14 @@ + +#ifndef _LINUX_CPU_ACCT_H +#define _LINUX_CPU_ACCT_H + +#include <linux/container.h> +#include <asm/cputime.h> + +#ifdef CONFIG_CONTAINER_CPUACCT +extern void cpuacct_charge (struct task_struct *,cputime_t cputime); +#else +static void inline cpuacct_charge (struct task_struct *p,cputime_t cputime) {} +#endif + +#endif
init/Kconfig +7 -0
@@ -337,6 +337,13 @@ config SYSFS_DEPRECATED If you are using a distro that was released in 2006 or later, it should be safe to say N here. +config CONTAINER_CPUACCT + bool "Simple CPU accounting container subsystem" + select CONTAINERS + help + Provides a simple Resource Controller for monitoring the + total CPU consumed by the tasks in a container + config RELAY bool "Kernel->user space relay support (formerly relayfs)" help
kernel/Makefile +1 -0
@@ -38,6 +38,7 @@ obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_COMPAT) += compat.o obj-$(CONFIG_CONTAINERS) += container.o obj-$(CONFIG_CPUSETS) += cpuset.o +obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o obj-$(CONFIG_IKCONFIG) += configs.o obj-$(CONFIG_STOP_MACHINE) += stop_machine.o obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
kernel/cpu_acct.c new file +185 -0
185 lines not on this page · raw message
kernel/sched.c +11 -3
@@ -59,6 +59,7 @@#include <linux/kprobes.h> #include <linux/delayacct.h> #include <linux/reciprocal_div.h> +#include <linux/cpu_acct.h> #include <asm/tlb.h> #include <asm/unistd.h>
@@ -2847,9 +2848,13 @@ void account_user_time(struct task_struc {struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;cputime64_t tmp; +struct rq *rq =this_rq (); p->utime =cputime_add (p->utime, cputime); +if (p != rq->idle) +cpuacct_charge (p, cputime); +/* Add user time to cpustat. */ tmp =cputime_to_cputime64 (cputime);if (TASK_NICE (p) >0 )
@@ -2879,9 +2884,10 @@ void account_system_time(struct task_str cpustat->irq =cputime64_add (cpustat->irq, tmp); else if (softirq_count()) cpustat->softirq = cputime64_add(cpustat->softirq, tmp); - else if (p != rq->idle) + else if (p != rq->idle) { cpustat->system =cputime64_add (cpustat->system, tmp); - else if (atomic_read(&rq->nr_iowait) > 0) +cpuacct_charge (p, cputime); + } else if (atomic_read(&rq->nr_iowait) > 0) cpustat->iowait =cputime64_add (cpustat->iowait, tmp); else cpustat->idle = cputime64_add(cpustat->idle, tmp);
@@ -2906,8 +2912,10 @@ void account_steal_time(struct task_stru cpustat->iowait =cputime64_add (cpustat->iowait, tmp); else cpustat->idle = cputime64_add(cpustat->idle, tmp); - }else + }else { cpustat->steal =cputime64_add (cpustat->steal, tmp); +cpuacct_charge (p, -tmp); + } }/*
--
Andrew Morton
maintainerRe: [PATCH 02/10] Containers(V10): Example CPU accounting subsystem
Quoted 26 lines · On Tue, 29 May 2007 06:01:06 -0700 [email protected] wrote:
This example demonstrates how to use the generic container subsystem for a simple resource tracker that counts, for the processes in a container, the total CPU time used and the %CPU used in the last complete 10 second interval.
...
kernel/cpu_acct.c new file +13
-0@@ -0,0 +1,185 @@ +/* +* kernel/cpu_acct.c - CPU accounting container subsystem +* +* Copyright (C) Google Inc, 2006 +* +* Developed by Paul Menage ([email protected]) and Balbir Singh +* ([email protected]) +* +*/ +static u64 cpuusage_read (struct container *cont, +struct cftype *cft) +{ +struct cpuacct *ca =container_ca (cont);
kernel/cpu_acct.c
Quoted 10 lines
kernel/cpu_acct.c new file +10
-0+u64 time; + +spin_lock_irq (&ca->lock); +cpuusage_update (ca); + time =cputime64_to_jiffies64 (ca->time); +spin_unlock_irq (&ca->lock); + +/* Convert 64-bit jiffies to seconds */ + time *=1000 ; +do_div (time,HZ );
hm, we have jiffies_to_lotsofthings, but we don't appear to have a jiffies_to_seconds. How odd.
PATCH 3/10 Containers(V10): Add tasks file interface
In reply to [email protected]This patch adds the per-directory "tasks" file for containerfs mounts; this allows the user to determine which tasks are members of a container by reading a container's "tasks", and to move a task into a container by writing its pid to its "tasks".
--- include/linux/container.h | 10 + kernel/container.c | 335 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 345 insertions(+)
include/linux/container.h +10 -0
@@ -128,6 +128,16 @@ int container_is_removed(const struct coint container_path (const struct container *cont,char *buf,int buflen); +int __container_task_count (const struct container *cont); +static inline int container_task_count (const struct container *cont) +{ +int task_count; +rcu_read_lock (); + task_count =__container_task_count (cont); +rcu_read_unlock (); +return task_count; +} +/* Return true if the container is a descendant of the current container */ int container_is_descendant (const struct container *cont);
kernel/container.c +335 -0
359 lines not on this page · raw message
--
Re: [PATCH 03/10] Containers(V10): Add tasks file interface
Hello Paul !
Quoted 4 lines · [email protected] wrote:
This patch adds the per-directory "tasks" file for containerfs mounts; this allows the user to determine which tasks are members of a container by reading a container's "tasks", and to move a task into a container by writing its pid to its "tasks".
here's a small fix for 2.6.22-rc4-mm2.
C.
when there's no tasks in a container, opening
<containerfs>/<container>/tasks
spits the following warning because we are trying to kmalloc(0).
WARNING: at mm/slab.c:777 __find_general_cachep() [<c0102b01>] show_trace_log_lvl+0x1a/0x2f [<c0103627>] show_trace+0x12/0x14 [<c010363e>] dump_stack+0x15/0x17 [<c0148040>] __kmalloc+0x56/0xf3 [<c012d030>] container_tasks_open+0x56/0x131 [<c012cdf8>] container_file_open+0x32/0x3a [<c014964d>] __dentry_open+0x99/0x13c [<c0149771>] nameidata_to_filp+0x27/0x37 [<c01497b4>] do_filp_open+0x33/0x3b [<c0149801>] do_sys_open+0x45/0xc9 [<c01498bd>] sys_open+0x1c/0x1e [<c0102540>] syscall_call+0x7/0xb
--- kernel/container.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-)
kernel/container.c +19 -14
@@ -1651,21 +1651,26 @@ static int container_tasks_open(struct i* show up until sometime later on. */ npids =container_task_count (cont); - pidarray =kmalloc (npids *sizeof (pid_t),GFP_KERNEL ); -if (!pidarray) - goto err1; - - npids =pid_array_load (pidarray, npids, cont); -sort (pidarray, npids,sizeof (pid_t), cmppid,NULL ); - -/* Call pid_array_to_buf() twice, first just to get bufsz */ - ctr->bufsz =pid_array_to_buf (&c,sizeof (c), pidarray, npids) +1 ; - ctr->buf =kmalloc (ctr->bufsz,GFP_KERNEL ); -if (!ctr->buf) - goto err2; - ctr->bufsz =pid_array_to_buf (ctr->buf, ctr->bufsz, pidarray, npids); +if (npids) { + pidarray =kmalloc (npids *sizeof (pid_t),GFP_KERNEL ); +if (!pidarray) + goto err1; + + npids =pid_array_load (pidarray, npids, cont); +sort (pidarray, npids,sizeof (pid_t), cmppid,NULL ); + +/* Call pid_array_to_buf() twice, first just to get bufsz */ + ctr->bufsz =pid_array_to_buf (&c,sizeof (c), pidarray, npids) +1 ; + ctr->buf =kmalloc (ctr->bufsz,GFP_KERNEL ); +if (!ctr->buf) + goto err2; + ctr->bufsz =pid_array_to_buf (ctr->buf, ctr->bufsz, pidarray, npids); -kfree (pidarray); +kfree (pidarray); + }else { + ctr->buf =0 ; + ctr->bufsz =0 ; + } file->private_data = ctr;return 0 ;
Re: [PATCH 03/10] Containers(V10): Add tasks file interface
Quoted 7 lines · Cedric Le Goater
when there's no tasks in a container, opening
<containerfs>/<container>/tasks
spits the following warning because we are trying to kmalloc(0).
I guess I'm not opposed to this change - but isn't there still discussion going on about whether kmalloc(0) should actually produce a warning or not?
Thanks,
Paul
PATCH 4/10 Containers(V10): Add fork/exit hooks
In reply to [email protected]This patch adds the necessary hooks to the fork() and exit() paths to ensure that new children inherit their parent's container assignments, and that exiting processes release reference counts on their containers.
--- include/linux/container.h | 6 ++ kernel/container.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/exit.c | 2 kernel/fork.c | 14 ++++- 4 files changed, 148 insertions(+), 2 deletions(-)
kernel/exit.c +2 -0
@@ -32,6 +32,7 @@#include <linux/taskstats_kern.h> #include <linux/delayacct.h> #include <linux/cpuset.h> +#include <linux/container.h> #include <linux/syscalls.h> #include <linux/signal.h> #include <linux/posix-timers.h>
@@ -935,6 +936,7 @@ fastcall void do_exit(long code)__exit_fs (tsk);exit_thread ();cpuset_exit (tsk); +container_exit (tsk,1 );exit_keys (tsk);if (group_dead && tsk->signal->leader)
kernel/fork.c +12 -2
@@ -30,6 +30,7 @@#include <linux/capability.h> #include <linux/cpu.h> #include <linux/cpuset.h> +#include <linux/container.h> #include <linux/security.h> #include <linux/swap.h> #include <linux/syscalls.h>
@@ -963,6 +964,7 @@ static struct task_struct *copy_process( {int retval;struct task_struct *p =NULL ; +int container_callbacks_done =0 ;if ((clone_flags & (CLONE_NEWNS |CLONE_FS )) == (CLONE_NEWNS |CLONE_FS ))return ERR_PTR (-EINVAL );
@@ -1063,12 +1065,13 @@ static struct task_struct *copy_process( p->io_wait =NULL ; p->audit_context =NULL ;cpuset_fork (p); +container_fork (p);#ifdef CONFIG_NUMA p->mempolicy =mpol_copy (p->mempolicy);if (IS_ERR (p->mempolicy)) { retval =PTR_ERR (p->mempolicy); p->mempolicy =NULL ; - goto bad_fork_cleanup_cpuset; + goto bad_fork_cleanup_container; }mpol_fix_fork_child_flag (p);#endif
@@ -1178,6 +1181,12 @@ static struct task_struct *copy_process(/* Perform scheduler related setup. Assign this task to a CPU. */ sched_fork (p, clone_flags); +/* Now that the task is set up, run container callbacks if +* necessary. We need to run them before the task is visible +* on the tasklist. */ +container_fork_callbacks (p); + container_callbacks_done =1 ; +/* Need tasklist lock for parent etc handling! */ write_lock_irq (&tasklist_lock);
@@ -1300,9 +1309,10 @@ bad_fork_cleanup_security: bad_fork_cleanup_policy:#ifdef CONFIG_NUMA mpol_free (p->mempolicy); -bad_fork_cleanup_cpuset: +bad_fork_cleanup_container:#endif cpuset_exit (p); +container_exit (p, container_callbacks_done);delayacct_tsk_free (p);if (p->binfmt)module_put (p->binfmt->module);
include/linux/container.h +6 -0
@@ -18,6 +18,9 @@extern int container_init_early (void );extern int container_init (void );extern void container_init_smp (void ); +extern void container_fork (struct task_struct *p); +extern void container_fork_callbacks (struct task_struct *p); +extern void container_exit (struct task_struct *p,int run_callbacks);extern struct file_operations proc_container_operations;
@@ -199,6 +202,9 @@ int container_path(const struct containestatic inline int container_init_early (void ) {return 0 ; }static inline int container_init (void ) {return 0 ; }static inline void container_init_smp (void ) {} +static inline void container_fork (struct task_struct *p) {} +static inline void container_fork_callbacks (struct task_struct *p) {} +static inline void container_exit (struct task_struct *p,int callbacks) {}static inline void container_lock (void ) {}static inline void container_unlock (void ) {}
kernel/container.c +128 -0
137 lines not on this page · raw message
--
PATCH 5/10 Containers(V10): Add container_clone() interface
In reply to [email protected]This patch adds support for container_clone(), a speculative interface to creating new containers intended to be used for systems such as namespace unsharing.
--- include/linux/container.h | 2 kernel/container.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+)
kernel/container.c +123 -0
@@ -1616,3 +1616,126 @@ void container_exit(struct task_struct * tsk->containers = init_task.containers;task_unlock (tsk); } + +static atomic_t namecnt; +static void get_unused_name (char *buf) +{ +sprintf (buf,"node%d" ,atomic_inc_return (&namecnt)); +} + +/** +* container_clone - duplicate the current container in the hierarchy +* that the given subsystem is attached to, and move this task into +* the new child +*/ +int container_clone (struct task_struct *tsk,struct container_subsys *subsys) +{ +struct dentry *dentry; +int ret =0 ; +char nodename[32 ]; +struct container *parent, *child; +struct inode *inode; +struct css_group *cg; +struct containerfs_root *root; + +/* We shouldn't be called by an unregistered subsystem */ +BUG_ON (!subsys->active); + +/* First figure out what hierarchy and container we're dealing +* with, and pin them so we can drop container_mutex */ +mutex_lock (&container_mutex); + again: + root = subsys->root; +if (root == &rootnode) { +printk (KERN_INFO +"Not cloning container for unused subsystem %s\n" , + subsys->name); +mutex_unlock (&container_mutex); +return 0 ; + } + cg = &tsk->containers; + parent =task_container (tsk, subsys->subsys_id); +/* Pin the hierarchy */ +atomic_inc (&parent->root->sb->s_active); + +mutex_unlock (&container_mutex); + +/* Now do the VFS work to create a container */ +get_unused_name (nodename); + inode = parent->dentry->d_inode; + +/* Hold the parent directory mutex across this operation to +* stop anyone else deleting the new container */ +mutex_lock (&inode->i_mutex); + dentry =container_get_dentry (parent->dentry, nodename); +if (IS_ERR (dentry)) { +printk (KERN_INFO +"Couldn't allocate dentry for %s: %ld\n" , nodename, +PTR_ERR (dentry)); + ret =PTR_ERR (dentry); + goto out_release; + } + +/* Create the container directory, which also creates the container */ + ret =vfs_mkdir (inode, dentry,S_IFDIR |0755 ); + child =__d_cont (dentry); +dput (dentry); +if (ret) { +printk (KERN_INFO +"Failed to create container %s: %d\n" , nodename, + ret); + goto out_release; + } + +if (!child) { +printk (KERN_INFO +"Couldn't find new container %s\n" , nodename); + ret = -ENOMEM ; + goto out_release; + } + +/* The container now exists. Retake container_mutex and check +* that we're still in the same state that we thought we +* were. */ +mutex_lock (&container_mutex); +if ((root != subsys->root) || + (parent !=task_container (tsk, subsys->subsys_id))) { +/* Aargh, we raced ... */ +mutex_unlock (&inode->i_mutex); + +deactivate_super (parent->root->sb); +/* The container is still accessible in the VFS, but +* we're not going to try to rmdir() it at this +* point. */ +printk (KERN_INFO +"Race in container_clone() - leaking container %s\n" , + nodename); + goto again; + } + +/* All seems fine. Finish by moving the task into the new container */ + ret =attach_task (child, tsk); +mutex_unlock (&container_mutex); + + out_release: +mutex_unlock (&inode->i_mutex); +deactivate_super (parent->root->sb); +return ret; +} + +/* See if "cont" is a descendant of the current task's container in +* the appropriate hierarchy */ + +int container_is_descendant (const struct container *cont) +{ +int ret; +struct container *target; +int subsys_id; +get_first_subsys (cont,NULL , &subsys_id); + target =task_container (current, subsys_id); +while (cont != target && cont!= cont->top_container) { + cont = cont->parent; + } + ret = (cont == target); +return ret; +}
include/linux/container.h +2 -0
@@ -197,6 +197,8 @@ static inline struct container* task_conint container_path (const struct container *cont,char *buf,int buflen); +int container_clone (struct task_struct *tsk,struct container_subsys *ss); +#else /* !CONFIG_CONTAINERS */ static inline int container_init_early (void ) {return 0 ; }
--
Andrew Morton
maintainerRe: [PATCH 05/10] Containers(V10): Add container_clone() interface
kernel/container.c void container_exit(struct task_struct * · line 1624
Quoted 12 lines · On Tue, 29 May 2007 06:01:09 -0700 [email protected] wrote:
This patch adds support for
container_clone(), a speculative interface to creating new containers intended to be used for systems such as namespace unsharing....
kernel/container.c +6
-0+ +static atomic_t namecnt; +static void get_unused_name (char *buf) +{ +sprintf (buf,"node%d" ,atomic_inc_return (&namecnt)); +}
A stupid thing, but a sufficiently determined attacker could cause this to wrap.
Quoted 41 lines
kernel/container.c +41
-0+/** +* container_clone - duplicate the current container in the hierarchy +* that the given subsystem is attached to, and move this task into +* the new child +*/ +int container_clone (struct task_struct *tsk,struct container_subsys *subsys) +{ +struct dentry *dentry; +int ret =0 ; +char nodename[32 ]; +struct container *parent, *child; +struct inode *inode; +struct css_group *cg; +struct containerfs_root *root; + +/* We shouldn't be called by an unregistered subsystem */ +BUG_ON (!subsys->active); + +/* First figure out what hierarchy and container we're dealing +* with, and pin them so we can drop container_mutex */ +mutex_lock (&container_mutex); + again: + root = subsys->root; +if (root == &rootnode) { +printk (KERN_INFO +"Not cloning container for unused subsystem %s\n" , + subsys->name); +mutex_unlock (&container_mutex); +return 0 ; + } + cg = &tsk->containers; + parent =task_container (tsk, subsys->subsys_id); +/* Pin the hierarchy */ +atomic_inc (&parent->root->sb->s_active); + +mutex_unlock (&container_mutex); + +/* Now do the VFS work to create a container */ +get_unused_name (nodename); + inode = parent->dentry->d_inode; +
kernel/container.c void container_exit(struct task_struct * · line 1676
Quoted 10 lines
kernel/container.c +10
-0+/* Hold the parent directory mutex across this operation to +* stop anyone else deleting the new container */ +mutex_lock (&inode->i_mutex); + dentry =container_get_dentry (parent->dentry, nodename); +if (IS_ERR (dentry)) { +printk (KERN_INFO +"Couldn't allocate dentry for %s: %ld\n" , nodename, +PTR_ERR (dentry)); + ret =PTR_ERR (dentry); + goto out_release;
Which I guess could cause a failure here.
Perhaps we could go back and rerun the get_unused_name() if EEXIST, if we're bothered.
I hope this is the biggest problem ;)
Quoted 48 lines
kernel/container.c +48
-0+ } + +/* Create the container directory, which also creates the container */ + ret =vfs_mkdir (inode, dentry,S_IFDIR |0755 ); + child =__d_cont (dentry); +dput (dentry); +if (ret) { +printk (KERN_INFO +"Failed to create container %s: %d\n" , nodename, + ret); + goto out_release; + } + +if (!child) { +printk (KERN_INFO +"Couldn't find new container %s\n" , nodename); + ret = -ENOMEM ; + goto out_release; + } + +/* The container now exists. Retake container_mutex and check +* that we're still in the same state that we thought we +* were. */ +mutex_lock (&container_mutex); +if ((root != subsys->root) || + (parent !=task_container (tsk, subsys->subsys_id))) { +/* Aargh, we raced ... */ +mutex_unlock (&inode->i_mutex); + +deactivate_super (parent->root->sb); +/* The container is still accessible in the VFS, but +* we're not going to try to rmdir() it at this +* point. */ +printk (KERN_INFO +"Race in container_clone() - leaking container %s\n" , + nodename); + goto again; + } + +/* All seems fine. Finish by moving the task into the new container */ + ret =attach_task (child, tsk); +mutex_unlock (&container_mutex); + + out_release: +mutex_unlock (&inode->i_mutex); +deactivate_super (parent->root->sb); +return ret; +}
Re: [PATCH 05/10] Containers(V10): Add container_clone() interface
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Tue, 29 May 2007 06:01:09 -0700 [email protected] wrote:
This patch adds support for
container_clone(), a speculative interface to creating new containers intended to be used for systems such as namespace unsharing....
Quoted 9 lines
kernel/container.c void container_exit(struct task_struct * · line 1624
Quoted 6 lines
kernel/container.c +6
-0+ +static atomic_t namecnt; +static void get_unused_name (char *buf) +{ +sprintf (buf,"node%d" ,atomic_inc_return (&namecnt)); +}A stupid thing, but a sufficiently determined attacker could cause this to wrap.
Yeah, this was very consciously done as a "just make it work for now" naming system. If we want to stick with this naming, then I suppose we could do a global bitmap.
But imo this naming is not very convenient - it would be nicer if we
a) allowed users to specify a name (not sure how that would work logistically)
b) made the namecnt variable for automatically named containers be per-directory. I'd much rather see
/containers/node1/node1
/containers/node2
than
/containers/node1/node3
/containers/node2
(assuming /node2 was created between /node1 and /node1/node1 or /node1/node3)
thanks,
-serge
PATCH 6/10 Containers(V10): Add procfs interface
In reply to [email protected]This patch adds:
/proc/containers - general system info
/proc/*/container - per-task container membership info
--- fs/proc/base.c | 7 ++ kernel/container.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+)
fs/proc/base.c +7 -0
@@ -68,6 +68,7 @@#include <linux/security.h> #include <linux/ptrace.h> #include <linux/seccomp.h> +#include <linux/container.h> #include <linux/cpuset.h> #include <linux/audit.h> #include <linux/poll.h>
@@ -2028,6 +2029,9 @@ static const struct pid_entry tgid_base_#ifdef CONFIG_CPUSETS REG("cpuset", S_IRUGO, cpuset),#endif +#ifdef CONFIG_CONTAINERS +REG ("container" ,S_IRUGO , container), +#endifINF ("oom_score" ,S_IRUGO , oom_score),REG ("oom_adj" ,S_IRUGO |S_IWUSR , oom_adjust), #ifdef CONFIG_AUDITSYSCALL
@@ -2319,6 +2323,9 @@ static const struct pid_entry tid_base_s#ifdef CONFIG_CPUSETS REG("cpuset", S_IRUGO, cpuset),#endif +#ifdef CONFIG_CONTAINERS +REG ("container" ,S_IRUGO , container), +#endifINF ("oom_score" ,S_IRUGO , oom_score),REG ("oom_adj" ,S_IRUGO |S_IWUSR , oom_adjust), #ifdef CONFIG_AUDITSYSCALL
kernel/container.c +128 -0
150 lines not on this page · raw message
--
PATCH 7/10 Containers(V10): Make cpusets a client of containers
In reply to [email protected]This patch removes the filesystem support logic from the cpusets system and makes cpusets a container subsystem
--- Documentation/cpusets.txt | 91 +-- fs/proc/base.c | 4 include/linux/container_subsys.h | 6 include/linux/cpuset.h | 12 include/linux/mempolicy.h | 12 include/linux/sched.h | 3 init/Kconfig | 6 kernel/cpuset.c | 1151 +++++---------------------------------- kernel/exit.c | 2 kernel/fork.c | 3 mm/mempolicy.c | 2 11 files changed, 241 insertions(+), 1051 deletions(-)
Documentation/cpusets.txt +42 -49
@@ -7,6 +7,7 @@ Written by [email protected] Portions Copyright (c) 2004-2006 Silicon Graphics, Inc. Modified by Paul Jackson <[email protected]> Modified by Christoph Lameter <[email protected]> +Modified by Paul Menage <[email protected]> CONTENTS: =========
@@ -16,10 +17,9 @@ CONTENTS: 1.2 Why are cpusets needed ? 1.3 How are cpusets implemented ? 1.4 What are exclusive cpusets ? - 1.5 What does notify_on_release do ? - 1.6 What is memory_pressure ? - 1.7 What is memory spread ? - 1.8 How do I use cpusets ? + 1.5 What is memory_pressure ? + 1.6 What is memory spread ? + 1.7 How do I use cpusets ? 2. Usage Examples and Syntax 2.1 Basic Usage 2.2 Adding/removing cpus
@@ -43,18 +43,19 @@ hierarchy visible in a virtual file syst hooks, beyond what is already present, required to manage dynamic job placement on large systems. -Each task has a pointer to a cpuset. Multiple tasks may reference -the same cpuset. Requests by a task, using the sched_setaffinity(2) -system call to include CPUs in its CPU affinity mask, and using the -mbind(2) and set_mempolicy(2) system calls to include Memory Nodes -in its memory policy, are both filtered through that tasks cpuset, -filtering out any CPUs or Memory Nodes not in that cpuset. The -scheduler will not schedule a task on a CPU that is not allowed in -its cpus_allowed vector, and the kernel page allocator will not -allocate a page on a node that is not allowed in the requesting tasks -mems_allowed vector. +Cpusets use the generic container subsystem described in +Documentation/container.txt. -User level code may create and destroy cpusets by name in the cpuset +Requests by a task, using the sched_setaffinity(2) system call to +include CPUs in its CPU affinity mask, and using the mbind(2) and +set_mempolicy(2) system calls to include Memory Nodes in its memory +policy, are both filtered through that tasks cpuset, filtering out any +CPUs or Memory Nodes not in that cpuset. The scheduler will not +schedule a task on a CPU that is not allowed in its cpus_allowed +vector, and the kernel page allocator will not allocate a page on a +node that is not allowed in the requesting tasks mems_allowed vector. + +User level code may create and destroy cpusets by name in the container virtual file system, manage the attributes and permissions of these cpusets and which CPUs and Memory Nodes are assigned to each cpuset, specify and query to which cpuset a task is assigned, and list the
@@ -114,7 +115,7 @@ Cpusets extends these two mechanisms as - Cpusets are sets of allowed CPUs and Memory Nodes, known to the kernel. - Each task in the system is attached to a cpuset, via a pointer - in the task structure to a reference counted cpuset structure. + in the task structure to a reference counted container structure. - Calls to sched_setaffinity are filtered to just those CPUs allowed in that tasks cpuset. - Calls to mbind and set_mempolicy are filtered to just
@@ -144,15 +145,10 @@ into the rest of the kernel, none in per - in page_alloc.c, to restrict memory to allowed nodes. - in vmscan.c, to restrict page recovery to the current cpuset. -In addition a new file system, of type "cpuset" may be mounted, -typically at /dev/cpuset, to enable browsing and modifying the cpusets -presently known to the kernel. No new system calls are added for -cpusets - all support for querying and modifying cpusets is via -this cpuset file system. - -Each task under /proc has an added file named 'cpuset', displaying -the cpuset name, as the path relative to the root of the cpuset file -system. +You should mount the "container" filesystem type in order to enable +browsing and modifying the cpusets presently known to the kernel. No +new system calls are added for cpusets - all support for querying and +modifying cpusets is via this cpuset file system. The /proc/<pid>/status file for each task has two added lines, displaying the tasks cpus_allowed (on which CPUs it may be scheduled)
@@ -162,16 +158,15 @@ in the format seen in the following exam Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff Mems_allowed: ffffffff,ffffffff -Each cpuset is represented by a directory in the cpuset file system -containing the following files describing that cpuset: +Each cpuset is represented by a directory in the container file system +containing (on top of the standard container files) the following +files describing that cpuset: - cpus: list of CPUs in that cpuset - mems: list of Memory Nodes in that cpuset - memory_migrate flag: if set, move pages to cpusets nodes - cpu_exclusive flag: is cpu placement exclusive? - mem_exclusive flag: is memory placement exclusive? - - tasks: list of tasks (by pid) attached to that cpuset - - notify_on_release flag: run /sbin/cpuset_release_agent on exit? - memory_pressure: measure of how much paging pressure in cpuset In addition, the root cpuset only has the following file:
@@ -236,21 +231,7 @@ such as requests from interrupt handlers outside even a mem_exclusive cpuset. -1.5 What does notify_on_release do ? ------------------------------------- - -If the notify_on_release flag is enabled (1) in a cpuset, then whenever -the last task in the cpuset leaves (exits or attaches to some other -cpuset) and the last child cpuset of that cpuset is removed, then -the kernel runs the command /sbin/cpuset_release_agent, supplying the -pathname (relative to the mount point of the cpuset file system) of the -abandoned cpuset. This enables automatic removal of abandoned cpusets. -The default value of notify_on_release in the root cpuset at system -boot is disabled (0). The default value of other cpusets at creation -is the current value of their parents notify_on_release setting. - - -1.6 What is memory_pressure ? +1.5 What is memory_pressure ? ----------------------------- The memory_pressure of a cpuset provides a simple per-cpuset metric of the rate that the tasks in a cpuset are attempting to free up in
@@ -307,7 +288,7 @@ the tasks in the cpuset, in units of rec times 1000. -1.7 What is memory spread ? +1.6 What is memory spread ? --------------------------- There are two boolean flag files per cpuset that control where the kernel allocates pages for the file system buffers and related in
@@ -378,7 +359,7 @@ data set, the memory allocation across t can become very uneven. -1.8 How do I use cpusets ? +1.7 How do I use cpusets ? -------------------------- In order to minimize the impact of cpusets on critical kernel
@@ -468,7 +449,7 @@ than stress the kernel. To start a new job that is to be contained within a cpuset, the steps are: 1) mkdir /dev/cpuset - 2) mount -t cpuset none /dev/cpuset + 2) mount -t container -ocpuset cpuset /dev/cpuset 3) Create the new cpuset by doing mkdir's and write's (or echo's) in the /dev/cpuset virtual file system. 4) Start a task that will be the "founding father" of the new job.
@@ -480,7 +461,7 @@ For example, the following sequence of c named "Charlie", containing just CPUs 2 and 3, and Memory Node 1, and then start a subshell 'sh' in that cpuset: - mount -t cpuset none /dev/cpuset + mount -t container -ocpuset cpuset /dev/cpuset cd /dev/cpuset mkdir Charlie cd Charlie
@@ -512,7 +493,7 @@ Creating, modifying, using the cpusets c virtual filesystem. To mount it, type: -# mount -t cpuset none /dev/cpuset +# mount -t container -o cpuset cpuset /dev/cpuset Then under /dev/cpuset you can find a tree that corresponds to the tree of the cpusets in the system. For instance, /dev/cpuset
@@ -555,6 +536,18 @@ To remove a cpuset, just use rmdir: This will fail if the cpuset is in use (has cpusets inside, or has processes attached). +Note that for legacy reasons, the "cpuset" filesystem exists as a +wrapper around the container filesystem. + +The command + +mount -t cpuset X /dev/cpuset + +is equivalent to + +mount -t container -ocpuset X /dev/cpuset +echo "/sbin/cpuset_release_agent" > /dev/cpuset/release_agent + 2.2 Adding/removing cpus ------------------------
include/linux/cpuset.h +8 -4
@@ -11,6 +11,7 @@#include <linux/sched.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/container.h> #ifdef CONFIG_CPUSETS
@@ -19,8 +20,6 @@ extern int number_of_cpusets; /* How manextern int cpuset_init_early (void );extern int cpuset_init (void );extern void cpuset_init_smp (void ); -extern void cpuset_fork (struct task_struct *p); -extern void cpuset_exit (struct task_struct *p);extern cpumask_t cpuset_cpus_allowed (struct task_struct *p);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)
@@ -75,13 +74,13 @@ static inline int cpuset_do_slab_mem_sprextern void cpuset_track_online_nodes (void ); +extern int current_cpuset_is_being_rebound (void ); +#else /* !CONFIG_CPUSETS */ static inline int cpuset_init_early (void ) {return 0 ; }static inline int cpuset_init (void ) {return 0 ; }static inline void cpuset_init_smp (void ) {} -static inline void cpuset_fork (struct task_struct *p) {} -static inline void cpuset_exit (struct task_struct *p) {}static inline cpumask_t cpuset_cpus_allowed (struct task_struct *p) {
@@ -146,6 +145,11 @@ static inline int cpuset_do_slab_mem_sprstatic inline void cpuset_track_online_nodes (void ) {} +static inline int current_cpuset_is_being_rebound (void ) +{ +return 0 ; +} +#endif /* !CONFIG_CPUSETS */ #endif /* _LINUX_CPUSET_H */
include/linux/mempolicy.h +0 -12
@@ -148,14 +148,6 @@ extern void mpol_rebind_task(struct task const nodemask_t *new);extern void mpol_rebind_mm (struct mm_struct *mm,nodemask_t *new);extern void mpol_fix_fork_child_flag (struct task_struct *p); -#define set_cpuset_being_rebound (x) (cpuset_being_rebound = (x)) - -#ifdef CONFIG_CPUSETS -#define current_cpuset_is_being_rebound () \ - (cpuset_being_rebound == current->cpuset) -#else -#define current_cpuset_is_being_rebound () 0 -#endif extern struct mempolicy default_policy; extern struct zonelist *huge_zonelist(struct vm_area_struct *vma,
@@ -173,8 +165,6 @@ static inline void check_highest_zone(enint do_migrate_pages (struct mm_struct *mm,const nodemask_t *from_nodes,const nodemask_t *to_nodes,int flags); -extern void *cpuset_being_rebound;/* Trigger mpol_copy vma rebind */ -#else struct mempolicy {};
@@ -253,8 +243,6 @@ static inline void mpol_fix_fork_child_f { } -#define set_cpuset_being_rebound (x) do {} while (0) -static inline struct zonelist *huge_zonelist (struct vm_area_struct *vma,unsigned long addr,gfp_t gfp_flags) {
include/linux/sched.h +0 -3
@@ -782,8 +782,6 @@ static inline int above_background_load( }struct io_context ;/* See blkdev.h */ -struct cpuset ; -#define NGROUPS_SMALL 32#define NGROUPS_PER_BLOCK ((int)(PAGE_SIZE / sizeof(gid_t)))struct group_info {
@@ -1130,7 +1128,6 @@ struct task_struct {short il_next;#endif #ifdef CONFIG_CPUSETS -struct cpuset *cpuset;nodemask_t mems_allowed;int cpuset_mems_generation;int cpuset_mem_spread_rotor;
init/Kconfig +6 -0
@@ -309,6 +309,7 @@ config CONTAINERS config CPUSETS bool "Cpuset support" depends on SMP + select CONTAINERS help This option will let you create and manage CPUSETs which allow dynamically partitioning a system into sets of CPUs and
@@ -344,6 +345,11 @@ config CONTAINER_CPUACCT Provides a simple Resource Controller for monitoring the total CPU consumed by the tasks in a container +config PROC_PID_CPUSET + bool "Include legacy /proc/<pid>/cpuset file" + depends on CPUSETS + default y + config RELAY bool "Kernel->user space relay support (formerly relayfs)" help
kernel/cpuset.c +177 -974
1,634 lines not on this page · raw message
kernel/exit.c +0 -2
@@ -31,7 +31,6 @@#include <linux/mempolicy.h> #include <linux/taskstats_kern.h> #include <linux/delayacct.h> -#include <linux/cpuset.h> #include <linux/container.h> #include <linux/syscalls.h> #include <linux/signal.h>
@@ -935,7 +934,6 @@ fastcall void do_exit(long code)__exit_files (tsk);__exit_fs (tsk);exit_thread (); -cpuset_exit (tsk);container_exit (tsk,1 );exit_keys (tsk);
kernel/fork.c +0 -3
@@ -29,7 +29,6 @@#include <linux/nsproxy.h> #include <linux/capability.h> #include <linux/cpu.h> -#include <linux/cpuset.h> #include <linux/container.h> #include <linux/security.h> #include <linux/swap.h>
@@ -1064,7 +1063,6 @@ static struct task_struct *copy_process( p->io_context =NULL ; p->io_wait =NULL ; p->audit_context =NULL ; -cpuset_fork (p);container_fork (p);#ifdef CONFIG_NUMA p->mempolicy =mpol_copy (p->mempolicy);
@@ -1311,7 +1309,6 @@ bad_fork_cleanup_policy:mpol_free (p->mempolicy); bad_fork_cleanup_container:#endif -cpuset_exit (p);container_exit (p, container_callbacks_done);delayacct_tsk_free (p);if (p->binfmt)
mm/mempolicy.c +0 -2
@@ -1311,7 +1311,6 @@ EXPORT_SYMBOL(alloc_pages_current);* keeps mempolicies cpuset relative after its cpuset moves. See * further kernel/cpuset.c update_nodemask(). */ -void *cpuset_being_rebound;/* Slow path of a mempolicy copy */ struct mempolicy *__mpol_copy (struct mempolicy *old)
@@ -1910,4 +1909,3 @@ out: m->version = (vma != priv->tail_vma) ? vma->vm_start :0 ;return 0 ; } -
fs/proc/base.c +2 -2
@@ -2026,7 +2026,7 @@ static const struct pid_entry tgid_base_#ifdef CONFIG_SCHEDSTATS INF("schedstat", S_IRUGO, pid_schedstat),#endif -#ifdef CONFIG_CPUSETS +#ifdef CONFIG_PROC_PID_CPUSET REG("cpuset", S_IRUGO, cpuset),#endif #ifdef CONFIG_CONTAINERS
@@ -2320,7 +2320,7 @@ static const struct pid_entry tid_base_s#ifdef CONFIG_SCHEDSTATS INF("schedstat", S_IRUGO, pid_schedstat),#endif -#ifdef CONFIG_CPUSETS +#ifdef CONFIG_PROC_PID_CPUSET REG("cpuset", S_IRUGO, cpuset),#endif #ifdef CONFIG_CONTAINERS
include/linux/container_subsys.h +6 -0
@@ -13,4 +13,10 @@ SUBSYS(cpuacct)/* */ +#ifdef CONFIG_CPUSETS +SUBSYS(cpuset) +#endif + +/* */ +/* */
--
PATCH 8/10 Containers(V10): Share css_group arrays between tasks with same container memberships
In reply to [email protected]This patch replaces the struct css_group embedded in task_struct with a pointer; all tasks that have the same set of memberships across all hierarchies will share a css_group object, and will be linked via their css_groups field to the "tasks" list_head in the css_group.
Assuming that many tasks share the same container assignments, this reduces overall space usage and keeps the size of the task_struct down (three pointers added to task_struct compared to a non-containers kernel, no matter how many subsystems are registered).
--- Documentation/containers.txt | 14 + include/linux/container.h | 93 ++++++- include/linux/sched.h | 33 -- kernel/container.c | 524 ++++++++++++++++++++++++++++++++++++------- kernel/cpuset.c | 15 - 5 files changed, 553 insertions(+), 126 deletions(-)
include/linux/container.h +83 -10
@@ -29,6 +29,14 @@ extern void container_unlock(void);struct containerfs_root ; +/* Define the enumeration of all container subsystems */ +#define SUBSYS (_x) _x ## _subsys_id, +enum container_subsys_id { +#include <linux/container_subsys.h> + CONTAINER_SUBSYS_COUNT +}; +#undef SUBSYS +/* Per-subsystem/per-container state maintained by the system. */ struct container_subsys_state {/* The container that this subsystem is attached to. Useful
@@ -85,6 +93,54 @@ struct container {struct containerfs_root *root;struct container *top_container; + +/* +* List of cg_container_links pointing at css_groups with +* tasks in this container. Protected by css_group_lock +*/ +struct list_head css_groups; +}; + +/* A css_group is a structure holding pointers to a set of +* container_subsys_state objects. This saves space in the task struct +* object and speeds up fork()/exit(), since a single inc/dec and a +* list_add()/del() can bump the reference count on the entire +* container set for a task. +*/ + +struct css_group { + +/* Reference count */ +struct kref ref; + +/* +* List running through all container groups. Protected by +* css_group_lock +*/ +struct list_head list; + +/* +* List running through all tasks using this container +* group. Protected by css_group_lock +*/ +struct list_head tasks; + +/* +* List of cg_container_link objects on link chains from +* containers referenced from this css_group. Protected by +* css_group_lock +*/ +struct list_head cg_links; + +/* Set of subsystem states, one for each subsystem. NULL for +* subsystems that aren't part of this hierarchy. These +* pointers reduce the number of dereferences required to get +* from a task to its state for a given container, but result +* in increased space usage if tasks are in wildly different +* groupings across different hierarchies. This array is +* immutable after creation */ +struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT ]; + };/* struct cftype:
@@ -111,6 +167,10 @@ struct cftype {ssize_t (*read) (struct container *cont,struct cftype *cft,struct file *file,char __user *buf,size_t nbytes,loff_t *ppos); +/* +* read_uint() is a shortcut for the common case of returning a +* single integer. Use it in place of read() +*/ u64 (*read_uint) (struct container *cont,struct cftype *cft); ssize_t (*write) (struct container *cont, struct cftype *cft, struct file *file,
@@ -131,15 +191,7 @@ int container_is_removed(const struct coint container_path (const struct container *cont,char *buf,int buflen); -int __container_task_count (const struct container *cont); -static inline int container_task_count (const struct container *cont) -{ -int task_count; -rcu_read_lock (); - task_count =__container_task_count (cont); -rcu_read_unlock (); -return task_count; -} +int container_task_count (const struct container *cont);/* Return true if the container is a descendant of the current container */ int container_is_descendant (const struct container *cont);
@@ -186,7 +238,7 @@ static inline struct container_subsys_ststatic inline struct container_subsys_state *task_subsys_state (struct task_struct *task,int subsys_id) { -return rcu_dereference (task->containers.subsys[subsys_id]); +return rcu_dereference (task->containers->subsys[subsys_id]); } static inline struct container* task_container(struct task_struct *task,
@@ -199,6 +251,27 @@ int container_path(const struct containeint container_clone (struct task_struct *tsk,struct container_subsys *ss); +/* A container_iter should be treated as an opaque object */ +struct container_iter { +struct list_head *cg_link; +struct list_head *task; +}; + +/* To iterate across the tasks in a container: +* +* 1) call container_iter_start to intialize an iterator +* +* 2) call container_iter_next() to retrieve member tasks until it +* returns NULL or until you want to end the iteration +* +* 3) call container_iter_end() to destroy the iterator. +*/ +void container_iter_start (struct container *cont,struct container_iter *it); +struct task_struct *container_iter_next (struct container *cont, +struct container_iter *it); +void container_iter_end (struct container *cont,struct container_iter *it); + +#else /* !CONFIG_CONTAINERS */ static inline int container_init_early (void ) {return 0 ; }
include/linux/sched.h +4 -29
@@ -849,34 +849,6 @@ struct sched_class {void (*task_new) (struct rq *rq,struct task_struct *p); }; -#ifdef CONFIG_CONTAINERS - -#define SUBSYS (_x) _x ## _subsys_id, -enum container_subsys_id { -#include <linux/container_subsys.h> - CONTAINER_SUBSYS_COUNT -}; -#undef SUBSYS - -/* A css_group is a structure holding pointers to a set of -* container_subsys_state objects. -*/ - -struct css_group { - -/* Set of subsystem states, one for each subsystem. NULL for -* subsystems that aren't part of this hierarchy. These -* pointers reduce the number of dereferences required to get -* from a task to its state for a given container, but result -* in increased space usage if tasks are in wildly different -* groupings across different hierarchies. This array is -* immutable after creation */ -struct container_subsys_state *subsys[CONTAINER_SUBSYS_COUNT ]; - -}; - -#endif /* CONFIG_CONTAINERS */ -struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;
@@ -1133,7 +1105,10 @@ struct task_struct {int cpuset_mem_spread_rotor;#endif #ifdef CONFIG_CONTAINERS -struct css_group containers; +/* Container info protected by css_group_lock */ +struct css_group *containers; +/* cg_list protected by css_group_lock and tsk->alloc_lock */ +struct list_head cg_list;#endif struct robust_list_head __user *robust_list;#ifdef CONFIG_COMPAT
kernel/container.c +448 -79
710 lines not on this page · raw message
kernel/cpuset.c +8 -7
@@ -593,12 +593,13 @@ static int update_nodemask(struct cpuset {struct cpuset trialcs;nodemask_t oldmem; -struct task_struct *g, *p; +struct task_struct *p;struct mm_struct **mmarray;int i, n, ntasks;int migrate;int fudge;int retval; +struct container_iter it;/* top_cpuset.mems_allowed tracks node_online_map; it's read-only */ if (cs == &top_cpuset)
@@ -659,7 +660,7 @@ static int update_nodemask(struct cpusetif (!mmarray) goto done;read_lock (&tasklist_lock);/* block fork */ -if (__container_task_count (cs->css.container) <= ntasks) +if (container_task_count (cs->css.container) <= ntasks)break ;/* got enough */ read_unlock (&tasklist_lock);/* try again */ kfree (mmarray);
@@ -668,21 +669,21 @@ static int update_nodemask(struct cpuset n =0 ;/* Load up mmarray[] with mm reference for each task in cpuset. */ - do_each_thread(g, p) { +container_iter_start (cs->css.container, &it); +while ((p =container_iter_next (cs->css.container, &it))) {struct mm_struct *mm;if (n >= ntasks) {printk (KERN_WARNING "Cpuset mempolicy rebind incomplete.\n" ); -continue ; +break ; } -if (task_cs (p) != cs) -continue ; mm =get_task_mm (p);if (!mm)continue ; mmarray[n++] = mm; - }while_each_thread (g, p); + } +container_iter_end (cs->css.container, &it);read_unlock (&tasklist_lock);/*
Documentation/containers.txt +13 -1
@@ -176,7 +176,9 @@ Containers extends the kernel as follows subsystem state is something that's expected to happen frequently and in performance-critical code, whereas operations that require a task's actual container assignments (in particular, moving between - containers) are less common. + containers) are less common. A linked list runs through the cg_list + field of each task_struct using the css_group, anchored at + css_group->tasks. - A container hierarchy filesystem can be mounted for browsing and manipulation from user space.
@@ -252,6 +254,16 @@ linear search to locate an appropriate e very efficient. A future version will use a hash table for better performance. +To allow access from a container to the css_groups (and hence tasks) +that comprise it, a set of cg_container_link objects form a lattice; +each cg_container_link is linked into a list of cg_container_links for +a single container on its cont_link_list field, and a list of +cg_container_links for a single css_group on its cg_link_list. + +Thus the set of tasks in a container can be listed by iterating over +each css_group that references the container, and sub-iterating over +each css_group's task set. + The use of a Linux virtual file system (vfs) to represent the container hierarchy provides for a familiar permission and name space for containers, with a minimum of additional kernel code.
--
PATCH 9/10 Containers(V10): Simple debug info subsystem
In reply to [email protected]This example subsystem exports debugging information as an aid to diagnosing refcount leaks, etc, in the container framework.
--- include/linux/container_subsys.h | 4 + init/Kconfig | 10 ++++ kernel/Makefile | 1 kernel/container_debug.c | 89 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 104 insertions(+)
include/linux/container_subsys.h +4 -0
@@ -19,4 +19,8 @@ SUBSYS(cpuset)/* */ +#ifdef CONFIG_CONTAINER_DEBUG +SUBSYS(debug) +#endif +/* */
init/Kconfig +10 -0
@@ -306,6 +306,16 @@ config LOG_BUF_SHIFT config CONTAINERS bool +config CONTAINER_DEBUG + bool "Example debug container subsystem" + select CONTAINERS + help + This option enables a simple container subsystem that + exports useful debugging information about the containers + framework + + Say N if unsure + config CPUSETS bool "Cpuset support" depends on SMP
kernel/container_debug.c new file +89 -0
89 lines not on this page · raw message
kernel/Makefile +1 -0
@@ -37,6 +37,7 @@ obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_COMPAT) += compat.o obj-$(CONFIG_CONTAINERS) += container.o +obj-$(CONFIG_CONTAINER_DEBUG) += container_debug.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o obj-$(CONFIG_IKCONFIG) += configs.o
--
PATCH 10/10 Containers(V10): Support for automatic userspace release agents
In reply to [email protected]This patch adds the following files to the container filesystem:
notify_on_release - configures/reports whether the container subsystem should attempt to run a release script when this container becomes unused
release_agent - configures/reports the release agent to be used for this hierarchy (top level in each hierarchy only)
releasable - reports whether this container would have been auto-released if notify_on_release was true and a release agent was configured (mainly useful for debugging)
To avoid locking issues, invoking the userspace release agent is done via a workqueue task; containers that need to have their release agents invoked by the workqueue task are linked on to a list.
When the "cpuset" filesystem is mounted, it automatically sets the hierarchy's release agent to be /sbin/cpuset_release_agent for backward-compatibility with existing cpusets users.
--- include/linux/container.h | 15 + kernel/container.c | 364 ++++++++++++++++++++++++++++++++++++++++++---- kernel/cpuset.c | 5 3 files changed, 348 insertions(+), 36 deletions(-)
include/linux/container.h +10 -5
@@ -64,11 +64,7 @@ static inline void css_get(struct contai* css_put() should be called to release a reference taken by * css_get() */ - -static inline void css_put (struct container_subsys_state *css) -{ -atomic_dec (&css->refcnt); -} +void css_put (struct container_subsys_state *css);struct container {unsigned long flags;/* "unsigned long" so bitops work */
@@ -99,6 +95,13 @@ struct container {* tasks in this container. Protected by css_group_lock */ struct list_head css_groups; + +/* +* Linked list running through all containers that can +* potentially be reaped by the release agent. Protected by +* container_mutex +*/ +struct list_head release_list; };/* A css_group is a structure holding pointers to a set of
@@ -271,6 +274,8 @@ struct task_struct *container_iter_next( struct container_iter *it);void container_iter_end (struct container *cont,struct container_iter *it); +void container_set_release_agent_path (struct container_subsys *ss, +const char *path);#else /* !CONFIG_CONTAINERS */
kernel/container.c +333 -31
522 lines not on this page · raw message
kernel/cpuset.c +5 -0
@@ -275,6 +275,11 @@ static int cpuset_get_sb(struct file_sys unused_dev_name, "cpuset", mnt);put_filesystem (container_fs); +if (!ret) { +container_set_release_agent_path ( + &cpuset_subsys, +"/sbin/cpuset_release_agent" ); + } }return ret; }
--
Andrew Morton
maintainerQuoted 11 lines · On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
This is an update to my multi-hierarchy multi-subsystem generic process containers patch.
...
Still TODO:
...
- lots more testing
So how do we do this?
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on each setuid(), toss the task into that. Or something along those lines?
Quoted 6 lines · Andrew Morton
So how do we do this?
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on eachsetuid(), toss the task into that. Or something along those lines?
How about a container for each thread group, pgrp, session, and user?
-- wli
Quoted 10 lines · William Lee Irwin III
Quoted 6 lines · Andrew Morton
So how do we do this?
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on eachsetuid(), toss the task into that. Or something along those lines?How about a container for each thread group, pgrp, session, and user?
I've been thinking about this, and figured that it could be quite useful to be able to mount a container tree that groups tasks by userid or thread group - for doing per-user resource controls, for example, without having to write a controller that specifically handles the per-user case.
One option would be to add a mount option, something like
mount -t container -ogroupkey=<X>
where X could be one of: uid, gid, pgrp, sid, tgid
And put hooks in the various places where these ids could change, in order to move tasks between contaners as appropriate. But after some thought it seems to me that this is putting complexity in the kernel that probably doesn't belong there, and additionally is probably not sufficently flexible for some real-life situations. (E.g. the user wants all users in the "student" group to be lumped into the same container, but each user in the "professor" group gets their own container).
So maybe this would be better handled in userspace? Have a daemon listing on a process connector socket, and move processes between containers based on notifications from the connector and user-defined rules.
We'd probably also want to add some new connector events, such as PROC_EVENT_PGRP, and PROC_EVENT_SID
A simple daemon that handles the case where we're classifying based on a single key with no complex rules shouldn't be too hard to write.
It also sounds rather like the classification engine that ResourceGroups had originally in the kernel and moved to userspace, so I'll take a look at that and see if it's adaptable for this.
Paul
Quoted 7 lines · Paul Menage
So maybe this would be better handled in userspace? Have a daemon listing on a process connector socket, and move processes between containers based on notifications from the connector and user-defined rules.
We'd probably also want to add some new connector events, such as
PROC_EVENT_PGRP, andPROC_EVENT_SID
Yep, this is what I did to test fair-user scheduling on top of my patches.
Dhaval has a working program, which listens for UID change events and moves the task to approp. container. I will review that and have it posted soon.
Signature · 2 lines
Regards, vatsa
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
This is an update to my multi-hierarchy multi-subsystem generic process containers patch.
...
Still TODO:
...
- lots more testing
Quoted 7 lines
So how do we do this?
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on each
setuid(), toss the task into that. Or something along those lines?
Please, lets get the RSS controller in. It's ready, been tested and commented on widely.
I'll also start porting my containerstats patches on top of -v10.
Signature · 4 lines
Thanks, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 16 lines · Balbir Singh
Quoted 15 lines · Andrew Morton
Quoted 11 lines · On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
This is an update to my multi-hierarchy multi-subsystem generic process containers patch.
...
Still TODO:
...
- lots more testing
So how do we do this?
Quoted 7 lines
Quoted 5 lines
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on each
setuid(), toss the task into that. Or something along those lines?Please, lets get the RSS controller in. It's ready, been tested
It is not 100% ready yet actually :) I am working on it right now and hope to get ready till tomorrow.
Quoted 1 line
and commented on widely.
Yup :) Balbir, thanks for testing, your patches are already in.
Quoted 2 lines
I'll also start porting my containerstats patches on top of -v10.
Quoted 10 lines · Pavel Emelianov
Quoted 6 lines
Quoted 5 lines
Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on each
setuid(), toss the task into that. Or something along those lines?Please, lets get the RSS controller in. It's ready, been tested
It is not 100% ready yet actually :) I am working on it right now and hope to get ready till tomorrow.
By ready, I meant ready for inclusion as a concept/approach.
Quoted 4 lines
Quoted 1 line
and commented on widely.
Yup :) Balbir, thanks for testing, your patches are already in.
Thanks for including them.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Hi Paul.
I have faced a warning during testing your patches. The testcase is simple:
# ssh to the node mount -t container none /cnt/rss/ -o rss mkdir /cnt/rss/0 /bin/echo $$ > /cnt/rss/0/tasks # exit with ^d and ssh again rmdir /cnt/rss/0 dmesg
BUG: at mm/slab.c:777 __find_general_cachep() [<c04656c8>] __kmalloc+0x3f/0xa5 [<c0440e3a>] container_tasks_open+0x56/0x11f [<c0440bcc>] container_file_open+0x0/0x36 [<c0440bfb>] container_file_open+0x2f/0x36 [<c0467a12>] __dentry_open+0xc1/0x178 [<c0467b43>] nameidata_to_filp+0x24/0x33 [<c0467b84>] do_filp_open+0x32/0x39 [<c04678eb>] get_unused_fd+0x50/0xb6 [<c0467bcd>] do_sys_open+0x42/0xbe [<c0467c82>] sys_open+0x1c/0x1e [<c0404c12>] sysenter_past_esp+0x5f/0x85 [<c05b0000>] __xfrm_policy_check+0x11a/0x4f6
The bug seems to be here:
static int container_tasks_open(struct inode *unused, struct file *file)
{
...
npids = container_task_count(cont);
pidarray = kmalloc(npids * sizeof(pid_t), GFP_KERNEL);
if (!pidarray)
goto err1;
...
}The npids happened to be 0 and kmalloc warns that size is zero.
Thanks,
Pavel.
Hi Paul,
I've got two problems working with this patchset:
1. A task can't join a cpuset unless 'cpus' and 'mems' are set. These don't seem to automatically inherit the parent's values. So when I do
mount -t container -o ns,cpuset nsproxy /containers (unshare a namespace)
the unshare fails because container_clone() created a new cpuset container but the task couldn't automatically enter that new cpuset.
2. I can't delete containers because of the files they contain, and am not allowed to delete those files by hand.
thanks,
-serge
What you describe, Serge, sounds like semantics carried over from cpusets.
Quoted 1 line · Serge
A task can't join a cpuset unless 'cpus' and 'mems' are set.
Yup - don't want to run a task in a cpuset that lacks cpu, or lacks memory. Hard to run without those.
Quoted 1 line
These don't seem to automatically inherit the parent's values
Yup - early in the life of cpusets, a created cpuset inherited the cpus and mems of its parent. But that broke the exclusive property big time. You will recall that a cpu_exclusive or mem_exclusive cpuset cannot overlap the cpus or memory, respectively, of any of its sibling cpusets.
So we changed it to creating new cpusets empty of cpus or memory.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 7 lines · Paul Jackson
Yup - early in the life of cpusets, a created cpuset inherited the cpus and mems of its parent. But that broke the exclusive property big time. You will recall that a
cpu_exclusiveormem_exclusivecpuset cannot overlap the cpus or memory, respectively, of any of its sibling cpusets.
Maybe we could make it a per-cpuset option whether children should inherit mems/cpus or not?
Paul
Quoted 2 lines · Paul M
Maybe we could make it a per-cpuset option whether children should inherit mems/cpus or not?
I suppose, if those needing inherited mems/cpus need it bad enough.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 11 lines · Paul Menage
Quoted 7 lines · Paul Jackson
Yup - early in the life of cpusets, a created cpuset inherited the cpus and mems of its parent. But that broke the exclusive property big time. You will recall that a
cpu_exclusiveormem_exclusivecpuset cannot overlap the cpus or memory, respectively, of any of its sibling cpusets.Maybe we could make it a per-cpuset option whether children should inherit mems/cpus or not?
The values can be changed after the cpuset is populated, right? So really these are just defaults? Would it then make sense to just default to (parent_set - sibling_exclusive_set) for a new sibling's value?
An option is fine with me, but without such an option at all, cpusets could not be applied to namespaces...
thanks,
-serge
Quoted 3 lines
Would it then make sense to just
default to (parent_set-sibling_exclusive_set) for a new sibling's value?
Which could well be empty, which in turn puts one back in the position of dealing with a newborn cpuset that is empty (of cpus or of memory), or else it introduces a new and odd constraint on when cpusets can be created (only when there are non-exclusive cpus and mems available.)
Quoted 2 lines
An option is fine with me, but without such an option at all, cpusets could not be applied to namespaces...
I wasn't paying close enough attention to understand why you couldn't do it in two steps - make the container, and then populate it with resources.
But if indeed that's not possible, then I guess we need some sort of option specifying whether to create kids empty, or inheriting.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 9 lines · Paul Jackson
Quoted 3 lines
Would it then make sense to just
default to (parent_set-sibling_exclusive_set) for a new sibling's value?Which could well be empty, which in turn puts one back in the position of dealing with a newborn cpuset that is empty (of cpus or of memory), or else it introduces a new and odd constraint on when cpusets can be created (only when there are non-exclusive cpus and mems available.)
Quoted 6 lines
Quoted 2 lines
An option is fine with me, but without such an option at all, cpusets could not be applied to namespaces...
I wasn't paying close enough attention to understand why you couldn't do it in two steps - make the container, and then populate it with resources.
Sorry, please clarify - are you saying that now you do understand, or that I should explain?
Quoted 2 lines
But if indeed that's not possible, then I guess we need some sort of option specifying whether to create kids empty, or inheriting.
Paul (uh, Menage :) should I do a patch for this or have you got it already?
thanks,
-serge
Quoted 6 lines
Quoted 3 lines
I wasn't paying close enough attention to understand why you couldn't do it in two steps - make the container, and then populate it with resources.
Sorry, please clarify - are you saying that now you do understand, or that I should explain?
Could you explain -- I still don't understand why you need this option. I still don't understand why you can't do it in two steps - make the container, then add cpu/mem separately.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 10 lines · Paul Jackson
Quoted 6 lines
Quoted 3 lines
I wasn't paying close enough attention to understand why you couldn't do it in two steps - make the container, and then populate it with resources.
Sorry, please clarify - are you saying that now you do understand, or that I should explain?
Could you explain -- I still don't understand why you need this option. I still don't understand why you can't do it in two steps - make the container, then add cpu/mem separately.
Sure - the key is that the ns subsystem uses container_clone() to automatically create a new container (on sys_unshare() or clone(2) with certain flags) and move the current task into it. Let's say we have done
mount -t container -o ns,cpuset nsproxy /containers
and we, as task 875, happen to be in the topmost container:
/containers/
Now we fork task 999 which does an unshare(CLONE_NEWNS), or we just clone(CLONE_NEWNS). This will create
/containers/node_999
and move task 999 into that container. Except that when it tries attach_task() it is refused by cpuset. So the container_clone() fails, and in turn the sys_unshare() or clone() fails. A login making use of the pam_namespace.so library would fail this way with the ns and cpuset subsystems composed.
We could special case this by having kernel/container.c:container_clone() check whether one of the subsystems is cpusets and, if so, setting the defaults for mems and cpus, but that is kind of ugly. I suppose as a cleaner alternative we could add a container_subsys->inherit_defaults() handler, to be called at container_clone(), and for cpusets this would set cpus and mems to the parent values - sibling exclusive values. If that comes to nothing, then the attach_task() is still refused, and the unshare() or clone() fails, but this time with good reason.
thanks,
-serge
Quoted 6 lines
I suppose as a cleaner alternative we could
add acontainer_subsys->inherit_defaults()handler, to be called atcontainer_clone(), and for cpusets this would set cpus and mems to the parent values - sibling exclusive values. If that comes to nothing, then theattach_task()is still refused, and theunshare()orclone()fails, but this time with good reason.
Unfortunately, I haven't spent the time I should thinking about container cloning, namespaces and such.
I don't know, for the workloads that matter to me, when, how or if this container cloning will be used.
I'm tempted to suggest the following.
First, I am assuming that the classic method of creating cpuset children will still work, such as the following (which can fail for certain combinations of exclusive cpus or mems):
cd /dev/cpuset/foobar
mkdir foochild
cp cpus foochild
cp mems foochild
echo $$ > foochild/tasks
Second, given that, how about you fail the unshare() or clone() anytime that the instance to be cloned has any sibling cpusets with any exclusive flags set.
The exclusive property is not really on friendly terms with cloning.
Now if the above classic code must be encoded using cloning under the covers, then we've got problems, probably more problems than just this.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 24 lines · Paul Jackson
Quoted 6 lines
I suppose as a cleaner alternative we could
add acontainer_subsys->inherit_defaults()handler, to be called atcontainer_clone(), and for cpusets this would set cpus and mems to the parent values - sibling exclusive values. If that comes to nothing, then theattach_task()is still refused, and theunshare()orclone()fails, but this time with good reason.Unfortunately, I haven't spent the time I should thinking about container cloning, namespaces and such.
I don't know, for the workloads that matter to me, when, how or if this container cloning will be used.
I'm tempted to suggest the following.
First, I am assuming that the classic method of creating cpuset children will still work, such as the following (which can fail for certain combinations of exclusive cpus or mems):
cd/dev/cpuset/foobar
mkdir foochild
cp cpus foochild
cp mems foochild
echo $$ > foochild/tasks
Quoted 3 lines
Second, given that, how about you fail the
unshare()orclone()anytime that the instance to be cloned has any sibling cpusets with any exclusive flags set.
The below patch (on top of my previous patch) does basically that. But I wasn't able to test it, bc i wasn't able to set cpus_exclusive...
For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 also have to have it exclusively?
If so, then clearly this approach won't work, since if any container has exclusive cpus, then every container will have siblings with exclusive cpus, and unshare still isn't possible on the system.
Quoted 10 lines
The exclusive property is not really on friendly terms with cloning.
Now if the above classic code must be encoded using cloning under the covers, then we've got problems, probably more problems than just this.
Signature · 3 lines
I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <[email protected]> 1.925.600.0401
thanks,
-serge
Quoted 1 line
From
821de58b6ba446e50225606e907baac00130586cMon Sep 17 00:00:00 2001
From: Serge E. Hallyn <[email protected]>
Date: Thu, 7 Jun 2007 13:53:43 -0400
Subject: [PATCH 1/1] containers: implement subsys->auto_setup
container_clone() in one step creates a new container and moves the current task into it. Since cpusets do not automatically fill in the allowed cpus and mems, and do not allow a task to be attached without these filled in, composing the ns subsystem, which uses container_clone(), and the cpuset subsystem, results in sys_unshare() (and clone(CLONE_NEWNS)) always being denied.
To allow the two subsystems to be meaningfully composed, implement subsystem->auto_setup, called from container_clone() after creating the new container.
Only the cpuset_auto_setup() is currently implemented. If any sibling containers have exclusive cpus or mems, then the cpus and mems are not filled in for the new container, meaning that unshare/clone(CLONE_NEWNS) will be denied. However so long as no siblings have exclusive cpus or mems, the new container's cpus and mems are inherited from the parent container.
--- Documentation/containers.txt | 7 +++++++ include/linux/container.h | 1 + kernel/container.c | 7 +++++++ kernel/cpuset.c | 21 +++++++++++++++++++++ 4 files changed, 36 insertions(+), 0 deletions(-)
Documentation/containers.txt +7 -0
@@ -514,6 +514,13 @@ include/linux/container.h for details). Note that although this method can return an error code, the error code is currently not always handled well. +void auto_setup(struct container_subsys *ss, struct container *cont) + +Called at container_clone() to do any paramater initialization +which might be required before a task could attach. For example +in cpusets, no task may attach before 'cpus' and 'mems' are +set up. + void bind(struct container_subsys *ss, struct container *root) LL=callback_mutex
include/linux/container.h +1 -0
@@ -213,6 +213,7 @@ struct container_subsys {void (*exit)(struct container_subsys *ss,struct task_struct *task);int (*populate)(struct container_subsys *ss,struct container *cont); +void (*auto_setup)(struct container_subsys *ss,struct container *cont);void (*bind)(struct container_subsys *ss,struct container *root);int subsys_id;int active;
kernel/container.c +7 -0
@@ -2316,6 +2316,7 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)struct inode *inode;struct css_group *cg;struct containerfs_root *root; +struct container_subsys *ss;/* We shouldn't be called by an unregistered subsystem */ BUG_ON (!subsys->active);
@@ -2397,6 +2398,12 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys) goto again; } +/* do any required auto-setup */ +for_each_subsys (root, ss) { +if (ss->auto_setup) + ss->auto_setup (ss, child); + } +/* All seems fine. Finish by moving the task into the new container */ ret =attach_task (child, tsk);mutex_unlock (&container_mutex);
kernel/cpuset.c +21 -0
@@ -1189,6 +1189,26 @@ int cpuset_populate(struct container_subsys *ss, struct container *cont)return 0 ; } +void cpuset_auto_setup (struct container_subsys *ss, +struct container *container) +{ +struct container *parent, *child; +struct cpuset *cs, *parent_cs; + + parent = container->parent; +list_for_each_entry (child, &parent->children, sibling) { + cs =container_cs (child); +if (is_mem_exclusive (cs) ||is_cpu_exclusive (cs)) +return ; + } + cs =container_cs (container); + parent_cs =container_cs (parent); + + cs->mems_allowed = parent_cs->mems_allowed; + cs->cpus_allowed = parent_cs->cpus_allowed; +return ; +} +/* * cpuset_create - create a cpuset * parent: cpuset that will be parent of the new cpuset.
@@ -1249,6 +1269,7 @@ struct container_subsys cpuset_subsys = { .can_attach = cpuset_can_attach, .attach = cpuset_attach, .populate = cpuset_populate, + .auto_setup = cpuset_auto_setup, .subsys_id = cpuset_subsys_id, .early_init = 1, };
Signature · 1 line
1.5.1.1.GIT
Quoted 2 lines
For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 also have to have it exclusively?
Yes.
Quoted 3 lines
If so, then clearly this approach won't work, since if any container has exclusive cpus, then every container will have siblings with exclusive cpus, and unshare still isn't possible on the system.
Well, if I'm following you, not exactly.
If we have some exclusive flags set, then every top level container will have exclusive siblings, but further down the hierarchy, some subtree might be entirely free of any exclusive settings. Then nodes below the top of that subtree would not have exclusive set, and would not have any exclusive siblings.
But, overall, yeah, exclusive is no friend of container cloning.
I just wish I had been thinking harder about how container cloning will impact my life, and the lives of the customers in my cpuset intensive corner of the world.
There are certainly a whole bunch of people who will never have any need for exclusive cpusets.
Perhaps (speculating wildly from great ignorance) there are a whole bunch of people who will never have need for container cloning.
And perhaps, hoping to get lucky here, the set of people who need both at the same time on the same system is sufficiently close to empty that we can just tell them tough toenails - you cannot do both at once.
How wide spread will be the use of container cloning, if it proceeds as envisioned?
The set of people using exclusive cpusets is roughly some subset of those running multiple, cpuset isolated, non-cooperating jobs on big iron, usually with the aid of a batch scheduler. Well, that's what I am aware of anyway. If there are any other friends of exclusive cpusets lurking here, you might want to speak up, before I sell your interests down the river.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 29 lines · Paul Jackson
Quoted 2 lines
For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 also have to have it exclusively?
Yes.
Quoted 3 lines
If so, then clearly this approach won't work, since if any container has exclusive cpus, then every container will have siblings with exclusive cpus, and unshare still isn't possible on the system.
Well, if I'm following you, not exactly.
If we have some exclusive flags set, then every top level container will have exclusive siblings, but further down the hierarchy, some subtree might be entirely free of any exclusive settings. Then nodes below the top of that subtree would not have exclusive set, and would not have any exclusive siblings.
But, overall, yeah, exclusive is no friend of container cloning.
I just wish I had been thinking harder about how container cloning will impact my life, and the lives of the customers in my cpuset intensive corner of the world.
There are certainly a whole bunch of people who will never have any need for exclusive cpusets.
Perhaps (speculating wildly from great ignorance) there are a whole bunch of people who will never have need for container cloning.
Quoted 6 lines
And perhaps, hoping to get lucky here, the set of people who need both at the same time on the same system is sufficiently close to empty that we can just tell them tough toenails - you cannot do both at once.
How wide spread will be the use of container cloning, if it proceeds as envisioned?
It's not just container cloning, but all namespace unsharing. So uses include (1) providing 'polyinstantiated directory' functionality, i.e. private per-user /tmp's or per-security-level /tmp and /home's. (2) any virtual server usage (3) hpc checkpoint/restart users.
Quoted 3 lines
The set of people using exclusive cpusets is roughly some subset of those running multiple, cpuset isolated, non-cooperating jobs on big iron, usually with the aid of a batch scheduler.
Unfortunately I would imagine these users to be very intereseted in providing checkpoint/restart/migrate functionality.
Quoted 9 lines
Well, that's what
I am aware of anyway. If there are any other friends of exclusive cpusets lurking here, you might want to speak up, before I sell your interests down the river.Signature · 3 lines
I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <[email protected]> 1.925.600.0401
Can you explain to me, though, why it should be that if /cpusets/set0 has access to cpus 0-8, and /cpusets/set0/set1 has exclusive access to cpus 0-2, and /cpusets/set0/set2 has exclusive access to cpus 3-4, why i a process in /cpusets/set0 creates /cpusets/set0/set3 through container_clone, it would be unsafe to have it automatically get cpus 5-8?
Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4 later, those cpus can just be taken away from set3?
thanks,
-serge
Quoted 6 lines
Quoted 3 lines
The set of people using exclusive cpusets is roughly some subset of those running multiple, cpuset isolated, non-cooperating jobs on big iron, usually with the aid of a batch scheduler.
Unfortunately I would imagine these users to be very intereseted in providing checkpoint/restart/migrate functionality.
Yup - such customers are very interested in checkpoint, restart, and migrate functionality.
Quoted 2 lines
Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4 later, those cpus can just be taken away from set3?
Yeah - that works, so far as I know (which isn't all that far ..')
But both:
1) that (using whatever cpus are still available) and
2) my other idea, of not allowing any cloning of cpusets with exclusive siblings at all,
looked a little ugly to me.
For example, such customers as above would not appreciate having their checkpoint/restart/migrate fail in any case where there weren't spare non-exclusive cpus, which for users of the exclusive flag, is often the more common case.
My rule of thumb when doing ugly stuff is to constrain it as best I can -- minimize what it allows. This led me to prefer (2) above over (1) above.
Perhaps there's a better way to think of this ... When we clone like this for checkpoint/restart/migrate functionality, perhaps we are not really starting up a new, separate, competing job that should have its resources isolated and separated from the original.
Perhaps instead we are firing up a convenient alter-ego of the original job, which will be co-operatively using the same resources by default. If that's the normal case, then it seems wrong to force the clone onto disjoint CPUs, or fail for lack thereof.
So perhaps we should refine the meaning of 'exclusive', from:
- no overlapping siblings
to:
- no overlapping siblings other than clones of ones self.
Then default to cloning right on the same CPU resources as the original, possibly with both original and clone marked exclusive.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 27 lines · Paul Jackson
Quoted 6 lines
Quoted 3 lines
The set of people using exclusive cpusets is roughly some subset of those running multiple, cpuset isolated, non-cooperating jobs on big iron, usually with the aid of a batch scheduler.
Unfortunately I would imagine these users to be very intereseted in providing checkpoint/restart/migrate functionality.
Yup - such customers are very interested in checkpoint, restart, and migrate functionality.
Quoted 2 lines
Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4 later, those cpus can just be taken away from set3?
Yeah - that works, so far as I know (which isn't all that far ..')
But both:
1) that (using whatever cpus are still available) and
2) my other idea, of not allowing any cloning of cpusets with exclusive siblings at all,looked a little ugly to me.
For example, such customers as above would not appreciate having their checkpoint/restart/migrate fail in any case where there weren't spare non-exclusive cpus, which for users of the exclusive flag, is often the more common case.
Quoted 8 lines
My rule of thumb when doing ugly stuff is to constrain it as best I can -- minimize what it allows. This led me to prefer (2) above over (1) above.
Perhaps there's a better way to think of this ... When we clone like this for checkpoint/restart/migrate functionality, perhaps we are not really starting up a new, separate, competing job that should have its resources isolated and separated from the original.
Depends on whether the cpus are allocated to a customer or to a job.
For the most part I would expect any job to be restart either on a different machine, or at a different time, but of course it doesn't have to be that way.
Quoted 9 lines
Perhaps instead we are firing up a convenient alter-ego of the original job, which will be co-operatively using the same resources by default. If that's the normal case, then it seems wrong to force the clone onto disjoint CPUs, or fail for lack thereof.
So perhaps we should refine the meaning of 'exclusive', from:
- no overlapping siblings
to:
- no overlapping siblings other than clones of ones self.
I'm not sure that clones of self will happen often enough to make a special case for them :)
Anyway the patch I sent is simple enough, and if users end up demanding the ability to better deal with exclusive cpusets, the patch will be simple enough to extend by changing cpuset_auto_setup(), so let's stick with that patch since it's your preference (IIUC).
Quoted 2 lines
Then default to cloning right on the same CPU resources as the original, possibly with both original and clone marked exclusive.
Thanks,
-serge
Quoted 6 lines · Serge E. Hallyn
Anyway the patch I sent is simple enough, and if users end up demanding the ability to better deal with exclusive cpusets, the patch will be simple enough to extend by changing
cpuset_auto_setup(), so let's stick with that patch since it's your preference (IIUC).
Sounds good to me, although I think my preference would be to extend the "create()" subsystem callback with a "struct task_struct *clone_task" parameter that indicates that clone_task is cloning its own container; a subsystem like cpusets that needs to do additional setup at that point could inherit settings either from the parent or from clone_task's container (or something else) as desired. (It could also do permission checking based on properties of clone_task, etc).
Paul
Quoted 8 lines · Paul Menage
Quoted 6 lines · Serge E. Hallyn
Anyway the patch I sent is simple enough, and if users end up demanding the ability to better deal with exclusive cpusets, the patch will be simple enough to extend by changing
cpuset_auto_setup(), so let's stick with that patch since it's your preference (IIUC).
Quoted 7 lines
Sounds good to me, although I think my preference would be to extend the "
create()" subsystem callback with a "structtask_struct*clone_task" parameter that indicates thatclone_taskis cloning its own container; a subsystem like cpusets that needs to do additional setup at that point could inherit settings either from the parent or from clone_task's container (or something else) as desired. (It could also do permission checking based on properties ofclone_task, etc).
The problem is container_clone() doesn't call ->create explicitly, it does vfs_mkdir. So we have no real way of passing in clone_task.
-serge
Quoted 4 lines · Serge E. Hallyn
The problem is
container_clone()doesn't call ->create explicitly, it doesvfs_mkdir. So we have no real way of passing inclone_task.
Good point.
Looking at vfs_mkdir(), it's pretty simple, and really the only bits that apply to container_clone() are the call to ->mkdir() and possibly the call to fsnotify_mkdir(). (I think that's maybe how you did it originally?)
Maybe it would make sense to just call container_create() at that point directly, which would allow us more parameters.
Paul
Quoted 6 lines · Paul Menage
Quoted 4 lines · Serge E. Hallyn
The problem is
container_clone()doesn't call ->create explicitly, it doesvfs_mkdir. So we have no real way of passing inclone_task.
Quoted 6 lines
Good point.
Looking at
vfs_mkdir(), it's pretty simple, and really the only bits that apply tocontainer_clone()are the call to ->mkdir() and possibly the call tofsnotify_mkdir(). (I think that's maybe how you did it originally?)
Yes it was.
Quoted 2 lines
Maybe it would make sense to just call
container_create()at that point directly, which would allow us more parameters.
I do fear that that could become a maintenance nightmare. For instance right now there's the call to fsnotify_mkdir(). Other such hooks might be placed at vfs_mkdir, which we'd then likely want to have placed in our container_mkdir() and container_clone() fns. And of course may_create() is static inline in fs/namei.c. It's trivial, but still if it changes we'd want to change the version in kernel/container.c as well.
What would be the main advantage of doing it this way? Do you consider the extra subys->auto_setup() hook to be avoidable bloat?
thanks,
-serge
Quoted 8 lines · Serge E. Hallyn
I do fear that that could become a maintenance nightmare. For instance right now there's the call to
fsnotify_mkdir(). Other such hooks might be placed atvfs_mkdir, which we'd then likely want to have placed in ourcontainer_mkdir()andcontainer_clone()fns. And of coursemay_create()is static inline infs/namei.c. It's trivial, but still if it changes we'd want to change the version inkernel/container.cas well.
Do we need to actually need to respect may_create() in container_clone()? I guess it would provide a way for root to control which processes could unshare namespaces.
Quoted 4 lines
What would be the main advantage of doing it this way? Do you consider the extra
subys->auto_setup()hook to be avoidable bloat?
I was thinking that it would be nice to be able to atomically set up the resources in the new container at the point when it's created rather than later. But I guess this way can work too. Can we call it something like "clone()" rather than "auto_setup()"?
Paul
Quoted 19 lines · Paul Menage
Quoted 8 lines · Serge E. Hallyn
I do fear that that could become a maintenance nightmare. For instance right now there's the call to
fsnotify_mkdir(). Other such hooks might be placed atvfs_mkdir, which we'd then likely want to have placed in ourcontainer_mkdir()andcontainer_clone()fns. And of coursemay_create()is static inline infs/namei.c. It's trivial, but still if it changes we'd want to change the version inkernel/container.cas well.Do we need to actually need to respect
may_create()incontainer_clone()? I guess it would provide a way for root to control which processes could unshare namespaces.Quoted 4 lines
What would be the main advantage of doing it this way? Do you consider the extra
subys->auto_setup()hook to be avoidable bloat?
Quoted 6 lines
I was thinking that it would be nice to be able to atomically set up the resources in the new container at the point when it's created rather than later. But I guess this way can work too. Can we call it something like "
clone()" rather than "auto_setup()"?Paul
clone() implies it does the actual cloning, so how about post_clone() as in the patch below?
I'm still not saying I'm entirely opposed to moving the vfs_mkdir logic straight into container_clone() - it's more that I would expect other people to object when they saw that. So if you decide you don't like the end result with this patch, let me know and I'll give that a shot.
Paul (Jackson), is this comment added in cpusets close enough to what you were asking for?
thanks,
-serge
Quoted 1 line
From
c2f1a39b231f06cb524c6e95d74de6ddee286f25Mon Sep 17 00:00:00 2001
From: Serge E. Hallyn <[email protected]>
Date: Fri, 8 Jun 2007 15:36:59 -0400
Subject: [PATCH 4/4] containers: minor clone cleanup
rename auto_setup() to post_clone(), and comment the cpusets version.
--- Documentation/containers.txt | 10 +++++----- include/linux/container.h | 2 +- kernel/container.c | 4 ++-- kernel/cpuset.c | 20 ++++++++++++++++++-- 4 files changed, 26 insertions(+), 10 deletions(-)
Documentation/containers.txt +5 -5
@@ -514,12 +514,12 @@ include/linux/container.h for details). Note that although this method can return an error code, the error code is currently not always handled well. -void auto_setup(struct container_subsys *ss, struct container *cont) +void post_clone(struct container_subsys *ss, struct container *cont) -Called at container_clone() to do any paramater initialization -which might be required before a task could attach. For example -in cpusets, no task may attach before 'cpus' and 'mems' are -set up. +Called at the end of container_clone() to do any paramater +initialization which might be required before a task could attach. For +example in cpusets, no task may attach before 'cpus' and 'mems' are set +up. void bind(struct container_subsys *ss, struct container *root) LL=callback_mutex
include/linux/container.h +1 -1
@@ -213,7 +213,7 @@ struct container_subsys {void (*exit)(struct container_subsys *ss,struct task_struct *task);int (*populate)(struct container_subsys *ss,struct container *cont); -void (*auto_setup)(struct container_subsys *ss,struct container *cont); +void (*post_clone)(struct container_subsys *ss,struct container *cont);void (*bind)(struct container_subsys *ss,struct container *root);int subsys_id;int active;
kernel/container.c +2 -2
@@ -2400,8 +2400,8 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)/* do any required auto-setup */ for_each_subsys (root, ss) { -if (ss->auto_setup) - ss->auto_setup (ss, child); +if (ss->post_clone) + ss->post_clone (ss, child); }/* All seems fine. Finish by moving the task into the new container */
kernel/cpuset.c +18 -2
@@ -1189,7 +1189,23 @@ int cpuset_populate(struct container_subsys *ss, struct container *cont)return 0 ; } -void cpuset_auto_setup (struct container_subsys *ss, +/* +* post_clone() is called at the end of container_clone(). +* 'container' was just created automatically as a result of +* a container_clone(), and the current task is about to +* be moved into 'container'. +* +* Currently we refuse to set up the container - thereby +* refusing the task to be entered, and as a result refusing +* the sys_unshare() or clone() which initiated it - if any +* sibling cpusets have exclusive cpus or mem. +* +* If this becomes a problem for some users who wish to +* allow that scenario, then cpuset_post_clone() could be +* changed to grant parent->cpus_allowed-sibling_cpus_exclusive +* (and likewise for mems) to the new container. +*/ +void cpuset_post_clone (struct container_subsys *ss,struct container *container) {struct container *parent, *child;
@@ -1269,7 +1285,7 @@ struct container_subsys cpuset_subsys = { .can_attach = cpuset_can_attach, .attach = cpuset_attach, .populate = cpuset_populate, - .auto_setup = cpuset_auto_setup, + .post_clone = cpuset_post_clone, .subsys_id = cpuset_subsys_id, .early_init = 1, };
Signature · 1 line
1.5.1.1.GIT
Quoted 2 lines · Serge
Paul (Jackson), is this comment added in cpusets close enough to what you were asking for?
This comment?
+ * Currently we refuse to set up the container - thereby
+ * refusing the task to be entered, and as a result refusing
+ * the sys_unshare() or clone() which initiated it - if any
+ * sibling cpusets have exclusive cpus or mem.
+ *
+ * If this becomes a problem for some users who wish to
+ * allow that scenario, then cpuset_post_clone() could be
+ * changed to grant parent->cpus_allowed-sibling_cpus_exclusive
+ * (and likewise for mems) to the new container.
Nice - thanks.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 4 lines
Anyway the patch I sent is simple enough, and if users end up demanding the ability to better deal with exclusive cpusets, the patch will be simple enough to extend by changing
cpuset_auto_setup(), so let's stick with that patch since it's your preference (IIUC).
Yeah - probably so.
When someone gets serious about things like checkpoint, restart, and migrate functionality, based on this container cloning, working with cpusets, they will probably have to revisit this interaction with exclusive cpusets.
Perhaps a comment could be put in the code, saying something like the above, so whomever does this will realize they are traveling in unchartered territory.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 4 lines · Serge E. Hallyn
2. I can't delete containers because of the files they contain, and am not allowed to delete those files by hand.
You should be able to delete a container with rmdir as long as it's not in use - its control files will get cleaned up automatically.
If you're getting an EBUSY error that means that either there are still tasks running in the container (look in the "tasks" file) or else there's a reference counting bug somewhere.
Can you post an example to reproduce the problem?
Paul
Quoted 6 lines · Paul Menage
Quoted 4 lines · Serge E. Hallyn
2. I can't delete containers because of the files they contain, and am not allowed to delete those files by hand.
Quoted 8 lines
You should be able to delete a container with rmdir as long as it's not in use - its control files will get cleaned up automatically.
If you're getting an EBUSY error that means that either there are still tasks running in the container (look in the "tasks" file) or else there's a reference counting bug somewhere.
Can you post an example to reproduce the problem?
here is an excerpt:
[root@linuz11 root]# mount -t container -ocpuset cpuset /containers/ [root@linuz11 root]# ls /containers/
cpu_exclusive memory_pressure mems tasks cpus memory_pressure_enabled notify_on_release mem_exclusive memory_spread_page releasable memory_migrate memory_spread_slab release_agent
[root@linuz11 root]# mkdir /containers/1
[root@linuz11 root]# echo 0 > /containers/1/mems [root@linuz11 root]# echo 0 > /containers/1/cpus [root@linuz11 root]# sh
sh-2.05b# echo $$ > /containers/1/tasks
sh-2.05b# cat /containers/1/tasks
4325
4326
sh-2.05b# exit
[root@linuz11 root]# ls /containers/1/tasks /containers/1/tasks
[root@linuz11 root]# rm -rf /containers/1
rm: cannot remove `/containers/1/memory_spread_slab': Operation not permitted
rm: cannot remove `/containers/1/memory_spread_page': Operation not permitted
rm: cannot remove `/containers/1/memory_pressure': Operation not permitted
rm: cannot remove `/containers/1/memory_migrate': Operation not permitted
rm: cannot remove `/containers/1/mem_exclusive': Operation not permitted
rm: cannot remove `/containers/1/cpu_exclusive': Operation not permitted
rm: cannot remove `/containers/1/mems': Operation not permitted
rm: cannot remove `/containers/1/cpus': Operation not permitted
rm: cannot remove `/containers/1/releasable': Operation not permitted
rm: cannot remove `/containers/1/notify_on_release': Operation not permitted
rm: cannot remove `/containers/1/tasks': Operation not permitted
Ah, I see the second time I typed 'ls /containers/1/tasks' instead of cat. When I then used cat, the file was empty, and I got an oops just like Pavel reported. I bet if I solve the problem he reported, then I solve my problem :)
thanks,
-serge
Quoted 1 line · Serge E. Hallyn
root@linuz11 root]# rm -rf /containers/1
Just use "rmdir /containers/1" here.
Quoted 6 lines
Ah, I see the second time I typed 'ls /containers/1/tasks' instead of cat. When I then used cat, the file was empty, and I got an oops just like Pavel reported. I bet if I solve the problem he reported, then I solve my problem :)
As far as I could see, Pavel's problem wasn't actually an Oops, it was a WARN_ON() when allocating a zero length chunk of memory. There's ongoing discussion as to whether this counts as a problem with the allocators or the kmalloc() code, since it used to be OK to allocate a zero-length chunk.
Paul
Quoted 4 lines · Paul Menage
Quoted 1 line · Serge E. Hallyn
root@linuz11 root]# rm -rf /containers/1
Just use "rmdir /containers/1" here.
Hmm. Ok, that works... Odd, I thought rm -rf used to work in the past, but i'm likely wrong.
thanks,
-serge
Quoted 13 lines
Quoted 5 lines
Ah, I see the second time I typed 'ls /containers/1/tasks' instead of cat. When I then used cat, the file was empty, and I got an oops just like Pavel reported. I bet if I solve the problem he reported, then I solve my problem :)
As far as I could see, Pavel's problem wasn't actually an Oops, it was a
WARN_ON()when allocating a zero length chunk of memory. There's ongoing discussion as to whether this counts as a problem with the allocators or thekmalloc()code, since it used to be OK to allocate a zero-length chunk.Paul
Quoted 2 lines · Serge
Odd, I thought rm -rf used to work in the past,
but i'm likely wrong.
I'm pretty sure it never worked.
And I've probably tested it myself, every few months, since the birth of cpusets, when I forget and type it again, and then stare dumbly at the screen wondering what all the complaining is about.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 1 line
[root@linuz11 root]# rm -rf /containers/1
No - not 'rm -fr'.
'rmdir'
Remove the cpuset directory, not start bottom up trying to remove the files first.
The poor 'rm -fr' command doesn't understand the rather odd nature of cpuset file systems, which have all files coming and going automagically in the middle of the night, and only allow mkdir and rmdir operations, not creat or unlink.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401