Per container statistics (containerstats)
Hi, Andrew/Paul,
Here's the latest version of containerstats ported to v10. Could you please consider it for inclusion
Changelog
1. Instead of parsing long container path's use the dentry to match the container for which stats are required. The user space application opens the container directory and passes the file descriptor, which is used to determine the container for which stats are required. This approach was suggested by Paul Menage
This patch is inspired by the discussion at http://lkml.org/lkml/2007/4/11/187
and implements per container statistics as suggested by Andrew Morton in http://lkml.org/lkml/2007/4/11/263. The patch is on top of 2.6.21-mm1 with Paul's containers v9 patches (forward ported)
This patch implements per container statistics infrastructure and re-uses code from the taskstats interface. A new set of container operations are registered with commands and attributes. It should be very easy to *extend* per container statistics, by adding members to the containerstats structure.
The current model for containerstats is a pull, a push model (to post statistics on interesting events), should be very easy to add. Currently user space requests for statistics by passing the container file descriptor. Statistics about the state of all the tasks in the container is returned to user space.
TODO's/NOTE:
This patch provides an infrastructure for implementing container statistics. Based on the needs of each controller, we can incrementally add more statistics, event based support for notification of statistics, accumulation of taskstats into container statistics in the future.
Sample output
# ./containerstats -C /container/a sleeping 2, blocked 0, running 1, stopped 0, uninterruptible 0
# ./containerstats -C /container/ sleeping 154, blocked 0, running 0, stopped 0, uninterruptible 0
If the approach looks good, I'll enhance and post the user space utility for the same
Feedback, comments, test results are always welcome!
---
Documentation/accounting/containerstats.txt | 27 ++++++++++ include/linux/Kbuild | 1 include/linux/container.h | 8 +++ include/linux/containerstats.h | 70 ++++++++++++++++++++++++++++ include/linux/delayacct.h | 11 ++++ kernel/container.c | 63 +++++++++++++++++++++++++ kernel/sched.c | 4 + kernel/taskstats.c | 66 ++++++++++++++++++++++++++ 8 files changed, 250 insertions(+)
Documentation/accounting/containerstats.txt new file +27 -0
@@ -0,0 +1,27 @@ +Containerstats is inspired by the discussion at +http://lkml.org/lkml/2007/4/11/187 and implements per container statistics as +suggested by Andrew Morton in http://lkml.org/lkml/2007/4/11/263. + +Per container statistics infrastructure re-uses code from the taskstats +interface. A new set of container operations are registered with commands +and attributes specific to containers. It should be very easy to +extend per container statistics, by adding members to the containerstats +structure. + +The current model for containerstats is a pull, a push model (to post +statistics on interesting events), should be very easy to add. Currently +user space requests for statistics by passing the container path. +Statistics about the state of all the tasks in the container is returned to +user space. + +NOTE: We currently rely on delay accounting for extracting information +about tasks blocked on I/O. If CONFIG_TASK_DELAY_ACCT is disabled, this +information will not be available. + +To extract container statistics a utility very similar to getdelays.c +has been developed, the sample output of the utility is shown below + +~/balbir/containerstats # ./containerstats -C "/container/a" +sleeping 1, blocked 0, running 1, stopped 0, uninterruptible 0 +~/balbir/containerstats # ./containerstats -C "/container" +sleeping 155, blocked 0, running 1, stopped 0, uninterruptible 2
include/linux/container.h renamed from include/linux/container.h~containers-taskstats +8 -0
@@ -12,6 +12,7 @@#include <linux/kref.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/containerstats.h> #ifdef CONFIG_CONTAINERS
@@ -21,6 +22,8 @@ extern void container_init_smp(void);extern void container_fork (struct task_struct *p);extern void container_fork_callbacks (struct task_struct *p);extern void container_exit (struct task_struct *p,int run_callbacks); +extern int containerstats_build (struct containerstats *stats, +struct dentry *dentry);extern struct file_operations proc_container_operations;
@@ -288,6 +291,11 @@ static inline void container_exit(structstatic inline void container_lock (void ) {}static inline void container_unlock (void ) {} +static inline int containerstats_build (struct containerstats *stats, +struct dentry *dentry) +{ +return -EINVAL ; +}#endif /* !CONFIG_CONTAINERS */
include/linux/containerstats.h new file +70 -0
70 lines not on this page · raw message
include/linux/delayacct.h renamed from include/linux/delayacct.h~containers-taskstats +11 -0
@@ -26,6 +26,7 @@* Used to set current->delays->flags */ #define DELAYACCT_PF_SWAPIN 0x00000001/* I am doing a swapin */ +#define DELAYACCT_PF_BLKIO 0x00000002/* I am waiting on IO */ #ifdef CONFIG_TASK_DELAY_ACCT
@@ -39,6 +40,14 @@ extern void __delayacct_blkio_end(void);extern int __delayacct_add_tsk (struct taskstats *,struct task_struct *);extern __u64 __delayacct_blkio_ticks (struct task_struct *); +static inline int delayacct_is_task_waiting_on_io (struct task_struct *p) +{ +if (p->delays) +return (p->delays->flags &DELAYACCT_PF_BLKIO ); +else +return 0 ; +} +static inline void delayacct_set_flag (int flag) {if (current->delays)
@@ -116,6 +125,8 @@ static inline int delayacct_add_tsk(stru {return 0 ; }static inline __u64 delayacct_blkio_ticks (struct task_struct *tsk) {return 0 ; } +static inline int delayacct_is_task_waiting_on_io (struct task_struct *p) +{return 0 ; }#endif /* CONFIG_TASK_DELAY_ACCT */ #endif
include/linux/Kbuild renamed from include/linux/Kbuild~containers-taskstats +1 -0
@@ -47,6 +47,7 @@ header-y += coff.h header-y += comstats.h header-y += consolemap.h header-y += const.h +header-y += containerstats.h header-y += cycx_cfm.h header-y += dlm_device.h header-y += dlm_netlink.h
kernel/container.c renamed from kernel/container.c~containers-taskstats +63 -0
75 lines not on this page · raw message
kernel/sched.c renamed from kernel/sched.c~containers-taskstats +4 -0
@@ -4280,11 +4280,13 @@ void __sched io_schedule(void) {struct rq *rq = &__raw_get_cpu_var (runqueues); +delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();atomic_inc (&rq->nr_iowait);schedule ();atomic_dec (&rq->nr_iowait);delayacct_blkio_end (); +delayacct_clear_flag (DELAYACCT_PF_BLKIO ); }EXPORT_SYMBOL (io_schedule);
@@ -4293,11 +4295,13 @@ long __sched io_schedule_timeout(long tistruct rq *rq = &__raw_get_cpu_var (runqueues);long ret; +delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();atomic_inc (&rq->nr_iowait); ret =schedule_timeout (timeout);atomic_dec (&rq->nr_iowait);delayacct_blkio_end (); +delayacct_clear_flag (DELAYACCT_PF_BLKIO );return ret; }
kernel/taskstats.c renamed from kernel/taskstats.c~containers-taskstats +66 -0
98 lines not on this page · raw message
diff -puN kernel/cpuset.c~containers-taskstats kernel/cpuset.c _
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 55 lines · On Wed, 6 Jun 2007 17:28:13 +0530 Balbir Singh <[email protected]>…
Hi, Andrew/Paul,
Here's the latest version of containerstats ported to v10. Could you please consider it for inclusion
Changelog
1. Instead of parsing long container path's use the dentry to match the container for which stats are required. The user space application opens the container directory and passes the file descriptor, which is used to determine the container for which stats are required. This approach was suggested by Paul Menage
This patch is inspired by the discussion at http://lkml.org/lkml/2007/4/11/187
and implements per container statistics as suggested by Andrew Morton in http://lkml.org/lkml/2007/4/11/263. The patch is on top of 2.6.21-mm1 with Paul's containers v9 patches (forward ported)This patch implements per container statistics infrastructure and re-uses code from the taskstats interface. A new set of container operations are registered with commands and attributes. It should be very easy to *extend* per container statistics, by adding members to the containerstats structure.
The current model for containerstats is a pull, a push model (to post statistics on interesting events), should be very easy to add. Currently user space requests for statistics by passing the container file descriptor. Statistics about the state of all the tasks in the container is returned to user space.
TODO's/NOTE:
This patch provides an infrastructure for implementing container statistics. Based on the needs of each controller, we can incrementally add more statistics, event based support for notification of statistics, accumulation of taskstats into container statistics in the future.
Sample output
# ./containerstats -C /container/a sleeping 2, blocked 0, running 1, stopped 0, uninterruptible 0# ./containerstats -C /container/ sleeping 154, blocked 0, running 0, stopped 0, uninterruptible 0If the approach looks good, I'll enhance and post the user space utility for the same
Feedback, comments, test results are always welcome!
Signed-off-by Balbir Singh <[email protected]>---
Quoted 9 lines
Documentation/accounting/containerstats.txt | 27 ++++++++++ include/linux/Kbuild | 1 include/linux/container.h | 8 +++ include/linux/containerstats.h | 70 ++++++++++++++++++++++++++++ include/linux/delayacct.h | 11 ++++ kernel/container.c | 63 +++++++++++++++++++++++++ kernel/sched.c | 4 + kernel/taskstats.c | 66 ++++++++++++++++++++++++++ 8 files changed, 250 insertions(+)
I'd have hoped to see containerstats.c in here.
Quoted 12 lines
diff -puN
/dev/nullinclude/linux/containerstats.hinclude/linux/containerstats.h new file +8
-0@@ -0,0 +1,70 @@ +/* containerstats.h - exporting per-container statistics +* +* __ Copyright IBM Corporation, 2007 +* Author Balbir Singh <[email protected]> +* +* This program is free software; you can redistribute it and/or modify it +* under the terms of version 2.1 of the GNU Lesser General Public License +* as published by the Free Software Foundation.
include/linux/containerstats.h line 18
Quoted 10 lines
include/linux/containerstats.h new file +10
-0+* +* This program is distributed in the hope that it would be useful, but +* WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +*/ + +#ifndef _LINUX_CONTAINERSTATS_H +#define _LINUX_CONTAINERSTATS_H + +#include <linux/taskstats.h>
I don't understand the relationship between containerstats and taskstats. afacit it's using the same genetlink channel?
Quoted 17 lines
include/linux/containerstats.h new file +17
-0+/* +* Data shared between user space and kernel space on a per container +* basis. This data is shared using taskstats. +* +* Most of these states are derived by looking at the task->state value +* For the nr_io_wait state, a flag in the delay accounting structure +* indicates that the task is waiting on IO +* +* Each member is aligned to a 8 byte boundary. +*/ +struct containerstats { +__u64 nr_sleeping;/* Number of tasks sleeping */ +__u64 nr_running;/* Number of tasks running */ +__u64 nr_stopped;/* Number of tasks in stopped state */ +__u64 nr_uninterruptible;/* Number of tasks in uninterruptible */ +/* state */ +__u64 nr_io_wait;/* Number of tasks waiting on IO */
include/linux/containerstats.h line 46
Quoted 10 lines
include/linux/containerstats.h new file +10
-0+}; + +/* +* Commands sent from userspace +* Not versioned. New commands should only be inserted at the enum's end +* prior to __CONTAINERSTATS_CMD_MAX +*/ + +enum { +CONTAINERSTATS_CMD_UNSPEC = __TASKSTATS_CMD_MAX,/* Reserved */
This seems to mean that the containerstats commands all get renumbered if we add new taskstats commands. That would be bad?
Quoted 8 lines
include/linux/containerstats.h new file +8
-0+*/ +int containerstats_build (struct containerstats *stats,struct dentry *dentry) +{ +int ret = -EINVAL ; +struct task_struct *g, *p; +struct container *cont, *root_cont; +struct container *src_cont; +int subsys_id;
include/linux/containerstats.h line 63
Quoted 10 lines
include/linux/containerstats.h new file +10
-0+struct containerfs_root *root; + +/* +* Validate dentry by checking the superblock operations +*/ +if (dentry->d_sb->s_op != &container_ops) + goto err; + + ret =0 ; + src_cont = (struct container *)dentry->d_fsdata;
Unneeded cast.
include/linux/containerstats.h
Quoted 8 lines
include/linux/containerstats.h new file +8
-0+rcu_read_lock (); + +for_each_root (root) { +if (!root->subsys_bits) +continue ; + root_cont = &root->top_container; +get_first_subsys (root_cont,NULL , &subsys_id); + do_each_thread(g, p) {
this needs tasklist_lock?
Quoted 35 lines
include/linux/containerstats.h new file +28
-0+ cont =task_container (p, subsys_id); +if (cont == src_cont) { +switch (p->state) { +case TASK_RUNNING : + stats->nr_running++; +break ; +case TASK_INTERRUPTIBLE : + stats->nr_sleeping++; +break ; +case TASK_UNINTERRUPTIBLE : + stats->nr_uninterruptible++; +break ; +case TASK_STOPPED : + stats->nr_stopped++; +break ; +default : +if (delayacct_is_task_waiting_on_io (p)) + stats->nr_io_wait++; +break ; + } + } + }while_each_thread (g, p); + } +rcu_read_unlock (); +err: +return ret; +} +static int cmppid (const void *a,const void *b) {return *(pid_t *)a - *(pid_t *)b;diff -puN
kernel/sched.c~containers-taskstatskernel/sched.c
kernel/sched.c void __sched io_schedule(void) · line 4284
Quoted 5 lines
kernel/sched.c renamed from kernel/sched.c~containers-taskstats +1
-0@@ -4280,11 +4280,13 @@ void __sched io_schedule(void) {struct rq *rq = &__raw_get_cpu_var (runqueues); +delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();
Would it be suitable and appropriate to embed the delayacct_set_flag() call inside delayacct_blkio_start()?
Quoted 3 lines · Andrew Morton
I'd have hoped to see
containerstats.cin here.
The current statistics code is really small, so it fit into taskstats.c. May be in the future, we could re-factor it and move it out.
Quoted 8 lines
include/linux/containerstats.h line 18
Quoted 4 lines
include/linux/containerstats.h new file +4
-0+#ifndef _LINUX_CONTAINERSTATS_H +#define _LINUX_CONTAINERSTATS_H + +#include <linux/taskstats.h> I don't understand the relationship between containerstats and taskstats. afacit it's using the same genetlink channel?
I've registered containerstats_ops with the same family as taskstats to reuse the taskstats interface.
Quoted 6 lines
include/linux/containerstats.h line 46
Quoted 2 lines
include/linux/containerstats.h new file +2
-0+enum { +CONTAINERSTATS_CMD_UNSPEC = __TASKSTATS_CMD_MAX,/* Reserved */ This seems to mean that the containerstats commands all get renumbered if we add new taskstats commands. That would be bad?
As per comment above, since we register containerstats_ops with the taskstats family, the commands need to be unique, hence we start where taskstats ended (left off)
Quoted 12 lines
Quoted 12 lines
+ */
kernel/container.c renamed from kernel/container.c~containers-taskstats +11
-0+int containerstats_build (struct containerstats *stats,struct dentry *dentry) +{ +int ret = -EINVAL ; +struct task_struct *g, *p; +struct container *cont, *root_cont; +struct container *src_cont; +int subsys_id; +struct containerfs_root *root; + +/* +* Validate dentry by checking the superblock operations
Quoted 9 lines
kernel/container.c static int pid_array_load(pid_t *pidarra · line 1636
Quoted 6 lines
kernel/container.c renamed from kernel/container.c~containers-taskstats +6
-0+*/ +if (dentry->d_sb->s_op != &container_ops) + goto err; + + ret =0 ; + src_cont = (struct container *)dentry->d_fsdata;Unneeded cast.
Will remove
Quoted 11 lines
kernel/container.c static int pid_array_load(pid_t *pidarra · line 1644
Quoted 8 lines
kernel/container.c renamed from kernel/container.c~containers-taskstats +8
-0+rcu_read_lock (); + +for_each_root (root) { +if (!root->subsys_bits) +continue ; + root_cont = &root->top_container; +get_first_subsys (root_cont,NULL , &subsys_id); + do_each_thread(g, p) {this needs
tasklist_lock?
rcu_read_lock() should be fine. From Eric's patch at
2.6.17-mm2 - proc-remove-tasklist_lock-from-proc_pid_readdir.patch
The patch mentions that "We don't need the tasklist_lock to safely iterate through processes anymore."
Quoted 38 lines
Quoted 38 lines
kernel/container.c renamed from kernel/container.c~containers-taskstats +28
-0+ cont =task_container (p, subsys_id); +if (cont == src_cont) { +switch (p->state) { +case TASK_RUNNING : + stats->nr_running++; +break ; +case TASK_INTERRUPTIBLE : + stats->nr_sleeping++; +break ; +case TASK_UNINTERRUPTIBLE : + stats->nr_uninterruptible++; +break ; +case TASK_STOPPED : + stats->nr_stopped++; +break ; +default : +if (delayacct_is_task_waiting_on_io (p)) + stats->nr_io_wait++; +break ; + } + } + }while_each_thread (g, p); + } +rcu_read_unlock (); +err: +return ret; +} +static int cmppid (const void *a,const void *b) {return *(pid_t *)a - *(pid_t *)b;diff -puN
kernel/sched.c~containers-taskstatskernel/sched.ckernel/sched.c renamed from kernel/sched.c~containers-taskstats +0
-0@@ -4280,11 +4280,13 @@ void __sched io_schedule(void) {struct rq *rq = &__raw_get_cpu_var (runqueues);
Quoted 6 lines
kernel/sched.c void __sched io_schedule(void) · line 4284
Quoted 2 lines
kernel/sched.c renamed from kernel/sched.c~containers-taskstats +1
-0+delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();Would it be suitable and appropriate to embed the
delayacct_set_flag()call insidedelayacct_blkio_start()?
Yes, I should have done that, will do.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 9 lines · Balbir Singh
Quoted 2 lines
this needs
tasklist_lock?
rcu_read_lock()should be fine. From Eric's patch at2.6.17-mm2 - proc-remove-tasklist_lock-from-proc_pid_readdir.patch
The patch mentions that "We don't need the
tasklist_lockto safely iterate through processes anymore."
Containers V10 includes an iterator interface for listing the member tasks of a container, which avoids scanning the entire tasklist. Downside is that it currently requires taking a read_lock on a global lock, but I hope to improve on that in the future.
Paul
Quoted 7 lines · On Fri, 08 Jun 2007 07:51:12 +0530 Balbir Singh <[email protected]>…
Quoted 3 lines · Andrew Morton
I'd have hoped to see
containerstats.cin here.The current statistics code is really small, so it fit into
taskstats.c. May be in the future, we could re-factor it and move it out.
I was referring to your userspace tool which reads this stuff. The one which you described in the changelog.
Quoted 12 lines
Quoted 11 lines
kernel/container.c static int pid_array_load(pid_t *pidarra · line 1644
Quoted 8 lines
kernel/container.c renamed from kernel/container.c~containers-taskstats +8
-0+rcu_read_lock (); + +for_each_root (root) { +if (!root->subsys_bits) +continue ; + root_cont = &root->top_container; +get_first_subsys (root_cont,NULL , &subsys_id); + do_each_thread(g, p) {this needs
tasklist_lock?
Quoted 7 lines
rcu_read_lock()should be fine. From Eric's patch at2.6.17-mm2 - proc-remove-tasklist_lock-from-proc_pid_readdir.patch
The patch mentions that "We don't need the
tasklist_lockto safely iterate through processes anymore."
oh, OK. rcu_read_lock() is the new lock_kernel() - always hard to tell what it's locking.
Quoted 11 lines · Andrew Morton
Quoted 5 lines · On Fri, 08 Jun 2007 07:51:12 +0530 Balbir Singh <[email protected]>…
Quoted 2 lines · Andrew Morton
I'd have hoped to see
containerstats.cin here.The current statistics code is really small, so it fit into
taskstats.c. May be in the future, we could re-factor it and move it out.I was referring to your userspace tool which reads this stuff. The one which you described in the changelog.
Ahh.. yes.. I'll make the changes and repost with the tool.
Quoted 12 lines
Quoted 12 lines
Quoted 10 lines
kernel/container.c static int pid_array_load(pid_t *pidarra · line 1644
Quoted 8 lines
kernel/container.c renamed from kernel/container.c~containers-taskstats +8
-0+rcu_read_lock (); + +for_each_root (root) { +if (!root->subsys_bits) +continue ; + root_cont = &root->top_container; +get_first_subsys (root_cont,NULL , &subsys_id); + do_each_thread(g, p) {this needs
tasklist_lock?
rcu_read_lock()should be fine. From Eric's patch at
Quoted 9 lines
Quoted 5 lines
2.6.17-mm2 - proc-remove-tasklist_lock-from-proc_pid_readdir.patch
The patch mentions that "We don't need the
tasklist_lockto safely iterate through processes anymore."oh, OK.
rcu_read_lock()is the newlock_kernel()- always hard to tell what it's locking.
:-) I'll add a comment to make it clearer.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 8 lines · Balbir Singh
Quoted 8 lines
Quoted 8 lines
return *(pid_t *)a - *(pid_t *)b;
diff -puNkernel/sched.c~containers-taskstatskernel/sched.ckernel/sched.c renamed from kernel/sched.c~containers-taskstats +0
-0@@ -4280,11 +4280,13 @@ void __sched io_schedule(void) {struct rq *rq = &__raw_get_cpu_var (runqueues);
Quoted 9 lines
Quoted 5 lines
kernel/sched.c void __sched io_schedule(void) · line 4284
Quoted 2 lines
kernel/sched.c renamed from kernel/sched.c~containers-taskstats +1
-0+delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();Would it be suitable and appropriate to embed the
delayacct_set_flag()call insidedelayacct_blkio_start()?Yes, I should have done that, will do.
My last reply was wrong, the flag needs to be set outside so that we know the context that initiated block I/O. We use this flag to track the time used by the swap block I/O.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 14 lines · Balbir Singh
Quoted 13 lines · Balbir Singh
Quoted 13 lines
kernel/sched.c void __sched io_schedule(void) · line 4284
Quoted 10 lines
return *(pid_t *)a - *(pid_t *)b;
diff -puNkernel/sched.c~containers-taskstatskernel/sched.ckernel/sched.c renamed from kernel/sched.c~containers-taskstats +1
-0@@ -4280,11 +4280,13 @@ void __sched io_schedule(void) {struct rq *rq = &__raw_get_cpu_var (runqueues); +delayacct_set_flag (DELAYACCT_PF_BLKIO );delayacct_blkio_start ();Would it be suitable and appropriate to embed the
delayacct_set_flag()call insidedelayacct_blkio_start()?
Quoted 8 lines
Quoted 3 lines
Yes, I should have done that, will do.
My last reply was wrong, the flag needs to be set outside so that we know the context that initiated block I/O. We use this flag to track the time used by the swap block I/O.
Please ignore this email.. it escaped my drafts folder... my fault.. apologies.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL