Skip to contentSkip to conversation

per-task delay accounting

PATCH In review Shailabh Nagar 5 participants 22 messages Apr 22, 2006 – Apr 28, 2006

Subsystems (11)
lore
Share & tools
lore
#0 lore raw

cover per-task delay accounting

Here are the delay accounting patches again. I'm not using the earlier email thread due to code being refactored a bit.

The previous posting

    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0603.3/1776.html

of these patches elicited several review comments from Andrew Morton all of which have been addressed.

The other main thread of the comments was whether other accounting stakeholders would be ok with this interface. Towards this end, I'd posted an overview of what the other packages do (which didn't seem to make the archives) and some of the stakeholders responded.

I'll repost the analysis as a reply to this post. Meanwhile, here's the list of the stakeholders identified by Andrew and a summary of status of their comments.

1. CSA accounting/PAGG/JOB: Jay Lan <[email protected]>

Raised several points

       http://www.uwsg.indiana.edu/hypermail/linux/kernel/0604.1/0397.html

all of which have been addressed in this set of patches.

2. per-process IO statistics: Levent Serinol <[email protected]>

No reponse.
I'd ascertained that its needs are a subset of CSA.

3. per-cpu time statistics: Erich Focht <[email protected]>

No response.
I'd ascertained that its needs can be met by taskstats interface whenever these statistics are submitted for inclusion.

4. Microstate accounting: Peter Chubb <[email protected]>

Mentioned overlap of patches with delay accounting
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0603.3/2286.html

and also that a /proc interface was preferable due to convenience. My position is that the netlink interface is a superset of /proc due to former's ability to supply exit-time data.

5. ELSA: Guillaume Thouvenin <[email protected]>

Confirmed that ELSA is not a direct user of a new kernel statistics interface since it is a consumer of CSA or BSD accounting's statistics.

6. pnotify: Jes Sorensen <[email protected]>
(taken over pnotify from Erik Jacobson)

Informed over private email that pnotify replacement is being worked on.

I'd ascertained that pnotify (or its replacemenent) will not be concerned with exporting data to userspace or collecting any stats. Thats left to the kernel module that uses pnotify to get notifications. CSA is one expected user of pnotify. Hence CSA's concerns are the only ones relevant to pnotify as well.

7. Scalable statistics counters with /proc reporting:
Ravikiran G Thirumalai, Dipankar Sarma <[email protected]>

Confirmed these counters aren't relevant to this discussion.

--Shailabh

Series

delayacct-setup.patch
delayacct-blkio-swapin.patch
delayacct-schedstats.patch
genetlink-utils.patch
taskstats-setup.patch
delayacct-taskstats.patch
delayacct-doc.patch
delayacct-procfs.patch

#1 lore raw

PATCH 1/8 Setup

Changelog

Fixes comments by akpm
- unnecessary initialization of delayacct_on
- use kmem_cache_zalloc
- redundant check in __delayacct_tsk_exit

delayacct-setup.patch

Initialization code related to collection of per-task "delay" statistics which measure how long it had to wait for cpu, sync block io, swapping etc. The collection of statistics and the interface are in other patches. This patch sets up the data structures and allows the statistics collection to be disabled through a kernel boot paramater.

 Documentation/kernel-parameters.txt |    2
 include/linux/delayacct.h           |   69 ++++++++++++++++++++++++++++
 include/linux/sched.h               |   21 ++++++++
 include/linux/time.h                |   10 ++++
 init/Kconfig                        |   13 +++++
 init/main.c                         |    2
 kernel/Makefile                     |    1
 kernel/delayacct.c                  |   87 ++++++++++++++++++++++++++++++++++++
 kernel/exit.c                       |    3 +
 kernel/fork.c                       |    2
 10 files changed, 210 insertions(+)
Documentation/kernel-parameters.txt +2 -0
Index: linux-2.6.17-rc1/Documentation/kernel-parameters.txt
===================================================================
@@ -430,6 +430,8 @@ running once the system is up.
 			Format: <area>[,<node>]
 			See also Documentation/networking/decnet.txt.
 
+	delayacct	[KNL] Enable per-task delay accounting
+
 	devfs=		[DEVFS]
 			See Documentation/filesystems/devfs/boot-options.
 
kernel/Makefile +1 -0
Index: linux-2.6.17-rc1/kernel/Makefile
===================================================================
@@ -38,6 +38,7 @@ obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_SECCOMP) += seccomp.o
 obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
 obj-$(CONFIG_RELAY) += relay.o
+obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
include/linux/delayacct.h new file +69 -0

69 lines not on this page · raw message

include/linux/sched.h +21 -0
Index: linux-2.6.17-rc1/include/linux/sched.h
===================================================================
@@ -536,6 +536,24 @@ struct sched_info {
 extern struct file_operations proc_schedstat_operations;
 #endif
 
+#ifdef CONFIG_TASK_DELAY_ACCT
+struct task_delay_info {
+	spinlock_t	lock;
+	unsigned int	flags;	/* Private per-task flags */
+
+	/* For each stat XXX, add following, aligned appropriately
+	 *
+	 * struct timespec XXX_start, XXX_end;
+	 * u64 XXX_delay;
+	 * u32 XXX_count;
+	 *
+	 * Atomicity of updates to XXX_delay, XXX_count protected by
+	 * single lock above (split into XXX_lock if contention is an issue).
+	 */
+};
+#endif
+
+
 enum idle_type
 {
 	SCHED_IDLE,
@@ -882,6 +900,9 @@ struct task_struct {
 
 	atomic_t fs_excl;	/* holding fs exclusive resources */
 	struct rcu_head rcu;
+#ifdef	CONFIG_TASK_DELAY_ACCT
+	struct task_delay_info *delays;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
init/Kconfig +13 -0
Index: linux-2.6.17-rc1/init/Kconfig
===================================================================
@@ -150,6 +150,19 @@ config BSD_PROCESS_ACCT_V3
 	  for processing it. A preliminary version of these tools is available
 	  at <http://www.physik3.uni-rostock.de/tim/kernel/utils/acct/>.
 
+config TASK_DELAY_ACCT
+	bool "Enable per-task delay accounting (EXPERIMENTAL)"
+	help
+	  Collect information on time spent by a task waiting for system
+	  resources like cpu, synchronous block I/O completion and swapping
+	  in pages. Such statistics can help in setting a task's priorities
+	  relative to other tasks for cpu, io, rss limits etc.
+
+	  Unlike BSD process accounting, this information is available
+	  continuously during the lifetime of a task.
+
+	  Say N if unsure.
+
 config SYSCTL
 	bool "Sysctl support"
 	---help---
init/main.c +2 -0
Index: linux-2.6.17-rc1/init/main.c
===================================================================
@@ -47,6 +47,7 @@
 #include <linux/rmap.h>
 #include <linux/mempolicy.h>
 #include <linux/key.h>
+#include <linux/delayacct.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -541,6 +542,7 @@ asmlinkage void __init start_kernel(void
 	proc_root_init();
 #endif
 	cpuset_init();
+	delayacct_init();
 
 	check_bugs();
 
kernel/delayacct.c new file +87 -0

87 lines not on this page · raw message

kernel/fork.c +2 -0
Index: linux-2.6.17-rc1/kernel/fork.c
===================================================================
@@ -44,6 +44,7 @@
 #include <linux/rmap.h>
 #include <linux/acct.h>
 #include <linux/cn_proc.h>
+#include <linux/delayacct.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>
@@ -989,6 +990,7 @@ static task_t *copy_process(unsigned lon
 		goto bad_fork_cleanup_put_domain;
 
 	p->did_exec = 0;
+	delayacct_tsk_init(p);	/* Must remain after dup_task_struct() */
 	copy_flags(clone_flags, p);
 	p->pid = pid;
 	retval = -EFAULT;
kernel/exit.c +3 -0
Index: linux-2.6.17-rc1/kernel/exit.c
===================================================================
@@ -34,6 +34,7 @@
 #include <linux/mutex.h>
 #include <linux/futex.h>
 #include <linux/compat.h>
+#include <linux/delayacct.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -893,6 +894,7 @@ fastcall NORET_TYPE void do_exit(long co
 				preempt_count());
 
 	acct_update_integrals(tsk);
+
 	if (tsk->mm) {
 		update_hiwater_rss(tsk->mm);
 		update_hiwater_vm(tsk->mm);
@@ -909,6 +911,7 @@ fastcall NORET_TYPE void do_exit(long co
 	if (unlikely(tsk->compat_robust_list))
 		compat_exit_robust_list(tsk);
 #endif
+	delayacct_tsk_exit(tsk);
 	exit_mm(tsk);
 
 	exit_sem(tsk);
include/linux/time.h +10 -0
Index: linux-2.6.17-rc1/include/linux/time.h
===================================================================
@@ -68,6 +68,16 @@ extern unsigned long mktime(const unsign
 extern void set_normalized_timespec(struct timespec *ts, time_t sec, long nsec);
 
 /*
+ * sub = end - start, in normalized form
+ */
+static inline void timespec_sub(struct timespec *start, struct timespec *end,
+				struct timespec *sub)
+{
+	set_normalized_timespec(sub, end->tv_sec - start->tv_sec,
+				end->tv_nsec - start->tv_nsec);
+}
+
+/*
  * Returns true if the timespec is norm, false if denorm:
  */
 #define timespec_valid(ts) \
#2 lore raw

Re: [Patch 1/8] Setup

include/linux/delayacct.h line 5

Quoted 10 lines · On Fri, 21 Apr 2006 22:23:25 -0400 Shailabh Nagar wrote:

Index: linux-2.6.17-rc1/include/linux/delayacct.h
===================================================================

include/linux/delayacct.h new file +5 -0
@@ -0,0 +1,69 @@
+/* delayacct.h - per-task delay accounting
+ */
+
+#ifndef _LINUX_TASKDELAYS_H
+#define _LINUX_TASKDELAYS_H

Probably _LINUX_DELAYACCT_H.
Or if I add linux/taskdelays.h, what #include guard should I use?

---
~Randy

#3 lore raw

Re: [Patch 1/8] Setup

Quoted 4 lines · Randy.Dunlap

On Fri, 21 Apr 2006 22:23:25 -0400 Shailabh Nagar wrote:

Quoted 16 lines

include/linux/delayacct.h line 5

Quoted 12 lines

Index: linux-2.6.17-rc1/include/linux/delayacct.h
===================================================================

include/linux/delayacct.h new file +5 -0
@@ -0,0 +1,69 @@
+/* delayacct.h - per-task delay accounting
+ */
+
+#ifndef _LINUX_TASKDELAYS_H
+#define _LINUX_TASKDELAYS_H

Probably _LINUX_DELAYACCT_H.

Yup. Hangover from old name...will fix.

Quoted 6 lines

Or if I add linux/taskdelays.h, what #include guard should I use?

---
~Randy

#4 lore raw

PATCH 2/8 Sync block I/O and swapin delay collection

In reply to Shailabh Nagar

Changelog

Fixes comments by akpm
- avoid creating new per-process flag PF_SWAPIN

delayacct-blkio-swapin.patch

Collect per-task block I/O delay statistics.

Unlike earlier iterations of the delay accounting patches, now delays are only collected for the actual I/O waits rather than try and cover the delays seen in I/O submission paths.

Account separately for block I/O delays
incurred as a result of swapin page faults whose frequency can be affected by the task/process' rss limit. Hence swapin delays can act as feedback for rss limit changes independent of I/O priority changes.

 include/linux/delayacct.h |   25 +++++++++++++++++++++++++
 include/linux/sched.h     |    6 ++++++
 kernel/delayacct.c        |   19 +++++++++++++++++++
 kernel/sched.c            |    5 +++++
 mm/memory.c               |    4 ++++
 5 files changed, 59 insertions(+)
include/linux/delayacct.h +25 -0
Index: linux-2.6.17-rc1/include/linux/delayacct.h
===================================================================
@@ -19,6 +19,13 @@
 
 #include <linux/sched.h>
 
+/*
+ * Per-task flags relevant to delay accounting
+ * maintained privately to avoid exhausting similar flags in sched.h:PF_*
+ * Used to set current->delays->flags
+ */
+#define DELAYACCT_PF_SWAPIN	0x00000001	/* I am doing a swapin */
+
 #ifdef CONFIG_TASK_DELAY_ACCT
 
 extern int delayacct_on;	/* Delay accounting turned on/off */
@@ -26,6 +33,8 @@ extern kmem_cache_t *delayacct_cache;
 extern void delayacct_init(void);
 extern void __delayacct_tsk_init(struct task_struct *);
 extern void __delayacct_tsk_exit(struct task_struct *);
+extern void __delayacct_blkio_start(void);
+extern void __delayacct_blkio_end(void);
 
 static inline void delayacct_set_flag(int flag)
 {
@@ -53,6 +62,18 @@ static inline void delayacct_tsk_exit(st
 		__delayacct_tsk_exit(tsk);
 }
 
+static inline void delayacct_blkio_start(void)
+{
+	if (current->delays)
+		__delayacct_blkio_start();
+}
+
+static inline void delayacct_blkio_end(void)
+{
+	if (current->delays)
+		__delayacct_blkio_end();
+}
+
 #else
 static inline void delayacct_set_flag(int flag)
 {}
@@ -64,6 +85,10 @@ static inline void delayacct_tsk_init(st
 {}
 static inline void delayacct_tsk_exit(struct task_struct *tsk)
 {}
+static inline void delayacct_blkio_start(void)
+{}
+static inline void delayacct_blkio_end(void)
+{}
 #endif /* CONFIG_TASK_DELAY_ACCT */
 
 #endif
kernel/delayacct.c +19 -0
Index: linux-2.6.17-rc1/kernel/delayacct.c
===================================================================
@@ -85,3 +85,22 @@ static inline void delayacct_end(struct
 	spin_unlock(&current->delays->lock);
 }
 
+void __delayacct_blkio_start(void)
+{
+	delayacct_start(&current->delays->blkio_start);
+}
+
+void __delayacct_blkio_end(void)
+{
+	if (current->delays->flags & DELAYACCT_PF_SWAPIN)
+		/* Swapin block I/O */
+		delayacct_end(&current->delays->blkio_start,
+			      &current->delays->blkio_end,
+			      &current->delays->swapin_delay,
+			      &current->delays->swapin_count);
+	else	/* Other block I/O */
+		delayacct_end(&current->delays->blkio_start,
+			      &current->delays->blkio_end,
+			      &current->delays->blkio_delay,
+			      &current->delays->blkio_count);
+}
include/linux/sched.h +6 -0
Index: linux-2.6.17-rc1/include/linux/sched.h
===================================================================
@@ -550,6 +550,12 @@ struct task_delay_info {
 	 * Atomicity of updates to XXX_delay, XXX_count protected by
 	 * single lock above (split into XXX_lock if contention is an issue).
 	 */
+
+	struct timespec blkio_start, blkio_end;	/* Shared by blkio, swapin */
+	u64 blkio_delay;	/* wait for sync block io completion */
+	u64 swapin_delay;	/* wait for swapin block io completion */
+	u32 blkio_count;
+	u32 swapin_count;
 };
 #endif
 
kernel/sched.c +5 -0
Index: linux-2.6.17-rc1/kernel/sched.c
===================================================================
@@ -50,6 +50,7 @@
 #include <linux/times.h>
 #include <linux/acct.h>
 #include <linux/kprobes.h>
+#include <linux/delayacct.h>
 #include <asm/tlb.h>
 
 #include <asm/unistd.h>
@@ -4144,9 +4145,11 @@ void __sched io_schedule(void)
 {
 	struct runqueue *rq = &per_cpu(runqueues, raw_smp_processor_id());
 
+	delayacct_blkio_start();
 	atomic_inc(&rq->nr_iowait);
 	schedule();
 	atomic_dec(&rq->nr_iowait);
+	delayacct_blkio_end();
 }
 
 EXPORT_SYMBOL(io_schedule);
@@ -4156,9 +4159,11 @@ long __sched io_schedule_timeout(long ti
 	struct runqueue *rq = &per_cpu(runqueues, raw_smp_processor_id());
 	long ret;
 
+	delayacct_blkio_start();
 	atomic_inc(&rq->nr_iowait);
 	ret = schedule_timeout(timeout);
 	atomic_dec(&rq->nr_iowait);
+	delayacct_blkio_end();
 	return ret;
 }
 
mm/memory.c +4 -0
Index: linux-2.6.17-rc1/mm/memory.c
===================================================================
@@ -48,6 +48,7 @@
 #include <linux/rmap.h>
 #include <linux/module.h>
 #include <linux/init.h>
+#include <linux/delayacct.h>
 
 #include <asm/pgalloc.h>
 #include <asm/uaccess.h>
@@ -1880,6 +1881,7 @@ static int do_swap_page(struct mm_struct
 
 	entry = pte_to_swp_entry(orig_pte);
 again:
+	delayacct_set_flag(DELAYACCT_PF_SWAPIN);
 	page = lookup_swap_cache(entry);
 	if (!page) {
  		swapin_readahead(entry, address, vma);
@@ -1892,6 +1894,7 @@ again:
 			page_table = pte_offset_map_lock(mm, pmd, address, &ptl);
 			if (likely(pte_same(*page_table, orig_pte)))
 				ret = VM_FAULT_OOM;
+			delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
 			goto unlock;
 		}
 
@@ -1903,6 +1906,7 @@ again:
 
 	mark_page_accessed(page);
 	lock_page(page);
+	delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
 	if (!PageSwapCache(page)) {
 		/* Page migration has occured */
 		unlock_page(page);
#5 lore raw

PATCH 3/8 cpu delay collection via schedstats

In reply to Shailabh Nagar

Changelog

Fixes comments by akpm
- comments about locking used in rq_sched_info_arrive/depart

No fix needed/possible
- redundant extern declaration of delayacct_on in sched.h suggested location (delayacct.h) cannot be used as it includes sched.h extern declaration moved to where its needed
- move unlikely declaration inside sched_info_on
Function only returns constants. Cannot be done.
- removal of #if defined in sched_fork (Dave Hansen) Refactoring suggested does not work if only SCHEDSTATS is configured

delayacct-shedstats.patch

Make the task-related schedstats functions
callable by delay accounting even if schedstats
collection isn't turned on. This removes the
dependency of delay accounting on schedstats.

 include/linux/sched.h |   21 +++++++++++++++---
 kernel/sched.c        |   56 ++++++++++++++++++++++++++++++++++----------------
 2 files changed, 56 insertions(+), 21 deletions(-)
include/linux/sched.h +17 -4
Index: linux-2.6.17-rc1/include/linux/sched.h
===================================================================
@@ -521,7 +521,7 @@ typedef struct prio_array prio_array_t;
 struct backing_dev_info;
 struct reclaim_state;
 
-#ifdef CONFIG_SCHEDSTATS
+#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 struct sched_info {
 	/* cumulative counters */
 	unsigned long	cpu_time,	/* time spent on the cpu */
@@ -532,9 +532,11 @@ struct sched_info {
 	unsigned long	last_arrival,	/* when we last ran on a cpu */
 			last_queued;	/* when we were last queued to run */
 };
+#endif /* defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT) */
 
+#ifdef CONFIG_SCHEDSTATS
 extern struct file_operations proc_schedstat_operations;
-#endif
+#endif /* CONFIG_SCHEDSTATS */
 
 #ifdef CONFIG_TASK_DELAY_ACCT
 struct task_delay_info {
@@ -557,8 +559,19 @@ struct task_delay_info {
 	u32 blkio_count;
 	u32 swapin_count;
 };
-#endif
+#endif	/* CONFIG_TASK_DELAY_ACCT */
 
+static inline int sched_info_on(void)
+{
+#ifdef CONFIG_SCHEDSTATS
+	return 1;
+#elif defined(CONFIG_TASK_DELAY_ACCT)
+	extern int delayacct_on;
+	return delayacct_on;
+#else
+	return 0;
+#endif
+}
 
 enum idle_type
 {
@@ -744,7 +757,7 @@ struct task_struct {
 	cpumask_t cpus_allowed;
 	unsigned int time_slice, first_time_slice;
 
-#ifdef CONFIG_SCHEDSTATS
+#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 	struct sched_info sched_info;
 #endif
 
kernel/sched.c +39 -17

119 lines not on this page · raw message

#6 lore raw

PATCH 4/8 Utilities for genetlink usage

In reply to Shailabh Nagar

genetlink-utils.patch

Two utilities for simplifying usage of NETLINK_GENERIC interface.

 include/net/genetlink.h |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+)
include/net/genetlink.h +20 -0
Index: linux-2.6.17-rc1/include/net/genetlink.h
===================================================================
@@ -150,4 +150,24 @@ static inline int genlmsg_unicast(struct
 	return nlmsg_unicast(genl_sock, skb, pid);
 }
 
+/**
+ * gennlmsg_data - head of message payload
+ * @gnlh: genetlink messsage header
+ */
+static inline void *genlmsg_data(const struct genlmsghdr *gnlh)
+{
+       return ((unsigned char *) gnlh + GENL_HDRLEN);
+}
+
+/**
+ * genlmsg_len - length of message payload
+ * @gnlh: genetlink message header
+ */
+static inline int genlmsg_len(const struct genlmsghdr *gnlh)
+{
+       struct nlmsghdr *nlh = (struct nlmsghdr *)((unsigned char *)gnlh -
+                                                   NLMSG_HDRLEN);
+       return (nlh->nlmsg_len - GENL_HDRLEN - NLMSG_HDRLEN);
+}
+
 #endif	/* __NET_GENERIC_NETLINK_H */

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

#7 lore raw

PATCH 5/8 taskstats interface

In reply to Shailabh Nagar

Changelog

Fixes comments by [email protected]
- separate out taskstats interface from delay accounting completely including separate documentation
- permit different accounting subsystems to fill in parts of common structure separately before common taskstats code sends it out on genetlink
- send common structure to userspace after update_hiwater_rss and before exit_mm in do_exit

Fixes comments by akpm
- comment to indicate locking used for taskstats struct
- whitespace issues
- unnecessary use of constant taskstats_version
- uninline fill_pid(), fill_tgid()
- unnecessary cast to pid_t in taskstats_send_stats()
- too early evaluation of thread_group_empty() in taskstats_exit_pid
- returning -EFAULT on genl_register_family failure in taskstats_init
- comment for late_initcall of taskstats_init

No fix needed
- moving kmem_cache_free of tsk->delays outside the exit mutex (mutex shifted and tsk->delays freeing being done elsewhere now)
- __delayacct_add_tsk returning -EINVAL if delay accounting isn't enabled user should know that no values can be returned returning zero would be misleading
- combining fill_pid(), fill_tgid() into a common function combined code convoluted and less readable

taskstats-setup.patch

Create a "taskstats" interface based on generic netlink (NETLINK_GENERIC family), for getting statistics of tasks and thread groups during their lifetime and when they exit. The interface is intended for use by multiple accounting packages though it is being created in the context of delay accounting.

This patch creates the interface without populating the fields of the data that is sent to the user in response to a command or upon the exit of a task. Each accounting package interested in using taskstats has to provide an additional patch to add its stats to the common structure.

 Documentation/accounting/taskstats.txt |  146 +++++++++++++++
 include/linux/taskstats.h              |   85 ++++++++
 include/linux/taskstats_kern.h         |   55 +++++
 init/Kconfig                           |   15 +
 init/main.c                            |    2
 kernel/Makefile                        |    1
 kernel/exit.c                          |    7
 kernel/taskstats.c                     |  321 +++++++++++++++++++++++++++++++++
 8 files changed, 629 insertions(+), 3 deletions(-)
include/linux/taskstats.h new file +85 -0
Index: linux-2.6.17-rc1/include/linux/taskstats.h
===================================================================
@@ -0,0 +1,85 @@
+/* taskstats.h - exporting per-task statistics
+ *
+ * Copyright (C) Shailabh Nagar, IBM Corp. 2006
+ *           (C) Balbir Singh,   IBM Corp. 2006
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of version 2.1 of the GNU Lesser General Public License
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ */
+
+#ifndef _LINUX_TASKSTATS_H
+#define _LINUX_TASKSTATS_H
+
+/* Format for per-task data returned to userland when
+ *	- a task exits
+ *	- listener requests stats for a task
+ *
+ * The struct is versioned. Newer versions should only add fields to
+ * the bottom of the struct to maintain backward compatibility.
+ *
+ *
+ * To add new fields
+ *	a) bump up TASKSTATS_VERSION
+ *	b) add comment indicating new version number at end of struct
+ *	c) add new fields after version comment; maintain 64-bit alignment
+ */
+
+#define TASKSTATS_VERSION	1
+
+struct taskstats {
+
+	/* Version 1 */
+
+	int filler_avoids_empty_struct_warnings;
+};
+
+
+#define TASKSTATS_LISTEN_GROUP	0x1
+
+/*
+ * Commands sent from userspace
+ * Not versioned. New commands should only be inserted at the enum's end
+ * prior to __TASKSTATS_CMD_MAX
+ */
+
+enum {
+	TASKSTATS_CMD_UNSPEC = 0,	/* Reserved */
+	TASKSTATS_CMD_GET,		/* user->kernel request/get-response */
+	TASKSTATS_CMD_NEW,		/* kernel->user event */
+	__TASKSTATS_CMD_MAX,
+};
+
+#define TASKSTATS_CMD_MAX (__TASKSTATS_CMD_MAX - 1)
+
+enum {
+	TASKSTATS_TYPE_UNSPEC = 0,	/* Reserved */
+	TASKSTATS_TYPE_PID,		/* Process id */
+	TASKSTATS_TYPE_TGID,		/* Thread group id */
+	TASKSTATS_TYPE_STATS,		/* taskstats structure */
+	TASKSTATS_TYPE_AGGR_PID,	/* contains pid + stats */
+	TASKSTATS_TYPE_AGGR_TGID,	/* contains tgid + stats */
+	__TASKSTATS_TYPE_MAX,
+};
+
+#define TASKSTATS_TYPE_MAX (__TASKSTATS_TYPE_MAX - 1)
+
+enum {
+	TASKSTATS_CMD_ATTR_UNSPEC = 0,
+	TASKSTATS_CMD_ATTR_PID,
+	TASKSTATS_CMD_ATTR_TGID,
+	__TASKSTATS_CMD_ATTR_MAX,
+};
+
+#define TASKSTATS_CMD_ATTR_MAX (__TASKSTATS_CMD_ATTR_MAX - 1)
+
+/* NETLINK_GENERIC related info */
+
+#define TASKSTATS_GENL_NAME	"TASKSTATS"
+#define TASKSTATS_GENL_VERSION	0x1
+
+#endif /* _LINUX_TASKSTATS_H */
init/Kconfig +12 -3
Index: linux-2.6.17-rc1/init/Kconfig
===================================================================
@@ -150,6 +150,18 @@ config BSD_PROCESS_ACCT_V3
 	  for processing it. A preliminary version of these tools is available
 	  at <http://www.physik3.uni-rostock.de/tim/kernel/utils/acct/>.
 
+config TASKSTATS
+	bool "Export task/process statistics through netlink (EXPERIMENTAL)"
+	default n
+	help
+	  Export selected statistics for tasks/processes through the
+	  generic netlink interface. Unlike BSD process accounting, the
+	  statistics are available during the lifetime of tasks/processes as
+	  responses to commands. Like BSD accounting, they are sent to user
+	  space on task exit.
+
+	  Say N if unsure.
+
 config TASK_DELAY_ACCT
 	bool "Enable per-task delay accounting (EXPERIMENTAL)"
 	help
@@ -158,9 +170,6 @@ config TASK_DELAY_ACCT
 	  in pages. Such statistics can help in setting a task's priorities
 	  relative to other tasks for cpu, io, rss limits etc.
 
-	  Unlike BSD process accounting, this information is available
-	  continuously during the lifetime of a task.
-
 	  Say N if unsure.
 
 config SYSCTL
kernel/Makefile +1 -0
Index: linux-2.6.17-rc1/kernel/Makefile
===================================================================
@@ -39,6 +39,7 @@ obj-$(CONFIG_SECCOMP) += seccomp.o
 obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
 obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
+obj-$(CONFIG_TASKSTATS) += taskstats.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/taskstats.c new file +321 -0

321 lines not on this page · raw message

kernel/exit.c +7 -0
Index: linux-2.6.17-rc1/kernel/exit.c
===================================================================
@@ -35,6 +35,7 @@
 #include <linux/futex.h>
 #include <linux/compat.h>
 #include <linux/delayacct.h>
+#include <linux/taskstats_kern.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -847,6 +848,7 @@ static void exit_notify(struct task_stru
 fastcall NORET_TYPE void do_exit(long code)
 {
 	struct task_struct *tsk = current;
+	struct taskstats *tidstats, *tgidstats;
 	int group_dead;
 
 	profile_task_exit(tsk);
@@ -893,6 +895,8 @@ fastcall NORET_TYPE void do_exit(long co
 				current->comm, current->pid,
 				preempt_count());
 
+	taskstats_exit_alloc(&tidstats, &tgidstats);
+
 	acct_update_integrals(tsk);
 
 	if (tsk->mm) {
@@ -911,7 +915,10 @@ fastcall NORET_TYPE void do_exit(long co
 	if (unlikely(tsk->compat_robust_list))
 		compat_exit_robust_list(tsk);
 #endif
+	taskstats_exit_send(tsk, tidstats, tgidstats);
+	taskstats_exit_free(tidstats, tgidstats);
 	delayacct_tsk_exit(tsk);
+
 	exit_mm(tsk);
 
 	exit_sem(tsk);
include/linux/taskstats_kern.h new file +55 -0

55 lines not on this page · raw message

Documentation/accounting/taskstats.txt new file +146 -0

147 lines not on this page · raw message

init/main.c +2 -0
Index: linux-2.6.17-rc1/init/main.c
===================================================================
@@ -47,6 +47,7 @@
 #include <linux/rmap.h>
 #include <linux/mempolicy.h>
 #include <linux/key.h>
+#include <linux/taskstats_kern.h>
 #include <linux/delayacct.h>
 
 #include <asm/io.h>
@@ -542,6 +543,7 @@ asmlinkage void __init start_kernel(void
 	proc_root_init();
 #endif
 	cpuset_init();
+	taskstats_init_early();
 	delayacct_init();
 
 	check_bugs();
#8 lore raw

Re: [Patch 5/8] taskstats interface

Hi Shailabh,

Thanks for your effort in taskstats interface! Really appreciated! I think this interface can offer a good foundation for other packages to build on.

Here are a few more comments:

1) You mentioned the "version number within the (taskstats) structure" in taskstats.txt and a few other places, but i do not see that field defined in struct taskstats in taskstats.h?

2) In taskstats.txt "Extending taskstats" section, you mentioned two ways to extend the interface. The second method looks like a method to encoureage other package developers to create their own interface (ie, not taskstats) based on generic netlink to avoid reading large
number
of fields not interested to other particular applications? I will be
fine
with this as long as it is understood and agreed.

Alternatively, you may have considered the pros and cons of #ifdef fields specific to only one accounting package in the struct taskstats. If you do, care to share your thoughts? Specific payload information can be carried in the version field. I am sure the version number of
struct
taskstats does not need 64 bits. With the version number and payload info, application can surely interpret the taskstats data correctly.

3) In taskstats.txt "Usage" section, you mentioned "... in the Advanced Usage section below...", but that section does not exist.

4) In do_exit() routine, you do:
+ taskstats_exit_alloc(&tidstats, &tgidstats);

The tidstats and tgidstats are checked in taskstats_exit_send() in taskstats.c for allocation failure, but a lot has been processed before the check. The allocation failure happens when system is stressed in memory. I think we want to do the check earlier?

Regards,
- jay

#9 lore raw

Re: [Patch 5/8] taskstats interface

Quoted 6 lines · Jay Lan

Hi Shailabh,

Thanks for your effort in taskstats interface! Really appreciated! I think this interface can offer a good foundation for other packages to build on.

Quoted 7 lines

Here are a few more comments:

1) You mentioned the "version number within the (taskstats) structure" in taskstats.txt and a few other places, but i do not see that field defined in struct taskstats in taskstats.h?

Missed out on that. Need to add it back in.

Quoted 10 lines

2) In taskstats.txt "Extending taskstats" section, you mentioned two ways to extend the interface. The second method looks like a method to encoureage other package developers to create their own interface (ie, not taskstats) based on generic netlink to avoid reading large
number
of fields not interested to other particular applications? I will be
fine
with this as long as it is understood and agreed.

Yes, the second method is for other packages, which have very little in common with the struct taskstats to extend the stats returned (using netlink attribs to extend rather than extending the structure).

Quoted 4 lines

Alternatively, you may have considered the pros and cons of #ifdef fields specific to only one accounting package in the struct taskstats. If you do, care to share your thoughts?

I'd rather avoid doing an #ifdef'ed definition of the fields based on configuration of one or the other accounting package...it'll add complexity for userspace parsing of the structure.

Its quite acceptable to have the fields have zero as content if the corresponding package isn't configured.

Quoted 7 lines

Specific payload information
can be carried in the version field. I am sure the version number of
struct
taskstats does not need 64 bits. With the version number and payload info, application can surely interpret the taskstats data correctly.

By "payload info" you mean some sort of bitmask (or encoding) which specifies which fields are present or absent ? I suppose that could be done but it adds unnecessary complexity ? e.g once delay accounting is there, all six to eight fields corresponding to it will be present...I don't see much value in further being able to configure cpu delays, mem delays etc. separately. Is that different for CSA ?

Quoted 4 lines

3) In taskstats.txt "Usage" section, you mentioned "... in the Advanced Usage section below...", but that section does not exist.

Thanks for pointing it out. Should replace it with "per-tgid stats section".

Quoted 9 lines

4) In do_exit() routine, you do:
+ taskstats_exit_alloc(&tidstats, &tgidstats);

The tidstats and tgidstats are checked in taskstats_exit_send() in taskstats.c for allocation failure, but a lot has been processed before the check. The allocation failure happens when system is stressed in memory. I think we want to do the check earlier?

Since accounting is non-critical, I didn't see the need for doing the check earlier if we're not going to do anything about it. The first use of the allocated structure is in the taskstats_exit_send() where filling of the stats is not done if allocation failed. What would you suggest we do, on allocation failure, if the check is performed immediately after the alloc ?

--Shailabh

Quoted 6 lines

Regards,
- jay

#10 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Quoted 8 lines · Shailabh Nagar
Quoted 6 lines · Jay Lan

Hi Shailabh,

Thanks for your effort in taskstats interface! Really appreciated! I think this interface can offer a good foundation for other packages to build on.

Quoted 8 lines
Quoted 7 lines

Here are a few more comments:

1) You mentioned the "version number within the (taskstats) structure" in taskstats.txt and a few other places, but i do not see that field defined in struct taskstats in taskstats.h?

Missed out on that. Need to add it back in.

There is a version field in genl_family as well. That can be used for versioning as well. When we user space tool queries for the family id, it can obtain and interpret the version information.

Quoted 11 lines
Quoted 11 lines

2) In taskstats.txt "Extending taskstats" section, you mentioned two ways to extend the interface. The second method looks like a method to encoureage other package developers to create their own interface (ie, not taskstats) based on generic netlink to avoid reading large
number
of fields not interested to other particular applications? I will be
fine
with this as long as it is understood and agreed.

Quoted 4 lines

Yes, the second method is for other packages, which have very little in common with the struct
taskstats to extend the stats returned (using netlink attribs to extend rather than extending the structure).

The second method will require the following

1. An API to return the length of data it wants to fill in
2. Another API to fill in the statistics along with the type -
Like Shailabh mentioned, this will require creating a new TASKSTATS_TYPE_XXXX

Quoted 5 lines
Quoted 5 lines

Alternatively, you may have considered the pros and cons of #ifdef fields specific to only one accounting package in the struct taskstats. If you do, care to share your thoughts?

Quoted 8 lines

I'd rather avoid doing an #ifdef'ed definition of the fields based on configuration of one or the other
accounting package...it'll add complexity for userspace parsing of the structure.

Its quite acceptable to have the fields have zero as content if the corresponding package isn't configured.

I agree with Shailabh, building in knowledge of other subsystems into taskstats.h might not be the best choice.

Quoted 8 lines
Quoted 8 lines

Specific payload information
can be carried in the version field. I am sure the version number of
struct
taskstats does not need 64 bits. With the version number and payload info, application can surely interpret the taskstats data correctly.

Quoted 7 lines

By "payload info" you mean some sort of bitmask (or encoding) which specifies which fields are present
or absent ? I suppose that could be done but it adds unnecessary complexity ? e.g once delay accounting is there,
all six to eight fields corresponding to it will be present...I don't see much value in further being able to configure
cpu delays, mem delays etc. separately. Is that different for CSA ?

Netlink attributes can be used to determine which attribute types are present in the payload. libnl does a great job of providing a good set of APIs to determine all attribute types present. This is one of the biggest advantages I see of genetlink (attributes are optional and can co-exist simultaneously)

Quoted 36 lines
Quoted 6 lines

3) In taskstats.txt "Usage" section, you mentioned "... in the Advanced Usage section below...", but that section does not exist.

Thanks for pointing it out. Should replace it with "per-tgid stats section".

Quoted 9 lines

4) In do_exit() routine, you do:
+ taskstats_exit_alloc(&tidstats, &tgidstats);

The tidstats and tgidstats are checked in taskstats_exit_send() in taskstats.c for allocation failure, but a lot has been processed before the check. The allocation failure happens when system is stressed in memory. I think we want to do the check earlier?

Since accounting is non-critical, I didn't see the need for doing the check earlier if we're not going to do
anything about it. The first use of the allocated structure is in the taskstats_exit_send() where filling of the
stats is not done if allocation failed. What would you suggest we do, on allocation failure, if the check is
performed immediately after the alloc ?

--Shailabh

Quoted 6 lines

Regards,
- jay

<snip>

					<---	Balbir
#11 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Quoted 19 lines · Balbir Singh
Quoted 17 lines · Shailabh Nagar
Quoted 13 lines · Jay Lan

Hi Shailabh,

Thanks for your effort in taskstats interface! Really appreciated! I think this interface can offer a good foundation for other packages to build on.

Here are a few more comments:

1) You mentioned the "version number within the (taskstats) structure" in taskstats.txt and a few other places, but i do not see that field defined in struct taskstats in taskstats.h?

Quoted 6 lines
Quoted 1 line

Missed out on that. Need to add it back in.

There is a version field in genl_family as well. That can be used for versioning as well. When we user space tool queries for the family id, it can obtain and interpret the version information.

Hi Shailabh and Balbir,

Are TASKSTATS_GENL_VERSION and TASKSTATS_VERSION the same thing? If they are meant to serve different purposes, we still need it.

Quoted 32 lines
Quoted 13 lines
Quoted 12 lines

2) In taskstats.txt "Extending taskstats" section, you mentioned two ways to extend the interface. The second method looks like a method to encoureage other package developers to create their own interface
(ie, not taskstats) based on generic netlink to avoid reading large number
of fields not interested to other particular applications? I will be
fine
with this as long as it is understood and agreed.

Quoted 2 lines

Yes, the second method is for other packages, which have very little in common with the struct taskstats to extend the stats returned (using netlink attribs to extend rather than extending the structure).

The second method will require the following

1. An API to return the length of data it wants to fill in
2. Another API to fill in the statistics along with the type -
Like Shailabh mentioned, this will require creating a new TASKSTATS_TYPE_XXXX

Quoted 8 lines
Quoted 4 lines

Alternatively, you may have considered the pros and cons of #ifdef fields specific to only one accounting package in the struct taskstats. If you do, care to share your thoughts?

I'd rather avoid doing an #ifdef'ed definition of the fields based on configuration of one or the other accounting package...it'll add complexity for userspace parsing of the structure.

Quoted 20 lines
Quoted 2 lines

Its quite acceptable to have the fields have zero as content if the corresponding package isn't configured.

I agree with Shailabh, building in knowledge of other subsystems into taskstats.h might not be the best choice.

Quoted 12 lines
Quoted 7 lines

Specific payload information
can be carried in the version field. I am sure the version number of
struct
taskstats does not need 64 bits. With the version number and payload info, application can surely interpret the taskstats data correctly.

By "payload info" you mean some sort of bitmask (or encoding) which specifies which fields are present or absent ? I suppose that could be done but it adds unnecessary complexity ? e.g once delay accounting is there, all six to eight fields corresponding to it will be present...I don't see much value in further being able to configure cpu delays, mem delays etc. separately. Is that different for CSA ?

I was thinking of a bitmask thing. But instead of keying specific fields, one bit may be used to key delay accounting, and another bit for CSA, el at. This way you do not need to have CSA-specifc fields in the payload and applications know how to correctly interpret the payload. Taskstats and application do not need to have knowledge of accounting packages, only need to set the bitmasks correctly.

When we start sending sys stats of each tasks to userland, that is s lot of data. Note that BSD accounting even uses encode_comp_t() routine to compress data into a 13-bit fraction with 3-bit exponent field to shrink its size. Even though you do not need to care about those zero's in taskstats, they still need to be delievered through netlink socket.

I must admit that this may create a point of failure due to the payload info not set correctly according to the CONFIG flags.

The idea was to eliminate the need of #2 methods, but maybe #2 method is better...

I am a little confused after reading Balbir's reply. It seems to me that Shailabh suggested to create a different struct to contain stats data. Is that also what Balbir talked about? If a different package builds a different taskstat-like interface as suggested in #2, would the data travel on the same socket as delay accounting?

Quoted 32 lines

Netlink attributes can be used to determine which attribute types are present in the payload. libnl does a great job of providing a good set of APIs to determine all attribute types present. This is one of the biggest advantages I see of genetlink (attributes are optional and can co-exist simultaneously)

Quoted 23 lines
Quoted 5 lines

3) In taskstats.txt "Usage" section, you mentioned "... in the Advanced Usage section below...", but that section does not exist.

Thanks for pointing it out. Should replace it with "per-tgid stats section".

Quoted 9 lines

4) In do_exit() routine, you do:
+ taskstats_exit_alloc(&tidstats, &tgidstats);

The tidstats and tgidstats are checked in taskstats_exit_send() in taskstats.c for allocation failure, but a lot has been processed before the check. The allocation failure happens when system is stressed in memory. I think we want to do the check earlier?

Since accounting is non-critical, I didn't see the need for doing the check earlier if we're not going to do anything about it. The first use of the allocated structure is in the taskstats_exit_send() where filling of the stats is not done if allocation failed. What would you suggest we do, on allocation failure, if the check is performed immediately after the alloc ?

I would suggest to do the check at the beginning of taskstats_exit_send() before mutex_lock(&taskstats_exit_mutex).

Regards,
- jay

#12 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Hi Jay,

Quoted 5 lines

Hi Shailabh and Balbir,

Are TASKSTATS_GENL_VERSION and TASKSTATS_VERSION the same thing? If they are meant to serve different purposes, we still need it.

Yes, thats true. But for now from what I can see, one version should be sufficient.

<snip>

Quoted 7 lines

I was thinking of a bitmask thing. But instead of keying specific fields, one bit may be used to key delay accounting, and another bit for CSA, el at. This way you do not need to have CSA-specifc fields in the payload and applications know how to correctly interpret the payload. Taskstats and application do not need to have knowledge of accounting packages, only need to set the bitmasks correctly.

Yes, but scanning the entire payload for various types is also feasible. It is a bit slow, but feasible and generally the recommended approach for dealing with genetlink types. What you are saying is still possible, the application can ignore types it does not understand.

Quoted 6 lines

When we start sending sys stats of each tasks to userland, that is s lot of data. Note that BSD accounting even uses encode_comp_t() routine to compress data into a 13-bit fraction with 3-bit exponent field to shrink its size. Even though you do not need to care about those zero's in taskstats, they still need to be delievered through netlink socket.

Yes, thats true. We can leave the decision of compressing, etc to the specific subsystem. It can encode it and the user level application can decode the data.

Quoted 7 lines

I must admit that this may create a point of failure due to the payload info not set correctly according to the CONFIG flags.

The idea was to eliminate the need of #2 methods, but maybe #2 method is better...

Quoted 6 lines

I am a little confused after reading Balbir's reply. It seems to me that Shailabh suggested to create a different struct to contain stats data. Is that also what Balbir talked about? If a different package builds a different taskstat-like interface as suggested in #2, would the data travel on the same socket as delay accounting?

Sorry for the confusion. Yes, even I would recommend creating a different struct for the stats data. The data will pass over the same socket as delay accounting (separate sockets can be used, but that would become inefficient).

Quoted 3 lines

I would suggest to do the check at the beginning of taskstats_exit_send() before mutex_lock(&taskstats_exit_mutex).

Good suggestion, we can move the check to that point.

Quoted 3 lines

Regards,
- jay

--

					Warm Regards, 
					<---	Balbir
#13 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Hi Balbir,

Quoted 7 lines · Balbir Singh
Quoted 3 lines

Are TASKSTATS_GENL_VERSION and TASKSTATS_VERSION the same thing? If they are meant to serve different purposes, we still need it.

Yes, thats true. But for now from what I can see, one version should be sufficient.

If we envision a need of it in the future, we'd better put it in today. It would be nice to have the revision number at beginning of the struct. Shailabh's instruction says to add new field after existing fields.

Quoted 27 lines

<snip>

Quoted 7 lines

I was thinking of a bitmask thing. But instead of keying specific fields, one bit may be used to key delay accounting, and another bit for CSA, el at. This way you do not need to have CSA-specifc fields in the payload and applications know how to correctly interpret the payload. Taskstats and application do not need to have knowledge of accounting packages, only need to set the bitmasks correctly.

Yes, but scanning the entire payload for various types is also feasible. It is a bit slow, but feasible and generally the recommended approach for dealing with genetlink types. What you are saying is still possible, the application can ignore types it does not understand.

Quoted 6 lines

When we start sending sys stats of each tasks to userland, that is s lot of data. Note that BSD accounting even uses encode_comp_t() routine to compress data into a 13-bit fraction with 3-bit exponent field to shrink its size. Even though you do not need to care about those zero's in taskstats, they still need to be delievered through netlink socket.

Quoted 3 lines

Yes, thats true. We can leave the decision of compressing, etc to the specific subsystem. It can encode it and the user level application can decode the data.

I am sorry that i did not make myself clear. My suggestion of using the bitmask payload info is to be combined with #ifdef CONFIG_* to eliminate unnecessary fields from the traffic. I am concerned about losing data due to application not reading data fast enough.

Well, we can revisit this suggestion when we start losing data though. ;-)

Regards,
- jay

#14 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Quoted 5 lines

If we envision a need of it in the future, we'd better put it in today. It would be nice to have the revision number at beginning of the struct. Shailabh's instruction says to add new field after existing fields.

Yes, true. It does not hurt to have a version number for taskstats. I will add it in.

<snip>

Quoted 8 lines

I am sorry that i did not make myself clear. My suggestion of using the bitmask payload info is to be combined with #ifdef CONFIG_* to eliminate unnecessary fields from the traffic. I am concerned about losing data due to application not reading data fast enough.

Well, we can revisit this suggestion when we start losing data though. ;-)

Like Shailabh said #ifdef CONFIG_* adds complexity for userspace parsing of the structure, but if it helps avoid sending unnecessary data we can consider using that approach.

Would something like the structure below be useful?

struct csastats {
#if defined(CONFIG_CSA) || defined(CONFIG_CSA_MODULE)
char acctent[sizeof(struct acctcsa) +
sizeof(struct acctmem) + sizeof(struct acctio)];
int filled;
#endif
};

The filled member can be a bool or an int to indicate that the structure contains meaningful data and the CONFIG_* is used to control the inclusion of meaningful fields. Instead of using a bitmap we use the filled member.

Is this what you had in mind?

Signature · 1 line
					<---	Balbir
#15 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Quoted 37 lines · Balbir Singh
Quoted 6 lines

If we envision a need of it in the future, we'd better put it in today. It would be nice to have the revision number at beginning of the struct. Shailabh's instruction says to add new field after existing fields.

Yes, true. It does not hurt to have a version number for taskstats. I will add it in.

<snip>

Quoted 8 lines

I am sorry that i did not make myself clear. My suggestion of using the bitmask payload info is to be combined with #ifdef CONFIG_* to eliminate unnecessary fields from the traffic. I am concerned about losing data due to application not reading data fast enough.

Well, we can revisit this suggestion when we start losing data though. ;-)

Like Shailabh said #ifdef CONFIG_* adds complexity for userspace parsing of the structure, but if it helps avoid sending unnecessary data we can consider using that approach.

Would something like the structure below be useful?

struct csastats {
#if defined(CONFIG_CSA) || defined(CONFIG_CSA_MODULE)
char acctent[sizeof(struct acctcsa) +
sizeof(struct acctmem) + sizeof(struct acctio)];
int filled;
#endif
};

Quoted 7 lines

The filled member can be a bool or an int to indicate that the structure contains meaningful data and the CONFIG_* is used to control the inclusion of meaningful fields. Instead of using a bitmap we use the filled member.

Is this what you had in mind?

No exactly. The payload information must be always available for application.

On a second thought, the idea of one big taskstats struct with many #ifconfig is not really a good idea. My goal is to cut down unnecessary data being transfered throught the socket.

Here is my Take 2. We can have a taskstats header containing taskstats version and other general fields useful to more than one taskstats application including a payload information. Then, we define accounting subsystem specific structs for delayacct, csa, etc. The kernel/{delayacct.c,csa.c,etc.c} set the payload information and fill the buffer with desired subsystem structs. The header thus contain enough information to tell applications how to map the data following the header.

Would IBM propose more accounting subsystems besides delayacct? If we only see delayacct and csa on the horizon, this scheme is really not necessary since delayacct does not have as much data (as csa :)) and csa can use part of the delayacct data. You gain more than csa can benefit from this. ;-) I guess i just speak from design point of view. :)

But, if one day somebody who does not need a paycheck decides to convert BSD accounting to use taskstats interface, this can be helpful.

Thanks,
- jay

#16 lore raw

Re: [Lse-tech] Re: [Patch 5/8] taskstats interface

Quoted 8 lines
Quoted 2 lines

Is this what you had in mind?

No exactly. The payload information must be always available for application.

On a second thought, the idea of one big taskstats struct with many #ifconfig is not really a good idea. My goal is to cut down unnecessary data being transfered throught the socket.

Yes, so we agree that #ifdef CONFIG_* is not good.

Quoted 9 lines

Here is my Take 2. We can have a taskstats header containing taskstats version and other general fields useful to more than one taskstats application including a payload information. Then, we define accounting subsystem specific structs for delayacct, csa, etc. The kernel/{delayacct.c,csa.c,etc.c} set the payload information and fill the buffer with desired subsystem structs. The header thus contain enough information to tell applications how to map the data following the header.

I agree with this suggestion.

Each netlink attribute contains the following fields (also referred to as TLV)

+----+--------+------+
|Type| length | value|
+----+--------+------+

The type is meant to serve the purpose of the header you describe. The type value can be used by the application to map the data. getdelays.c is a sample application posted in the previous patches, it interprets data based on type.

Quoted 8 lines

Would IBM propose more accounting subsystems besides delayacct? If we only see delayacct and csa on the horizon, this scheme is really not necessary since delayacct does not have as much data (as csa :)) and csa can use part of the delayacct data. You gain more than csa can benefit from this. ;-) I guess i just speak from design point of view. :)

Quoted 4 lines

But, if one day somebody who does not need a paycheck decides to convert BSD accounting to use taskstats interface, this can be helpful.

Yes, I think in the long term it would be more useful to use the scheme of adding subsystem structs. taskstats.txt explains the process of extending taskstats. Point #2 is the same as what we have just discussed. Could you please see if the text needs any changes based on our discussions so far (taskstats.txt was posted in the delayacct-doc.patch).

Quoted 4 lines

Thanks,
- jay

Signature · 1 line
					<---	Balbir
#17 lore raw

PATCH 6/8 delay accounting usage of taskstats interface

In reply to Shailabh Nagar

Changelog

Fixes comments by akpm (on earlier patch now incorporated here)
- detailed comments on atomicity rules of accounting fields
- replace use of nsec_t

delayacct-taskstats.patch

Usage of taskstats interface by delay accounting.

 include/linux/delayacct.h |   11 ++++++++++
 include/linux/taskstats.h |   48 +++++++++++++++++++++++++++++++++++++++++++++-
 init/Kconfig              |    1
 kernel/delayacct.c        |   42 ++++++++++++++++++++++++++++++++++++++++
 kernel/taskstats.c        |    9 +++++++-
 5 files changed, 109 insertions(+), 2 deletions(-)
include/linux/delayacct.h +11 -0
Index: linux-2.6.17-rc1/include/linux/delayacct.h
===================================================================
@@ -18,6 +18,7 @@
 #define _LINUX_TASKDELAYS_H
 
 #include <linux/sched.h>
+#include <linux/taskstats_kern.h>
 
 /*
  * Per-task flags relevant to delay accounting
@@ -35,6 +36,7 @@ extern void __delayacct_tsk_init(struct
 extern void __delayacct_tsk_exit(struct task_struct *);
 extern void __delayacct_blkio_start(void);
 extern void __delayacct_blkio_end(void);
+extern int __delayacct_add_tsk(struct taskstats *, struct task_struct *);
 
 static inline void delayacct_set_flag(int flag)
 {
@@ -74,6 +76,13 @@ static inline void delayacct_blkio_end(v
 		__delayacct_blkio_end();
 }
 
+static inline int delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
+{
+	if (!tsk->delays)
+		return -EINVAL;
+	return __delayacct_add_tsk(d, tsk);
+}
+
 #else
 static inline void delayacct_set_flag(int flag)
 {}
@@ -89,6 +98,8 @@ static inline void delayacct_blkio_start
 {}
 static inline void delayacct_blkio_end(void)
 {}
+static inline int delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
+{ return 0; }
 #endif /* CONFIG_TASK_DELAY_ACCT */
 
 #endif
kernel/delayacct.c +42 -0

45 lines not on this page · raw message

kernel/taskstats.c +8 -1
Index: linux-2.6.17-rc1/kernel/taskstats.c
===================================================================
@@ -18,6 +18,7 @@
 
 #include <linux/kernel.h>
 #include <linux/taskstats_kern.h>
+#include <linux/delayacct.h>
 #include <net/genetlink.h>
 #include <asm/atomic.h>
 
@@ -119,7 +120,9 @@ static int fill_pid(pid_t pid, struct ta
 	 *		goto err;
 	 */
 
-err:
+	rc = delayacct_add_tsk(stats, tsk);
+
+	/* Define err: label here if needed */
 	put_task_struct(tsk);
 	return rc;
 
@@ -151,6 +154,10 @@ static int fill_tgid(pid_t tgid, struct
 		 *		break;
 		 */
 
+		rc = delayacct_add_tsk(stats, tsk);
+		if (rc)
+			break;
+
 	} while_each_thread(first, tsk);
 	read_unlock(&tasklist_lock);
 
init/Kconfig +1 -0
Index: linux-2.6.17-rc1/init/Kconfig
===================================================================
@@ -164,6 +164,7 @@ config TASKSTATS
 
 config TASK_DELAY_ACCT
 	bool "Enable per-task delay accounting (EXPERIMENTAL)"
+	depends on TASKSTATS
 	help
 	  Collect information on time spent by a task waiting for system
 	  resources like cpu, synchronous block I/O completion and swapping
include/linux/taskstats.h +47 -1

54 lines not on this page · raw message

#18 lore raw

PATCH 7/8 documentation

In reply to Shailabh Nagar

delayacct-doc.patch

Some documentation for delay accounting.

 Documentation/accounting/delay-accounting.txt |  115 +++++++
 Documentation/accounting/getdelays.c          |  376 ++++++++++++++++++++++++++
 Documentation/accounting/taskstats.txt        |    2
 3 files changed, 493 insertions(+)
Documentation/accounting/delay-accounting.txt new file +115 -0
Index: linux-2.6.17-rc1/Documentation/accounting/delay-accounting.txt
===================================================================
@@ -0,0 +1,115 @@
+Delay accounting
+----------------
+
+Tasks encounter delays in execution when they wait
+for some kernel resource to become available e.g. a
+runnable task may wait for a free CPU to run on.
+
+The per-task delay accounting functionality measures
+the delays experienced by a task while
+
+a) waiting for a CPU (while being runnable)
+b) completion of synchronous block I/O initiated by the task
+c) swapping in pages
+
+and makes these statistics available to userspace through
+the taskstats interface.
+
+Such delays provide feedback for setting a task's cpu priority,
+io priority and rss limit values appropriately. Long delays for
+important tasks could be a trigger for raising its corresponding priority.
+
+The functionality, through its use of the taskstats interface, also provides
+delay statistics aggregated for all tasks (or threads) belonging to a
+thread group (corresponding to a traditional Unix process). This is a commonly
+needed aggregation that is more efficiently done by the kernel.
+
+Userspace utilities, particularly resource management applications, can also
+aggregate delay statistics into arbitrary groups. To enable this, delay
+statistics of a task are available both during its lifetime as well as on its
+exit, ensuring continuous and complete monitoring can be done.
+
+
+Interface
+---------
+
+Delay accounting uses the taskstats interface which is described
+in detail in a separate document in this directory. Taskstats returns a
+generic data structure to userspace corresponding to per-pid and per-tgid
+statistics. The delay accounting functionality populates specific fields of
+this structure. See
+     include/linux/taskstats.h
+for a description of the fields pertaining to delay accounting.
+It will generally be in the form of counters returning the cumulative
+delay seen for cpu, sync block I/O, swapin etc.
+
+Taking the difference of two successive readings of a given
+counter (say cpu_delay_total) for a task will give the delay
+experienced by the task waiting for the corresponding resource
+in that interval.
+
+When a task exits, records containing the per-task and per-process statistics
+are sent to userspace without requiring a command. More details are given in
+the taskstats interface description.
+
+The getdelays.c userspace utility in this directory allows simple commands to
+be run and the corresponding delay statistics to be displayed. It also serves
+as an example of using the taskstats interface.
+
+Usage
+-----
+
+Compile the kernel with
+	CONFIG_TASK_DELAY_ACCT=y
+	CONFIG_TASKSTATS=y
+
+Enable the accounting at boot time by adding
+the following to the kernel boot options
+	delayacct
+
+and after the system has booted up, use a utility
+similar to  getdelays.c to access the delays
+seen by a given task or a task group (tgid).
+The utility also allows a given command to be
+executed and the corresponding delays to be
+seen.
+
+General format of the getdelays command
+
+getdelays [-t tgid] [-p pid] [-c cmd...]
+
+
+Get delays, since system boot, for pid 10
+# ./getdelays -p 10
+(output similar to next case)
+
+Get sum of delays, since system boot, for all pids with tgid 5
+# ./getdelays -t 5
+
+
+CPU	count	real total	virtual total	delay total
+	7876	92005750	100000000	24001500
+IO	count	delay total
+	0	0
+MEM	count	delay total
+	0	0
+
+Get delays seen in executing a given simple command
+# ./getdelays -c ls /
+
+bin   data1  data3  data5  dev  home  media  opt   root  srv        sys  usr
+boot  data2  data4  data6  etc  lib   mnt    proc  sbin  subdomain  tmp  var
+
+
+CPU	count	real total	virtual total	delay total
+	6	4000250		4000000		0
+IO	count	delay total
+	0	0
+MEM	count	delay total
+	0	0
+
+
+
+
+
+
Documentation/accounting/getdelays.c new file +376 -0

376 lines not on this page · raw message

Documentation/accounting/taskstats.txt +2 -0
Index: linux-2.6.17-rc1/Documentation/accounting/taskstats.txt
===================================================================
@@ -39,6 +39,8 @@ belongs (the task does not need to be th
 per-tgid stats to be sent for each exiting task is explained in the Advanced
 Usage section below.
 
+getdelays.c is a simple utility demonstrating usage of the taskstats interface
+for reporting delay accounting statistics.
 
 Interface
 ---------
#19 lore raw

PATCH 8/8 /proc export of aggregated block I/O delays

In reply to Shailabh Nagar

Changelog
Fixed comments by akpm
- use __u64 for delayacct_blkio_ticks() return type
- redundant check for tsk->delays in __delayacct_blkio_ticks()

delayacct-procfs.patch

Export I/O delays seen by a task through /proc/<tgid>/stats for use in top etc.

Note that delays for I/O done for swapping in pages (swapin I/O) is clubbed together with all other I/O here (this is not the case in the netlink interface where the swapin I/O is kept distinct)

 fs/proc/array.c           |    6 ++++--
 include/linux/delayacct.h |   10 ++++++++++
 kernel/delayacct.c        |   12 ++++++++++++
 3 files changed, 26 insertions(+), 2 deletions(-)
fs/proc/array.c +4 -2
Index: linux-2.6.17-rc1/fs/proc/array.c
===================================================================
@@ -75,6 +75,7 @@
 #include <linux/times.h>
 #include <linux/cpuset.h>
 #include <linux/rcupdate.h>
+#include <linux/delayacct.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
@@ -412,7 +413,7 @@ static int do_task_stat(struct task_stru
 
 	res = sprintf(buffer,"%d (%s) %c %d %d %d %d %d %lu %lu \
 %lu %lu %lu %lu %lu %ld %ld %ld %ld %d 0 %llu %lu %ld %lu %lu %lu %lu %lu \
-%lu %lu %lu %lu %lu %lu %lu %lu %d %d %lu %lu\n",
+%lu %lu %lu %lu %lu %lu %lu %lu %d %d %lu %lu %llu\n",
 		task->pid,
 		tcomm,
 		state,
@@ -456,7 +457,8 @@ static int do_task_stat(struct task_stru
 		task->exit_signal,
 		task_cpu(task),
 		task->rt_priority,
-		task->policy);
+		task->policy,
+		delayacct_blkio_ticks(task));
 	if(mm)
 		mmput(mm);
 	return res;
include/linux/delayacct.h +10 -0
Index: linux-2.6.17-rc1/include/linux/delayacct.h
===================================================================
@@ -37,6 +37,7 @@ extern void __delayacct_tsk_exit(struct
 extern void __delayacct_blkio_start(void);
 extern void __delayacct_blkio_end(void);
 extern int __delayacct_add_tsk(struct taskstats *, struct task_struct *);
+extern __u64 __delayacct_blkio_ticks(struct task_struct *);
 
 static inline void delayacct_set_flag(int flag)
 {
@@ -83,6 +84,13 @@ static inline int delayacct_add_tsk(stru
 	return __delayacct_add_tsk(d, tsk);
 }
 
+static inline __u64 delayacct_blkio_ticks(struct task_struct *tsk)
+{
+	if (tsk->delays)
+		return __delayacct_blkio_ticks(tsk);
+	return 0;
+}
+
 #else
 static inline void delayacct_set_flag(int flag)
 {}
@@ -100,6 +108,8 @@ static inline void delayacct_blkio_end(v
 {}
 static inline int delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
 { return 0; }
+static inline __u64 delayacct_blkio_ticks(struct task_struct *tsk)
+{ return 0; }
 #endif /* CONFIG_TASK_DELAY_ACCT */
 
 #endif
kernel/delayacct.c +12 -0
Index: linux-2.6.17-rc1/kernel/delayacct.c
===================================================================
@@ -146,3 +146,15 @@ int __delayacct_add_tsk(struct taskstats
 
 	return 0;
 }
+
+__u64 __delayacct_blkio_ticks(struct task_struct *tsk)
+{
+	__u64 ret;
+
+	spin_lock(&tsk->delays->lock);
+	ret = nsec_to_clock_t(tsk->delays->blkio_delay +
+				tsk->delays->swapin_delay);
+	spin_unlock(&tsk->delays->lock);
+	return ret;
+}
+
#20 lore raw

Re: [Lse-tech] [Patch 8/8] /proc export of aggregated block I/O delays

Quoted 4 lines · Shailabh Nagar

Changelog
Fixed comments by akpm
- use __u64 for delayacct_blkio_ticks() return type
- redundant check for tsk->delays in __delayacct_blkio_ticks()

I think these basic statistics in /proc are quite useful. Hopefully top etc. would learn quickly about them too so that normal people can actually make use of it.

-Andi

#21 lore raw
In reply to Shailabh Nagar

Here's a repost of my overview of the other stakeholders. For some reason, lkml keeps rejecting this and its earlier post wasn't archived either. Retrying.

Following Andrew's suggestion, here's my quick overview of the various other accounting packages that have been proposed on lse-tech with a focus on whether they can utilize the netlink-based taskstats interface being proposed by the delay accounting patches.

Please note that unification of statistics *collection* is not being discussed since that kind of merger can be done as these patches get accepted, if at all, into the kernel. To try and unify right away would hold every patch (esp. delay accounting !) hostage to the problems in every other patch unnecessarily. As long as the interface can be unified, the merger of the collection bits can always happen without affecting user space.

Stakeholders of each of these patches, on cc, are requested to please correct any misunderstandings of what their patches do so we can make forward progress.

--Shailabh

Summary

The following can use the taskstats netlink-based interface by extending the returned data structure

- Comprehensive System Accounting
- per-process I/O stats
- Microstate accounting
- per cpu time stats

The following patches' interface needs are independent of taskstats or subsumed by one of above:
- Enhanced Linux System Accounting
- pnotify
- scalable statistics counters

Details
(please correct if these are misunderstood)

1. Comprehensive System Accounting (Jay Lan)
--------------------------------------------

- Collect various per-task statistics and write an accounting record containing
these stats at task exit. Interface similar to BSD process accounting but the accounting record structure is quite different.

- CSA could utilize some stats collected/exported by delay accounting blkio wait time
cpu run time for task
- CSA only needs data to be available at task exit, not during the task's lifetime. Moreover, at task exit, it needs the accounting record to be written to a file.
- CSA could utilize delay accounting's taskstats netlink interface to gather task data at exit through
a userspace utility that then writes it out to its expected file.

To do so, CSA would need the taskstats struct to be extended with whatever additional stats it needs. The additional stats could be selectively exported only on task exit to avoid imposing a space burden on users of delay accounting who query a process's statistics during its lifetime.

Collection of the additional stats needed by CSA may be tied to pnotify and job patches which are still being reviewed/considered for acceptance. As such, unification in the collection of stats can be deferred until status of pnotify/job/CSA patches becomes more clear.

2. per-process I/O statistics (Levent Serinol)
----------------------------------------------

- Exports task->{rchar,wchar} through /proc/tgid/iostat (earlier version proposed export through /proc/tgid/stats)

- No new stats collection. Just export of existing task fields
- Problem with accepting the patch stems from the accuracy of the statistics
in these fields. The fields are updated only in three cases today (sys_read/write, sys_readv/writev, do_sendfile)
so they aren't accurate. async I/O, memory-mapped I/O is not counted at the very least).

CSA patches also export these fields through their accounting record but don't appear to be doing anything to improve accuracy of collection (or maybe it doesn't matter to them).
BSD accounting, which ought to be using the sum of these fields for its ac_io field, doesn't (it hardcodes the output to zero).

When the fate of task->rchar/wchar is decided, based on CSA's needs, those fields can be easily added to taskstats.

3. per-cpu time statistics (Erich Focht)
----------------------------------------

- Collects time spent by a task on each cpu of a system and exports it through new interface /proc/tgid/cpu

- Statistic is needed for performance analysis/debugging (like schedstats) and not for production systems.
- Unsure why push for acceptance was abandoned. Possibly due to one or more of:
space overhead of allocating NR_CPUS variables in task_struct time overhead of collecting the data ?

- Can use taskstats interface to export the data by adding needed fields to struct taskstats and bumping up the version.

4. Microstate accounting (Peter Chubb)
--------------------------------------

- Measure time spent by a thread in various interesting states, while accounting for interrupts, and export through /proc/tid/msa and through a syscall interface

- Interesting states have some overlap with delay accounting
- Exporting of per-task stats can be done through taskstats netlink interface

5. Enhanced Linux System Accounting (Guillaume Thouvenine)
----------------------------------------------------------

- Group tasks at a user level into "jobs" and aggregate, at user level, per-task statistics collected by CSA and/or BSD process accounting.

- ELSA does not introduce any new requirement for either collection or export of statistics from the kernel. It can use either BSD and/or CSA's method of using an accounting file.

- ELSA needs notification of forks and exits which it can already get through the process events connector in the kernel.

Hence ELSA's needs are either met by the kernel today or are a strict subset of CSA (since BSD accounting is already there).

6. pnotify (Erik Jacobson)
--------------------------

- Infrastructure for kernel modules to be notified when an event (like fork/exit/exec)
happens to a task. Also provides some per-task data for the modules' convenience