Skip to contentSkip to conversation

vfs: Add a trace point in the mark_inode_dirty function

PATCH In review Arjan van de Ven 6 participants 11 messages Nov 26, 2010 – Nov 30, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfs: Add a trace point in the mark_inode_dirty function

long time overdue to clean this guy up;
this patch adds a very basic trace point for tracking who dirties an inode (this is the guy who is the real cause for a disk to spin up, so PowerTOP would like to know this...)

compared to earlier patches the trace point is done a lot simpler, and consistent with how others (ext4 etc) do their vfs level tracepoints; with a device number and inode number

Quoted 1 line

From 880a17e27f0b90d32c0c0b9cf4cd9b4e5d779c73 Mon Sep 17 00:00:00 2001

From: Arjan van de Ven <[email protected]>
Date: Fri, 26 Nov 2010 12:18:03 -0800
Subject: [PATCH] vfs: Add a trace point in the mark_inode_dirty function

PowerTOP would like to be able to show who is keeping the disk busy by dirtying data. The most logical spot for this is in the vfs in the mark_inode_dirty() function, doing this on the block level is not possible because by the time the IO hits the block layer the guilty party can no longer be found ("kjournald" and "pdflush" are not useful answers to "who caused this file to be dirty).

The trace point follows the same logic/style as the block_dump code and pretty much dumps the same data, just not to dmesg (and thus to /var/log/messages) but via the trace events streams.

Eventually we should be able to phase out the block dump code, but that's for later on after a transition time.

---
 fs/fs-writeback.c          |    4 ++++
 fs/inode.c                 |    4 ++++
 include/trace/events/vfs.h |   35 +++++++++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/events/vfs.h
fs/fs-writeback.c +4 -0
index 3d06ccc..6c93517 100644
@@ -27,6 +27,7 @@
 #include <linux/backing-dev.h>
 #include <linux/buffer_head.h>
 #include <linux/tracepoint.h>
+#include <trace/events/vfs.h>
 #include "internal.h"
 
 /*
@@ -942,6 +943,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 			sb->s_op->dirty_inode(inode);
 	}
 
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES))
+		trace_inode_dirty(inode);
+
 	/*
 	 * make sure that changes are seen by all cpus before we test i_state
 	 * -- mikulas
fs/inode.c +4 -0
index ae2727a..e8b1d42 100644
@@ -1708,3 +1708,7 @@ void inode_init_owner(struct inode *inode, const struct inode *dir,
 	inode->i_mode = mode;
 }
 EXPORT_SYMBOL(inode_init_owner);
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/vfs.h>
+
include/trace/events/vfs.h new file +35 -0
new file mode 100644
index 0000000..7e5c135
@@ -0,0 +1,35 @@
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM vfs
+
+#if !defined(_TRACE_VFS_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_VFS_H
+
+/*
+ * Tracepoint for dirtying an inode:
+ */
+TRACE_EVENT(inode_dirty,
+
+	TP_PROTO(struct inode *inode),
+
+	TP_ARGS(inode),
+
+	TP_STRUCT__entry(
+		__field(	int,	dev_major		)
+		__field(	int,	dev_minor		)
+		__field(	ino_t,	ino			)
+	),
+
+	TP_fast_assign(
+		__entry->dev_major = MAJOR(inode->i_sb->s_dev);
+		__entry->dev_minor = MINOR(inode->i_sb->s_dev);
+		__entry->ino	   = inode->i_ino;
+	),
+
+	TP_printk("dev %d,%d ino %lu ", __entry->dev_major, __entry->dev_minor,
+		  (unsigned long) __entry->ino)
+);
+
+#endif /* _TRACE_VFS_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
Signature · 7 lines
1.7.2.3


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#1 lore raw

Looks generally good to me.

Two sugestions:

- also trace the flags value and decode it using __print_flags in the output
- use a dev_t instead of the split major/minor to follow the way the block layer, writeback and xfs tracepoints track the block device. Seems like this was copied from the recent conversion of ext4 away from it's braindead string printing, which didn't follow the existing way either.

#2 lore raw
Quoted 4 lines · On Sun, 28 Nov 2010 12:52:56 -0500 Christoph Hellwig <[email protected]> wrote:

Looks generally good to me.

Two sugestions:

Quoted 8 lines

- also trace the flags value and decode it using __print_flags in the output
- use a dev_t instead of the split major/minor to follow the way the block layer, writeback and xfs tracepoints track the block device. Seems like this was copied from the recent conversion of ext4 away from it's braindead string printing, which didn't follow the
existing way either.

ok how about this?

only question left is if we should trave every dirty, or only those that change the flags.

Right now the patch only traces the non-trivial (eg changing) ones... but opinions welcome (otherwise you get MANY duplicate, basically no-op trace events)

Quoted 1 line

From 3950d3c04a6bf8ccf9ff912a49bdd242a2fe9e47 Mon Sep 17 00:00:00 2001

From: Arjan van de Ven <[email protected]>
Date: Fri, 26 Nov 2010 12:18:03 -0800
Subject: [PATCH] vfs: Add a trace point in the mark_inode_dirty function

PowerTOP would like to be able to show who is keeping the disk busy by dirtying data. The most logical spot for this is in the vfs in the mark_inode_dirty() function, doing this on the block level is not possible because by the time the IO hits the block layer the guilty party can no longer be found ("kjournald" and "pdflush" are not useful answers to "who caused this file to be dirty).

The trace point follows the same logic/style as the block_dump code and pretty much dumps the same data, just not to dmesg (and thus to /var/log/messages) but via the trace events streams.

Eventually we should be able to phase out the block dump code, but that's for later on after a transition time.

---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28 ++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 0 deletions(-)
fs/fs-writeback.c +3 -0
index 3d06ccc..62e33cc 100644
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 	if ((inode->i_state & flags) == flags)
 		return;
 
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+
 	if (unlikely(block_dump))
 		block_dump___mark_inode_dirty(inode);
 
include/linux/fs.h +12 -0
index c9e06cc..25935e1 100644
@@ -1676,6 +1676,18 @@ struct super_operations {
 
 #define I_DIRTY (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES)
 
+#define INODE_DIRTY_FLAGS \
+	{ I_DIRTY_SYNC,		"DIRTY-SYNC" }, \
+	{ I_DIRTY_DATASYNC,	"DIRTY-DATASYNC" }, \
+	{ I_DIRTY_PAGES,	"DIRTY-PAGES" }, \
+	{ I_NEW,		"NEW" }, \
+	{ I_WILL_FREE,		"WILL-FREE" }, \
+	{ I_FREEING,		"FREEING" }, \
+	{ I_CLEAR,		"CLEAR" }, \
+	{ I_SYNC,		"SYNC" }, \
+	{ I_REFERENCED,		"REFERENCED" }
+
+
 extern void __mark_inode_dirty(struct inode *, int);
 static inline void mark_inode_dirty(struct inode *inode)
 {
include/trace/events/writeback.h +28 -0
index 89a2b2d..5c80875 100644
@@ -186,6 +186,34 @@ DEFINE_EVENT(writeback_congest_waited_template, writeback_wait_iff_congested,
 	TP_ARGS(usec_timeout, usec_delayed)
 );
 
+/*
+ * Tracepoint for dirtying an inode; used by PowerTOP
+ */
+TRACE_EVENT(writeback_inode_dirty,
+
+	TP_PROTO(struct inode *inode, int flags),
+
+	TP_ARGS(inode, flags),
+
+	TP_STRUCT__entry(
+		__field(	__kernel_dev_t,	dev		)
+		__field(	ino_t,		ino		)
+		__field(	u32,		flags		)
+	),
+
+	TP_fast_assign(
+		__entry->dev	= inode->i_sb->s_dev;
+		__entry->ino	= inode->i_ino;
+		__entry->flags  = flags;
+	),
+
+	TP_printk("dev %d:%d ino %lu flags %d %s", MAJOR(__entry->dev), MINOR(__entry->dev),
+		  (unsigned long) __entry->ino,
+		  __entry->flags,
+		  __print_flags(__entry->flags, "|", INODE_DIRTY_FLAGS)
+	)
+);
+
 #endif /* _TRACE_WRITEBACK_H */
 
 /* This part must be outside protection */
Signature · 8 lines
1.7.2.3



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#3 lore raw
Quoted 12 lines
---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28 ++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 0 deletions(-)
fs/fs-writeback.c +0 -0
index 3d06ccc..62e33cc 100644

fs/fs-writeback.c void __mark_inode_dirty(struct inode *inode, int flags) · line 957

Quoted 6 lines
fs/fs-writeback.c +3 -0
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 	if ((inode->i_state & flags) == flags)
 		return;
 
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+

Why can't we move this branch into TP_fast_assign()?

#4 lore raw
Quoted 16 lines · On Mon, 29 Nov 2010 10:41:51 +0900 (JST) KOSAKI Motohiro…
Quoted 16 lines
---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28

++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 0 deletions(-)

fs/fs-writeback.c +0 -0
index 3d06ccc..62e33cc 100644
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode,
 int flags) if ((inode->i_state & flags) == flags)
 		return;
 
Quoted 6 lines

fs/fs-writeback.c void __mark_inode_dirty(struct inode *inode, int flags) · line 957

Quoted 4 lines
fs/fs-writeback.c +3 -0
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
 I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+

Why can't we move this branch into TP_fast_assign()?

not really because then the tracepoint is already in process of being emitted... no way to retract it anymore.

#5 lore raw
Quoted 26 lines
Quoted 16 lines · On Mon, 29 Nov 2010 10:41:51 +0900 (JST) KOSAKI Motohiro…
Quoted 16 lines
---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28

++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 0 deletions(-)

fs/fs-writeback.c +0 -0
index 3d06ccc..62e33cc 100644
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode,
 int flags) if ((inode->i_state & flags) == flags)
 		return;
 
Quoted 6 lines

fs/fs-writeback.c void __mark_inode_dirty(struct inode *inode, int flags) · line 957

Quoted 4 lines
fs/fs-writeback.c +3 -0
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
 I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+

Why can't we move this branch into TP_fast_assign()?

Quoted 2 lines

not really because then the tracepoint is already in process of being emitted... no way to retract it anymore.

I'm not tracing expert. but Steven said we can it in past. (cc him)

http://www.spinics.net/lists/linux-ext4/msg20045.html

maintainer
#6 lore raw

[ Adding Ingo, Peter and Mathieu ]

Quoted 26 lines · KOSAKI Motohiro
Quoted 26 lines
Quoted 22 lines · On Mon, 29 Nov 2010 10:41:51 +0900 (JST) KOSAKI Motohiro…
Quoted 13 lines
---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28

++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 0 deletions(-)

fs/fs-writeback.c +0 -0
index 3d06ccc..62e33cc 100644

fs/fs-writeback.c void __mark_inode_dirty(struct inode *inode, · line 958

Quoted 7 lines
fs/fs-writeback.c +3 -0
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode,
 int flags) if ((inode->i_state & flags) == flags)
 		return;
 
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
 I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+

Why can't we move this branch into TP_fast_assign()?

Quoted 7 lines
Quoted 2 lines

not really because then the tracepoint is already in process of being emitted... no way to retract it anymore.

I'm not tracing expert. but Steven said we can it in past. (cc him)

http://www.spinics.net/lists/linux-ext4/msg20045.html

Yes, this came up before. Currently, if you add the if statement in the trcacepoint, you just wasted space. But if we can add a discard_entry, or better yet, I can add a TRACE_EVENT_CONDITION() that adds an if statement to check. Something like:

	TRACE_EVENT_CONDITION(name,
		[...]
	TP_condition(
			if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
			    I_DIRTY_PAGES))
				return 1;
			else
				return 0;
		    ),

Have this run on the parameters and not the entry fields (because the entry fields are from the ring buffer, and to use them, we must first write to the ring buffer (something we want to avoid).

We could also make this code work with "pre-filters". That is, filters on the tracepoints that access the parameters and not the final entries. This would let us circumvent allocating ring buffer space when the filter states to skip the entry.

-- Steve

#7 lore raw
Quoted 42 lines · * Steven Rostedt ([email protected]) wrote:

[ Adding Ingo, Peter and Mathieu ]

Quoted 26 lines · KOSAKI Motohiro
Quoted 26 lines
Quoted 22 lines · On Mon, 29 Nov 2010 10:41:51 +0900 (JST) KOSAKI Motohiro…
Quoted 13 lines
---
 fs/fs-writeback.c                |    3 +++
 include/linux/fs.h               |   12 ++++++++++++
 include/trace/events/writeback.h |   28

++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 0 deletions(-)

fs/fs-writeback.c +0 -0
index 3d06ccc..62e33cc 100644

fs/fs-writeback.c void __mark_inode_dirty(struct inode *inode, · line 958

Quoted 7 lines
fs/fs-writeback.c +3 -0
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode,
 int flags) if ((inode->i_state & flags) == flags)
 		return;
 
+	if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
 I_DIRTY_PAGES))
+		trace_writeback_inode_dirty(inode, flags);
+

Why can't we move this branch into TP_fast_assign()?

Quoted 7 lines
Quoted 2 lines

not really because then the tracepoint is already in process of being emitted... no way to retract it anymore.

I'm not tracing expert. but Steven said we can it in past. (cc him)

http://www.spinics.net/lists/linux-ext4/msg20045.html

Yes, this came up before. Currently, if you add the if statement in the trcacepoint, you just wasted space. But if we can add a discard_entry, or better yet, I can add a TRACE_EVENT_CONDITION() that adds an if statement to check. Something like:

Quoted 9 lines
	TRACE_EVENT_CONDITION(name,
		[...]
	TP_condition(
			if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
			    I_DIRTY_PAGES))
				return 1;
			else
				return 0;
		    ),

Yep, something like this would be really useful for us too in user-space. One example use-case is to let a telecom application filter by "call ID" when following one call across multiple telecom switches. The kind of condition we would have is:

TRACE_EVENT_CONDITION(name,
[...]
TP_condition(
if (unlikely(call_id == monitored_call_id))

 				return 1;
                        if (unlikely(!call_id_monitoring))
                                return 1;

return 0;
),

So in this case, "call_id" is received as parameter, but "monitored_call_id" and "call_id_monitoring" are global variables.

One question that strikes me is: would you declare this outside of the TRACE_EVENT() or inside it ? How would you match the TRACE_EVENT() and the TRACE_EVENT_CONDITION() if they are separate, by name ?

The dynamic "pre-filters" will also be useful, but I think this kind of statically defined conditions will answer a large amount of our requirements, letting these filtering expressions be designed by application developers and used by operators/engineers.

Thanks,

Mathieu

Quoted 15 lines

Have this run on the parameters and not the entry fields (because the entry fields are from the ring buffer, and to use them, we must first write to the ring buffer (something we want to avoid).

We could also make this code work with "pre-filters". That is, filters on the tracepoints that access the parameters and not the final entries. This would let us circumvent allocating ring buffer space when the filter states to skip the entry.

-- Steve

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
maintainer
#8 lore raw
Quoted 4 lines · Mathieu Desnoyers

One question that strikes me is: would you declare this outside of the TRACE_EVENT() or inside it ? How would you match the TRACE_EVENT() and the TRACE_EVENT_CONDITION() if they are separate, by name ?

TRACE_EVENT_CONDITION() would just add the condition field and TRACE_EVENT() will just have a null condition:

#define TRACE_EVENT(name, proto, args, tstruct, assign, print) \
	TRACE_EVENT_CONDITION(name, proto, args, , tsturct, \
		assign, print)

where that ", ," is the condition expression. (I also left out the PARAMS() that would be required here too.)

-- Steve

#9 lore raw
In reply to Steven Rostedt
Quoted 5 lines

Yes, this came up before. Currently, if you add the if statement in the trcacepoint, you just wasted space. But if we can add a discard_entry, or better yet, I can add a TRACE_EVENT_CONDITION() that adds an if statement to check. Something like:

Quoted 9 lines
	TRACE_EVENT_CONDITION(name,
		[...]
	TP_condition(
			if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC |
			    I_DIRTY_PAGES))
				return 1;
			else
				return 0;
		    ),

I love this idea. :)

For MM, struct page has a lot of union hack. then, page tracer strongly need conditional trace point.

Quoted 15 lines

Have this run on the parameters and not the entry fields (because the entry fields are from the ring buffer, and to use them, we must first write to the ring buffer (something we want to avoid).

We could also make this code work with "pre-filters". That is, filters on the tracepoints that access the parameters and not the final entries. This would let us circumvent allocating ring buffer space when the filter states to skip the entry.

-- Steve

#10 lore raw
In reply to Arjan van de Ven
Quoted 73 lines · Arjan van de Ven
Quoted 4 lines · On Sun, 28 Nov 2010 12:52:56 -0500 Christoph Hellwig <[email protected]> wrote:

Looks generally good to me.

Two sugestions:

Quoted 8 lines

 - also trace the flags value and decode it using __print_flags in the output
 - use a dev_t instead of the split major/minor to follow the way the block layer, writeback and xfs tracepoints track the block device. Seems like this was copied from the recent conversion of ext4 away from it's braindead string printing, which didn't follow the existing way either.

ok how about this?

only question left is if we should trave every dirty, or only those that change the flags.

Right now the patch only traces the non-trivial (eg changing) ones... but opinions welcome (otherwise you get MANY duplicate, basically no-op trace events)

From 3950d3c04a6bf8ccf9ff912a49bdd242a2fe9e47 Mon Sep 17 00:00:00 2001
From: Arjan van de Ven <[email protected]>
Date: Fri, 26 Nov 2010 12:18:03 -0800
Subject: [PATCH] vfs: Add a trace point in the mark_inode_dirty function

PowerTOP would like to be able to show who is keeping the disk busy by dirtying data. The most logical spot for this is in the vfs in the mark_inode_dirty() function, doing this on the block level is not possible because by the time the IO hits the block layer the guilty party can no longer be found ("kjournald" and "pdflush" are not useful answers to "who caused this file to be dirty).

The trace point follows the same logic/style as the block_dump code and pretty much dumps the same data, just not to dmesg (and thus to /var/log/messages) but via the trace events streams.

Eventually we should be able to phase out the block dump code, but that's for later on after a transition time.

---
 fs/fs-writeback.c                |    3 +++ include/linux/fs.h               |   12 ++++++++++++ include/trace/events/writeback.h |   28 ++++++++++++++++++++++++++++

 3 files changed, 43 insertions(+), 0 deletions(-)
fs/fs-writeback.c +3 -0
index 3d06ccc..62e33cc 100644
@@ -952,6 +952,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
        if ((inode->i_state & flags) == flags)
                return;
 
+       if (flags & (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES))
+               trace_writeback_inode_dirty(inode, flags);
+
        if (unlikely(block_dump))
                block_dump___mark_inode_dirty(inode);
 
include/linux/fs.h +0 -0
index c9e06cc..25935e1 100644
@@ -1676,6 +1676,18 @@ struct super_operations {
 
  #define I_DIRTY (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES)
 

include/linux/fs.h struct super_operations { · line 1688

Quoted 10 lines
include/linux/fs.h +10 -0
+#define INODE_DIRTY_FLAGS \
+       { I_DIRTY_SYNC,         "DIRTY-SYNC" }, \
+       { I_DIRTY_DATASYNC,     "DIRTY-DATASYNC" }, \
+       { I_DIRTY_PAGES,        "DIRTY-PAGES" }, \
+       { I_NEW,                "NEW" }, \
+       { I_WILL_FREE,          "WILL-FREE" }, \
+       { I_FREEING,            "FREEING" }, \
+       { I_CLEAR,              "CLEAR" }, \
+       { I_SYNC,               "SYNC" }, \
+       { I_REFERENCED,         "REFERENCED" }

Can you change these to exactly the same name as the flags ("I_DIRTY_SYNC",..). Arbitrarily changing them a little bit is unnecessary.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html