Skip to contentSkip to conversation

blktrace: last round of fixes

PATCH In review Li Zefan 3 participants 16 messages Mar 27, 2009 – Mar 30, 2009

Subsystems (2)
lore
Share & tools
lore

Li Zefan

author
#0 lore raw

v2 cover blktrace: last round of fixes

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)
author
#1 lore raw

PATCH v2 1/7 trace: make 'mem' argument of trace_seq_putmem() const

I passed a const value to trace_seq_putmem(), and I got compile warning.

---
 kernel/trace/trace_output.c |    6 +++---
 kernel/trace/trace_output.h |    5 +++--
 2 files changed, 6 insertions(+), 5 deletions(-)
kernel/trace/trace_output.c +3 -3
index 6595074..d72b9a6 100644
@@ -137,7 +137,7 @@ int trace_seq_putc(struct trace_seq *s, unsigned char c)
 	return 1;
 }
 
-int trace_seq_putmem(struct trace_seq *s, void *mem, size_t len)
+int trace_seq_putmem(struct trace_seq *s, const void *mem, size_t len)
 {
 	if (len > ((PAGE_SIZE - 1) - s->len))
 		return 0;
@@ -148,10 +148,10 @@ int trace_seq_putmem(struct trace_seq *s, void *mem, size_t len)
 	return len;
 }
 
-int trace_seq_putmem_hex(struct trace_seq *s, void *mem, size_t len)
+int trace_seq_putmem_hex(struct trace_seq *s, const void *mem, size_t len)
 {
 	unsigned char hex[HEX_CHARS];
-	unsigned char *data = mem;
+	const unsigned char *data = mem;
 	int i, j;
 
 #ifdef __BIG_ENDIAN
kernel/trace/trace_output.h +3 -2
index 46fb961..e0bde39 100644
@@ -31,8 +31,9 @@ extern ssize_t trace_seq_to_user(struct trace_seq *s, char __user *ubuf,
 				 size_t cnt);
 extern int trace_seq_puts(struct trace_seq *s, const char *str);
 extern int trace_seq_putc(struct trace_seq *s, unsigned char c);
-extern int trace_seq_putmem(struct trace_seq *s, void *mem, size_t len);
-extern int trace_seq_putmem_hex(struct trace_seq *s, void *mem, size_t len);
+extern int trace_seq_putmem(struct trace_seq *s, const void *mem, size_t len);
+extern int trace_seq_putmem_hex(struct trace_seq *s, const void *mem,
+				size_t len);
 extern void *trace_seq_reserve(struct trace_seq *s, size_t len);
 extern int trace_seq_path(struct trace_seq *s, struct path *path);
 extern int seq_print_userip_objs(const struct userstack_entry *entry,
Signature · 1 line
1.5.4.rc3
author
#2 lore raw

PATCH v2 2/7 blktrace: make classic output more classic

In reply to Li Zefan

In the classic user-space blktrace, the output timestamp is sec.nsec not sec.usec.

---
 kernel/trace/blktrace.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/trace/blktrace.c +4 -4
index d43cdac..5b28f0f 100644
@@ -994,8 +994,8 @@ static void get_pdu_remap(const struct trace_entry *ent,
 static int blk_log_action_iter(struct trace_iterator *iter, const char *act)
 {
 	char rwbs[6];
-	unsigned long long ts  = ns2usecs(iter->ts);
-	unsigned long usec_rem = do_div(ts, USEC_PER_SEC);
+	unsigned long long ts  = iter->ts;
+	unsigned long nsec_rem = do_div(ts, NSEC_PER_SEC);
 	unsigned secs	       = (unsigned long)ts;
 	const struct trace_entry *ent = iter->ent;
 	const struct blk_io_trace *t = (const struct blk_io_trace *)ent;
@@ -1003,9 +1003,9 @@ static int blk_log_action_iter(struct trace_iterator *iter, const char *act)
 	fill_rwbs(rwbs, t);
 
 	return trace_seq_printf(&iter->seq,
-				"%3d,%-3d %2d %5d.%06lu %5u %2s %3s ",
+				"%3d,%-3d %2d %5d.%09lu %5u %2s %3s ",
 				MAJOR(t->device), MINOR(t->device), iter->cpu,
-				secs, usec_rem, ent->pid, act, rwbs);
+				secs, nsec_rem, ent->pid, act, rwbs);
 }
 
 static int blk_log_action_seq(struct trace_seq *s, const struct blk_io_trace *t,
Signature · 1 line
1.5.4.rc3
author
#3 lore raw

PATCH v2 3/7 blktrace: fix blk_probes_ref chaos

In reply to Li Zefan

ioctl-based blktrace allocates bt and registers tracepoints when ioctl(BLKTRACESETUP), and do all cleanups when ioctl(BLKTRACETEARDOWN).

while ftrace-based blktrace allocates/frees bt when:

  # echo 1/0 > /sys/block/sda/sda1/trace/enable

and registers/unregisters tracepoints when:

  # echo blk/nop > /debugfs/tracing/current_tracer
or
  # echo 1/0 > /debugfs/tracing/tracing_enable

The separatation of allocation and registeration causes 2 problems:
1. current user-space blktrace still calls ioctl(TEARDOWN) when ioctl(SETUP) failed:
# echo 1 > /sys/block/sda/sda1/trace/enable
# blktrace /dev/sda
BLKTRACESETUP: Device or resource busy
^C
and now blk_probes_ref == -1

2. Another way to make blk_probes_ref == -1:

     # plugin sdb && mount sdb1
     # echo 1 > /sys/block/sdb/sdb1/trace/enable
     # remove sdb

This patch does the allocation and registeration when writing sdaX/trace/enable.

---
 kernel/trace/blktrace.c |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)
kernel/trace/blktrace.c +7 -8
index 5b28f0f..8d6bd12 100644
@@ -478,7 +478,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 		goto err;
 	}
 
-	if (atomic_add_return(1, &blk_probes_ref) == 1)
+	if (atomic_inc_return(&blk_probes_ref) == 1)
 		blk_register_tracepoints();
 
 	return 0;
@@ -1091,8 +1091,6 @@ static void blk_tracer_print_header(struct seq_file *m)
 
 static void blk_tracer_start(struct trace_array *tr)
 {
-	if (atomic_add_return(1, &blk_probes_ref) == 1)
-		blk_register_tracepoints();
 	trace_flags &= ~TRACE_ITER_CONTEXT_INFO;
 }
 
@@ -1107,15 +1105,10 @@ static int blk_tracer_init(struct trace_array *tr)
 static void blk_tracer_stop(struct trace_array *tr)
 {
 	trace_flags |= TRACE_ITER_CONTEXT_INFO;
-	if (atomic_dec_and_test(&blk_probes_ref))
-		blk_unregister_tracepoints();
 }
 
 static void blk_tracer_reset(struct trace_array *tr)
 {
-	if (!atomic_read(&blk_probes_ref))
-		return;
-
 	blk_tracer_enabled = false;
 	blk_tracer_stop(tr);
 }
@@ -1254,6 +1247,9 @@ static int blk_trace_remove_queue(struct request_queue *q)
 	if (bt == NULL)
 		return -EINVAL;
 
+	if (atomic_dec_and_test(&blk_probes_ref))
+		blk_unregister_tracepoints();
+
 	kfree(bt);
 	return 0;
 }
@@ -1280,6 +1276,9 @@ static int blk_trace_setup_queue(struct request_queue *q, dev_t dev)
 		return -EBUSY;
 	}
 
+	if (atomic_inc_return(&blk_probes_ref) == 1)
+		blk_register_tracepoints();
+
 	return 0;
 }
 
Signature · 1 line
1.5.4.rc3
author
#4 lore raw

PATCH v3 3/7 blktrace: fix blk_probes_ref chaos

Some changes that should have been in this patch, went into [PATCH 4/7]..

Something went wrong when I made those patches.. :(

========

ioctl-based blktrace allocates bt and registers tracepoints when ioctl(BLKTRACESETUP), and do all cleanups when ioctl(BLKTRACETEARDOWN).

while ftrace-based blktrace allocates/frees bt when:

  # echo 1/0 > /sys/block/sda/sda1/trace/enable

and registers/unregisters tracepoints when:

  # echo blk/nop > /debugfs/tracing/current_tracer
or
  # echo 1/0 > /debugfs/tracing/tracing_enable

The separatation of allocation and registeration causes 2 problems:
1. current user-space blktrace still calls ioctl(TEARDOWN) when ioctl(SETUP) failed:
# echo 1 > /sys/block/sda/sda1/trace/enable
# blktrace /dev/sda
BLKTRACESETUP: Device or resource busy
^C
and now blk_probes_ref == -1

2. Another way to make blk_probes_ref == -1:

     # plugin sdb && mount sdb1
     # echo 1 > /sys/block/sdb/sdb1/trace/enable
     # remove sdb

This patch does the allocation and registeration when writing sdaX/trace/enable.

---
 kernel/trace/blktrace.c |   19 +++++++++----------
 1 files changed, 9 insertions(+), 10 deletions(-)
kernel/trace/blktrace.c +9 -10
index 5b28f0f..acf8454 100644
@@ -478,7 +478,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 		goto err;
 	}
 
-	if (atomic_add_return(1, &blk_probes_ref) == 1)
+	if (atomic_inc_return(&blk_probes_ref) == 1)
 		blk_register_tracepoints();
 
 	return 0;
@@ -1091,8 +1091,7 @@ static void blk_tracer_print_header(struct seq_file *m)
 
 static void blk_tracer_start(struct trace_array *tr)
 {
-	if (atomic_add_return(1, &blk_probes_ref) == 1)
-		blk_register_tracepoints();
+	blk_tracer_enabled = true;
 	trace_flags &= ~TRACE_ITER_CONTEXT_INFO;
 }
 
@@ -1100,23 +1099,17 @@ static int blk_tracer_init(struct trace_array *tr)
 {
 	blk_tr = tr;
 	blk_tracer_start(tr);
-	blk_tracer_enabled = true;
 	return 0;
 }
 
 static void blk_tracer_stop(struct trace_array *tr)
 {
+	blk_tracer_enabled = false;
 	trace_flags |= TRACE_ITER_CONTEXT_INFO;
-	if (atomic_dec_and_test(&blk_probes_ref))
-		blk_unregister_tracepoints();
 }
 
 static void blk_tracer_reset(struct trace_array *tr)
 {
-	if (!atomic_read(&blk_probes_ref))
-		return;
-
-	blk_tracer_enabled = false;
 	blk_tracer_stop(tr);
 }
 
@@ -1254,6 +1247,9 @@ static int blk_trace_remove_queue(struct request_queue *q)
 	if (bt == NULL)
 		return -EINVAL;
 
+	if (atomic_dec_and_test(&blk_probes_ref))
+		blk_unregister_tracepoints();
+
 	kfree(bt);
 	return 0;
 }
@@ -1280,6 +1276,9 @@ static int blk_trace_setup_queue(struct request_queue *q, dev_t dev)
 		return -EBUSY;
 	}
 
+	if (atomic_inc_return(&blk_probes_ref) == 1)
+		blk_register_tracepoints();
+
 	return 0;
 }
 
Signature · 1 line
1.5.4.rc3
author
#5 lore raw

PATCH v2 4/7 blktrace: fix memory leak when freeing struct blk_io_trace

In reply to Li Zefan

When mixing the use of ioctl-based blktrace and ftrace-based blktrace, we can leak memory in this way:

  # btrace /dev/sda > /dev/null &
  # echo 0 > /sys/block/sda/sda1/trace/enable
now we leak bt->dropped_file, bt->msg_file, bt->rchan...
---
 kernel/trace/blktrace.c |   29 ++++++++++++-----------------
 1 files changed, 12 insertions(+), 17 deletions(-)
kernel/trace/blktrace.c +12 -17
index 8d6bd12..2f21d77 100644
@@ -247,7 +247,7 @@ record_it:
 static struct dentry *blk_tree_root;
 static DEFINE_MUTEX(blk_tree_mutex);
 
-static void blk_trace_cleanup(struct blk_trace *bt)
+static void blk_trace_free(struct blk_trace *bt)
 {
 	debugfs_remove(bt->msg_file);
 	debugfs_remove(bt->dropped_file);
@@ -255,6 +255,11 @@ static void blk_trace_cleanup(struct blk_trace *bt)
 	free_percpu(bt->sequence);
 	free_percpu(bt->msg_data);
 	kfree(bt);
+}
+
+static void blk_trace_cleanup(struct blk_trace *bt)
+{
+	blk_trace_free(bt);
 	if (atomic_dec_and_test(&blk_probes_ref))
 		blk_unregister_tracepoints();
 }
@@ -410,11 +415,11 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 		if (buts->name[i] == '/')
 			buts->name[i] = '_';
 
-	ret = -ENOMEM;
 	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
 	if (!bt)
-		goto err;
+		return -ENOMEM;
 
+	ret = -ENOMEM;
 	bt->sequence = alloc_percpu(unsigned long);
 	if (!bt->sequence)
 		goto err;
@@ -483,17 +488,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 
 	return 0;
 err:
-	if (bt) {
-		if (bt->msg_file)
-			debugfs_remove(bt->msg_file);
-		if (bt->dropped_file)
-			debugfs_remove(bt->dropped_file);
-		free_percpu(bt->sequence);
-		free_percpu(bt->msg_data);
-		if (bt->rchan)
-			relay_close(bt->rchan);
-		kfree(bt);
-	}
+	blk_trace_free(bt);
 	return ret;
 }
 
@@ -1091,6 +1086,7 @@ static void blk_tracer_print_header(struct seq_file *m)
 
 static void blk_tracer_start(struct trace_array *tr)
 {
+	blk_tracer_enabled = true;
 	trace_flags &= ~TRACE_ITER_CONTEXT_INFO;
 }
 
@@ -1098,18 +1094,17 @@ static int blk_tracer_init(struct trace_array *tr)
 {
 	blk_tr = tr;
 	blk_tracer_start(tr);
-	blk_tracer_enabled = true;
 	return 0;
 }
 
 static void blk_tracer_stop(struct trace_array *tr)
 {
+	blk_tracer_enabled = false;
 	trace_flags |= TRACE_ITER_CONTEXT_INFO;
 }
 
 static void blk_tracer_reset(struct trace_array *tr)
 {
-	blk_tracer_enabled = false;
 	blk_tracer_stop(tr);
 }
 
@@ -1250,7 +1245,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
 	if (atomic_dec_and_test(&blk_probes_ref))
 		blk_unregister_tracepoints();
 
-	kfree(bt);
+	blk_trace_free(bt);
 	return 0;
 }
 
Signature · 1 line
1.5.4.rc3
author
#6 lore raw

PATCH v3 4/7 blktrace: fix memory leak when freeing struct blk_io_trace

Quoted 5 lines

static void blk_tracer_start(struct trace_array *tr) {

(quoted diff) +1 -0
+	blk_tracer_enabled = true;
 	trace_flags &= ~TRACE_ITER_CONTEXT_INFO;
 }

...

Review

Quoted 5 lines

static void blk_tracer_reset(struct trace_array *tr) {

(quoted diff) +0 -1
-	blk_tracer_enabled = false;
 	blk_tracer_stop(tr);
 }

Those changes should have been in [PATCH 3/7]..

==========

[PATCH] blktrace: fix memory leak when freeing struct blk_io_trace

When mixing the use of ioctl-based blktrace and ftrace-based blktrace, we can leak memory in this way:

  # btrace /dev/sda > /dev/null &
  # echo 0 > /sys/block/sda/sda1/trace/enable
now we leak bt->dropped_file, bt->msg_file, bt->rchan...
---
 kernel/trace/blktrace.c |   25 ++++++++++---------------
 1 files changed, 10 insertions(+), 15 deletions(-)
kernel/trace/blktrace.c +10 -15
index acf8454..2f21d77 100644
@@ -247,7 +247,7 @@ record_it:
 static struct dentry *blk_tree_root;
 static DEFINE_MUTEX(blk_tree_mutex);
 
-static void blk_trace_cleanup(struct blk_trace *bt)
+static void blk_trace_free(struct blk_trace *bt)
 {
 	debugfs_remove(bt->msg_file);
 	debugfs_remove(bt->dropped_file);
@@ -255,6 +255,11 @@ static void blk_trace_cleanup(struct blk_trace *bt)
 	free_percpu(bt->sequence);
 	free_percpu(bt->msg_data);
 	kfree(bt);
+}
+
+static void blk_trace_cleanup(struct blk_trace *bt)
+{
+	blk_trace_free(bt);
 	if (atomic_dec_and_test(&blk_probes_ref))
 		blk_unregister_tracepoints();
 }
@@ -410,11 +415,11 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 		if (buts->name[i] == '/')
 			buts->name[i] = '_';
 
-	ret = -ENOMEM;
 	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
 	if (!bt)
-		goto err;
+		return -ENOMEM;
 
+	ret = -ENOMEM;
 	bt->sequence = alloc_percpu(unsigned long);
 	if (!bt->sequence)
 		goto err;
@@ -483,17 +488,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 
 	return 0;
 err:
-	if (bt) {
-		if (bt->msg_file)
-			debugfs_remove(bt->msg_file);
-		if (bt->dropped_file)
-			debugfs_remove(bt->dropped_file);
-		free_percpu(bt->sequence);
-		free_percpu(bt->msg_data);
-		if (bt->rchan)
-			relay_close(bt->rchan);
-		kfree(bt);
-	}
+	blk_trace_free(bt);
 	return ret;
 }
 
@@ -1250,7 +1245,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
 	if (atomic_dec_and_test(&blk_probes_ref))
 		blk_unregister_tracepoints();
 
-	kfree(bt);
+	blk_trace_free(bt);
 	return 0;
 }
 
Signature · 1 line
1.5.4.rc3
author
#7 lore raw

PATCH v2 5/7 blktrace: extract duplidate code

In reply to Li Zefan

blk_trace_event_print() and blk_tracer_print_line() share most of the code.

text data bss dec hex filename
8605 393 12 9010 2332 kernel/trace/blktrace.o.orig
text data bss dec hex filename
8555 393 12 8960 2300 kernel/trace/blktrace.o

This patch also prepares for the next patch, that prints out BLK_TN_MESSAGE.

---
 kernel/trace/blktrace.c |   62 ++++++++++++++++++++++------------------------
 1 files changed, 30 insertions(+), 32 deletions(-)
kernel/trace/blktrace.c +30 -32
index 2f21d77..c103b0c 100644
@@ -986,29 +986,31 @@ static void get_pdu_remap(const struct trace_entry *ent,
 	r->sector = be64_to_cpu(sector);
 }
 
-static int blk_log_action_iter(struct trace_iterator *iter, const char *act)
+typedef int (blk_log_action_t) (struct trace_iterator *iter, const char *act);
+
+static int blk_log_action_classic(struct trace_iterator *iter, const char *act)
 {
 	char rwbs[6];
 	unsigned long long ts  = iter->ts;
 	unsigned long nsec_rem = do_div(ts, NSEC_PER_SEC);
 	unsigned secs	       = (unsigned long)ts;
-	const struct trace_entry *ent = iter->ent;
-	const struct blk_io_trace *t = (const struct blk_io_trace *)ent;
+	const struct blk_io_trace *t = te_blk_io_trace(iter->ent);
 
 	fill_rwbs(rwbs, t);
 
 	return trace_seq_printf(&iter->seq,
 				"%3d,%-3d %2d %5d.%09lu %5u %2s %3s ",
 				MAJOR(t->device), MINOR(t->device), iter->cpu,
-				secs, nsec_rem, ent->pid, act, rwbs);
+				secs, nsec_rem, iter->ent->pid, act, rwbs);
 }
 
-static int blk_log_action_seq(struct trace_seq *s, const struct blk_io_trace *t,
-			      const char *act)
+static int blk_log_action(struct trace_iterator *iter, const char *act)
 {
 	char rwbs[6];
+	const struct blk_io_trace *t = te_blk_io_trace(iter->ent);
+
 	fill_rwbs(rwbs, t);
-	return trace_seq_printf(s, "%3d,%-3d %2s %3s ",
+	return trace_seq_printf(&iter->seq, "%3d,%-3d %2s %3s ",
 				MAJOR(t->device), MINOR(t->device), act, rwbs);
 }
 
@@ -1129,22 +1131,25 @@ static const struct {
 	[__BLK_TA_REMAP]	= {{  "A", "remap" },	   blk_log_remap },
 };
 
-static enum print_line_t blk_trace_event_print(struct trace_iterator *iter,
-					       int flags)
+static enum print_line_t print_one_line(struct trace_iterator *iter,
+					bool classic)
 {
 	struct trace_seq *s = &iter->seq;
-	const struct blk_io_trace *t = (struct blk_io_trace *)iter->ent;
-	const u16 what = t->action & ((1 << BLK_TC_SHIFT) - 1);
+	const struct blk_io_trace *t;
+	u16 what;
 	int ret;
+	bool long_act;
+	blk_log_action_t *log_action;
 
-	if (!trace_print_context(iter))
-		return TRACE_TYPE_PARTIAL_LINE;
+	t	   = te_blk_io_trace(iter->ent);
+	what	   = t->action & ((1 << BLK_TC_SHIFT) - 1);
+	long_act   = !!(trace_flags & TRACE_ITER_VERBOSE);
+	log_action = classic ? &blk_log_action_classic : &blk_log_action;
 
 	if (unlikely(what == 0 || what >= ARRAY_SIZE(what2act)))
 		ret = trace_seq_printf(s, "Bad pc action %x\n", what);
 	else {
-		const bool long_act = !!(trace_flags & TRACE_ITER_VERBOSE);
-		ret = blk_log_action_seq(s, t, what2act[what].act[long_act]);
+		ret = log_action(iter, what2act[what].act[long_act]);
 		if (ret)
 			ret = what2act[what].print(s, iter->ent);
 	}
@@ -1152,6 +1157,15 @@ static enum print_line_t blk_trace_event_print(struct trace_iterator *iter,
 	return ret ? TRACE_TYPE_HANDLED : TRACE_TYPE_PARTIAL_LINE;
 }
 
+static enum print_line_t blk_trace_event_print(struct trace_iterator *iter,
+					       int flags)
+{
+	if (!trace_print_context(iter))
+		return TRACE_TYPE_PARTIAL_LINE;
+
+	return print_one_line(iter, false);
+}
+
 static int blk_trace_synthesize_old_trace(struct trace_iterator *iter)
 {
 	struct trace_seq *s = &iter->seq;
@@ -1177,26 +1191,10 @@ blk_trace_event_print_binary(struct trace_iterator *iter, int flags)
 
 static enum print_line_t blk_tracer_print_line(struct trace_iterator *iter)
 {
-	const struct blk_io_trace *t;
-	u16 what;
-	int ret;
-
 	if (!(blk_tracer_flags.val & TRACE_BLK_OPT_CLASSIC))
 		return TRACE_TYPE_UNHANDLED;
 
-	t = (const struct blk_io_trace *)iter->ent;
-	what = t->action & ((1 << BLK_TC_SHIFT) - 1);
-
-	if (unlikely(what == 0 || what >= ARRAY_SIZE(what2act)))
-		ret = trace_seq_printf(&iter->seq, "Bad pc action %x\n", what);
-	else {
-		const bool long_act = !!(trace_flags & TRACE_ITER_VERBOSE);
-		ret = blk_log_action_iter(iter, what2act[what].act[long_act]);
-		if (ret)
-			ret = what2act[what].print(&iter->seq, iter->ent);
-	}
-
-	return ret ? TRACE_TYPE_HANDLED : TRACE_TYPE_PARTIAL_LINE;
+	return print_one_line(iter, true);
 }
 
 static struct tracer blk_tracer __read_mostly = {
Signature · 1 line
1.5.4.rc3
author
#8 lore raw

PATCH v2 6/7 blktrace: print out BLK_TN_MESSAGE properly

In reply to Li Zefan

Before this patch:

 # cat trace
         make-5383  [001]   741.240059:   8,7    P   N [make]
 __trace_note_message: cfq1074
 # echo 1 > options/blk_classic
 # cat trace
   8,7    1     0.692221252     0  C   W 130411392 + 1024 [0]
 Bad pc action 6361
 Bad pc action 283d
 # echo 0 > options/blk_classic
 # echo bin > trace_options
 # cat trace_pipe | blkparse -i -
 (can't parse messages generated by blk_add_trace_msg())

After this patch:

 # cat trace
      <idle>-0     [001]   187.600933:   8,7    C   W 145220224 + 8 [0]
      <idle>-0     [001]   187.600946:   8,7    m   N cfq1076 complete
 # echo 1 > options/blk_classic
 # cat trace
   8,7    1     0.256378996   238  I   W 113190728 + 8 [pdflush]
   8,7    1     0.256378998   238  m   N cfq1076 insert_request
 # echo 0 > options/blk_classic
 # echo bin > trace_options
 # cat trace_pipe | blkparse -i -
  8,7    1        0    22.973250293     0  C   W 102770576 + 8 [0]
  8,7    1        0    22.973259213     0  m   N cfq1076 complete
---
 kernel/trace/blktrace.c |   80 +++++++++++++++++++++++++++++++++++-----------
 1 files changed, 61 insertions(+), 19 deletions(-)
kernel/trace/blktrace.c +61 -19
index c103b0c..947c5b3 100644
@@ -59,22 +59,39 @@ static void trace_note(struct blk_trace *bt, pid_t pid, int action,
 		       const void *data, size_t len)
 {
 	struct blk_io_trace *t;
+	struct ring_buffer_event *event = NULL;
+	int pc = 0;
+	int cpu = smp_processor_id();
+	bool blk_tracer = blk_tracer_enabled;
+
+	if (blk_tracer) {
+		pc = preempt_count();
+		event = trace_buffer_lock_reserve(blk_tr, TRACE_BLK,
+						  sizeof(*t) + len,
+						  0, pc);
+		if (!event)
+			return;
+		t = ring_buffer_event_data(event);
+		goto record_it;
+	}
 
 	if (!bt->rchan)
 		return;
 
 	t = relay_reserve(bt->rchan, sizeof(*t) + len);
 	if (t) {
-		const int cpu = smp_processor_id();
-
 		t->magic = BLK_IO_TRACE_MAGIC | BLK_IO_TRACE_VERSION;
 		t->time = ktime_to_ns(ktime_get());
+record_it:
 		t->device = bt->dev;
 		t->action = action;
 		t->pid = pid;
 		t->cpu = cpu;
 		t->pdu_len = len;
 		memcpy((void *) t + sizeof(*t), data, len);
+
+		if (blk_tracer)
+			trace_buffer_unlock_commit(blk_tr, event, 0, pc);
 	}
 }
 
@@ -110,14 +127,8 @@ void __trace_note_message(struct blk_trace *bt, const char *fmt, ...)
 	unsigned long flags;
 	char *buf;
 
-	if (blk_tracer_enabled) {
-		va_start(args, fmt);
-		ftrace_vprintk(fmt, args);
-		va_end(args);
-		return;
-	}
-
-	if (!bt->msg_data)
+	if (unlikely(bt->trace_state != Blktrace_running &&
+		     !blk_tracer_enabled))
 		return;
 
 	local_irq_save(flags);
@@ -168,9 +179,9 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 	unsigned long *sequence;
 	pid_t pid;
 	int cpu, pc = 0;
+	bool blk_tracer = blk_tracer_enabled;
 
-	if (unlikely(bt->trace_state != Blktrace_running &&
-		     !blk_tracer_enabled))
+	if (unlikely(bt->trace_state != Blktrace_running && !blk_tracer))
 		return;
 
 	what |= ddir_act[rw & WRITE];
@@ -185,7 +196,7 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 		return;
 	cpu = raw_smp_processor_id();
 
-	if (blk_tracer_enabled) {
+	if (blk_tracer) {
 		tracing_record_cmdline(current);
 
 		pc = preempt_count();
@@ -235,7 +246,7 @@ record_it:
 		if (pdu_len)
 			memcpy((void *) t + sizeof(*t), pdu_data, pdu_len);
 
-		if (blk_tracer_enabled) {
+		if (blk_tracer) {
 			trace_buffer_unlock_commit(blk_tr, event, 0, pc);
 			return;
 		}
@@ -922,6 +933,11 @@ static void fill_rwbs(char *rwbs, const struct blk_io_trace *t)
 	int i = 0;
 	int tc = t->action >> BLK_TC_SHIFT;
 
+	if (t->action == BLK_TN_MESSAGE) {
+		rwbs[i++] = 'N';
+		goto out;
+	}
+
 	if (tc & BLK_TC_DISCARD)
 		rwbs[i++] = 'D';
 	else if (tc & BLK_TC_WRITE)
@@ -939,7 +955,7 @@ static void fill_rwbs(char *rwbs, const struct blk_io_trace *t)
 		rwbs[i++] = 'S';
 	if (tc & BLK_TC_META)
 		rwbs[i++] = 'M';
-
+out:
 	rwbs[i] = '\0';
 }
 
@@ -1074,6 +1090,17 @@ static int blk_log_split(struct trace_seq *s, const struct trace_entry *ent)
 				get_pdu_int(ent), cmd);
 }
 
+static int blk_log_msg(struct trace_seq *s, const struct trace_entry *ent)
+{
+	int ret;
+	const struct blk_io_trace *t = te_blk_io_trace(ent);
+
+	ret = trace_seq_putmem(s, t + 1, t->pdu_len);
+	if (ret)
+		return trace_seq_putc(s, '\n');
+	return ret;
+}
+
 /*
  * struct tracer operations
  */
@@ -1146,6 +1173,13 @@ static enum print_line_t print_one_line(struct trace_iterator *iter,
 	long_act   = !!(trace_flags & TRACE_ITER_VERBOSE);
 	log_action = classic ? &blk_log_action_classic : &blk_log_action;
 
+	if (t->action == BLK_TN_MESSAGE) {
+		ret = log_action(iter, long_act ? "message" : "m");
+		if (ret)
+			ret = blk_log_msg(s, iter->ent);
+		goto out;
+	}
+
 	if (unlikely(what == 0 || what >= ARRAY_SIZE(what2act)))
 		ret = trace_seq_printf(s, "Bad pc action %x\n", what);
 	else {
@@ -1153,7 +1187,7 @@ static enum print_line_t print_one_line(struct trace_iterator *iter,
 		if (ret)
 			ret = what2act[what].print(s, iter->ent);
 	}
-
+out:
 	return ret ? TRACE_TYPE_HANDLED : TRACE_TYPE_PARTIAL_LINE;
 }
 
@@ -1253,11 +1287,16 @@ static int blk_trace_remove_queue(struct request_queue *q)
 static int blk_trace_setup_queue(struct request_queue *q, dev_t dev)
 {
 	struct blk_trace *old_bt, *bt = NULL;
+	int ret = -ENOMEM;
 
 	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
 	if (!bt)
 		return -ENOMEM;
 
+	bt->msg_data = __alloc_percpu(BLK_TN_MAX_MSG, __alignof__(char));
+	if (!bt->msg_data)
+		goto free_bt;
+
 	bt->dev = dev;
 	bt->act_mask = (u16)-1;
 	bt->end_lba = -1ULL;
@@ -1265,14 +1304,17 @@ static int blk_trace_setup_queue(struct request_queue *q, dev_t dev)
 	old_bt = xchg(&q->blk_trace, bt);
 	if (old_bt != NULL) {
 		(void)xchg(&q->blk_trace, old_bt);
-		kfree(bt);
-		return -EBUSY;
+		ret = -EBUSY;
+		goto free_bt;
 	}
 
 	if (atomic_inc_return(&blk_probes_ref) == 1)
 		blk_register_tracepoints();
-
 	return 0;
+
+free_bt:
+	blk_trace_free(bt);
+	return ret;
 }
 
 /*
Signature · 1 line
1.5.4.rc3
author
#9 lore raw

PATCH v2 7/7 blktrace: support filtering of BLK_TN_MESSAGE

In reply to Li Zefan

I found large bunk of the output of blktrace are BLK_TN_MESSAGE events:

% blktrace -d /dev/sda -o - | blkparse -i - -o ~/blk.log (after some seconds)
% cat ~/blk.log | wc -l
69041
% cat ~/blk.log | grep "m N" | wc -l
38232

It will be useful to be able to filter those events.

---
 kernel/trace/blktrace.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
kernel/trace/blktrace.c +4 -0
index 947c5b3..dcabe59 100644
@@ -131,6 +131,9 @@ void __trace_note_message(struct blk_trace *bt, const char *fmt, ...)
 		     !blk_tracer_enabled))
 		return;
 
+	if (!(bt->act_mask & BLK_TC_NOTIFY))
+		return;
+
 	local_irq_save(flags);
 	buf = per_cpu_ptr(bt->msg_data, smp_processor_id());
 	va_start(args, fmt);
@@ -1366,6 +1369,7 @@ static const struct {
 	{ BLK_TC_COMPLETE,	"complete"	},
 	{ BLK_TC_FS,		"fs"		},
 	{ BLK_TC_PC,		"pc"		},
+	{ BLK_TC_NOTIFY,	"notify"	},
 	{ BLK_TC_AHEAD,		"ahead"		},
 	{ BLK_TC_META,		"meta"		},
 	{ BLK_TC_DISCARD,	"discard"	},
Signature · 1 line
1.5.4.rc3
Reviewed-by on the v2 series
#10 lore raw
In reply to Li Zefan
Quoted 9 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

Quoted 6 lines

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

thanks Li!

I think we can do the following ordering. First apply the fixes:

Quoted 3 lines

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace

These mostly relate to mixed use of ioctl and ftrace based block tracing - which is rare but indeed possible. The fixes and cleanups you did to refcounting look good.

These bits:

Quoted 3 lines

[PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 5/6] blktrace: extract duplidate code [PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

are optional improvements. The BLK_TN_MESSAGE filter addition - Jens, are you fine with that feature addition? (it certainly looks like a nice thing to me.)

Also, note that we'll get even more in-kernel filtering capabilities in blktrace if we convert the tracepoints to TRACE_EVENT().

Ingo

#11 lore raw
In reply to Li Zefan
Quoted 14 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

Quoted 5 lines
---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)

Li, Jens - what's your call for v2.6.30, can the blktrace changes go upstream or should i zap the whole thing?

There are no new -tip test failures here so it's green as far as i'm concerned. It's holding up the tracing tree so we need to decide one way or another.

Thanks,

Ingo

#12 lore raw
Quoted 23 lines · Ingo Molnar
Quoted 14 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

Quoted 5 lines
---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)
Quoted 6 lines

Li, Jens - what's your call for v2.6.30, can the blktrace changes go upstream or should i zap the whole thing?

There are no new -tip test failures here so it's green as far as i'm concerned. It's holding up the tracing tree so we need to decide one way or another.

If Li deems it ready, then it has my blessing as well.

Signature · 1 line
Jens Axboe
#13 lore raw
Quoted 24 lines · Jens Axboe
Quoted 23 lines · Ingo Molnar
Quoted 19 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)
Quoted 8 lines
Quoted 6 lines

Li, Jens - what's your call for v2.6.30, can the blktrace changes go upstream or should i zap the whole thing?

There are no new -tip test failures here so it's green as far as i'm concerned. It's holding up the tracing tree so we need to decide one way or another.

If Li deems it ready, then it has my blessing as well.

thanks Jens - i have added your Acked-by's as well.

Ingo

author
#14 lore raw
In reply to Ingo Molnar
Quoted 22 lines · Ingo Molnar
Quoted 14 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

Quoted 5 lines
---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)
Quoted 3 lines

Li, Jens - what's your call for v2.6.30, can the blktrace changes go upstream or should i zap the whole thing?

It should be find to merge tip/blktrace into .30, and we still have 2 month to fix remaining bugs, if any.

Quoted 4 lines

There are no new -tip test failures here so it's green as far as i'm concerned. It's holding up the tracing tree so we need to decide one way or another.

#15 lore raw
Quoted 23 lines · Li Zefan
Quoted 22 lines · Ingo Molnar
Quoted 19 lines · Li Zefan

This is the last round of fixes for blktrace, I hope. :)

Changelog v1 -> v2:
- fix a bug in "blktrace: make classic output more classic", and rebase other patches.
- re-order patches.
- a new patch for filtering BLK_TN_MESSAGE

[PATCH v2 1/6] trace: make 'mem' argument of trace_seq_putmem() const [PATCH v2 2/6] blktrace: make classic output more classic [PATCH v2 3/6] blktrace: fix blk_probes_ref chaos
[PATCH v2 4/6] blktrace: fix memory leak when freeing struct blk_io_trace [PATCH v2 5/6] blktrace: extract duplidate code
[PATCH v2 6/6] blktrace: print out BLK_TN_MESSAGE properly

---
 blktrace.c     |  188 +++++++++++++++++++++++++++++++++------------------------
 trace_output.c |    6 -
 trace_output.h |    5 -
 3 files changed, 117 insertions(+), 82 deletions(-)
Quoted 6 lines
Quoted 3 lines

Li, Jens - what's your call for v2.6.30, can the blktrace changes go upstream or should i zap the whole thing?

It should be fine to merge tip/blktrace into .30, and we still have 2 month to fix remaining bugs, if any.

ok.

Although regarding instrumentation patches we carry in -tip we aim to enforce perfect stability right at the point of the merge window.

One major point of instrumentation is to find kernel bugs ... and it's self-defeating if instrumentation is buggy itself! By the time the stable kernel is released the most important kernel bugs are fixed already so having stable instrumentation by that moment is too late.

So we've got extensive self-tests and other measures in place to ensure stability.

( The mixed blktrace ioctl/ftrace plugin usage model fell a bit
outside of the expectation of these stability tests though, hence this relatively slow and late stabilization curve there. It still looks all fine for .30 to me too. )

Ingo