Skip to contentSkip to conversation

blktrace2: Fully working variant... Needs testing... :-)

PATCH Alan D. Brunelle 3 participants 6 messages Feb 4, 2009 – Feb 6, 2009 Kbuild

lore
Share & tools
lore

Alan D. Brunelle

author
#0 lore raw

I'm seeing some positive results on my 16-way amd64 box (w/ 48 FC disks & 48 CCISS disks) - less intrusive blktrace()ing, resulting in more benchmark through put for example.

It seems to be pretty valgrind clean (only issue I've seen is in
inet_ntoa: man page says it uses static storage, but valgrind claims it uses malloc - nothing for us to be concerned with).

Anyways, I'm putting this out there whilst I do some more testing to verify things.

From 64e349af652d58213725673dd722df079f38c74c Mon Sep 17 00:00:00 2001
From: Alan D. Brunelle <[email protected]>
Date: Wed, 4 Feb 2009 15:55:31 -0500
Subject: [PATCH] Rewrote blktrace into blktrace2

Massive changes: mostly around the notion of having much fewer threads (instead of N(devs) X N(cpus) threads, we'll have just N(cpus)). This is very important for larger systems (with lots of devices to trace). (Obviously, large parts of this have been stolen from blktrace...)

o  Command line arguments for blktrace2 are identical w/ blktrace
o  Some of the output has been slightly modified
o  Added list_replace_init to btt/list.h -- needed in blktrace2

I'm concerned about pipeline mode, in particular:

blktrace2 -o - <dev> | blkparse -i -

seems to hiccup after an initial burst of data - blkparse doesn't spit out more data until a long time afterwards (typically at ^C). I'm seeing the same thing with blktrace, so I'm not _too_ concerned... :-)

Needs a lot of testing... :-)

Some possible clean up areas (more based upon testing...):

o Have some volatile variables: may _not_ be necessary.

o Use of mutex & condition variables _may_ be overkill. Used for modes where the main thread is combining things from the tracer threads to send out (e.g., piped mode or network clients that don't use sendfile).

o There is probably some redundant code that could be extracted and put into functions.

Have I mentioned this could use a lot more testing?

Signed-off-by Alan D. Brunelle <[email protected]>
---
 Makefile    |    5 +-
 blktrace2.c | 2467 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 btt/list.h  |   23 +
 3 files changed, 2494 insertions(+), 1 deletions(-)
Makefile +4 -1
index 623b84b..ad2836d 100644
@@ -1,7 +1,7 @@
 CC	= gcc
 CFLAGS	= -Wall -O2 -g -W
 ALL_CFLAGS = $(CFLAGS) -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-PROGS	= blkparse blktrace verify_blkparse blkrawverify blkiomon
+PROGS	= blkparse blktrace verify_blkparse blkrawverify blkiomon blktrace2
 LIBS	= -lpthread
 SCRIPTS	= btrace
 

@@ -28,6 +28,9 @@ blkparse: blkparse.o blkparse_fmt.o rbtree.o act_mask.o
 blktrace: blktrace.o act_mask.o
 	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^) $(LIBS)
 
+blktrace2: blktrace2.o act_mask.o
+	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^) $(LIBS)
+
 verify_blkparse: verify_blkparse.o
 	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^)
blktrace2.c new file +2467 -0

2,467 lines not on this page · raw message

btt/list.h +23 -0
index 363c888..71c4a65 100644
@@ -213,4 +213,27 @@ static inline void list_splice(struct list_head *list, struct list_head *head)
                 __list_splice(list, head);
 }
 
+/**
+ * list_replace - replace old entry by new one
+ * @old : the element to be replaced
+ * @new : the new element to insert
+ *
+ * If @old was empty, it will be overwritten.
+ */
+static inline void list_replace(struct list_head *old,
+				struct list_head *new)
+{
+	new->next = old->next;
+	new->next->prev = new;
+	new->prev = old->prev;
+	new->prev->next = new;
+}
+
+static inline void list_replace_init(struct list_head *old,
+					struct list_head *new)
+{
+	list_replace(old, new);
+	INIT_LIST_HEAD(old);
+}
+
 #endif
Signature · 1 line
1.5.6.3
author
#1 lore raw
Quoted 11 lines · Alan D. Brunelle

I'm seeing some positive results on my 16-way amd64 box (w/ 48 FC disks & 48 CCISS disks) - less intrusive blktrace()ing, resulting in more benchmark through put for example.

It seems to be pretty valgrind clean (only issue I've seen is in
inet_ntoa: man page says it uses static storage, but valgrind claims it uses malloc - nothing for us to be concerned with).

Anyways, I'm putting this out there whilst I do some more testing to verify things.

Some good news: doing my previously reported testing on the balanced configuration completed successfully. (mkfs on large numbers of CCISS disks, tracing to a large number of FC disks)

What is more, it appears to be a little better in terms of fewer drops & fewer drop cases - results below are in percent drops:

blktrace:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            4.4   0.0   0.0   0.0
    8|                                      1.5   0.0
   16|                                0.1         0.0
   32|                          0.8               0.0
   64|                    1.1
  128|              0.8
  256|        2.6
  512|  2.3
 1024|  0.5
 2048|  0.1
 4096|  0.0

blktrace2:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            0.2   0.0   0.0   0.0
    8|                                      0.1   0.0
   16|                                0.0         0.0
   32|                          0.0               0.0
   64|                    0.1
  128|              0.1
  256|        0.1
  512|  0.2
 1024|  0.0
 2048|  0.0
 4096|  0.0

The goal now will be to try and see if I can wiggle out the remaining 0.1 or 0.2% drops...

Alan

#2 lore raw
In reply to Alan D. Brunelle
Quoted 39 lines · Alan D. Brunelle
Quoted 11 lines · Alan D. Brunelle

I'm seeing some positive results on my 16-way amd64 box (w/ 48 FC disks & 48 CCISS disks) - less intrusive blktrace()ing, resulting in more benchmark through put for example.

It seems to be pretty valgrind clean (only issue I've seen is in
inet_ntoa: man page says it uses static storage, but valgrind claims it uses malloc - nothing for us to be concerned with).

Anyways, I'm putting this out there whilst I do some more testing to verify things.

Some good news: doing my previously reported testing on the balanced configuration completed successfully. (mkfs on large numbers of CCISS disks, tracing to a large number of FC disks)

What is more, it appears to be a little better in terms of fewer drops & fewer drop cases - results below are in percent drops:

blktrace:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            4.4   0.0   0.0   0.0
    8|                                      1.5   0.0
   16|                                0.1         0.0
   32|                          0.8               0.0
   64|                    1.1
  128|              0.8
  256|        2.6
  512|  2.3
 1024|  0.5
 2048|  0.1
 4096|  0.0

blktrace2:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

Quoted 12 lines
-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            0.2   0.0   0.0   0.0
    8|                                      0.1   0.0
   16|                                0.0         0.0
   32|                          0.0               0.0
   64|                    0.1
  128|              0.1
  256|        0.1
  512|  0.2
 1024|  0.0
 2048|  0.0
 4096|  0.0

That looks pretty good. As I mentioned earlier, I think the blktrace2 approach is sound. The existing scheme just doesn't scale to large number of spindles and CPUs, so it's a step in the right direction.

I'll be on vacation later today and 9 days forward, so once you feel confident in blktrace2, feel free to commit it. Commit it as blktrace.c though, we don't want two tools!

Quoted 2 lines

The goal now will be to try and see if I can wiggle out the remaining 0.1 or 0.2% drops...

That would be optimal, naturally :-)

Signature · 1 line
Jens Axboe
author
#3 lore raw
In reply to Alan D. Brunelle
Quoted 56 lines · Jens Axboe
Quoted 38 lines · Alan D. Brunelle
Quoted 11 lines · Alan D. Brunelle

I'm seeing some positive results on my 16-way amd64 box (w/ 48 FC disks & 48 CCISS disks) - less intrusive blktrace()ing, resulting in more benchmark through put for example.

It seems to be pretty valgrind clean (only issue I've seen is in
inet_ntoa: man page says it uses static storage, but valgrind claims it uses malloc - nothing for us to be concerned with).

Anyways, I'm putting this out there whilst I do some more testing to verify things.

Some good news: doing my previously reported testing on the balanced configuration completed successfully. (mkfs on large numbers of CCISS disks, tracing to a large number of FC disks)

What is more, it appears to be a little better in terms of fewer drops & fewer drop cases - results below are in percent drops:

blktrace:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            4.4   0.0   0.0   0.0
    8|                                      1.5   0.0
   16|                                0.1         0.0
   32|                          0.8               0.0
   64|                    1.1
  128|              0.8
  256|        2.6
  512|  2.3
 1024|  0.5
 2048|  0.1
 4096|  0.0

blktrace2:

-b 4 8 16 32 64 128 256 512 1024 2048 4096

Quoted 12 lines
-n   |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
    4|                                            0.2   0.0   0.0   0.0
    8|                                      0.1   0.0
   16|                                0.0         0.0
   32|                          0.0               0.0
   64|                    0.1
  128|              0.1
  256|        0.1
  512|  0.2
 1024|  0.0
 2048|  0.0
 4096|  0.0

That looks pretty good. As I mentioned earlier, I think the blktrace2 approach is sound. The existing scheme just doesn't scale to large number of spindles and CPUs, so it's a step in the right direction.

Quoted 9 lines

I'll be on vacation later today and 9 days forward, so once you feel confident in blktrace2, feel free to commit it. Commit it as blktrace.c though, we don't want two tools!

Quoted 2 lines

The goal now will be to try and see if I can wiggle out the remaining 0.1 or 0.2% drops...

That would be optimal, naturally :-)

OK, but I'm still working through some issues that testing is uncovering, blktrace works better at these things (simple stuff really), so I'll leave that alone until I'm really sure blktrace2 works as good...

Alan

#4 lore raw
In reply to Alan D. Brunelle
Quoted 5 lines · Alan D. Brunelle

OK, but I'm still working through some issues that testing is uncovering, blktrace works better at these things (simple stuff really), so I'll leave that alone until I'm really sure blktrace2 works as good...

Alan

I'm interested in testing this. How do I check out a copy?

Signature · 3 lines
M. Edward (Ed) Borasky

I've never met a happy clam. In fact, most of them were pretty steamed.
author
#5 lore raw
In reply to Alan D. Brunelle
Quoted 10 lines · M. Edward (Ed) Borasky
Quoted 5 lines · Alan D. Brunelle

OK, but I'm still working through some issues that testing is uncovering, blktrace works better at these things (simple stuff really), so I'll leave that alone until I'm really sure blktrace2 works as good...

Alan

I'm interested in testing this. How do I check out a copy?

Hi Ed -

I'm working a networking issue, but for now here's a patch that should apply to the standard blktrace tree, and when made will produce 'blktrace2' - which should be plug-for-plug compatible w/ blktrace. (Note: I'm in the middle of some stuff w/ this patch, so there may be errors - point them out & I'll work on it... :-) )

When I get this networking issue fixed, Jens has asked me to overwrite 'blktrace' itself.

Alan

From 9aff0c008c35b46870243ba022a4e31219796bfa Mon Sep 17 00:00:00 2001
From: Alan D. Brunelle <[email protected]>
Date: Fri, 6 Feb 2009 10:18:01 -0500
Subject: [PATCH] Temporary: blktrace2 updates

Should be plug-for-plug compatible w/ blktrace, however, network clean up is kinda fishy - the data looks OK, but I'm getting some polling issues at the end.

Signed-off-by Alan D. Brunelle <[email protected]>
---
 Makefile    |    5 +-
 blktrace2.c | 2525 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 btt/list.h  |   23 +
 3 files changed, 2552 insertions(+), 1 deletions(-)
Makefile +4 -1
index 623b84b..ad2836d 100644
@@ -1,7 +1,7 @@
 CC	= gcc
 CFLAGS	= -Wall -O2 -g -W
 ALL_CFLAGS = $(CFLAGS) -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-PROGS	= blkparse blktrace verify_blkparse blkrawverify blkiomon
+PROGS	= blkparse blktrace verify_blkparse blkrawverify blkiomon blktrace2
 LIBS	= -lpthread
 SCRIPTS	= btrace
 

@@ -28,6 +28,9 @@ blkparse: blkparse.o blkparse_fmt.o rbtree.o act_mask.o
 blktrace: blktrace.o act_mask.o
 	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^) $(LIBS)
 
+blktrace2: blktrace2.o act_mask.o
+	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^) $(LIBS)
+
 verify_blkparse: verify_blkparse.o
 	$(CC) $(ALL_CFLAGS) -o $@ $(filter %.o,$^)
blktrace2.c new file +2525 -0

2,525 lines not on this page · raw message

btt/list.h +23 -0
index 363c888..71c4a65 100644
@@ -213,4 +213,27 @@ static inline void list_splice(struct list_head *list, struct list_head *head)
                 __list_splice(list, head);
 }
 
+/**
+ * list_replace - replace old entry by new one
+ * @old : the element to be replaced
+ * @new : the new element to insert
+ *
+ * If @old was empty, it will be overwritten.
+ */
+static inline void list_replace(struct list_head *old,
+				struct list_head *new)
+{
+	new->next = old->next;
+	new->next->prev = new;
+	new->prev = old->prev;
+	new->prev->next = new;
+}
+
+static inline void list_replace_init(struct list_head *old,
+					struct list_head *new)
+{
+	list_replace(old, new);
+	INIT_LIST_HEAD(old);
+}
+
 #endif
Signature · 1 line
1.5.6.3