writeback: moving expire targets for background/kupdate works v2

---
 fs/fs-writeback.c |   16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

fs/fs-writeback.c +8 -8

@@ -251,8 +251,8 @@ static bool inode_dirtied_after(struct i
  * Move expired dirty inodes from @delaying_queue to @dispatch_queue.
  */
 static void move_expired_inodes(struct list_head *delaying_queue,
-			       struct list_head *dispatch_queue,
-				unsigned long *older_than_this)
+				struct list_head *dispatch_queue,
+				struct writeback_control *wbc)
 {
 	LIST_HEAD(tmp);
 	struct list_head *pos, *node;

@@ -262,8 +262,8 @@ static void move_expired_inodes(struct l
 
 	while (!list_empty(delaying_queue)) {
 		inode = wb_inode(delaying_queue->prev);
-		if (older_than_this &&
-		    inode_dirtied_after(inode, *older_than_this))
+		if (wbc->older_than_this &&
+		    inode_dirtied_after(inode, *wbc->older_than_this))
 			break;
 		if (sb && sb != inode->i_sb)
 			do_sb_sort = 1;

@@ -299,11 +299,11 @@ static void move_expired_inodes(struct l
  *                                           |
  *                                           +--> dequeue for IO
  */
-static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
+static void queue_io(struct bdi_writeback *wb, struct writeback_control *wbc)
 {
 	assert_spin_locked(&inode_wb_list_lock);
 	list_splice_init(&wb->b_more_io, &wb->b_io);
-	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
+	move_expired_inodes(&wb->b_dirty, &wb->b_io, wbc);
 }
 
 static int write_inode(struct inode *inode, struct writeback_control *wbc)

@@ -579,7 +579,7 @@ void writeback_inodes_wb(struct bdi_writ
 		wbc->wb_start = jiffies; /* livelock avoidance */
 	spin_lock(&inode_wb_list_lock);
 	if (!wbc->for_kupdate || list_empty(&wb->b_io))
-		queue_io(wb, wbc->older_than_this);
+		queue_io(wb, wbc);
 
 	while (!list_empty(&wb->b_io)) {
 		struct inode *inode = wb_inode(wb->b_io.prev);

@@ -606,7 +606,7 @@ static void __writeback_inodes_sb(struct
 
 	spin_lock(&inode_wb_list_lock);
 	if (!wbc->for_kupdate || list_empty(&wb->b_io))
-		queue_io(wb, wbc->older_than_this);
+		queue_io(wb, wbc);
 	writeback_sb_inodes(sb, wb, wbc, true);
 	spin_unlock(&inode_wb_list_lock);
 }

Christoph Hellwig

2011-05-04 11:04 UTC #2 lore raw

Re: [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes()

Quoted 2 lines · Wu Fengguang

No behavior change. This will add debug visibility to the code, for example, to dump the wbc contents when kprobing queue_io().

I don't think it's a good idea. The writeback_control should move back to just controlling per-inode writeback and not be passed to more routines dealing with high-level writeback.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wu Fengguang

author

2011-05-04 11:13 UTC #3 lore raw

Re: [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes()

Quoted 7 lines · Christoph Hellwig

Quoted 2 lines · Wu Fengguang
No behavior change. This will add debug visibility to the code, for example, to dump the wbc contents when kprobing queue_io().
I don't think it's a good idea. The writeback_control should move back to just controlling per-inode writeback and not be passed to more routines dealing with high-level writeback.

Good point. I can do without this patch.

Thanks,
Fengguang

Wu Fengguang

author

2011-04-20 08:03 UTC #4 lore raw

PATCH 2/6 writeback: introduce writeback_control.inodes_cleaned

The flusher works on dirty inodes in batches, and may quit prematurely if the batch of inodes happen to be metadata-only dirtied: in this case wbc->nr_to_write won't be decreased at all, which stands for "no pages written" but also mis-interpreted as "no progress".

So introduce writeback_control.inodes_cleaned to count the inodes get cleaned. A non-zero value means there are some progress on writeback, in which case more writeback can be tried.

about v1: The initial version was to count successful ->write_inode() calls. However it leads to busy loops for sync() over NFS, because NFS ridiculously returns 0 (success) while at the same time redirties the inode. The NFS case can be trivially fixed, however there may be more hidden bugs in other filesystems..

Acked-by Jan Kara <[email protected]>

---
 fs/fs-writeback.c         |    4 ++++
 include/linux/writeback.h |    1 +
 2 files changed, 5 insertions(+)

fs/fs-writeback.c +4 -0

@@ -454,6 +454,7 @@ writeback_single_inode(struct inode *ino
 			 * No need to add it back to the LRU.
 			 */
 			list_del_init(&inode->i_wb_list);
+			wbc->inodes_cleaned++;
 		}
 	}
 	inode_sync_complete(inode);

@@ -716,6 +717,7 @@ static long wb_writeback(struct bdi_writ
 		wbc.more_io = 0;
 		wbc.nr_to_write = write_chunk;
 		wbc.pages_skipped = 0;
+		wbc.inodes_cleaned = 0;
 
 		trace_wbc_writeback_start(&wbc, wb->bdi);
 		if (work->sb)

@@ -732,6 +734,8 @@ static long wb_writeback(struct bdi_writ
 		 */
 		if (wbc.nr_to_write <= 0)
 			continue;
+		if (wbc.inodes_cleaned)
+			continue;
 		/*
 		 * Didn't write everything and we don't have more IO, bail
 		 */

include/linux/writeback.h +1 -0

@@ -34,6 +34,7 @@ struct writeback_control {
 	long nr_to_write;		/* Write this many pages, and decrement
 					   this for each page written */
 	long pages_skipped;		/* Pages which were not written */
+	long inodes_cleaned;		/* # of inodes cleaned */
 
 	/*
 	 * For a_ops->writepages(): is start or end are non-zero then this is

Christoph Hellwig

2011-05-04 11:05 UTC #5 lore raw

Re: [PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned

Same here - this has nothing to do with actual page writeback and really should stay internal to fs/fs-writeback.c

Wu Fengguang

author

2011-05-04 11:11 UTC #6 lore raw

Re: [PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned

Quoted 2 lines · Christoph Hellwig

Same here - this has nothing to do with actual page writeback and really should stay internal to fs/fs-writeback.c

OK, I'll check how to constrain writeback_control to the minimal scope.

Thanks,
Fengguang

Christoph Hellwig

2011-05-04 11:16 UTC #7 lore raw

Re: [PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned

Quoted 5 lines · Wu Fengguang

Quoted 2 lines · Christoph Hellwig
Same here - this has nothing to do with actual page writeback and really should stay internal to fs/fs-writeback.c
OK, I'll check how to constrain writeback_control to the minimal scope.

The basic idea would be to pass the struct wb_writeback_work all the way down to writeback_sb_inodes, and initialize the struct writeback_control there. This requires adding a few more fields like more_io and older_than_this to struct wb_writeback_work, and redoing a lot of the tracing, but it immediately clean things up, e.g. suddently wbc.nr_to_write vs work->nr_pages starts to make sense, and instead of saving and restoring pages_skipped in writeback_sb_inodes it can always start with a clean zero value.

Wu Fengguang

author

2011-05-04 11:32 UTC #8 lore raw

Re: [PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned

Quoted 10 lines · Christoph Hellwig

Quoted 5 lines · Wu Fengguang
Quoted 2 lines · Christoph Hellwig
Same here - this has nothing to do with actual page writeback and really should stay internal to fs/fs-writeback.c
OK, I'll check how to constrain writeback_control to the minimal scope.
The basic idea would be to pass the struct wb_writeback_work all the way down to writeback_sb_inodes, and initialize the struct writeback_control there.

OK.

Quoted 1 line

This requires adding a few more fields like more_io and

We can in fact just kill more_io and test list_empty(b_more_io) instead.

Quoted 2 lines

older_than_this to struct wb_writeback_work, and redoing a lot of the tracing,

Yeah, older_than_this must still be set in wb_writeback(). The two trace points could be removed for now.

Quoted 4 lines

but it immediately clean things up, e.g. suddently wbc.nr_to_write vs work->nr_pages starts to make sense, and instead of saving and restoring pages_skipped in writeback_sb_inodes it can always start with a clean zero value.

That's right.

Thanks,
Fengguang

Wu Fengguang

author

2011-04-20 08:03 UTC #9 lore raw

PATCH 3/6 writeback: try more writeback as long as something was written

writeback_inodes_wb()/__writeback_inodes_sb() are not aggressive in that they only populate possibly a subset of eligible inodes into b_io at entrance time. When the queued set of inodes are all synced, they just return, possibly with all queued inode pages written but still wbc.nr_to_write > 0.

For kupdate and background writeback, there may be more eligible inodes sitting in b_dirty when the current set of b_io inodes are completed. So it is necessary to try another round of writeback as long as we made some progress in this round. When there are no more eligible inodes, no more inodes will be enqueued in queue_io(), hence nothing could/will be synced and we may safely bail.

For example, imagine 100 inodes

        i0, i1, i2, ..., i90, i91, i99

At queue_io() time, i90-i99 happen to be expired and moved to s_io for IO. When finished successfully, if their total size is less than MAX_WRITEBACK_PAGES, nr_to_write will be > 0. Then wb_writeback() will quit the background work (w/o this patch) while it's still over background threshold. This will be a fairly normal/frequent case I guess.

Jan raised the concern

I'm just afraid that in some pathological cases this could result in bad writeback pattern - like if there is some process which manages to dirty just a few pages while we are doing writeout, this looping could result in writing just a few pages in each round which is bad for fragmentation etc.

However it requires really strong timing to make that to (continuously) happen. In practice it's very hard to produce such a pattern even if there is such a possibility in theory. I actually tried to write 1 page per 1ms with this command

	write-and-fsync -n10000 -S 1000 -c 4096 /fs/test

and do sync(1) at the same time. The sync completes quickly on ext4, xfs, btrfs. The readers could try other write-and-sleep patterns and check if it can block sync for longer time.

CC Jan Kara <[email protected]>

---
 fs/fs-writeback.c |   16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

fs/fs-writeback.c +8 -8

@@ -730,23 +730,23 @@ static long wb_writeback(struct bdi_writ
 		wrote += write_chunk - wbc.nr_to_write;
 
 		/*
-		 * If we consumed everything, see if we have more
+		 * Did we write something? Try for more
+		 *
+		 * Dirty inodes are moved to b_io for writeback in batches.
+		 * The completion of the current batch does not necessarily
+		 * mean the overall work is done. So we keep looping as long
+		 * as made some progress on cleaning pages or inodes.
 		 */
-		if (wbc.nr_to_write <= 0)
+		if (wbc.nr_to_write < write_chunk)
 			continue;
 		if (wbc.inodes_cleaned)
 			continue;
 		/*
-		 * Didn't write everything and we don't have more IO, bail
+		 * No more inodes for IO, bail
 		 */
 		if (!wbc.more_io)
 			break;
 		/*
-		 * Did we write something? Try for more
-		 */
-		if (wbc.nr_to_write < write_chunk)
-			continue;
-		/*
 		 * Nothing written. Wait for some inode to
 		 * become available for writeback. Otherwise
 		 * we'll just busyloop.

Wu Fengguang

author

2011-04-20 08:03 UTC #10 lore raw

PATCH 4/6 writeback: the kupdate expire timestamp should be a moving target

Dynamically compute the dirty expire timestamp at queue_io() time.

writeback_control.older_than_this used to be determined at entrance to the kupdate writeback work. This _static_ timestamp may go stale if the kupdate work runs on and on. The flusher may then stuck with some old busy inodes, never considering newly expired inodes thereafter.

This has two possible problems:

- It is unfair for a large dirty inode to delay (for a long time) the writeback of small dirty inodes.

- As time goes by, the large and busy dirty inode may contain only _freshly_ dirtied pages. Ignoring newly expired dirty inodes risks delaying the expired dirty pages to the end of LRU lists, triggering the evil pageout(). Nevertheless this patch merely addresses part of the problem.

v2: keep policy changes inside wb_writeback() and keep the wbc.older_than_this visibility as suggested by Dave.

CC Dave Chinner <[email protected]>

Acked-by Jan Kara <[email protected]>

Signed-off-by Itaru Kitayama <[email protected]>

---
 fs/fs-writeback.c |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

fs/fs-writeback.c +6 -5

@@ -661,11 +661,6 @@ static long wb_writeback(struct bdi_writ
 	long write_chunk;
 	struct inode *inode;
 
-	if (wbc.for_kupdate) {
-		wbc.older_than_this = &oldest_jif;
-		oldest_jif = jiffies -
-				msecs_to_jiffies(dirty_expire_interval * 10);
-	}
 	if (!wbc.range_cyclic) {
 		wbc.range_start = 0;
 		wbc.range_end = LLONG_MAX;

@@ -714,6 +709,12 @@ static long wb_writeback(struct bdi_writ
 		if (work->for_background && !over_bground_thresh())
 			break;
 
+		if (work->for_kupdate) {
+			oldest_jif = jiffies -
+				msecs_to_jiffies(dirty_expire_interval * 10);
+			wbc.older_than_this = &oldest_jif;
+		}
+
 		wbc.more_io = 0;
 		wbc.nr_to_write = write_chunk;
 		wbc.pages_skipped = 0;

Wu Fengguang

author

2011-04-20 08:03 UTC #11 lore raw

PATCH 5/6 writeback: sync expired inodes first in background writeback

A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.

At each queue_io() time, first try enqueuing only newly expired inodes. If there are zero expired inodes to work with, then relax the rule and enqueue all dirty inodes.

This will help reduce the number of dirty pages encountered by page reclaim, eg. the pageout() calls. Normally older inodes contain older dirty pages, which are more close to the end of the LRU lists. So syncing older inodes first helps reducing the dirty pages reached by the page reclaim code.

More background: as Mel put it, "it makes sense to write old pages first to reduce the chances page reclaim is initiating IO."

Rik also presented the situation with a graph:

LRU head [*] dirty page
[ * * * * * * * * * * *]

Ideally, most dirty pages should lie close to the LRU tail instead of LRU head. That requires the flusher thread to sync old/expired inodes first (as there are obvious correlations between inode age and page age), and to give fair opportunities to newly expired inodes rather than sticking with some large eldest inodes (as larger inodes have weaker correlations in the inode<=>page ages).

This patch helps the flusher to meet both the above requirements.

Side effects: it might reduce the batch size and hence reduce inode_wb_list_lock hold time, but in turn make the cluster-by-partition logic in the same function less effective on reducing disk seeks.

v2: keep policy changes inside wb_writeback() and keep the wbc.older_than_this visibility as suggested by Dave.

CC Dave Chinner <[email protected]>

CC Jan Kara <[email protected]>

CC Rik van Riel <[email protected]>

In reply to Andrew Morton

---
 fs/fs-writeback.c |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

fs/fs-writeback.c +15 -1

@@ -709,7 +709,7 @@ static long wb_writeback(struct bdi_writ
 		if (work->for_background && !over_bground_thresh())
 			break;
 
-		if (work->for_kupdate) {
+		if (work->for_kupdate || work->for_background) {
 			oldest_jif = jiffies -
 				msecs_to_jiffies(dirty_expire_interval * 10);
 			wbc.older_than_this = &oldest_jif;

@@ -720,6 +720,7 @@ static long wb_writeback(struct bdi_writ
 		wbc.pages_skipped = 0;
 		wbc.inodes_cleaned = 0;
 
+retry:
 		trace_wbc_writeback_start(&wbc, wb->bdi);
 		if (work->sb)
 			__writeback_inodes_sb(work->sb, wb, &wbc);

@@ -743,6 +744,19 @@ static long wb_writeback(struct bdi_writ
 		if (wbc.inodes_cleaned)
 			continue;
 		/*
+		 * background writeback will start with expired inodes, and
+		 * if none is found, fallback to all inodes. This order helps
+		 * reduce the number of dirty pages reaching the end of LRU
+		 * lists and cause trouble to the page reclaim.
+		 */
+		if (work->for_background &&
+		    wbc.older_than_this &&
+		    list_empty(&wb->b_io) &&
+		    list_empty(&wb->b_more_io)) {
+			wbc.older_than_this = NULL;
+			goto retry;
+		}
+		/*
 		 * No more inodes for IO, bail
 		 */
 		if (!wbc.more_io)

Andrew Morton

maintainer

2011-04-20 23:40 UTC #12 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 28 lines · On Wed, 20 Apr 2011 16:03:41 +0800 Wu Fengguang <[email protected]> wrote:

A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.
At each queue_io() time, first try enqueuing only newly expired inodes. If there are zero expired inodes to work with, then relax the rule and enqueue all dirty inodes.
This will help reduce the number of dirty pages encountered by page reclaim, eg. the pageout() calls. Normally older inodes contain older dirty pages, which are more close to the end of the LRU lists. So syncing older inodes first helps reducing the dirty pages reached by the page reclaim code.
More background: as Mel put it, "it makes sense to write old pages first to reduce the chances page reclaim is initiating IO."
Rik also presented the situation with a graph:
LRU head [*] dirty page
[ * * * * * * * * * * *]
Ideally, most dirty pages should lie close to the LRU tail instead of LRU head. That requires the flusher thread to sync old/expired inodes first (as there are obvious correlations between inode age and page age), and to give fair opportunities to newly expired inodes rather than sticking with some large eldest inodes (as larger inodes have weaker correlations in the inode<=>page ages).

Quoted 5 lines

This patch helps the flusher to meet both the above requirements.
Side effects: it might reduce the batch size and hence reduce inode_wb_list_lock hold time, but in turn make the cluster-by-partition logic in the same function less effective on reducing disk seeks.

One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).

And the background writeback function and fsync() and msync() and everything else shouldn't cause starvation of expired pages, either. I guess you could say that the expired dirty pages become the highest-priority writeback item.

Are you testing for this failure scenario? If so, can you briefly describe the testing?

It would be hlpeful if you could explain how the current code implements this requirement?

Wu Fengguang

author

2011-04-21 01:14 UTC #13 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 37 lines · Andrew Morton

Quoted 28 lines · On Wed, 20 Apr 2011 16:03:41 +0800 Wu Fengguang <[email protected]> wrote:
A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.
At each queue_io() time, first try enqueuing only newly expired inodes. If there are zero expired inodes to work with, then relax the rule and enqueue all dirty inodes.
This will help reduce the number of dirty pages encountered by page reclaim, eg. the pageout() calls. Normally older inodes contain older dirty pages, which are more close to the end of the LRU lists. So syncing older inodes first helps reducing the dirty pages reached by the page reclaim code.
More background: as Mel put it, "it makes sense to write old pages first to reduce the chances page reclaim is initiating IO."
Rik also presented the situation with a graph:
LRU head [*] dirty page
[ * * * * * * * * * * *]
Ideally, most dirty pages should lie close to the LRU tail instead of LRU head. That requires the flusher thread to sync old/expired inodes first (as there are obvious correlations between inode age and page age), and to give fair opportunities to newly expired inodes rather than sticking with some large eldest inodes (as larger inodes have weaker correlations in the inode<=>page ages).
Quoted 5 lines
This patch helps the flusher to meet both the above requirements.
Side effects: it might reduce the batch size and hence reduce inode_wb_list_lock hold time, but in turn make the cluster-by-partition logic in the same function less effective on reducing disk seeks.

Quoted 5 lines

One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).

Right. This patch will exclude unexpired inodes from background work, as long as there are expired ones to work on. Which hopefully will provide better data safety in normal cases. After all the flusher is based on inodes and this is the most sane order it can take.

Given that the unexpired inodes will eventually expire after 30s, it won't exclude any inode from entering b_io for long time.

When there is a continually dirtied inode, let's check its internal pages writeback order first, and then examine the inter-inode order.

a) it's a small file (less than the write chunk size)

No problem at all, it will be fairly redirty_tail()ed after all the freshly dirtied inodes, waiting for the next turn

b) it's a large file, random writes

No optimal solution at all, because it will contain both old and new pages all over the places.

c) it's a large file, sequentially written to

Old pages will be served first (if not, we'll cycle around sooner or later to behind the write stream), we will be rightfully concentrating on old pages

d) it's a large file, reversely written to

This is rare. We'll work through a series of segments, eg. pages 90000-100000, pages 70000-80000, pages 60000-70000. Within each segment we'll work on youngest pages first..

Except for (d) there is no way we may wrongly keep concentrating on the newly-dirtied pages _inside_ one file.

As for _inter_ inode fairness, the last patch describes the scheme:

: A b_io refill will setup a _fixed_ work set with all currently eligible : inodes and start a new round of walking through b_io. The "fixed" work : set means no new inodes will be added to the work set during the walk. : Only when a complete walk over b_io is done, new inodes that are eligible : at the time will be enqueued and the walk will be started over. :
: This procedure provides fairness among the inodes because it guarantees : that each inode will be synced once and only once in each round. So all : inodes will be free from starvation.

In long turn, each inode will be honored a chance to write write_chunk pages _in fair turn_. However there is an implicit priority: the larger write_chunk, the more a large file can effectively write than small files.

Quoted 4 lines

And the background writeback function and fsync() and msync() and everything else shouldn't cause starvation of expired pages, either. I guess you could say that the expired dirty pages become the highest-priority writeback item.

fsync() and msync() is expected to finish in bounded time, since they work on a fixed set of PAGECACHE_TAG_TOWRITE pages. So won't be starved by them. If there are too many fsync()s, we have the more serious problem of the whole system coming to a halt.

Inside the background work, inter-inode fairness is guaranteed as described above.

Quoted 2 lines

Are you testing for this failure scenario? If so, can you briefly describe the testing?

Not yet.. But one possible scheme is to record the dirty time of each page in a debug kernel and expose them to user space. Then we can run any kind of workloads, and in the mean while run a background scanner to collect and report the distribution of dirty page ages.

Does it sound too heavy weight? Or we may start by reporting the dirty inode age first. To maintain a mapping->writeback_index_wrapped_when and a mapping->pages_dirtied_when to follow it (or just reuse/reset mapping->dirtied_when?). The former will be reset to jiffies on each full scan of the pages. range_whole=1 scan can maintain its start time in a local variable. Then we get an estimation "what's the max possible dirty page age this inode has?". There will sure be redirtied pages though..

Quoted 2 lines

It would be hlpeful if you could explain how the current code implements this requirement?

See above descriptions.

Thanks,
Fengguang

Wu Fengguang

author

2011-04-21 01:21 UTC #14 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 8 lines

Quoted 2 lines
Are you testing for this failure scenario? If so, can you briefly describe the testing?
Not yet.. But one possible scheme is to record the dirty time of each page in a debug kernel and expose them to user space. Then we can run any kind of workloads, and in the mean while run a background scanner to collect and report the distribution of dirty page ages.

Quoted 8 lines

Does it sound too heavy weight? Or we may start by reporting the dirty inode age first. To maintain a mapping->writeback_index_wrapped_when and a mapping->pages_dirtied_when to follow it (or just reuse/reset mapping->dirtied_when?). The former will be reset to jiffies on each full scan of the pages. range_whole=1 scan can maintain its start time in a local variable. Then we get an estimation "what's the max possible dirty page age this inode has?". There will sure be redirtied pages though..

Hmm the lighter scheme will fail the common "active sequential write to large file" case, because the full scan will never manage to come to an end..

Thanks,
Fengguang

Wu Fengguang

author

2011-04-24 03:15 UTC #15 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 5 lines

One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).

Sadly I do find the old pages that the flusher never get a chance to catch and write them out.

In the below case, if the task dirties pages fast enough at the end of file, writeback_index will never get a chance to wrap back. There may be various variations of this case.

file head
[ *** ==>***************]==>
old pages writeback_index fresh dirties

Ironically the current kernel relies on pageout() to catch these old pages, which is not only inefficient, but also not reliable. If a full LRU walk takes an hour, the old pages may stay dirtied for an hour.

We may have to do (conditional) tagged ->writepages to safeguard users from losing data he'd expect to be written hours ago.

Thanks,
Fengguang

Jan Kara

maintainer

2011-04-26 12:17 UTC #16 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 8 lines · Wu Fengguang

Quoted 5 lines
One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).
Sadly I do find the old pages that the flusher never get a chance to catch and write them out.

What kind of load do you use?

Quoted 4 lines

In the below case, if the task dirties pages fast enough at the end of file, writeback_index will never get a chance to wrap back. There may be various variations of this case.

Quoted 8 lines

file head
[ *** ==>***************]==>
old pages writeback_index fresh dirties
Ironically the current kernel relies on pageout() to catch these old pages, which is not only inefficient, but also not reliable. If a full LRU walk takes an hour, the old pages may stay dirtied for an hour.

Well, the kupdate behavior has always been just a best-effort thing. We
always tried to handle well common cases but didn't try to solve all of them. Unless we want to track dirty-age of every page (which we don't want because it's too expensive), there is really no way to make syncing of old pages 100% working for all the cases unless we do data-integrity type of writeback for the whole inode - but that could create new problems with stalling other files for too long I suspect.

Quoted 2 lines

We may have to do (conditional) tagged ->writepages to safeguard users from losing data he'd expect to be written hours ago.

Well, if the file is continuously written (and in your case it must be
even continuosly grown) I'd be content if we handle well the common case of linear append (that happens for log files etc.). If we can do well for more cases, even better but I'd be cautious not to disrupt some other more common cases.

Honza

Signature · 9 lines

Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wu Fengguang

author

2011-04-26 13:51 UTC #17 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 10 lines · Jan Kara

Quoted 8 lines · Wu Fengguang
Quoted 5 lines
One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).
Sadly I do find the old pages that the flusher never get a chance to catch and write them out.
What kind of load do you use?

Sorry I was just thinking about it and then got a _theoretic_ case.

Quoted 12 lines

Quoted 4 lines
In the below case, if the task dirties pages fast enough at the end of file, writeback_index will never get a chance to wrap back. There may be various variations of this case.
Quoted 8 lines
file head
[ *** ==>***************]==>
old pages writeback_index fresh dirties
Ironically the current kernel relies on pageout() to catch these old pages, which is not only inefficient, but also not reliable. If a full LRU walk takes an hour, the old pages may stay dirtied for an hour.

Quoted 7 lines

Well, the kupdate behavior has always been just a best-effort thing. We
always tried to handle well common cases but didn't try to solve all of them. Unless we want to track dirty-age of every page (which we don't want because it's too expensive), there is really no way to make syncing of old pages 100% working for all the cases unless we do data-integrity type of writeback for the whole inode - but that could create new problems with stalling other files for too long I suspect.

Yeah, it's a hard problem in general. The flusher works naturally in the coarse way..

Quoted 7 lines

Quoted 2 lines
We may have to do (conditional) tagged ->writepages to safeguard users from losing data he'd expect to be written hours ago.
Well, if the file is continuously written (and in your case it must be
even continuosly grown) I'd be content if we handle well the common case of linear append (that happens for log files etc.). If we can do well for more cases, even better but I'd be cautious not to disrupt some other more common cases.

I scratched a patch (totally untested) which will guarantee any kind of starvation inside an inode. Will this be too overweight?

Thanks,
Fengguang
---
Subject: writeback: livelock prevention inside actively dirtied files
Date: Tue Apr 26 21:35:47 CST 2011

- refresh dirtied_when on every full writeback_index cycle (pages may be skipped on SYNC_NONE, but as long as they are retried in next cycle..)

- do tagged sync when writeback_index not cycled for too long time (the arbitrarily 60s may lead to more page tagging overheads in "large dirty threshold but slow storage" system..)

---
 fs/fs-writeback.c       |    1 +
 include/linux/fs.h      |    1 +
 include/linux/pagemap.h |   16 ++++++++++++++++
 mm/page-writeback.c     |   24 ++++++++++++++++++------
 4 files changed, 36 insertions(+), 6 deletions(-)

fs/fs-writeback.c +1 -0

@@ -1110,6 +1110,7 @@ void __mark_inode_dirty(struct inode *in
 			spin_unlock(&inode->i_lock);
 			spin_lock(&bdi->wb.list_lock);
 			inode->dirtied_when = jiffies;
+			inode->i_mapping->writeback_cycle_time = jiffies;
 			list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
 			spin_unlock(&bdi->wb.list_lock);

include/linux/fs.h +1 -0

@@ -639,6 +639,7 @@ struct address_space {
 	unsigned int		truncate_count;	/* Cover race condition with truncate */
 	unsigned long		nrpages;	/* number of total pages */
 	pgoff_t			writeback_index;/* writeback starts here */
+	unsigned long		writeback_cycle_time;
 	const struct address_space_operations *a_ops;	/* methods */
 	unsigned long		flags;		/* error bits/gfp mask */
 	struct backing_dev_info *backing_dev_info; /* device readahead, etc */

mm/page-writeback.c +18 -6

@@ -835,6 +835,9 @@ void tag_pages_for_writeback(struct addr
 		cond_resched();
 		/* We check 'start' to handle wrapping when end == ~0UL */
 	} while (tagged >= WRITEBACK_TAG_BATCH && start);
+
+	mapping_set_tagged_sync(mapping);
+	mapping->writeback_cycle_time = jiffies;
 }
 EXPORT_SYMBOL(tag_pages_for_writeback);

@@ -872,7 +875,7 @@ int write_cache_pages(struct address_spa
 	pgoff_t end;		/* Inclusive */
 	pgoff_t done_index;
 	int range_whole = 0;
-	int tag;
+	int tag = PAGECACHE_TAG_DIRTY;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {

@@ -884,13 +887,19 @@ int write_cache_pages(struct address_spa
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
 	}
-	if (wbc->sync_mode == WB_SYNC_ALL)
-		tag = PAGECACHE_TAG_TOWRITE;
-	else
-		tag = PAGECACHE_TAG_DIRTY;
+	if (!index)
+		mapping->writeback_cycle_time = jiffies;
 
-	if (wbc->sync_mode == WB_SYNC_ALL)
+	if (wbc->sync_mode == WB_SYNC_ALL ||
+	    (!mapping_tagged_sync(mapping) &&
+	     jiffies - mapping->host->dirtied_when > 60 * HZ)) {
 		tag_pages_for_writeback(mapping, index, end);
+		tag = PAGECACHE_TAG_TOWRITE;
+	}
+
+	if (mapping_tagged_sync(mapping))
+		tag = PAGECACHE_TAG_TOWRITE;
+
 	done_index = index;
 	while (!done && (index <= end)) {
 		int i;

@@ -899,6 +908,9 @@ int write_cache_pages(struct address_spa
 			      min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
 		if (nr_pages == 0) {
 			done_index = 0;
+			mapping->dirtied_when = mapping->writeback_cycle_time;
+			if (tag == PAGECACHE_TAG_TOWRITE)
+				mapping_clear_tagged_sync(mapping);
 			break;
 		}

include/linux/pagemap.h +16 -0

@@ -24,6 +24,7 @@ enum mapping_flags {
 	AS_ENOSPC	= __GFP_BITS_SHIFT + 1,	/* ENOSPC on async write */
 	AS_MM_ALL_LOCKS	= __GFP_BITS_SHIFT + 2,	/* under mm_take_all_locks() */
 	AS_UNEVICTABLE	= __GFP_BITS_SHIFT + 3,	/* e.g., ramdisk, SHM_LOCK */
+	AS_TAGGED_SYNC	= __GFP_BITS_SHIFT + 4,	/* sync only tagged pages */
 };
 
 static inline void mapping_set_error(struct address_space *mapping, int error)

@@ -53,6 +54,21 @@ static inline int mapping_unevictable(st
 	return !!mapping;
 }
 
+static inline void mapping_set_tagged_sync(struct address_space *mapping)
+{
+	set_bit(AS_TAGGED_SYNC, &mapping->flags);
+}
+
+static inline void mapping_clear_tagged_sync(struct address_space *mapping)
+{
+	clear_bit(AS_TAGGED_SYNC, &mapping->flags);
+}
+
+static inline int mapping_tagged_sync(struct address_space *mapping)
+{
+	return test_bit(AS_TAGGED_SYNC, &mapping->flags);
+}
+
 static inline gfp_t mapping_gfp_mask(struct address_space * mapping)
 {
 	return (__force gfp_t)mapping->flags & __GFP_BITS_MASK;

Wu Fengguang

author

2011-04-26 13:59 UTC #18 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 94 lines · Wu Fengguang

Quoted 10 lines · Jan Kara
Quoted 8 lines · Wu Fengguang
Quoted 5 lines
One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).
Sadly I do find the old pages that the flusher never get a chance to catch and write them out.
What kind of load do you use?
Sorry I was just thinking about it and then got a _theoretic_ case.
Quoted 12 lines
Quoted 4 lines
In the below case, if the task dirties pages fast enough at the end of file, writeback_index will never get a chance to wrap back. There may be various variations of this case.
Quoted 8 lines
file head
[ *** ==>***************]==>
old pages writeback_index fresh dirties
Ironically the current kernel relies on pageout() to catch these old pages, which is not only inefficient, but also not reliable. If a full LRU walk takes an hour, the old pages may stay dirtied for an hour.
Quoted 7 lines
Well, the kupdate behavior has always been just a best-effort thing. We
always tried to handle well common cases but didn't try to solve all of them. Unless we want to track dirty-age of every page (which we don't want because it's too expensive), there is really no way to make syncing of old pages 100% working for all the cases unless we do data-integrity type of writeback for the whole inode - but that could create new problems with stalling other files for too long I suspect.
Yeah, it's a hard problem in general. The flusher works naturally in the coarse way..
Quoted 7 lines
Quoted 2 lines
We may have to do (conditional) tagged ->writepages to safeguard users from losing data he'd expect to be written hours ago.
Well, if the file is continuously written (and in your case it must be
even continuosly grown) I'd be content if we handle well the common case of linear append (that happens for log files etc.). If we can do well for more cases, even better but I'd be cautious not to disrupt some other more common cases.
I scratched a patch (totally untested) which will guarantee any kind of starvation inside an inode. Will this be too overweight?
Thanks,
Fengguang
---
Subject: writeback: livelock prevention inside actively dirtied files
Date: Tue Apr 26 21:35:47 CST 2011
- refresh dirtied_when on every full writeback_index cycle (pages may be skipped on SYNC_NONE, but as long as they are retried in next cycle..)
- do tagged sync when writeback_index not cycled for too long time (the arbitrarily 60s may lead to more page tagging overheads in "large dirty threshold but slow storage" system..)
Signed-off-by Wu Fengguang <[email protected]>
---
 fs/fs-writeback.c       |    1 +
 include/linux/fs.h      |    1 +
 include/linux/pagemap.h |   16 ++++++++++++++++
 mm/page-writeback.c     |   24 ++++++++++++++++++------
 4 files changed, 36 insertions(+), 6 deletions(-)
fs/fs-writeback.c +1 -0
@@ -1110,6 +1110,7 @@ void __mark_inode_dirty(struct inode *in
 			spin_unlock(&inode->i_lock);
 			spin_lock(&bdi->wb.list_lock);
 			inode->dirtied_when = jiffies;
+			inode->i_mapping->writeback_cycle_time = jiffies;
 			list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
 			spin_unlock(&bdi->wb.list_lock);
 
include/linux/fs.h +1 -0
@@ -639,6 +639,7 @@ struct address_space {
 	unsigned int		truncate_count;	/* Cover race condition with truncate */
 	unsigned long		nrpages;	/* number of total pages */
 	pgoff_t			writeback_index;/* writeback starts here */
+	unsigned long		writeback_cycle_time;
 	const struct address_space_operations *a_ops;	/* methods */
 	unsigned long		flags;		/* error bits/gfp mask */
 	struct backing_dev_info *backing_dev_info; /* device readahead, etc */

mm/page-writeback.c void tag_pages_for_writeback(struct addr · line 841

Quoted 7 lines

mm/page-writeback.c +3 -0

@@ -835,6 +835,9 @@ void tag_pages_for_writeback(struct addr
 		cond_resched();
 		/* We check 'start' to handle wrapping when end == ~0UL */
 	} while (tagged >= WRITEBACK_TAG_BATCH && start);
+
+	mapping_set_tagged_sync(mapping);
+	mapping->writeback_cycle_time = jiffies;
 }

Sorry the above "mapping->writeback_cycle_time = jiffies" is not correct and shall be removed. It should be updated iff (index == 0).

Thanks,
Fengguang

Quoted 12 lines

mm/page-writeback.c +1 -1

 EXPORT_SYMBOL(tag_pages_for_writeback);

@@ -872,7 +875,7 @@ int write_cache_pages(struct address_spa
 	pgoff_t end;		/* Inclusive */
 	pgoff_t done_index;
 	int range_whole = 0;
-	int tag;
+	int tag = PAGECACHE_TAG_DIRTY;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {

mm/page-writeback.c int write_cache_pages(struct address_spa · line 891

Quoted 9 lines

mm/page-writeback.c +2 -4

@@ -884,13 +887,19 @@ int write_cache_pages(struct address_spa
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
 	}
-	if (wbc->sync_mode == WB_SYNC_ALL)
-		tag = PAGECACHE_TAG_TOWRITE;
-	else
-		tag = PAGECACHE_TAG_DIRTY;
+	if (!index)
+		mapping->writeback_cycle_time = jiffies;

Wu Fengguang

author

2011-04-26 14:05 UTC #19 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 17 lines

I scratched a patch (totally untested) which will guarantee any kind of starvation inside an inode. Will this be too overweight?
Thanks,
Fengguang
---
Subject: writeback: livelock prevention inside actively dirtied files
Date: Tue Apr 26 21:35:47 CST 2011
- refresh dirtied_when on every full writeback_index cycle (pages may be skipped on SYNC_NONE, but as long as they are retried in next cycle..)
- do tagged sync when writeback_index not cycled for too long time (the arbitrarily 60s may lead to more page tagging overheads in "large dirty threshold but slow storage" system..)

Quoted 7 lines

---
 fs/fs-writeback.c       |    1 +
 include/linux/fs.h      |    1 +
 include/linux/pagemap.h |   16 ++++++++++++++++
 mm/page-writeback.c     |   24 ++++++++++++++++++------
 4 files changed, 36 insertions(+), 6 deletions(-)

FYI to make it easier, the above patch is actually based upon this one.
---
Subject: writeback: quit on wrap for .range_cyclic
Date: Mon Apr 25 14:58:19 CST 2011

Convert wbc.range_cyclic to new behavior: when past EOF, abort the writeback of the current inode, which will instruct writeback_single_inode() to redirty_tail() it.

This is the right behavior for
- sync writeback (is already so with range_whole)
we have scanned the inode address space, and don't care any more newly dirtied pages. So shall update its i_dirtied_when and exclude it from the todo list.
- periodic writeback
any more newly dirtied pages should be associated with a new expire time. This also prevents pointless IO for busy overwriters.
- background writeback
irrelevant because it generally don't care the dirty timestamp.

That should get rid of one inefficient IO pattern of .range_cyclic when writeback_index wraps, in which the submitted pages may be consisted of two distant ranges: submit [10000-10100], (wrap), submit [0-100].

---
 drivers/staging/pohmelfs/inode.c |   25 ++++++++-----------------
 fs/afs/write.c                   |    7 +++----
 fs/btrfs/extent_io.c             |   21 ++++++---------------
 fs/cifs/file.c                   |   15 +++------------
 fs/ext4/inode.c                  |   18 ++++--------------
 fs/gfs2/aops.c                   |   16 ++--------------
 mm/page-writeback.c              |   27 +++++----------------------
 7 files changed, 31 insertions(+), 98 deletions(-)

mm/page-writeback.c +5 ~~-22~~

@@ -868,35 +868,27 @@ int write_cache_pages(struct address_spa
 	int done = 0;
 	struct pagevec pvec;
 	int nr_pages;
-	pgoff_t uninitialized_var(writeback_index);
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
 	pgoff_t done_index;
-	int cycled;
 	int range_whole = 0;
 	int tag;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
-		writeback_index = mapping->writeback_index; /* prev offset */
-		index = writeback_index;
-		if (index == 0)
-			cycled = 1;
-		else
-			cycled = 0;
+		index = mapping->writeback_index; /* prev offset */
 		end = -1;
 	} else {
 		index = wbc->range_start >> PAGE_CACHE_SHIFT;
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		cycled = 1; /* ignore range_cyclic tests */
 	}
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		tag = PAGECACHE_TAG_TOWRITE;
 	else
 		tag = PAGECACHE_TAG_DIRTY;
-retry:
+
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		tag_pages_for_writeback(mapping, index, end);
 	done_index = index;

@@ -905,8 +897,10 @@ retry:
 
 		nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
 			      min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
-		if (nr_pages == 0)
+		if (nr_pages == 0) {
+			done_index = 0;
 			break;
+		}
 
 		for (i = 0; i < nr_pages; i++) {
 			struct page *page = pvec.pages[i];

@@ -998,17 +992,6 @@ continue_unlock:
 		pagevec_release(&pvec);
 		cond_resched();
 	}
-	if (!cycled && !done) {
-		/*
-		 * range_cyclic:
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		cycled = 1;
-		index = 0;
-		end = writeback_index - 1;
-		goto retry;
-	}
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = done_index;

drivers/staging/pohmelfs/inode.c +8 ~~-17~~

@@ -148,7 +148,6 @@ static int pohmelfs_writepages(struct ad
 	int nr_pages;
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
-	int scanned = 0;
 	int range_whole = 0;
 
 	if (wbc->range_cyclic) {

@@ -159,17 +158,18 @@ static int pohmelfs_writepages(struct ad
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && (index <= end)) {
 		unsigned int i = min(end - index, (pgoff_t)psb->trans_max_pages);
 		int path_len;
 		struct netfs_trans *trans;
 
 		err = pohmelfs_inode_has_dirty_pages(mapping, index);
-		if (!err)
+		if (!err) {
+			index = 0;
 			break;
+		}
 
 		err = pohmelfs_path_length(pi);
 		if (err < 0)

@@ -196,15 +196,16 @@ retry:
 		dprintk("%s: t: %p, nr_pages: %u, end: %lu, index: %lu, max: %u.\n",
 				__func__, trans, nr_pages, end, index, trans->page_num);
 
-		if (!nr_pages)
+		if (!nr_pages) {
+			index = 0;
 			goto err_out_reset;
+		}
 
 		err = pohmelfs_write_inode_create(inode, trans);
 		if (err)
 			goto err_out_reset;
 
 		err = 0;
-		scanned = 1;
 
 		for (i = 0; i < trans->page_num; i++) {
 			struct page *page = trans->pages[i];

@@ -214,7 +215,7 @@ retry:
 			if (unlikely(page->mapping != mapping))
 				goto out_continue;
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				goto out_continue;
 			}

@@ -262,16 +263,6 @@ err_out_reset:
 		break;
 	}
 
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
-		index = 0;
-		goto retry;
-	}
-
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;

fs/btrfs/extent_io.c +6 ~~-15~~

@@ -2464,10 +2464,9 @@ static int extent_write_cache_pages(stru
 	int done = 0;
 	int nr_to_write_done = 0;
 	struct pagevec pvec;
-	int nr_pages;
+	int nr_pages = 1;
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
-	int scanned = 0;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {

@@ -2476,16 +2475,14 @@ static int extent_write_cache_pages(stru
 	} else {
 		index = wbc->range_start >> PAGE_CACHE_SHIFT;
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && !nr_to_write_done && (index <= end) &&
 	       (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 			      PAGECACHE_TAG_DIRTY, min(end - index,
 				  (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
 		unsigned i;
 
-		scanned = 1;
 		for (i = 0; i < nr_pages; i++) {
 			struct page *page = pvec.pages[i];

@@ -2506,7 +2503,7 @@ retry:
 				continue;
 			}
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				unlock_page(page);
 				continue;

@@ -2543,15 +2540,9 @@ retry:
 		pagevec_release(&pvec);
 		cond_resched();
 	}
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
-		index = 0;
-		goto retry;
-	}
+	if (!nr_pages)
+		mapping->writeback_index = 0;
+
 	return ret;
 }

fs/cifs/file.c +3 ~~-12~~

@@ -1200,7 +1200,6 @@ static int cifs_writepages(struct addres
 	struct page *page;
 	struct pagevec pvec;
 	int rc = 0;
-	int scanned = 0;
 	int xid;
 
 	cifs_sb = CIFS_SB(mapping->host->i_sb);

@@ -1241,9 +1240,8 @@ static int cifs_writepages(struct addres
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && (index <= end) &&
 	       (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 			PAGECACHE_TAG_DIRTY,

@@ -1276,7 +1274,7 @@ retry:
 				break;
 			}
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				unlock_page(page);
 				break;

@@ -1403,15 +1401,8 @@ retry_write:
 
 		pagevec_release(&pvec);
 	}
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
+	if (!nr_pages)
 		index = 0;
-		goto retry;
-	}
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;

fs/ext4/inode.c +4 ~~-14~~

@@ -2893,7 +2893,7 @@ static int ext4_da_writepages(struct add
 	struct inode *inode = mapping->host;
 	int pages_written = 0;
 	unsigned int max_pages;
-	int range_cyclic, cycled = 1, io_done = 0;
+	int range_cyclic, io_done = 0;
 	int needed_blocks, ret = 0;
 	long desired_nr_to_write, nr_to_writebump = 0;
 	loff_t range_start = wbc->range_start;

@@ -2930,8 +2930,6 @@ static int ext4_da_writepages(struct add
 	range_cyclic = wbc->range_cyclic;
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index;
-		if (index)
-			cycled = 0;
 		wbc->range_start = index << PAGE_CACHE_SHIFT;
 		wbc->range_end  = LLONG_MAX;
 		wbc->range_cyclic = 0;

@@ -2974,7 +2972,6 @@ static int ext4_da_writepages(struct add
 		wbc->nr_to_write = desired_nr_to_write;
 	}
 
-retry:
 	if (wbc->sync_mode == WB_SYNC_ALL)
 		tag_pages_for_writeback(mapping, index, end);

@@ -3034,20 +3031,13 @@ retry:
 			pages_written += mpd.pages_written;
 			ret = 0;
 			io_done = 1;
-		} else if (wbc->nr_to_write)
+		} else if (wbc->nr_to_write > 0) {
 			/*
 			 * There is no more writeout needed
-			 * or we requested for a noblocking writeout
-			 * and we found the device congested
 			 */
+			index = 0;
 			break;
-	}
-	if (!io_done && !cycled) {
-		cycled = 1;
-		index = 0;
-		wbc->range_start = index << PAGE_CACHE_SHIFT;
-		wbc->range_end  = mapping->writeback_index - 1;
-		goto retry;
+		}
 	}
 
 	/* Update index */

fs/gfs2/aops.c +2 ~~-14~~

@@ -283,7 +283,7 @@ static int gfs2_write_jdata_pagevec(stru
 			continue;
 		}
 
-		if (!wbc->range_cyclic && page->index > end) {
+		if (page->index > end) {
 			ret = 1;
 			unlock_page(page);
 			continue;

@@ -335,7 +335,6 @@ static int gfs2_write_cache_jdata(struct
 	int nr_pages;
 	pgoff_t index;
 	pgoff_t end;
-	int scanned = 0;
 	int range_whole = 0;
 
 	pagevec_init(&pvec, 0);

@@ -347,15 +346,12 @@ static int gfs2_write_cache_jdata(struct
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
 
-retry:
 	 while (!done && (index <= end) &&
 		(nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 					       PAGECACHE_TAG_DIRTY,
 					       min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
-		scanned = 1;
 		ret = gfs2_write_jdata_pagevec(mapping, wbc, &pvec, nr_pages, end);
 		if (ret)
 			done = 1;

@@ -366,16 +362,8 @@ retry:
 		cond_resched();
 	}
 
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
+	if (!nr_pages)
 		index = 0;
-		goto retry;
-	}
-
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;
 	return ret;

fs/afs/write.c +3 -4

@@ -476,8 +476,10 @@ static int afs_writepages_region(struct 
 	do {
 		n = find_get_pages_tag(mapping, &index, PAGECACHE_TAG_DIRTY,
 				       1, &page);
-		if (!n)
+		if (!n) {
+			index = 0;
 			break;
+		}
 
 		_debug("wback %lx", page->index);

@@ -549,9 +551,6 @@ int afs_writepages(struct address_space 
 		start = mapping->writeback_index;
 		end = -1;
 		ret = afs_writepages_region(mapping, wbc, start, end, &next);
-		if (start > 0 && wbc->nr_to_write > 0 && ret == 0)
-			ret = afs_writepages_region(mapping, wbc, 0, start,
-						    &next);
 		mapping->writeback_index = next;
 	} else if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX) {
 		end = (pgoff_t)(LLONG_MAX >> PAGE_CACHE_SHIFT);

Wu Fengguang

author

2011-04-27 11:15 UTC #20 lore raw

Re: [PATCH 5/6] writeback: sync expired inodes first in background writeback

Quoted 39 lines · Wu Fengguang

Quoted 10 lines · Jan Kara
Quoted 8 lines · Wu Fengguang
Quoted 5 lines
One of the many requirements for writeback is that if userspace is continually dirtying pages in a particular file, that shouldn't cause the kupdate function to concentrate on that file's newly-dirtied pages, neglecting pages from other files which were less-recently dirtied. (and dirty nodes, etc).
Sadly I do find the old pages that the flusher never get a chance to catch and write them out.
What kind of load do you use?
Sorry I was just thinking about it and then got a _theoretic_ case.
Quoted 12 lines
Quoted 4 lines
In the below case, if the task dirties pages fast enough at the end of file, writeback_index will never get a chance to wrap back. There may be various variations of this case.
Quoted 8 lines
file head
[ *** ==>***************]==>
old pages writeback_index fresh dirties
Ironically the current kernel relies on pageout() to catch these old pages, which is not only inefficient, but also not reliable. If a full LRU walk takes an hour, the old pages may stay dirtied for an hour.
Quoted 7 lines
Well, the kupdate behavior has always been just a best-effort thing. We
always tried to handle well common cases but didn't try to solve all of them. Unless we want to track dirty-age of every page (which we don't want because it's too expensive), there is really no way to make syncing of old pages 100% working for all the cases unless we do data-integrity type of writeback for the whole inode - but that could create new problems with stalling other files for too long I suspect.
Yeah, it's a hard problem in general. The flusher works naturally in the coarse way..
Quoted 2 lines
Quoted 2 lines
We may have to do (conditional) tagged ->writepages to safeguard users from losing data he'd expect to be written hours ago.

Quoted 8 lines

Quoted 5 lines
Well, if the file is continuously written (and in your case it must be
even continuosly grown) I'd be content if we handle well the common case of linear append (that happens for log files etc.). If we can do well for more cases, even better but I'd be cautious not to disrupt some other more common cases.
I scratched a patch (totally untested) which will guarantee any kind of starvation inside an inode. Will this be too overweight?

Sorry please ignore this broken patch..

I cannot find the right time to call mapping_clear_tagged_sync() at all, given that several entities may be writing the same inode. Even if we make it a reference count, the background work may still abort at any time.

Thanks,
Fengguang

Quoted 118 lines

---
Subject: writeback: livelock prevention inside actively dirtied files
Date: Tue Apr 26 21:35:47 CST 2011

- refresh dirtied_when on every full writeback_index cycle (pages may be skipped on SYNC_NONE, but as long as they are retried in next cycle..)

- do tagged sync when writeback_index not cycled for too long time (the arbitrarily 60s may lead to more page tagging overheads in "large dirty threshold but slow storage" system..)

---
 fs/fs-writeback.c       |    1 +
 include/linux/fs.h      |    1 +
 include/linux/pagemap.h |   16 ++++++++++++++++
 mm/page-writeback.c     |   24 ++++++++++++++++++------
 4 files changed, 36 insertions(+), 6 deletions(-)

fs/fs-writeback.c +1 -0

@@ -1110,6 +1110,7 @@ void __mark_inode_dirty(struct inode *in
 			spin_unlock(&inode->i_lock);
 			spin_lock(&bdi->wb.list_lock);
 			inode->dirtied_when = jiffies;
+			inode->i_mapping->writeback_cycle_time = jiffies;
 			list_move(&inode->i_wb_list, &bdi->wb.b_dirty);
 			spin_unlock(&bdi->wb.list_lock);

include/linux/fs.h +1 -0

@@ -639,6 +639,7 @@ struct address_space {
 	unsigned int		truncate_count;	/* Cover race condition with truncate */
 	unsigned long		nrpages;	/* number of total pages */
 	pgoff_t			writeback_index;/* writeback starts here */
+	unsigned long		writeback_cycle_time;
 	const struct address_space_operations *a_ops;	/* methods */
 	unsigned long		flags;		/* error bits/gfp mask */
 	struct backing_dev_info *backing_dev_info; /* device readahead, etc */

mm/page-writeback.c +18 -6

@@ -835,6 +835,9 @@ void tag_pages_for_writeback(struct addr
 		cond_resched();
 		/* We check 'start' to handle wrapping when end == ~0UL */
 	} while (tagged >= WRITEBACK_TAG_BATCH && start);
+
+	mapping_set_tagged_sync(mapping);
+	mapping->writeback_cycle_time = jiffies;
 }
 EXPORT_SYMBOL(tag_pages_for_writeback);

@@ -872,7 +875,7 @@ int write_cache_pages(struct address_spa
 	pgoff_t end;		/* Inclusive */
 	pgoff_t done_index;
 	int range_whole = 0;
-	int tag;
+	int tag = PAGECACHE_TAG_DIRTY;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {

@@ -884,13 +887,19 @@ int write_cache_pages(struct address_spa
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
 	}
-	if (wbc->sync_mode == WB_SYNC_ALL)
-		tag = PAGECACHE_TAG_TOWRITE;
-	else
-		tag = PAGECACHE_TAG_DIRTY;
+	if (!index)
+		mapping->writeback_cycle_time = jiffies;
 
-	if (wbc->sync_mode == WB_SYNC_ALL)
+	if (wbc->sync_mode == WB_SYNC_ALL ||
+	    (!mapping_tagged_sync(mapping) &&
+	     jiffies - mapping->host->dirtied_when > 60 * HZ)) {
 		tag_pages_for_writeback(mapping, index, end);
+		tag = PAGECACHE_TAG_TOWRITE;
+	}
+
+	if (mapping_tagged_sync(mapping))
+		tag = PAGECACHE_TAG_TOWRITE;
+
 	done_index = index;
 	while (!done && (index <= end)) {
 		int i;

@@ -899,6 +908,9 @@ int write_cache_pages(struct address_spa
 			      min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
 		if (nr_pages == 0) {
 			done_index = 0;
+			mapping->dirtied_when = mapping->writeback_cycle_time;
+			if (tag == PAGECACHE_TAG_TOWRITE)
+				mapping_clear_tagged_sync(mapping);
 			break;
 		}

include/linux/pagemap.h +9 -0

@@ -24,6 +24,7 @@ enum mapping_flags {
 	AS_ENOSPC	= __GFP_BITS_SHIFT + 1,	/* ENOSPC on async write */
 	AS_MM_ALL_LOCKS	= __GFP_BITS_SHIFT + 2,	/* under mm_take_all_locks() */
 	AS_UNEVICTABLE	= __GFP_BITS_SHIFT + 3,	/* e.g., ramdisk, SHM_LOCK */
+	AS_TAGGED_SYNC	= __GFP_BITS_SHIFT + 4,	/* sync only tagged pages */
 };
 
 static inline void mapping_set_error(struct address_space *mapping, int error)

@@ -53,6 +54,21 @@ static inline int mapping_unevictable(st
 	return !!mapping;
 }
 
+static inline void mapping_set_tagged_sync(struct address_space *mapping)
+{
+	set_bit(AS_TAGGED_SYNC, &mapping->flags);
+}
+
+static inline void mapping_clear_tagged_sync(struct address_space *mapping)
+{
+	clear_bit(AS_TAGGED_SYNC, &mapping->flags);

include/linux/pagemap.h static inline int mapping_unevictable(st · line 74

Quoted 10 lines

include/linux/pagemap.h +7 -0

+}
+
+static inline int mapping_tagged_sync(struct address_space *mapping)
+{
+	return test_bit(AS_TAGGED_SYNC, &mapping->flags);
+}
+
 static inline gfp_t mapping_gfp_mask(struct address_space * mapping)
 {
 	return (__force gfp_t)mapping->flags & __GFP_BITS_MASK;

Wu Fengguang

author

2011-04-20 08:03 UTC #21 lore raw

PATCH 6/6 writeback: refill b_io iff empty

There is no point to carry different refill policies between for_kupdate and other type of works. Use a consistent "refill b_io iff empty" policy which can guarantee fairness in an easy to understand way.

A b_io refill will setup a _fixed_ work set with all currently eligible inodes and start a new round of walk through b_io. The "fixed" work set means no new inodes will be added to the work set during the walk. Only when a complete walk over b_io is done, new inodes that are eligible at the time will be enqueued and the walk be started over.

This procedure provides fairness among the inodes because it guarantees each inode to be synced once and only once at each round. So all inodes will be free from starvations.

This change relies on wb_writeback() to keep retrying as long as we made some progress on cleaning some pages and/or inodes. Without that ability, the old logic on background works relies on aggressively queuing all eligible inodes into b_io at every time. But that's not a guarantee.

The below test script completes a slightly faster now on XFS:

             2.6.39-rc3	  2.6.39-rc3-dyn-expire+

------------------------------------------------
all elapsed 256.043 252.367
stddev 24.381 12.530

tar elapsed      30.097       28.808
dd  elapsed      13.214       11.782

	#!/bin/zsh

	cp /c/linux-2.6.38.3.tar.bz2 /dev/shm/

	umount /dev/sda7
	mkfs.xfs -f /dev/sda7
	mount /dev/sda7 /fs

	echo 3 > /proc/sys/vm/drop_caches

	tic=$(cat /proc/uptime|cut -d' ' -f2)

	cd /fs
	time tar jxf /dev/shm/linux-2.6.38.3.tar.bz2 &
	time dd if=/dev/zero of=/fs/zero bs=1M count=1000 &

	wait
	sync
	tac=$(cat /proc/uptime|cut -d' ' -f2)
	echo elapsed: $((tac - tic))

It maintains roughly the same small vs. large file writeout shares, and offers large files better chances to be written in nice 4M chunks.

Analyzes from Dave Chinner in great details:

Let's say we have lots of inodes with 100 dirty pages being created, and one large writeback going on. We expire 8 new inodes for every 1024 pages we write back.

With the old code, we do:

	b_more_io (large inode) -> b_io (1l)
	8 newly expired inodes -> b_io (1l, 8s)

	writeback  large inode 1024 pages -> b_more_io

	b_more_io (large inode) -> b_io (8s, 1l)
	8 newly expired inodes -> b_io (8s, 1l, 8s)

	writeback  8 small inodes 800 pages
		   1 large inode 224 pages -> b_more_io

	b_more_io (large inode) -> b_io (8s, 1l)
	8 newly expired inodes -> b_io (8s, 1l, 8s)
	.....

Your new code:

	b_more_io (large inode) -> b_io (1l)
	8 newly expired inodes -> b_io (1l, 8s)

	writeback  large inode 1024 pages -> b_more_io
	(b_io == 8s)
	writeback  8 small inodes 800 pages

	b_io empty: (1800 pages written)
		b_more_io (large inode) -> b_io (1l)
		14 newly expired inodes -> b_io (1l, 14s)

	writeback  large inode 1024 pages -> b_more_io
	(b_io == 14s)
	writeback  10 small inodes 1000 pages
		   1 small inode 24 pages -> b_more_io (1l, 1s(24))
	writeback  5 small inodes 500 pages
	b_io empty: (2548 pages written)
		b_more_io (large inode) -> b_io (1l, 1s(24))
		20 newly expired inodes -> b_io (1l, 1s(24), 20s)
	......

Rough progression of pages written at b_io refill:

Old code:

	total	large file	% of writeback
	1024	224		21.9% (fixed)

New code:

	total	large file	% of writeback
	1800	1024		~55%
	2550	1024		~40%
	3050	1024		~33%
	3500	1024		~29%
	3950	1024		~26%
	4250	1024		~24%
	4500	1024		~22.7%
	4700	1024		~21.7%
	4800	1024		~21.3%
	4800	1024		~21.3%
	(pretty much steady state from here)

Ok, so the steady state is reached with a similar percentage of writeback to the large file as the existing code. Ok, that's good, but providing some evidence that is doesn't change the shared of writeback to the large should be in the commit message ;)

The other advantage to this is that we always write 1024 page chunks to the large file, rather than smaller "whatever remains" chunks.

CC Jan Kara <[email protected]>

---
 fs/fs-writeback.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

fs/fs-writeback.c +3 -2

@@ -579,7 +579,8 @@ void writeback_inodes_wb(struct bdi_writ
 	if (!wbc->wb_start)
 		wbc->wb_start = jiffies; /* livelock avoidance */
 	spin_lock(&inode_wb_list_lock);
-	if (!wbc->for_kupdate || list_empty(&wb->b_io))
+
+	if (list_empty(&wb->b_io))
 		queue_io(wb, wbc);
 
 	while (!list_empty(&wb->b_io)) {

@@ -606,7 +607,7 @@ static void __writeback_inodes_sb(struct
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	spin_lock(&inode_wb_list_lock);
-	if (!wbc->for_kupdate || list_empty(&wb->b_io))
+	if (list_empty(&wb->b_io))
 		queue_io(wb, wbc);
 	writeback_sb_inodes(sb, wb, wbc, true);
 	spin_unlock(&inode_wb_list_lock);

Wu Fengguang

author

2011-05-04 07:39 UTC #22 lore raw

Re: [PATCH 6/6] writeback: refill b_io iff empty

To help understand the behavior change, I wrote the writeback_queue_io trace event, and found very different patterns between
- vanilla kernel
- this patchset plus the sync livelock fixes

Basically the vanilla kernel each time pulls a random number of inodes from b_dirty, while the patched kernel tends to pull a fixed number of inodes (enqueue=1031) from b_dirty. The new behavior is very interesting...

The attached test script runs 1 dd and 1 tar concurrently on XFS, whose output can be found at the start of the trace files. The elapsed time is 289s for vanilla kernel and 270s for patched kernel.

Thanks,
Fengguang

Subject:
Date: Sat Apr 23 12:27:27 CST 2011

---
 fs/fs-writeback.c                |   16 +++++++++++-----
 include/trace/events/writeback.h |   23 +++++++++++++++++++++++
 2 files changed, 34 insertions(+), 5 deletions(-)

fs/fs-writeback.c +11 -5

@@ -250,15 +250,16 @@ static bool inode_dirtied_after(struct i
 /*
  * Move expired dirty inodes from @delaying_queue to @dispatch_queue.
  */
-static void move_expired_inodes(struct list_head *delaying_queue,
-				struct list_head *dispatch_queue,
-				struct writeback_control *wbc)
+static int move_expired_inodes(struct list_head *delaying_queue,
+			       struct list_head *dispatch_queue,
+			       struct writeback_control *wbc)
 {
 	LIST_HEAD(tmp);
 	struct list_head *pos, *node;
 	struct super_block *sb = NULL;
 	struct inode *inode;
 	int do_sb_sort = 0;
+	int moved = 0;
 
 	while (!list_empty(delaying_queue)) {
 		inode = wb_inode(delaying_queue->prev);

@@ -269,12 +270,13 @@ static void move_expired_inodes(struct l
 			do_sb_sort = 1;
 		sb = inode->i_sb;
 		list_move(&inode->i_wb_list, &tmp);
+		moved++;
 	}
 
 	/* just one sb in list, splice to dispatch_queue and we're done */
 	if (!do_sb_sort) {
 		list_splice(&tmp, dispatch_queue);
-		return;
+		goto out;
 	}
 
 	/* Move inodes from one superblock together */

@@ -286,6 +288,8 @@ static void move_expired_inodes(struct l
 				list_move(&inode->i_wb_list, dispatch_queue);
 		}
 	}
+out:
+	return moved;
 }
 
 /*

@@ -301,9 +305,11 @@ static void move_expired_inodes(struct l
  */
 static void queue_io(struct bdi_writeback *wb, struct writeback_control *wbc)
 {
+	int moved;
 	assert_spin_locked(&inode_wb_list_lock);
 	list_splice_init(&wb->b_more_io, &wb->b_io);
-	move_expired_inodes(&wb->b_dirty, &wb->b_io, wbc);
+	moved = move_expired_inodes(&wb->b_dirty, &wb->b_io, wbc);
+	trace_writeback_queue_io(wb, wbc, moved);
 }
 
 static int write_inode(struct inode *inode, struct writeback_control *wbc)

include/trace/events/writeback.h +23 -0

@@ -152,6 +152,29 @@ DEFINE_WBC_EVENT(wbc_balance_dirty_writt
 DEFINE_WBC_EVENT(wbc_balance_dirty_wait);
 DEFINE_WBC_EVENT(wbc_writepage);
 
+TRACE_EVENT(writeback_queue_io,
+	TP_PROTO(struct bdi_writeback *wb,
+		 struct writeback_control *wbc,
+		 int moved),
+	TP_ARGS(wb, wbc, moved),
+	TP_STRUCT__entry(
+		__array(char,	name, 32)
+		__field(int,	older)
+		__field(int,	moved)
+	),
+	TP_fast_assign(
+		strncpy(__entry->name, dev_name(wb->bdi->dev), 32);
+		__entry->older	= wbc->older_than_this ?
+				  (jiffies - *wbc->older_than_this) * 1000 / HZ
+				  : -1;
+		__entry->moved	= moved;
+	),
+	TP_printk("bdi %s: older=%d enqueue=%d",
+		__entry->name,
+		__entry->older,
+		__entry->moved)
+);
+
 DECLARE_EVENT_CLASS(writeback_congest_waited_template,
 
 	TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),

root@fat /home/wfg# bin/test-tar-dd.sh
umount: /dev/sda7: not mounted

meta-data=/dev/sda7              isize=256    agcount=4, agsize=6170464 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=24681856, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=12051, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

bin/test-tar-dd.sh:14: no such file or directory: /debug/tracing/events/writeback/writeback_single_inode/enable 18.95 134.64
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.5855 s, 90.5 MB/s
dd if=/dev/zero of=/fs/zero bs=1M count=1000 0.00s user 2.59s system 22% cpu 11.614 total tar jxf /dev/shm/linux-2.6.38.3.tar.bz2 12.08s user 4.43s system 55% cpu 29.862 total
59.05 401.19
elapsed: 270.38

# tracer: nop
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

       flush-8:0-2926  [005]    19.491518: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.491521: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.491592: writeback_queue_io: bdi 8:0: older=-1 enqueue=644
       flush-8:0-2926  [005]    19.511680: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.512392: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.513408: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.514321: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.521382: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.522103: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.530078: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.530833: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.534340: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.538572: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.542839: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.547076: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.551240: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.555429: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.559650: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.563863: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [005]    19.568111: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [002]    19.640092: writeback_queue_io: bdi 8:0: older=-1 enqueue=1008
       flush-8:0-2926  [000]    19.934320: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    20.182384: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.190345: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.191834: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.192749: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [003]    20.215325: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    20.570616: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.571424: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.572249: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.573045: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    20.759429: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    20.760391: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    20.761302: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    20.786554: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.790471: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.796459: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.814775: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.818366: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.822369: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.828361: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.838421: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.856283: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.894326: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.897156: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.901125: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.907124: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.918181: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.940281: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.942999: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.946983: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.952965: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.962948: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    20.968968: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.970181: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.971367: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    20.972552: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    20.985901: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.990851: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    20.997841: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.007815: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.025829: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.063984: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.066615: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.070600: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.076599: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.087573: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.109714: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.112475: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.116457: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.122441: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.132480: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.154576: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.157333: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.161318: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.167314: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [001]    21.171047: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [001]    21.172293: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [001]    21.173527: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    21.194252: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.235402: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.238087: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.242104: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.248052: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.258124: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.284649: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.286932: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.290919: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.296899: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.306889: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.324897: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.358728: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    21.386536: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.387106: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.387664: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.388216: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    21.429514: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.433491: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.439458: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.453498: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.456406: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.460391: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.466386: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.476358: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.494301: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.532203: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.535163: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.539150: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.545141: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.555114: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.581393: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.585008: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.588993: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.594989: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    21.602369: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.602941: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.603496: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    21.620920: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.663323: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.665759: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.669745: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.675738: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.693686: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.696662: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.700648: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.706628: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.716617: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.734610: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.774548: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.777412: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.781419: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.787369: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.797350: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    21.811137: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.811708: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.812340: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    21.812899: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    21.820301: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.824264: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.830246: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.840275: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.859186: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.899544: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.902039: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.906018: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.912003: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.931784: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.934923: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.938910: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.944901: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.954875: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    21.972824: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.011067: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.013682: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.017666: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.023657: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    22.026325: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.026895: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.027450: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.028007: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [000]    22.054316: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.057545: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.061539: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.067518: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.077512: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.097435: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.139106: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.143283: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.147265: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.153262: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
      flush-0:15-2236  [003]    22.163312: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [003]    22.163315: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [003]    22.163317: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
      flush-0:15-2236  [003]    22.163478: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [003]    22.163479: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
              dd-2924  [000]    22.170035: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.173185: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.177168: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.183192: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.186150: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.190132: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.196115: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    22.206099: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [004]    22.228903: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    22.230641: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    22.231589: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    22.240047: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.243969: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.249995: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.260005: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.282466: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.285833: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.289819: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.295836: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.305815: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.323755: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.362152: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.364591: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.368574: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.374588: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.384576: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.408997: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.411442: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.415435: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.421455: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.431422: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    22.436513: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.437719: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.438913: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.440097: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    22.454336: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.458297: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.464307: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.474290: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.492238: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.530624: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.533068: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.537051: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.543080: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.553052: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.575463: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.577929: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.581914: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.587927: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.597895: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.616851: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    22.645196: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.646391: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.647589: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    22.655710: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.659681: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.665687: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.675666: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.698097: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.700548: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.704533: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.710557: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.720506: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.738437: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.780324: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.783292: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.787280: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.793293: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.803331: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.810686: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.813198: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.817189: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.823201: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.833185: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.853122: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    22.862321: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.863507: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.864689: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    22.865871: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    22.892977: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.896937: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.902950: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.912930: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.935376: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.937812: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.941797: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.947807: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.957783: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    22.975742: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.010631: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.080903: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.083361: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.087346: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.093364: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.103343: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.121288: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.155180: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.229071: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    23.231074: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.231663: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.232218: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.232763: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    23.237897: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.247941: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.270407: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.272778: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.276759: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.282807: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.292806: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.310736: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.344636: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.414952: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.417336: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.421311: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.427381: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.437345: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.455290: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.493697: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.496082: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.500068: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.506121: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.516086: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.534046: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.567949: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    23.579537: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.580357: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.581158: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [001]    23.640649: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.644628: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.650603: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.660652: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.683112: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.685497: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.689481: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.695537: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.705493: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [001]    23.723461: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.757301: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.829786: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.833041: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.837040: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.843008: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.853090: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.872028: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.910340: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.912794: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.916793: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.922758: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.932835: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    23.951770: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    23.968721: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.969755: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.970877: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    23.972034: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    24.056456: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.059339: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.063324: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.069304: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.079376: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.097325: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.137789: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.140072: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.144073: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.150054: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.160097: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.178069: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.211958: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.282211: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.284662: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.288626: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.294626: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    24.301958: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.302684: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.303210: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    24.309589: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.313550: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.319529: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.329598: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.347549: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.381440: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.447236: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.553256: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.555789: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.564191: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.566766: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.570763: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.576814: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.586806: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.606660: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    24.636914: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.637444: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.637982: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.638613: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    24.645545: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.649506: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.655502: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.665479: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.683413: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.717401: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.797288: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.800045: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.804041: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.810009: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.824422: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.826960: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.830943: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.836925: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.846992: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.864858: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.898838: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    24.912196: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.912854: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    24.913378: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    24.969553: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.977807: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.980485: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.984468: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    24.990450: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.000446: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.019380: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.057649: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.060241: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.064221: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.070205: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.080254: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.102544: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.105097: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.109082: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.115075: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    25.124573: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.125175: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.125702: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.126270: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    25.180687: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.183853: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.187851: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.193835: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.204922: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.207778: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.211776: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.217748: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.227818: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.245678: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.284943: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.287533: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.291530: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.297501: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.307550: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    25.324915: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.325688: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.326667: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    25.331434: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.335381: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.341379: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.351403: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.369359: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.408223: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.411150: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.415133: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.421131: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.433167: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.455441: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.458004: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.462001: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.467969: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.478038: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.500270: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.502865: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.506849: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.512843: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.522892: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    25.537975: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.538533: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.539482: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.540605: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    25.581677: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.584612: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.588595: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.594577: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.604638: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.626905: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.629473: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.633457: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.639461: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.649418: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.667405: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.710301: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.713212: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.717199: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.723191: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.737539: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.740143: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.744113: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    25.746331: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.746869: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.747404: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.747929: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    25.758160: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.776117: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.814338: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.816892: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.820885: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.826846: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.836931: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.859270: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.861752: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.865740: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.871854: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.881706: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.899739: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.937853: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.940509: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.944492: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.950492: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    25.962050: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.962798: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    25.963325: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    25.983391: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.987360: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    25.993344: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.003410: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.021301: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.061588: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.064125: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.068112: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.074094: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.084151: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.106423: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.109001: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.112970: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.118966: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.129038: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.146936: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    26.170795: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.171389: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.172221: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.173344: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    26.185767: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.189747: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.195715: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.205764: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.228243: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.230613: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.234593: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.240589: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.250570: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.272892: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.275472: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.279462: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.285424: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.295508: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.313431: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.351620: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.354228: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.358221: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.364196: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.374259: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    26.385018: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.385567: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.386098: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    26.397109: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.401081: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.407064: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.417090: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.435070: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.473260: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.475850: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.479834: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.485820: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.495863: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.518386: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.520713: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.524694: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.530686: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.540744: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.562973: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.565573: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.569557: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.575538: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.585577: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    26.590955: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.591606: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.592223: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.592847: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    26.640591: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.643331: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.647325: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.653296: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.663362: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.685519: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.688193: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.692179: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.698165: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.708220: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.726164: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.767942: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.770934: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.774918: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.780900: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.790949: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    26.805607: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.806142: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    26.806664: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    26.813818: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.817798: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.823757: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.833839: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.851781: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.889973: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.892558: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.896556: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.902526: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.912575: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.936821: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.939411: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.943396: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.949395: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.960446: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.982589: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.985270: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.989254: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    26.995235: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.005289: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    27.020772: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.021458: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.022110: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.022756: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    27.059788: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.063028: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.067029: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.072994: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.083043: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.105317: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.107890: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.111874: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.117875: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.128905: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.146847: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
      flush-0:15-2236  [003]    27.147835: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [003]    27.147837: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [003]    27.147838: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
              dd-2924  [004]    27.185029: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.187642: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.191626: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.197624: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.207657: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.225619: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.263806: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.266398: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    27.269975: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.270547: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.271073: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.271668: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    27.282430: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.300327: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.334286: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.404365: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.406962: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.410946: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.416941: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.431265: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.433879: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.437864: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.443847: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.454915: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.472774: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.511740: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.514627: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.518613: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.524619: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.536642: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    27.554552: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    27.559462: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.560050: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.560609: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    27.597436: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.601358: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.607354: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.617380: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.635305: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.669223: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.735048: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.840764: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.843624: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.847593: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.853588: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.863656: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.881511: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.919954: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.923361: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.927346: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.933343: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    27.945397: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    27.959128: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.959970: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.960815: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    27.961659: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    27.995244: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.066062: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.068912: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.072910: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.078877: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.088952: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.108861: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.146792: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.149662: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.153661: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.159627: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.169665: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.187632: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    28.221484: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.298654: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.303188: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.307168: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.313250: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [004]    28.318308: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    28.319040: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    28.319606: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    28.328146: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.332107: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.338074: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.348142: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.366085: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.399983: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.470246: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.473655: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.477655: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.483634: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.493694: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.515558: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.518517: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.522515: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.528485: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.538567: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.556496: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.590391: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.656190: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    28.689116: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    28.690321: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    28.691531: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    28.692726: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    28.767809: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.771741: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.777711: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.787729: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.809691: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.812605: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.816589: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.822587: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.833640: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.851581: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.885485: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
       flush-8:0-2926  [000]    28.913598: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    28.914827: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    28.916034: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [004]    28.960225: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.964132: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.970123: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.980180: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [004]    28.998043: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [000]    29.036041: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.038910: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.042891: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.049003: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.058970: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.077930: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
              dd-2924  [002]    29.111824: writeback_queue_io: bdi 8:0: older=-1 enqueue=1031
             tar-2923  [001]    29.151841: writeback_queue_io: bdi 8:0: older=-1 enqueue=1038
       flush-8:0-2926  [000]    29.167378: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    29.168022: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    29.168674: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    29.169302: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [002]    29.186509: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.190441: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.196444: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.206423: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.228680: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.231307: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.235292: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.241423: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.251370: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.273377: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.276170: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.280160: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.286250: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.296235: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
              dd-2924  [002]    29.314179: writeback_queue_io: bdi 8:0: older=-1 enqueue=1002
       flush-8:0-2926  [000]    29.344570: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    29.345137: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    29.345692: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
              dd-2924  [002]    29.352985: writeback_queue_io: bdi 8:0: older=-1 enqueue=1020
       flush-8:0-2926  [004]    30.357613: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.358274: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.358910: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.359916: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.360948: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.361981: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.363002: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.363940: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.364613: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.365310: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.366024: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.366711: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.367400: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [004]    30.368090: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
             tar-2923  [003]    30.406614: writeback_queue_io: bdi 8:0: older=-1 enqueue=3212
      flush-0:15-2236  [000]    32.132363: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [000]    32.132367: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [000]    32.132370: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
      flush-0:15-2236  [000]    32.132451: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [000]    32.132453: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
       flush-8:0-2926  [002]    32.142432: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.143079: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.143705: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.144375: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.144960: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.145543: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.146114: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.146761: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.147327: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.147899: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.148565: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.149128: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.149708: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.150306: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.150871: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.151452: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.384772: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    32.385781: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.108661: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.109353: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.109966: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.110583: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.111194: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.111773: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.112361: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.112941: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.113530: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.114107: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.114705: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.115289: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.115865: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.116449: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.117028: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.117614: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.118197: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.118773: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.119354: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.293855: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.294470: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.295067: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.492814: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.493443: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.494024: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.494598: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.705300: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.706149: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.706733: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.921688: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.922297: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.922873: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    34.923422: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.130528: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.131113: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.131663: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.132219: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.345774: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.346368: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.346914: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.660761: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.661339: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.661889: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.662474: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.868558: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.869192: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    35.869756: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.084358: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.084930: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.085490: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.086073: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.285689: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.286257: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.286820: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.613493: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.614062: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.614626: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    36.615175: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    37.001846: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    37.002698: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    37.003253: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    37.003800: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
      flush-0:15-2236  [004]    37.116839: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.383966: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.384554: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.385108: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.808417: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.809034: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [000]    37.809452: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
      flush-0:15-2236  [004]    42.101397: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
       flush-8:0-2926  [002]    42.299344: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    42.299346: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2926  [002]    42.301467: writeback_queue_io: bdi 8:0: older=-1 enqueue=23904
      flush-0:15-2236  [004]    47.085941: writeback_queue_io: bdi 0:15: older=30000 enqueue=6
      flush-0:15-2236  [004]    47.085979: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    47.085981: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    47.085982: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
      flush-0:15-2236  [004]    48.937186: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
      flush-0:15-2236  [004]    48.937304: writeback_queue_io: bdi 0:15: older=-1 enqueue=7
      flush-0:15-2236  [004]    48.937330: writeback_queue_io: bdi 0:15: older=0 enqueue=7
       flush-8:0-2926  [003]    48.947592: writeback_queue_io: bdi 8:0: older=-1 enqueue=35715
      flush-0:15-2236  [004]    53.921831: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
       flush-8:0-2926  [003]    54.465097: writeback_queue_io: bdi 8:0: older=-1 enqueue=34518
       flush-8:0-2926  [003]    54.556999: writeback_queue_io: bdi 8:0: older=-1 enqueue=34271
       flush-8:0-2926  [003]    54.641888: writeback_queue_io: bdi 8:0: older=3 enqueue=34271
       flush-8:0-2926  [003]    54.733057: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
      flush-0:15-2236  [004]    54.733153: writeback_queue_io: bdi 0:15: older=0 enqueue=7
      flush-0:15-2236  [004]    54.733175: writeback_queue_io: bdi 0:15: older=0 enqueue=0
       flush-8:0-2926  [003]    54.735958: writeback_queue_io: bdi 8:0: older=2 enqueue=34271
       flush-8:0-2926  [003]    57.066536: writeback_queue_io: bdi 8:0: older=2340 enqueue=3
       flush-8:0-2926  [003]    57.066544: writeback_queue_io: bdi 8:0: older=2340 enqueue=0
      flush-0:15-2236  [004]    68.683073: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    73.667542: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    78.652162: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    83.636732: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2236  [004]    88.621260: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
root@fat /home/wfg# bin/test-tar-dd.sh
umount: /dev/sda7: not mounted
meta-data=/dev/sda7              isize=256    agcount=4, agsize=6170464 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=24681856, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=12051, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

bin/test-tar-dd.sh:14: no such file or directory: /debug/tracing/events/writeback/writeback_single_inode/enable 280.90 2235.98
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 14.8681 s, 70.5 MB/s
dd if=/dev/zero of=/fs/zero bs=1M count=1000 0.00s user 1.48s system 9% cpu 14.888 total
tar jxf /dev/shm/linux-2.6.38.3.tar.bz2 12.42s user 4.33s system 49% cpu 33.563 total
322.49 2524.47
elapsed: 288.99000000000024

# tracer: nop
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

       flush-8:0-2939  [006]   289.234869: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [006]   289.234872: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
       flush-8:0-2939  [000]   289.244973: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   289.249918: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   289.254722: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   289.259679: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   289.264460: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.270664: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.274682: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.279065: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.283450: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.287875: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.292278: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.297190: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.302320: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.307390: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
       flush-8:0-2939  [002]   289.311641: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.315871: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
       flush-8:0-2939  [002]   289.320143: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.324426: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
       flush-8:0-2939  [002]   289.328687: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [004]   289.399926: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.400601: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.402595: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.406672: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.414706: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.430701: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.462702: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.526711: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
             tar-2937  [000]   289.618775: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
              dd-2938  [004]   289.626643: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.738324: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.738611: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.740605: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.744693: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.752691: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
      flush-0:15-2212  [007]   289.763781: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   289.763813: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
              dd-2938  [004]   289.772835: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.773643: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.775615: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.779692: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.787702: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
       flush-8:0-2939  [002]   289.788072: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   289.803711: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [002]   289.891206: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [004]   289.896533: writeback_queue_io: bdi 8:0: older=-1 enqueue=6
              dd-2938  [004]   290.045651: writeback_queue_io: bdi 8:0: older=-1 enqueue=6
              dd-2938  [004]   290.047619: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   290.051702: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
              dd-2938  [004]   290.059724: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
             tar-2937  [000]   290.067294: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
       flush-8:0-2939  [002]   290.096955: writeback_queue_io: bdi 8:0: older=-1 enqueue=4
       flush-8:0-2939  [002]   290.175145: writeback_queue_io: bdi 8:0: older=-1 enqueue=2
              dd-2938  [004]   290.190908: writeback_queue_io: bdi 8:0: older=-1 enqueue=102
       flush-8:0-2939  [002]   290.381441: writeback_queue_io: bdi 8:0: older=-1 enqueue=24
       flush-8:0-2939  [002]   290.381576: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
       flush-8:0-2939  [003]   290.424721: writeback_queue_io: bdi 8:0: older=-1 enqueue=146
              dd-2938  [000]   290.436335: writeback_queue_io: bdi 8:0: older=-1 enqueue=110
             tar-2937  [002]   290.436340: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [000]   290.440647: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.442640: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.446674: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.454731: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.475084: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.476642: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.478640: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.482656: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.490670: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
              dd-2938  [000]   290.506731: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
       flush-8:0-2939  [003]   290.542023: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
       flush-8:0-2939  [003]   290.542791: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [000]   290.543801: writeback_queue_io: bdi 8:0: older=-1 enqueue=144
       flush-8:0-2939  [003]   290.645714: writeback_queue_io: bdi 8:0: older=-1 enqueue=255
              dd-2938  [000]   290.661384: writeback_queue_io: bdi 8:0: older=-1 enqueue=118
              dd-2938  [000]   290.662872: writeback_queue_io: bdi 8:0: older=-1 enqueue=145
             tar-2937  [000]   290.671348: writeback_queue_io: bdi 8:0: older=-1 enqueue=48
              dd-2938  [000]   291.093712: writeback_queue_io: bdi 8:0: older=-1 enqueue=280
              dd-2938  [000]   291.094604: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [003]   291.196888: writeback_queue_io: bdi 8:0: older=-1 enqueue=254
       flush-8:0-2939  [003]   291.322189: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
              dd-2938  [004]   291.344134: writeback_queue_io: bdi 8:0: older=-1 enqueue=40
             tar-2937  [000]   291.348827: writeback_queue_io: bdi 8:0: older=-1 enqueue=258
             tar-2937  [000]   291.350680: writeback_queue_io: bdi 8:0: older=-1 enqueue=297
             tar-2937  [000]   291.352682: writeback_queue_io: bdi 8:0: older=-1 enqueue=297
             tar-2937  [000]   291.356689: writeback_queue_io: bdi 8:0: older=-1 enqueue=297
             tar-2937  [000]   291.364708: writeback_queue_io: bdi 8:0: older=-1 enqueue=297
       flush-8:0-2939  [000]   291.374525: writeback_queue_io: bdi 8:0: older=-1 enqueue=298
       flush-8:0-2939  [000]   291.374613: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
       flush-8:0-2939  [000]   291.374656: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   291.376586: writeback_queue_io: bdi 8:0: older=-1 enqueue=293
       flush-8:0-2939  [000]   291.377524: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
       flush-8:0-2939  [000]   291.379490: writeback_queue_io: bdi 8:0: older=-1 enqueue=293
              dd-2938  [004]   291.386729: writeback_queue_io: bdi 8:0: older=-1 enqueue=51
       flush-8:0-2939  [000]   291.751020: writeback_queue_io: bdi 8:0: older=-1 enqueue=300
       flush-8:0-2939  [000]   291.889096: writeback_queue_io: bdi 8:0: older=-1 enqueue=2
              dd-2938  [000]   291.922994: writeback_queue_io: bdi 8:0: older=-1 enqueue=141
             tar-2937  [002]   291.923004: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
       flush-8:0-2939  [000]   291.943943: writeback_queue_io: bdi 8:0: older=-1 enqueue=305
             tar-2937  [000]   291.962703: writeback_queue_io: bdi 8:0: older=-1 enqueue=432
             tar-2937  [000]   291.965711: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
             tar-2937  [000]   291.970700: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
             tar-2937  [000]   291.979722: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
       flush-8:0-2939  [004]   291.989047: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
              dd-2938  [000]   291.989082: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [000]   291.990710: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
              dd-2938  [000]   291.993718: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
             tar-2937  [000]   291.996729: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
              dd-2938  [000]   291.999700: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
              dd-2938  [000]   292.009794: writeback_queue_io: bdi 8:0: older=-1 enqueue=440
              dd-2938  [003]   292.038604: writeback_queue_io: bdi 8:0: older=-1 enqueue=490
       flush-8:0-2939  [000]   292.078232: writeback_queue_io: bdi 8:0: older=-1 enqueue=142
       flush-8:0-2939  [001]   292.110171: writeback_queue_io: bdi 8:0: older=-1 enqueue=301
       flush-8:0-2939  [001]   293.075697: writeback_queue_io: bdi 8:0: older=-1 enqueue=5
       flush-8:0-2939  [001]   293.078623: writeback_queue_io: bdi 8:0: older=-1 enqueue=496
       flush-8:0-2939  [001]   293.080842: writeback_queue_io: bdi 8:0: older=-1 enqueue=12
       flush-8:0-2939  [001]   293.085045: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [001]   293.088241: writeback_queue_io: bdi 8:0: older=-1 enqueue=494
       flush-8:0-2939  [001]   293.090036: writeback_queue_io: bdi 8:0: older=-1 enqueue=27
       flush-8:0-2939  [001]   293.092962: writeback_queue_io: bdi 8:0: older=-1 enqueue=494
       flush-8:0-2939  [001]   293.094561: writeback_queue_io: bdi 8:0: older=-1 enqueue=12
       flush-8:0-2939  [001]   293.095628: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [001]   293.098615: writeback_queue_io: bdi 8:0: older=-1 enqueue=506
       flush-8:0-2939  [001]   293.099645: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [001]   293.102537: writeback_queue_io: bdi 8:0: older=-1 enqueue=506
       flush-8:0-2939  [001]   293.103571: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [001]   293.106481: writeback_queue_io: bdi 8:0: older=-1 enqueue=506
       flush-8:0-2939  [001]   293.107516: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [001]   293.110467: writeback_queue_io: bdi 8:0: older=-1 enqueue=506
       flush-8:0-2939  [001]   293.111475: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
              dd-2938  [003]   293.146707: writeback_queue_io: bdi 8:0: older=-1 enqueue=642
             tar-2937  [004]   293.153753: writeback_queue_io: bdi 8:0: older=-1 enqueue=277
       flush-8:0-2939  [000]   293.315851: writeback_queue_io: bdi 8:0: older=-1 enqueue=15
       flush-8:0-2939  [000]   293.338781: writeback_queue_io: bdi 8:0: older=-1 enqueue=379
              dd-2938  [004]   293.353306: writeback_queue_io: bdi 8:0: older=-1 enqueue=263
       flush-8:0-2939  [000]   293.353349: writeback_queue_io: bdi 8:0: older=-1 enqueue=8
       flush-8:0-2939  [000]   293.354913: writeback_queue_io: bdi 8:0: older=-1 enqueue=649
             tar-2937  [004]   293.356764: writeback_queue_io: bdi 8:0: older=-1 enqueue=394
       flush-8:0-2939  [000]   293.418244: writeback_queue_io: bdi 8:0: older=-1 enqueue=839
              dd-2938  [004]   293.426899: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [000]   293.555279: writeback_queue_io: bdi 8:0: older=-1 enqueue=657
      flush-0:15-2212  [007]   294.763913: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   294.763935: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
       flush-8:0-2939  [003]   295.266112: writeback_queue_io: bdi 8:0: older=30007 enqueue=0
       flush-8:0-2939  [003]   295.266716: writeback_queue_io: bdi 8:0: older=30007 enqueue=0
       flush-8:0-2939  [003]   295.267359: writeback_queue_io: bdi 8:0: older=30008 enqueue=0
       flush-8:0-2939  [003]   295.268065: writeback_queue_io: bdi 8:0: older=30009 enqueue=0
       flush-8:0-2939  [003]   295.268738: writeback_queue_io: bdi 8:0: older=30009 enqueue=0
       flush-8:0-2939  [003]   295.269423: writeback_queue_io: bdi 8:0: older=30010 enqueue=0
       flush-8:0-2939  [003]   295.270078: writeback_queue_io: bdi 8:0: older=30011 enqueue=0
       flush-8:0-2939  [003]   295.270734: writeback_queue_io: bdi 8:0: older=30011 enqueue=0
       flush-8:0-2939  [003]   295.271391: writeback_queue_io: bdi 8:0: older=30012 enqueue=0
       flush-8:0-2939  [003]   295.272063: writeback_queue_io: bdi 8:0: older=30013 enqueue=0
       flush-8:0-2939  [003]   295.272707: writeback_queue_io: bdi 8:0: older=30013 enqueue=0
       flush-8:0-2939  [003]   295.273389: writeback_queue_io: bdi 8:0: older=30014 enqueue=0
       flush-8:0-2939  [003]   295.274075: writeback_queue_io: bdi 8:0: older=30015 enqueue=0
       flush-8:0-2939  [003]   295.274733: writeback_queue_io: bdi 8:0: older=30015 enqueue=0
       flush-8:0-2939  [003]   295.275392: writeback_queue_io: bdi 8:0: older=30016 enqueue=0
              dd-2938  [002]   295.337571: writeback_queue_io: bdi 8:0: older=-1 enqueue=981
       flush-8:0-2939  [003]   295.530905: writeback_queue_io: bdi 8:0: older=-1 enqueue=600
       flush-8:0-2939  [007]   295.537538: writeback_queue_io: bdi 8:0: older=-1 enqueue=982
       flush-8:0-2939  [007]   295.545959: writeback_queue_io: bdi 8:0: older=-1 enqueue=346
       flush-8:0-2939  [007]   295.776000: writeback_queue_io: bdi 8:0: older=-1 enqueue=1452
              dd-2938  [000]   295.779745: writeback_queue_io: bdi 8:0: older=-1 enqueue=212
             tar-2937  [002]   295.779873: writeback_queue_io: bdi 8:0: older=-1 enqueue=25
              dd-2938  [000]   295.783513: writeback_queue_io: bdi 8:0: older=-1 enqueue=1355
       flush-8:0-2939  [007]   295.783516: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [007]   296.071577: writeback_queue_io: bdi 8:0: older=-1 enqueue=2883
       flush-8:0-2939  [007]   296.072234: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [007]   296.076483: writeback_queue_io: bdi 8:0: older=-1 enqueue=32
              dd-2938  [000]   296.090712: writeback_queue_io: bdi 8:0: older=-1 enqueue=1439
       flush-8:0-2939  [007]   297.340433: writeback_queue_io: bdi 8:0: older=-1 enqueue=2886
       flush-8:0-2939  [007]   297.341188: writeback_queue_io: bdi 8:0: older=-1 enqueue=7
       flush-8:0-2939  [007]   297.345204: writeback_queue_io: bdi 8:0: older=-1 enqueue=22
       flush-8:0-2939  [007]   297.356773: writeback_queue_io: bdi 8:0: older=-1 enqueue=2880
       flush-8:0-2939  [007]   297.358096: writeback_queue_io: bdi 8:0: older=-1 enqueue=30
       flush-8:0-2939  [007]   297.367081: writeback_queue_io: bdi 8:0: older=-1 enqueue=2920
       flush-8:0-2939  [007]   297.368869: writeback_queue_io: bdi 8:0: older=-1 enqueue=29
       flush-8:0-2939  [007]   297.369583: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
              dd-2938  [002]   297.399896: writeback_queue_io: bdi 8:0: older=-1 enqueue=1368
       flush-8:0-2939  [007]   297.962997: writeback_queue_io: bdi 8:0: older=-1 enqueue=1777
       flush-8:0-2939  [007]   297.966832: writeback_queue_io: bdi 8:0: older=-1 enqueue=1263
       flush-8:0-2939  [007]   297.971540: writeback_queue_io: bdi 8:0: older=-1 enqueue=42
       flush-8:0-2939  [007]   297.978605: writeback_queue_io: bdi 8:0: older=-1 enqueue=1801
       flush-8:0-2939  [007]   297.984734: writeback_queue_io: bdi 8:0: older=-1 enqueue=1356
       flush-8:0-2939  [007]   297.985728: writeback_queue_io: bdi 8:0: older=-1 enqueue=77
              dd-2938  [002]   297.990407: writeback_queue_io: bdi 8:0: older=-1 enqueue=1307
       flush-8:0-2939  [007]   297.992031: writeback_queue_io: bdi 8:0: older=-1 enqueue=568
       flush-8:0-2939  [007]   297.998370: writeback_queue_io: bdi 8:0: older=-1 enqueue=2756
       flush-8:0-2939  [007]   297.998599: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [002]   297.998646: writeback_queue_io: bdi 8:0: older=-1 enqueue=14
             tar-2937  [004]   297.999541: writeback_queue_io: bdi 8:0: older=-1 enqueue=434
       flush-8:0-2939  [007]   298.005972: writeback_queue_io: bdi 8:0: older=-1 enqueue=3030
       flush-8:0-2939  [007]   298.006175: writeback_queue_io: bdi 8:0: older=-1 enqueue=95
              dd-2938  [002]   298.006892: writeback_queue_io: bdi 8:0: older=-1 enqueue=40
       flush-8:0-2939  [007]   298.007243: writeback_queue_io: bdi 8:0: older=-1 enqueue=129
             tar-2937  [004]   298.007779: writeback_queue_io: bdi 8:0: older=-1 enqueue=280
       flush-8:0-2939  [007]   298.949789: writeback_queue_io: bdi 8:0: older=-1 enqueue=3175
       flush-8:0-2939  [007]   298.950422: writeback_queue_io: bdi 8:0: older=-1 enqueue=6
       flush-8:0-2939  [007]   298.962155: writeback_queue_io: bdi 8:0: older=-1 enqueue=2710
       flush-8:0-2939  [007]   298.964807: writeback_queue_io: bdi 8:0: older=-1 enqueue=571
       flush-8:0-2939  [007]   298.976086: writeback_queue_io: bdi 8:0: older=-1 enqueue=2798
       flush-8:0-2939  [007]   298.980770: writeback_queue_io: bdi 8:0: older=-1 enqueue=264
       flush-8:0-2939  [007]   298.982764: writeback_queue_io: bdi 8:0: older=-1 enqueue=314
       flush-8:0-2939  [007]   298.994973: writeback_queue_io: bdi 8:0: older=-1 enqueue=3099
       flush-8:0-2939  [007]   298.996633: writeback_queue_io: bdi 8:0: older=-1 enqueue=323
              dd-2938  [002]   299.004420: writeback_queue_io: bdi 8:0: older=-1 enqueue=2160
             tar-2937  [004]   299.004426: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
       flush-8:0-2939  [007]   299.007812: writeback_queue_io: bdi 8:0: older=-1 enqueue=1265
             tar-2937  [004]   299.008346: writeback_queue_io: bdi 8:0: older=-1 enqueue=224
              dd-2938  [002]   299.013535: writeback_queue_io: bdi 8:0: older=-1 enqueue=2427
       flush-8:0-2939  [007]   299.015495: writeback_queue_io: bdi 8:0: older=-1 enqueue=879
       flush-8:0-2939  [007]   299.016767: writeback_queue_io: bdi 8:0: older=-1 enqueue=468
              dd-2938  [002]   299.021580: writeback_queue_io: bdi 8:0: older=-1 enqueue=1721
       flush-8:0-2939  [007]   299.024831: writeback_queue_io: bdi 8:0: older=-1 enqueue=1514
       flush-8:0-2939  [007]   299.025548: writeback_queue_io: bdi 8:0: older=-1 enqueue=357
      flush-0:15-2212  [007]   299.764129: writeback_queue_io: bdi 0:15: older=30000 enqueue=2
      flush-0:15-2212  [007]   299.764146: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
       flush-8:0-2939  [007]   300.175524: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
       flush-8:0-2939  [007]   300.183204: writeback_queue_io: bdi 8:0: older=-1 enqueue=1875
       flush-8:0-2939  [007]   300.189489: writeback_queue_io: bdi 8:0: older=-1 enqueue=1707
       flush-8:0-2939  [007]   300.193487: writeback_queue_io: bdi 8:0: older=-1 enqueue=145
       flush-8:0-2939  [007]   300.200647: writeback_queue_io: bdi 8:0: older=-1 enqueue=1730
       flush-8:0-2939  [007]   300.208785: writeback_queue_io: bdi 8:0: older=-1 enqueue=1916
       flush-8:0-2939  [007]   300.214815: writeback_queue_io: bdi 8:0: older=-1 enqueue=1788
       flush-8:0-2939  [007]   300.217281: writeback_queue_io: bdi 8:0: older=-1 enqueue=343
       flush-8:0-2939  [007]   300.224290: writeback_queue_io: bdi 8:0: older=-1 enqueue=1608
              dd-2938  [002]   300.231330: writeback_queue_io: bdi 8:0: older=-1 enqueue=1792
       flush-8:0-2939  [007]   300.232698: writeback_queue_io: bdi 8:0: older=-1 enqueue=490
       flush-8:0-2939  [007]   300.232729: writeback_queue_io: bdi 8:0: older=-1 enqueue=12
             tar-2937  [004]   300.234232: writeback_queue_io: bdi 8:0: older=-1 enqueue=432
       flush-8:0-2939  [007]   300.241170: writeback_queue_io: bdi 8:0: older=-1 enqueue=3096
              dd-2938  [002]   300.241173: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [007]   300.243152: writeback_queue_io: bdi 8:0: older=-1 enqueue=962
             tar-2937  [004]   300.243993: writeback_queue_io: bdi 8:0: older=-1 enqueue=418
              dd-2938  [002]   300.249851: writeback_queue_io: bdi 8:0: older=-1 enqueue=2835
       flush-8:0-2939  [007]   300.256861: writeback_queue_io: bdi 8:0: older=-1 enqueue=3263
              dd-2938  [002]   300.257859: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
             tar-2937  [004]   300.257975: writeback_queue_io: bdi 8:0: older=-1 enqueue=35
             tar-2937  [004]   300.260115: writeback_queue_io: bdi 8:0: older=-1 enqueue=915
              dd-2938  [002]   300.267108: writeback_queue_io: bdi 8:0: older=-1 enqueue=3302
       flush-8:0-2939  [000]   300.385222: writeback_queue_io: bdi 8:0: older=-1 enqueue=436
       flush-8:0-2939  [004]   300.398842: writeback_queue_io: bdi 8:0: older=-1 enqueue=3303
       flush-8:0-2939  [004]   300.400631: writeback_queue_io: bdi 8:0: older=-1 enqueue=436
       flush-8:0-2939  [000]   302.112115: writeback_queue_io: bdi 8:0: older=-1 enqueue=146
       flush-8:0-2939  [003]   302.125713: writeback_queue_io: bdi 8:0: older=-1 enqueue=3193
       flush-8:0-2939  [003]   302.128185: writeback_queue_io: bdi 8:0: older=-1 enqueue=582
              dd-2938  [002]   302.176685: writeback_queue_io: bdi 8:0: older=-1 enqueue=33
       flush-8:0-2939  [003]   302.310786: writeback_queue_io: bdi 8:0: older=-1 enqueue=73
       flush-8:0-2939  [003]   302.324014: writeback_queue_io: bdi 8:0: older=-1 enqueue=3100
       flush-8:0-2939  [003]   302.327425: writeback_queue_io: bdi 8:0: older=-1 enqueue=716
              dd-2938  [002]   302.328409: writeback_queue_io: bdi 8:0: older=-1 enqueue=320
             tar-2937  [004]   302.328415: writeback_queue_io: bdi 8:0: older=-1 enqueue=1
       flush-8:0-2939  [003]   302.335538: writeback_queue_io: bdi 8:0: older=-1 enqueue=3002
             tar-2937  [004]   302.335578: writeback_queue_io: bdi 8:0: older=-1 enqueue=13
       flush-8:0-2939  [003]   302.338355: writeback_queue_io: bdi 8:0: older=-1 enqueue=1141
              dd-2938  [002]   302.338357: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
             tar-2937  [004]   302.345254: writeback_queue_io: bdi 8:0: older=-1 enqueue=3267
              dd-2938  [002]   302.347398: writeback_queue_io: bdi 8:0: older=-1 enqueue=1045
       flush-8:0-2939  [003]   302.352058: writeback_queue_io: bdi 8:0: older=-1 enqueue=2185
       flush-8:0-2939  [003]   302.354279: writeback_queue_io: bdi 8:0: older=-1 enqueue=588
       flush-8:0-2939  [003]   302.363556: writeback_queue_io: bdi 8:0: older=-1 enqueue=3230
       flush-8:0-2939  [003]   302.365739: writeback_queue_io: bdi 8:0: older=-1 enqueue=588
       flush-8:0-2939  [003]   303.227794: writeback_queue_io: bdi 8:0: older=-1 enqueue=1046
       flush-8:0-2939  [003]   303.238308: writeback_queue_io: bdi 8:0: older=-1 enqueue=2221
       flush-8:0-2939  [003]   303.244048: writeback_queue_io: bdi 8:0: older=-1 enqueue=1643
       flush-8:0-2939  [003]   303.244463: writeback_queue_io: bdi 8:0: older=-1 enqueue=80
       flush-8:0-2939  [003]   303.252856: writeback_queue_io: bdi 8:0: older=-1 enqueue=2141
       flush-8:0-2939  [003]   303.259314: writeback_queue_io: bdi 8:0: older=-1 enqueue=1748
       flush-8:0-2939  [003]   303.266725: writeback_queue_io: bdi 8:0: older=-1 enqueue=2193
       flush-8:0-2939  [003]   303.271326: writeback_queue_io: bdi 8:0: older=-1 enqueue=723
       flush-8:0-2939  [003]   303.275869: writeback_queue_io: bdi 8:0: older=-1 enqueue=1085
       flush-8:0-2939  [003]   303.287943: writeback_queue_io: bdi 8:0: older=-1 enqueue=2918
              dd-2938  [002]   303.288922: writeback_queue_io: bdi 8:0: older=-1 enqueue=213
             tar-2937  [001]   303.288961: writeback_queue_io: bdi 8:0: older=-1 enqueue=11
       flush-8:0-2939  [003]   303.291737: writeback_queue_io: bdi 8:0: older=-1 enqueue=1220
             tar-2937  [001]   303.291810: writeback_queue_io: bdi 8:0: older=-1 enqueue=32
       flush-8:0-2939  [003]   303.298751: writeback_queue_io: bdi 8:0: older=-1 enqueue=3199
       flush-8:0-2939  [003]   303.301163: writeback_queue_io: bdi 8:0: older=-1 enqueue=1058
              dd-2938  [002]   303.304720: writeback_queue_io: bdi 8:0: older=-1 enqueue=1715
       flush-8:0-2939  [003]   303.306021: writeback_queue_io: bdi 8:0: older=-1 enqueue=608
             tar-2937  [001]   303.306079: writeback_queue_io: bdi 8:0: older=-1 enqueue=18
       flush-8:0-2939  [003]   303.308391: writeback_queue_io: bdi 8:0: older=-1 enqueue=982
             tar-2937  [001]   303.308561: writeback_queue_io: bdi 8:0: older=-1 enqueue=76
       flush-8:0-2939  [003]   303.315582: writeback_queue_io: bdi 8:0: older=-1 enqueue=3260
              dd-2938  [002]   303.317096: writeback_queue_io: bdi 8:0: older=-1 enqueue=743
       flush-8:0-2939  [003]   303.319923: writeback_queue_io: bdi 8:0: older=-1 enqueue=623
              dd-2938  [000]   303.319959: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [000]   303.332228: writeback_queue_io: bdi 8:0: older=-1 enqueue=3389
       flush-8:0-2939  [003]   303.332231: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
              dd-2938  [000]   303.343272: writeback_queue_io: bdi 8:0: older=-1 enqueue=4012
              dd-2938  [000]   303.355275: writeback_queue_io: bdi 8:0: older=-1 enqueue=4012
              dd-2938  [000]   303.370342: writeback_queue_io: bdi 8:0: older=-1 enqueue=4012
       flush-8:0-2939  [003]   303.382795: writeback_queue_io: bdi 8:0: older=-1 enqueue=4012
       flush-8:0-2939  [003]   303.383462: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [003]   303.388168: writeback_queue_io: bdi 8:0: older=-1 enqueue=623
              dd-2938  [000]   303.392819: writeback_queue_io: bdi 8:0: older=-1 enqueue=1579
       flush-8:0-2939  [003]   303.397128: writeback_queue_io: bdi 8:0: older=-1 enqueue=2041
              dd-2938  [000]   303.523086: writeback_queue_io: bdi 8:0: older=-1 enqueue=3781
              dd-2938  [000]   303.524717: writeback_queue_io: bdi 8:0: older=-1 enqueue=231
       flush-8:0-2939  [003]   304.574804: writeback_queue_io: bdi 8:0: older=-1 enqueue=3782
       flush-8:0-2939  [003]   304.575192: writeback_queue_io: bdi 8:0: older=-1 enqueue=0
       flush-8:0-2939  [003]   304.577811: writeback_queue_io: bdi 8:0: older=-1 enqueue=244
       flush-8:0-2939  [003]   304.592993: writeback_queue_io: bdi 8:0: older=-1 enqueue=3870
      flush-0:15-2212  [007]   304.764258: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   304.764272: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
       flush-8:0-2939  [003]   305.435995: writeback_queue_io: bdi 8:0: older=30015 enqueue=0
       flush-8:0-2939  [003]   305.436635: writeback_queue_io: bdi 8:0: older=30016 enqueue=0
       flush-8:0-2939  [003]   305.437234: writeback_queue_io: bdi 8:0: older=30017 enqueue=0
       flush-8:0-2939  [003]   305.437824: writeback_queue_io: bdi 8:0: older=30017 enqueue=0
       flush-8:0-2939  [003]   305.438418: writeback_queue_io: bdi 8:0: older=30018 enqueue=0
       flush-8:0-2939  [003]   305.439006: writeback_queue_io: bdi 8:0: older=30018 enqueue=0
       flush-8:0-2939  [003]   305.439603: writeback_queue_io: bdi 8:0: older=30019 enqueue=0
       flush-8:0-2939  [003]   305.440198: writeback_queue_io: bdi 8:0: older=30020 enqueue=0
       flush-8:0-2939  [003]   305.440787: writeback_queue_io: bdi 8:0: older=30020 enqueue=0
       flush-8:0-2939  [003]   305.441651: writeback_queue_io: bdi 8:0: older=30021 enqueue=0
       flush-8:0-2939  [003]   305.442247: writeback_queue_io: bdi 8:0: older=30022 enqueue=0
       flush-8:0-2939  [003]   305.442839: writeback_queue_io: bdi 8:0: older=30022 enqueue=0
       flush-8:0-2939  [003]   305.443439: writeback_queue_io: bdi 8:0: older=30023 enqueue=0
       flush-8:0-2939  [003]   305.444029: writeback_queue_io: bdi 8:0: older=30023 enqueue=0
       flush-8:0-2939  [003]   305.444643: writeback_queue_io: bdi 8:0: older=30024 enqueue=0
       flush-8:0-2939  [000]   305.775150: writeback_queue_io: bdi 8:0: older=30354 enqueue=0
       flush-8:0-2939  [000]   305.775788: writeback_queue_io: bdi 8:0: older=30355 enqueue=0
       flush-8:0-2939  [000]   305.776388: writeback_queue_io: bdi 8:0: older=30356 enqueue=0
       flush-8:0-2939  [004]   306.167610: writeback_queue_io: bdi 8:0: older=30747 enqueue=0
       flush-8:0-2939  [004]   306.168516: writeback_queue_io: bdi 8:0: older=30748 enqueue=0
       flush-8:0-2939  [004]   306.169372: writeback_queue_io: bdi 8:0: older=30749 enqueue=0
       flush-8:0-2939  [004]   306.170248: writeback_queue_io: bdi 8:0: older=30750 enqueue=0
       flush-8:0-2939  [004]   306.554556: writeback_queue_io: bdi 8:0: older=31134 enqueue=0
       flush-8:0-2939  [004]   306.555794: writeback_queue_io: bdi 8:0: older=31135 enqueue=0
       flush-8:0-2939  [004]   306.556703: writeback_queue_io: bdi 8:0: older=31136 enqueue=0
       flush-8:0-2939  [004]   306.557613: writeback_queue_io: bdi 8:0: older=31137 enqueue=0
       flush-8:0-2939  [004]   306.793776: writeback_queue_io: bdi 8:0: older=31373 enqueue=0
       flush-8:0-2939  [004]   306.794939: writeback_queue_io: bdi 8:0: older=31374 enqueue=0
       flush-8:0-2939  [004]   306.795901: writeback_queue_io: bdi 8:0: older=31375 enqueue=0
       flush-8:0-2939  [004]   307.008637: writeback_queue_io: bdi 8:0: older=31588 enqueue=0
       flush-8:0-2939  [004]   307.009451: writeback_queue_io: bdi 8:0: older=31589 enqueue=0
       flush-8:0-2939  [004]   307.010196: writeback_queue_io: bdi 8:0: older=31589 enqueue=0
       flush-8:0-2939  [004]   307.011037: writeback_queue_io: bdi 8:0: older=31590 enqueue=0
       flush-8:0-2939  [004]   307.209647: writeback_queue_io: bdi 8:0: older=31789 enqueue=0
       flush-8:0-2939  [004]   307.210586: writeback_queue_io: bdi 8:0: older=31790 enqueue=0
       flush-8:0-2939  [004]   307.211503: writeback_queue_io: bdi 8:0: older=31791 enqueue=0
       flush-8:0-2939  [000]   307.439319: writeback_queue_io: bdi 8:0: older=32019 enqueue=0
       flush-8:0-2939  [000]   307.439940: writeback_queue_io: bdi 8:0: older=32019 enqueue=0
       flush-8:0-2939  [000]   307.440565: writeback_queue_io: bdi 8:0: older=32020 enqueue=0
       flush-8:0-2939  [000]   307.441152: writeback_queue_io: bdi 8:0: older=32020 enqueue=0
       flush-8:0-2939  [000]   307.648785: writeback_queue_io: bdi 8:0: older=32228 enqueue=0
       flush-8:0-2939  [000]   307.649393: writeback_queue_io: bdi 8:0: older=32229 enqueue=0
       flush-8:0-2939  [000]   307.649984: writeback_queue_io: bdi 8:0: older=32229 enqueue=0
       flush-8:0-2939  [000]   307.955109: writeback_queue_io: bdi 8:0: older=32534 enqueue=0
       flush-8:0-2939  [000]   307.955766: writeback_queue_io: bdi 8:0: older=32535 enqueue=0
       flush-8:0-2939  [000]   307.956363: writeback_queue_io: bdi 8:0: older=32536 enqueue=0
       flush-8:0-2939  [000]   307.956953: writeback_queue_io: bdi 8:0: older=32536 enqueue=0
       flush-8:0-2939  [000]   308.171085: writeback_queue_io: bdi 8:0: older=32750 enqueue=0
       flush-8:0-2939  [000]   308.171741: writeback_queue_io: bdi 8:0: older=32751 enqueue=0
       flush-8:0-2939  [000]   308.172308: writeback_queue_io: bdi 8:0: older=32752 enqueue=0
       flush-8:0-2939  [000]   308.376067: writeback_queue_io: bdi 8:0: older=32955 enqueue=0
       flush-8:0-2939  [000]   308.376700: writeback_queue_io: bdi 8:0: older=32956 enqueue=0
       flush-8:0-2939  [000]   308.377268: writeback_queue_io: bdi 8:0: older=32957 enqueue=0
       flush-8:0-2939  [000]   308.377831: writeback_queue_io: bdi 8:0: older=32957 enqueue=0
       flush-8:0-2939  [004]   308.563760: writeback_queue_io: bdi 8:0: older=33143 enqueue=0
       flush-8:0-2939  [004]   308.564571: writeback_queue_io: bdi 8:0: older=33144 enqueue=0
       flush-8:0-2939  [004]   308.565404: writeback_queue_io: bdi 8:0: older=33145 enqueue=0
       flush-8:0-2939  [004]   308.566098: writeback_queue_io: bdi 8:0: older=33145 enqueue=0
       flush-8:0-2939  [000]   308.704526: writeback_queue_io: bdi 8:0: older=33284 enqueue=0
       flush-8:0-2939  [000]   308.705118: writeback_queue_io: bdi 8:0: older=33284 enqueue=0
       flush-8:0-2939  [000]   308.705688: writeback_queue_io: bdi 8:0: older=33285 enqueue=0
       flush-8:0-2939  [004]   308.943857: writeback_queue_io: bdi 8:0: older=33523 enqueue=0
       flush-8:0-2939  [004]   308.944594: writeback_queue_io: bdi 8:0: older=33524 enqueue=0
       flush-8:0-2939  [004]   308.945156: writeback_queue_io: bdi 8:0: older=33524 enqueue=0
       flush-8:0-2939  [004]   308.945805: writeback_queue_io: bdi 8:0: older=33525 enqueue=0
       flush-8:0-2939  [000]   309.232903: writeback_queue_io: bdi 8:0: older=33812 enqueue=0
       flush-8:0-2939  [000]   309.233503: writeback_queue_io: bdi 8:0: older=33813 enqueue=0
       flush-8:0-2939  [000]   309.234070: writeback_queue_io: bdi 8:0: older=33813 enqueue=0
       flush-8:0-2939  [000]   309.583782: writeback_queue_io: bdi 8:0: older=34163 enqueue=0
       flush-8:0-2939  [000]   309.584377: writeback_queue_io: bdi 8:0: older=34164 enqueue=0
       flush-8:0-2939  [000]   309.584941: writeback_queue_io: bdi 8:0: older=34164 enqueue=0
       flush-8:0-2939  [000]   309.585514: writeback_queue_io: bdi 8:0: older=34165 enqueue=0
      flush-0:15-2212  [007]   309.764382: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
       flush-8:0-2939  [000]   309.958752: writeback_queue_io: bdi 8:0: older=34538 enqueue=0
       flush-8:0-2939  [000]   309.959332: writeback_queue_io: bdi 8:0: older=34539 enqueue=0
       flush-8:0-2939  [000]   309.959895: writeback_queue_io: bdi 8:0: older=34539 enqueue=0
       flush-8:0-2939  [000]   309.960508: writeback_queue_io: bdi 8:0: older=34540 enqueue=0
       flush-8:0-2939  [000]   310.288178: writeback_queue_io: bdi 8:0: older=34867 enqueue=0
       flush-8:0-2939  [000]   310.288749: writeback_queue_io: bdi 8:0: older=34868 enqueue=0
       flush-8:0-2939  [000]   310.289309: writeback_queue_io: bdi 8:0: older=34868 enqueue=0
       flush-8:0-2939  [000]   310.793020: writeback_queue_io: bdi 8:0: older=35372 enqueue=0
       flush-8:0-2939  [000]   310.793594: writeback_queue_io: bdi 8:0: older=35373 enqueue=0
       flush-8:0-2939  [000]   310.794156: writeback_queue_io: bdi 8:0: older=35373 enqueue=0
       flush-8:0-2939  [000]   310.794769: writeback_queue_io: bdi 8:0: older=35374 enqueue=0
      flush-0:15-2212  [007]   314.764559: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
       flush-8:0-2939  [004]   315.180482: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   315.182519: writeback_queue_io: bdi 8:0: older=-1 enqueue=24158
       flush-8:0-2939  [004]   315.305213: writeback_queue_io: bdi 8:0: older=-1 enqueue=1916
       flush-8:0-2939  [004]   315.362979: writeback_queue_io: bdi 8:0: older=-1 enqueue=674
       flush-8:0-2939  [004]   315.550202: writeback_queue_io: bdi 8:0: older=-1 enqueue=882
       flush-8:0-2939  [004]   315.627676: writeback_queue_io: bdi 8:0: older=-1 enqueue=859
       flush-8:0-2939  [004]   315.707167: writeback_queue_io: bdi 8:0: older=-1 enqueue=476
       flush-8:0-2939  [004]   315.794131: writeback_queue_io: bdi 8:0: older=-1 enqueue=689
       flush-8:0-2939  [004]   315.902549: writeback_queue_io: bdi 8:0: older=-1 enqueue=747
       flush-8:0-2939  [004]   315.983656: writeback_queue_io: bdi 8:0: older=-1 enqueue=725
       flush-8:0-2939  [004]   316.131605: writeback_queue_io: bdi 8:0: older=-1 enqueue=692
       flush-8:0-2939  [004]   316.211777: writeback_queue_io: bdi 8:0: older=-1 enqueue=749
       flush-8:0-2939  [004]   316.290287: writeback_queue_io: bdi 8:0: older=-1 enqueue=654
       flush-8:0-2939  [004]   316.393865: writeback_queue_io: bdi 8:0: older=-1 enqueue=752
       flush-8:0-2939  [004]   316.566939: writeback_queue_io: bdi 8:0: older=-1 enqueue=798
       flush-8:0-2939  [004]   316.649929: writeback_queue_io: bdi 8:0: older=-1 enqueue=734
       flush-8:0-2939  [004]   316.734868: writeback_queue_io: bdi 8:0: older=-1 enqueue=592
       flush-8:0-2939  [004]   316.896972: writeback_queue_io: bdi 8:0: older=-1 enqueue=726
       flush-8:0-2939  [004]   316.914074: writeback_queue_io: bdi 8:0: older=-1 enqueue=654
       flush-8:0-2939  [004]   317.100975: writeback_queue_io: bdi 8:0: older=-1 enqueue=887
       flush-8:0-2939  [004]   317.218250: writeback_queue_io: bdi 8:0: older=-1 enqueue=608
       flush-8:0-2939  [004]   317.306473: writeback_queue_io: bdi 8:0: older=-1 enqueue=782
       flush-8:0-2939  [004]   317.498823: writeback_queue_io: bdi 8:0: older=-1 enqueue=839
       flush-8:0-2939  [004]   317.620263: writeback_queue_io: bdi 8:0: older=-1 enqueue=661
       flush-8:0-2939  [004]   317.722236: writeback_queue_io: bdi 8:0: older=-1 enqueue=555
       flush-8:0-2939  [004]   317.794648: writeback_queue_io: bdi 8:0: older=-1 enqueue=299
       flush-8:0-2939  [004]   317.895347: writeback_queue_io: bdi 8:0: older=-1 enqueue=316
       flush-8:0-2939  [004]   317.983708: writeback_queue_io: bdi 8:0: older=-1 enqueue=398
       flush-8:0-2939  [004]   318.103976: writeback_queue_io: bdi 8:0: older=-1 enqueue=477
      flush-0:15-2212  [007]   319.764719: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   322.465207: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
      flush-0:15-2212  [007]   322.465247: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
       flush-8:0-2939  [004]   322.467453: writeback_queue_io: bdi 8:0: older=-1 enqueue=13012
       flush-8:0-2939  [004]   322.473793: writeback_queue_io: bdi 8:0: older=-1 enqueue=125
       flush-8:0-2939  [004]   322.480534: writeback_queue_io: bdi 8:0: older=-1 enqueue=221
       flush-8:0-2939  [004]   323.036278: writeback_queue_io: bdi 8:0: older=-1 enqueue=133
       flush-8:0-2939  [004]   323.050850: writeback_queue_io: bdi 8:0: older=-1 enqueue=166
       flush-8:0-2939  [004]   323.219196: writeback_queue_io: bdi 8:0: older=-1 enqueue=171
       flush-8:0-2939  [004]   323.226573: writeback_queue_io: bdi 8:0: older=-1 enqueue=208
       flush-8:0-2939  [004]   323.236098: writeback_queue_io: bdi 8:0: older=-1 enqueue=318
       flush-8:0-2939  [004]   323.401420: writeback_queue_io: bdi 8:0: older=-1 enqueue=246
       flush-8:0-2939  [004]   323.409997: writeback_queue_io: bdi 8:0: older=-1 enqueue=227
       flush-8:0-2939  [004]   323.683992: writeback_queue_io: bdi 8:0: older=-1 enqueue=229
       flush-8:0-2939  [004]   323.692427: writeback_queue_io: bdi 8:0: older=-1 enqueue=301
       flush-8:0-2939  [004]   323.697916: writeback_queue_io: bdi 8:0: older=-1 enqueue=215
       flush-8:0-2939  [004]   324.083378: writeback_queue_io: bdi 8:0: older=-1 enqueue=136
       flush-8:0-2939  [004]   324.089267: writeback_queue_io: bdi 8:0: older=-1 enqueue=172
       flush-8:0-2939  [004]   324.095232: writeback_queue_io: bdi 8:0: older=-1 enqueue=170
       flush-8:0-2939  [004]   324.102113: writeback_queue_io: bdi 8:0: older=-1 enqueue=201
       flush-8:0-2939  [004]   324.456413: writeback_queue_io: bdi 8:0: older=-1 enqueue=231
       flush-8:0-2939  [004]   324.462807: writeback_queue_io: bdi 8:0: older=-1 enqueue=209
       flush-8:0-2939  [004]   324.468340: writeback_queue_io: bdi 8:0: older=-1 enqueue=201
       flush-8:0-2939  [004]   324.743166: writeback_queue_io: bdi 8:0: older=-1 enqueue=271
       flush-8:0-2939  [004]   324.747173: writeback_queue_io: bdi 8:0: older=-1 enqueue=80
       flush-8:0-2939  [004]   324.754360: writeback_queue_io: bdi 8:0: older=-1 enqueue=220
       flush-8:0-2939  [004]   325.056758: writeback_queue_io: bdi 8:0: older=-1 enqueue=212
       flush-8:0-2939  [004]   325.063579: writeback_queue_io: bdi 8:0: older=-1 enqueue=231
       flush-8:0-2939  [004]   325.328492: writeback_queue_io: bdi 8:0: older=-1 enqueue=253
       flush-8:0-2939  [004]   325.333020: writeback_queue_io: bdi 8:0: older=-1 enqueue=152
       flush-8:0-2939  [004]   325.337244: writeback_queue_io: bdi 8:0: older=-1 enqueue=137
       flush-8:0-2939  [004]   325.627000: writeback_queue_io: bdi 8:0: older=-1 enqueue=273
       flush-8:0-2939  [004]   325.631667: writeback_queue_io: bdi 8:0: older=-1 enqueue=196
       flush-8:0-2939  [004]   325.997899: writeback_queue_io: bdi 8:0: older=-1 enqueue=477
       flush-8:0-2939  [004]   326.056377: writeback_queue_io: bdi 8:0: older=-1 enqueue=530
       flush-8:0-2939  [004]   326.179895: writeback_queue_io: bdi 8:0: older=-1 enqueue=2647
       flush-8:0-2939  [004]   326.195636: writeback_queue_io: bdi 8:0: older=-1 enqueue=2245
       flush-8:0-2939  [004]   326.250487: writeback_queue_io: bdi 8:0: older=-1 enqueue=446
       flush-8:0-2939  [004]   326.256807: writeback_queue_io: bdi 8:0: older=-1 enqueue=259
       flush-8:0-2939  [004]   326.438760: writeback_queue_io: bdi 8:0: older=-1 enqueue=787
       flush-8:0-2939  [004]   326.445477: writeback_queue_io: bdi 8:0: older=-1 enqueue=296
       flush-8:0-2939  [004]   326.451228: writeback_queue_io: bdi 8:0: older=-1 enqueue=235
       flush-8:0-2939  [004]   326.458182: writeback_queue_io: bdi 8:0: older=-1 enqueue=697
       flush-8:0-2939  [004]   326.729892: writeback_queue_io: bdi 8:0: older=-1 enqueue=305
       flush-8:0-2939  [004]   326.736946: writeback_queue_io: bdi 8:0: older=-1 enqueue=880
       flush-8:0-2939  [004]   326.795887: writeback_queue_io: bdi 8:0: older=-1 enqueue=379
       flush-8:0-2939  [004]   326.803055: writeback_queue_io: bdi 8:0: older=-1 enqueue=298
       flush-8:0-2939  [004]   326.809735: writeback_queue_io: bdi 8:0: older=-1 enqueue=842
       flush-8:0-2939  [004]   327.170602: writeback_queue_io: bdi 8:0: older=-1 enqueue=152
       flush-8:0-2939  [004]   327.179150: writeback_queue_io: bdi 8:0: older=-1 enqueue=651
       flush-8:0-2939  [004]   327.185108: writeback_queue_io: bdi 8:0: older=-1 enqueue=245
       flush-8:0-2939  [004]   327.450900: writeback_queue_io: bdi 8:0: older=-1 enqueue=539
       flush-8:0-2939  [004]   327.452250: writeback_queue_io: bdi 8:0: older=-1 enqueue=563
      flush-0:15-2212  [007]   327.465044: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
       flush-8:0-2939  [004]   327.813397: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
      flush-0:15-2212  [007]   327.813481: writeback_queue_io: bdi 0:15: older=-1 enqueue=8
       flush-8:0-2939  [004]   327.815030: writeback_queue_io: bdi 8:0: older=-1 enqueue=18709
       flush-8:0-2939  [004]   334.276274: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   339.276480: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   344.276650: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   349.276806: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   354.277001: writeback_queue_io: bdi 8:0: older=30000 enqueue=0
       flush-8:0-2939  [004]   359.277166: writeback_queue_io: bdi 8:0: older=30000 enqueue=22
      flush-0:15-2212  [007]   427.616627: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   432.616749: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   437.616987: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   442.617140: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   447.617320: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   452.617509: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   457.617695: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   462.617864: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   467.618021: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   472.618221: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   477.618426: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   482.618563: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   487.618753: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   492.618929: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   497.619130: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   502.619267: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   507.619428: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   512.619645: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   517.619821: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   522.619969: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   527.620173: writeback_queue_io: bdi 0:15: older=30000 enqueue=1
      flush-0:15-2212  [007]   532.620344: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   537.620528: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   542.620668: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   547.620863: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   552.621062: writeback_queue_io: bdi 0:15: older=30000 enqueue=0
      flush-0:15-2212  [007]   557.621244: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   562.621383: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   567.621558: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   572.621751: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   577.621952: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   582.622086: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   587.622272: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   592.622470: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   597.622641: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   602.622798: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   607.622996: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   612.623159: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   617.623365: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   622.623500: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   627.623714: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   632.623874: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   637.624067: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   642.624216: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   647.624399: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   652.624603: writeback_queue_io: bdi 0:15: older=30000 enqueue=3
      flush-0:15-2212  [007]   657.624764: writeback_queue_io: bdi 0:15: older=30000 enqueue=0

Jan Kara

maintainer

2011-05-05 16:37 UTC #23 lore raw

Re: [PATCH 6/6] writeback: refill b_io iff empty

Quoted 8 lines · Wu Fengguang

To help understand the behavior change, I wrote the writeback_queue_io trace event, and found very different patterns between
- vanilla kernel
- this patchset plus the sync livelock fixes
Basically the vanilla kernel each time pulls a random number of inodes from b_dirty, while the patched kernel tends to pull a fixed number of inodes (enqueue=1031) from b_dirty. The new behavior is very interesting...

This regularity is really strange. Did you have a chance to look more into
it? I find it highly unlikely that there would be exactly 1031 dirty inodes in b_dirty list every time you call move_expired_inodes()...

Honza

Signature · 9 lines

Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wu Fengguang

author

2011-05-05 16:47 UTC #24 lore raw

Re: [PATCH 6/6] writeback: refill b_io iff empty

Quoted 6 lines · Jan Kara

Quoted 5 lines · Wu Fengguang
To help understand the behavior change, I wrote the writeback_queue_io trace event, and found very different patterns between
- vanilla kernel
- this patchset plus the sync livelock fixes

Quoted 6 lines

Quoted 3 lines
Basically the vanilla kernel each time pulls a random number of inodes from b_dirty, while the patched kernel tends to pull a fixed number of inodes (enqueue=1031) from b_dirty. The new behavior is very interesting...
This regularity is really strange. Did you have a chance to look more into
it? I find it highly unlikely that there would be exactly 1031 dirty inodes in b_dirty list every time you call move_expired_inodes()...

Yeah that's the weird point. The other things I noticed are more regular "flusher - dd - flusher - dd - ..." writeout patterns after the patches. In vanilla kernel it behaves more randomly and there are many balance_dirty_pages() IOs from tar.

I'll try to collect more traces in ext4 tomorrow. Sorry it's too late for me now.

Thanks,
Fengguang

Wu Fengguang

author

2011-05-06 05:30 UTC #25 lore raw

Re: [PATCH 6/6] writeback: refill b_io iff empty

In reply to Jan Kara

Quoted 6 lines · Jan Kara

Quoted 5 lines · Wu Fengguang
To help understand the behavior change, I wrote the writeback_queue_io trace event, and found very different patterns between
- vanilla kernel
- this patchset plus the sync livelock fixes

Quoted 6 lines

Quoted 3 lines
Basically the vanilla kernel each time pulls a random number of inodes from b_dirty, while the patched kernel tends to pull a fixed number of inodes (enqueue=1031) from b_dirty. The new behavior is very interesting...
This regularity is really strange. Did you have a chance to look more into
it? I find it highly unlikely that there would be exactly 1031 dirty inodes in b_dirty list every time you call move_expired_inodes()...

Jan, I got some results for ext4. The total dd+tar+sync time is decreased from 177s to 167s. The other numbers are either raised or dropped.

dd-tar-ext4-patched:elapsed: 177.4900000000016
dd-tar-ext4-patched:elapsed: 166.40000000000146
dd-tar-ext4-patched:elapsed: 164.33000000000175
dd-tar-ext4-patched:elapsed: 176.36999999999898
dd-tar-ext4-patched:elapsed: 152.84000000000015

avg 167.486
stddev 8.996

dd-tar-ext4-vanilla:elapsed: 172.08999999999924
dd-tar-ext4-vanilla:elapsed: 170.50999999999931
dd-tar-ext4-vanilla:elapsed: 183.61999999999989
dd-tar-ext4-vanilla:elapsed: 187.69999999999982
dd-tar-ext4-vanilla:elapsed: 174.8100000000004

avg 177.746
stddev 6.731

dd-tar-ext4-patched:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.66s system 9% cpu 16.922 total
dd-tar-ext4-patched:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.70s system 8% cpu 19.681 total
dd-tar-ext4-patched:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.69s system 8% cpu 19.041 total
dd-tar-ext4-patched:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.69s system 9% cpu 18.632 total
dd-tar-ext4-patched:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.70s system 9% cpu 18.478 total

avg          0.000      1.688      8.600     18.551
stddev       0.000      0.015      0.490      0.915

dd-tar-ext4-vanilla:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.67s system 8% cpu 19.658 total
dd-tar-ext4-vanilla:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.67s system 8% cpu 18.663 total
dd-tar-ext4-vanilla:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.70s system 8% cpu 19.970 total
dd-tar-ext4-vanilla:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.66s system 7% cpu 20.985 total
dd-tar-ext4-vanilla:dd if=/dev/zero of=/fs/zero bs=1M count=1000  0.00s user 1.66s system 8% cpu 19.806 total

avg          0.000      1.672      7.800     19.816
stddev       0.000      0.015      0.400      0.741

dd-tar-ext4-patched:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.08s user 3.51s system 71% cpu 21.677 total
dd-tar-ext4-patched:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.07s user 3.73s system 80% cpu 19.679 total
dd-tar-ext4-patched:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.43s user 3.09s system 79% cpu 19.444 total
dd-tar-ext4-patched:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.52s user 3.14s system 80% cpu 19.510 total
dd-tar-ext4-patched:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.38s user 3.15s system 92% cpu 16.701 total

avg 12.296 3.324 80.400 19.402
stddev 0.186 0.252 6.711 1.585

dd-tar-ext4-vanilla:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.34s user 3.26s system 96% cpu 16.155 total
dd-tar-ext4-vanilla:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.70s user 3.52s system 97% cpu 16.694 total
dd-tar-ext4-vanilla:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.47s user 3.31s system 92% cpu 17.017 total
dd-tar-ext4-vanilla:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.36s user 3.19s system 86% cpu 17.885 total
dd-tar-ext4-vanilla:tar jxf /dev/shm/linux-2.6.38.3.tar.bz2  12.46s user 3.17s system 101% cpu 15.382 total

avg 12.466 3.290 94.400 16.627
stddev 0.128 0.125 5.083 0.838

patched trace-tar-dd-ext4-2.6.39-rc3+

       flush-8:0-3048  [000]  1902.672610: writeback_queue_io: bdi 8:0: older=4296543506 age=30000 enqueue=0
       flush-8:0-3048  [000]  1905.209570: writeback_queue_io: bdi 8:0: older=4296546051 age=30000 enqueue=0
       flush-8:0-3048  [000]  1907.294936: writeback_queue_io: bdi 8:0: older=4296548143 age=30000 enqueue=0
       flush-8:0-3048  [004]  1909.607301: writeback_queue_io: bdi 8:0: older=4296550462 age=30000 enqueue=0
       flush-8:0-3048  [003]  1912.290627: writeback_queue_io: bdi 8:0: older=4296553154 age=30000 enqueue=0
       flush-8:0-3048  [001]  1914.331197: writeback_queue_io: bdi 8:0: older=4296555201 age=30000 enqueue=0
      flush-0:15-2283  [002]  1924.729828: writeback_queue_io: bdi 0:15: older=4296565632 age=30000 enqueue=1
       flush-8:0-3048  [000]  1927.275838: writeback_queue_io: bdi 8:0: older=4296568186 age=30000 enqueue=0
       flush-8:0-3048  [000]  1927.277794: writeback_queue_io: bdi 8:0: older=4296568188 age=30000 enqueue=0
       flush-8:0-3048  [000]  1927.279504: writeback_queue_io: bdi 8:0: older=4296568189 age=30000 enqueue=0
       flush-8:0-3048  [000]  1927.279923: writeback_queue_io: bdi 8:0: older=4296568190 age=30000 enqueue=0
      flush-0:15-2283  [002]  1929.714335: writeback_queue_io: bdi 0:15: older=4296570632 age=30000 enqueue=2
      flush-0:15-2283  [002]  1929.979616: writeback_queue_io: bdi 0:15: older=4296600898 age=0 enqueue=10
      flush-0:15-2283  [002]  1929.979675: writeback_queue_io: bdi 0:15: older=4296600898 age=0 enqueue=10
       flush-8:0-3048  [004]  1929.981734: writeback_queue_io: bdi 8:0: older=4296600898 age=2 enqueue=13227
       flush-8:0-3048  [000]  1932.840150: writeback_queue_io: bdi 8:0: older=4296600898 age=2869 enqueue=0
       flush-8:0-3048  [000]  1932.840781: writeback_queue_io: bdi 8:0: older=4296603768 age=0 enqueue=0
       flush-8:0-3048  [000]  1932.840787: writeback_queue_io: bdi 8:0: older=4296573768 age=30000 enqueue=0
      flush-0:15-2283  [002]  1932.991221: writeback_queue_io: bdi 0:15: older=4296603919 age=0 enqueue=10
      flush-0:15-2283  [002]  1932.991282: writeback_queue_io: bdi 0:15: older=4296603919 age=0 enqueue=0
       flush-8:0-3048  [004]  1932.991596: writeback_queue_io: bdi 8:0: older=4296603919 age=0 enqueue=1
       flush-8:0-3048  [004]  1937.975765: writeback_queue_io: bdi 8:0: older=4296578919 age=30000 enqueue=0
       flush-8:0-3048  [004]  1942.960305: writeback_queue_io: bdi 8:0: older=4296583919 age=30000 enqueue=0
       flush-8:0-3048  [004]  1947.944925: writeback_queue_io: bdi 8:0: older=4296588919 age=30000 enqueue=0
       flush-8:0-3048  [004]  1952.929427: writeback_queue_io: bdi 8:0: older=4296593919 age=30000 enqueue=0
       flush-8:0-3048  [004]  1957.914031: writeback_queue_io: bdi 8:0: older=4296598919 age=30000 enqueue=0
       flush-8:0-3048  [004]  1962.898507: writeback_queue_io: bdi 8:0: older=4296603919 age=30000 enqueue=1
       flush-8:0-3048  [004]  1962.898518: writeback_queue_io: bdi 8:0: older=4296603919 age=30000 enqueue=0

vanilla trace-tar-dd-ext4-2.6.39-rc3

       flush-8:0-2911  [002]    53.756624: writeback_queue_io: bdi 8:0: older=4294690962 age=30000 enqueue=1
       flush-8:0-2911  [002]    53.756738: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=1184
      flush-0:15-2214  [001]    56.960369: writeback_queue_io: bdi 0:15: older=4294694176 age=30000 enqueue=0
      flush-0:15-2214  [001]    56.960373: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
       flush-8:0-2911  [004]    57.447062: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=4479
       flush-8:0-2911  [004]    57.482760: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=58
       flush-8:0-2911  [004]    57.492041: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=12
       flush-8:0-2911  [004]    57.613401: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=198
       flush-8:0-2911  [004]    61.569241: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=229
       flush-8:0-2911  [004]    61.589268: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=141
       flush-8:0-2911  [004]    61.607471: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=209
       flush-8:0-2911  [004]    61.619094: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [004]    61.629282: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
      flush-0:15-2214  [001]    61.945027: writeback_queue_io: bdi 0:15: older=4294699176 age=30000 enqueue=0
      flush-0:15-2214  [001]    61.945032: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
       flush-8:0-2911  [001]    63.715007: writeback_queue_io: bdi 8:0: older=4294698964 age=31987 enqueue=0
       flush-8:0-2911  [000]    66.469285: writeback_queue_io: bdi 8:0: older=4294698964 age=34750 enqueue=0
      flush-0:15-2214  [005]    66.929422: writeback_queue_io: bdi 0:15: older=4294704176 age=30000 enqueue=0
      flush-0:15-2214  [005]    66.929428: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
       flush-8:0-2911  [000]    68.377843: writeback_queue_io: bdi 8:0: older=4294698964 age=36664 enqueue=0
      flush-0:15-2214  [005]    71.913983: writeback_queue_io: bdi 0:15: older=4294709176 age=30000 enqueue=1
       flush-8:0-2911  [004]    74.365281: writeback_queue_io: bdi 8:0: older=4294711634 age=30000 enqueue=0
       flush-8:0-2911  [000]    74.821770: writeback_queue_io: bdi 8:0: older=4294711634 age=30458 enqueue=0
      flush-0:15-2214  [005]    76.898540: writeback_queue_io: bdi 0:15: older=4294714176 age=30000 enqueue=5
       flush-8:0-2911  [004]    77.158312: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=18938
       flush-8:0-2911  [004]    77.235169: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=185
       flush-8:0-2911  [004]    77.388776: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=455
       flush-8:0-2911  [004]    77.462484: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=187
       flush-8:0-2911  [004]    77.541086: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=97
       flush-8:0-2911  [004]    77.619482: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=126
       flush-8:0-2911  [004]    77.810862: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=265
       flush-8:0-2911  [006]    77.924553: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=209
       flush-8:0-2911  [003]    78.034276: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=234
       flush-8:0-2911  [003]    78.108989: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=148
       flush-8:0-2911  [003]    78.179240: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=162
       flush-8:0-2911  [003]    78.348585: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=406
       flush-8:0-2911  [003]    78.476980: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=36
       flush-8:0-2911  [003]    78.580621: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=144
       flush-8:0-2911  [003]    78.754257: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=366
       flush-8:0-2911  [003]    78.858451: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=184
       flush-8:0-2911  [003]    78.945059: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=228
       flush-8:0-2911  [003]    79.075494: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=267
       flush-8:0-2911  [003]    79.152004: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=56
       flush-8:0-2911  [003]    79.238366: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=145
      flush-0:15-2214  [005]    81.883169: writeback_queue_io: bdi 0:15: older=4294719176 age=30000 enqueue=1
      flush-0:15-2214  [005]    82.457961: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
      flush-0:15-2214  [005]    82.457985: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
       flush-8:0-2911  [000]    82.461064: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=6957
       flush-8:0-2911  [000]    82.465738: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.084199: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.091283: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.097138: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.101489: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.105795: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.111433: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.113795: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.118674: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.123061: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.127796: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.318828: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.335335: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.513142: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.671440: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.727269: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.800181: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.827915: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.890632: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.930052: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.950224: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    83.980423: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.022254: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.140787: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.312729: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.379059: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.420117: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.473221: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.501959: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.589525: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.696628: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.699403: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.792484: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    84.920198: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    85.003982: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    85.006321: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    85.008473: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    85.009997: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=0
       flush-8:0-2911  [000]    85.010003: writeback_queue_io: bdi 8:0: older=4294722312 age=30000 enqueue=0
      flush-0:15-2214  [005]    85.245018: writeback_queue_io: bdi 0:15: older=0 age=-1 enqueue=7
       flush-8:0-2911  [000]    85.245369: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=1
       flush-8:0-2911  [000]    90.229292: writeback_queue_io: bdi 8:0: older=4294727548 age=30000 enqueue=0
       flush-8:0-2911  [000]    95.213838: writeback_queue_io: bdi 8:0: older=4294732548 age=30000 enqueue=0
      flush-0:15-2214  [005]    98.671161: writeback_queue_io: bdi 0:15: older=4294736016 age=30000 enqueue=0
       flush-8:0-2911  [000]   100.198391: writeback_queue_io: bdi 8:0: older=4294737548 age=30000 enqueue=0
      flush-0:15-2214  [005]   103.655662: writeback_queue_io: bdi 0:15: older=4294741016 age=30000 enqueue=0
       flush-8:0-2911  [000]   105.182941: writeback_queue_io: bdi 8:0: older=4294742548 age=30000 enqueue=1
      flush-0:15-2214  [005]   108.640284: writeback_queue_io: bdi 0:15: older=4294746016 age=30000 enqueue=0
      flush-0:15-2214  [005]   113.624829: writeback_queue_io: bdi 0:15: older=4294751016 age=30000 enqueue=0
      flush-0:15-2214  [005]   118.609396: writeback_queue_io: bdi 0:15: older=4294756016 age=30000 enqueue=0
      flush-0:15-2214  [005]   123.593932: writeback_queue_io: bdi 0:15: older=4294761016 age=30000 enqueue=1
      flush-0:15-2214  [005]   128.578494: writeback_queue_io: bdi 0:15: older=4294766016 age=30000 enqueue=0
      flush-0:15-2214  [005]   133.563026: writeback_queue_io: bdi 0:15: older=4294771016 age=30000 enqueue=0
      flush-0:15-2214  [005]   138.547592: writeback_queue_io: bdi 0:15: older=4294776016 age=30000 enqueue=0
      flush-0:15-2214  [005]   143.532126: writeback_queue_io: bdi 0:15: older=4294781016 age=30000 enqueue=0
      flush-0:15-2214  [005]   148.516696: writeback_queue_io: bdi 0:15: older=4294786016 age=30000 enqueue=2
      flush-0:15-2214  [005]   153.501234: writeback_queue_io: bdi 0:15: older=4294791016 age=30000 enqueue=1
      flush-0:15-2214  [005]   158.485800: writeback_queue_io: bdi 0:15: older=4294796016 age=30000 enqueue=0
      flush-0:15-2214  [005]   163.470348: writeback_queue_io: bdi 0:15: older=4294801016 age=30000 enqueue=0
      flush-0:15-2214  [005]   168.454905: writeback_queue_io: bdi 0:15: older=4294806016 age=30000 enqueue=0
      flush-0:15-2214  [005]   173.439445: writeback_queue_io: bdi 0:15: older=4294811016 age=30000 enqueue=0
      flush-0:15-2214  [005]   178.424001: writeback_queue_io: bdi 0:15: older=4294816016 age=30000 enqueue=2
      flush-0:15-2214  [005]   183.408546: writeback_queue_io: bdi 0:15: older=4294821016 age=30000 enqueue=1
      flush-0:15-2214  [005]   188.393102: writeback_queue_io: bdi 0:15: older=4294826016 age=30000 enqueue=0
      flush-0:15-2214  [005]   193.377653: writeback_queue_io: bdi 0:15: older=4294831016 age=30000 enqueue=0
      flush-0:15-2214  [005]   198.362203: writeback_queue_io: bdi 0:15: older=4294836016 age=30000 enqueue=0
      flush-0:15-2214  [005]   203.346749: writeback_queue_io: bdi 0:15: older=4294841016 age=30000 enqueue=0
      flush-0:15-2214  [005]   208.331306: writeback_queue_io: bdi 0:15: older=4294846016 age=30000 enqueue=2
      flush-0:15-2214  [005]   213.315844: writeback_queue_io: bdi 0:15: older=4294851016 age=30000 enqueue=1
      flush-0:15-2214  [005]   218.300404: writeback_queue_io: bdi 0:15: older=4294856016 age=30000 enqueue=0
      flush-0:15-2214  [005]   223.284955: writeback_queue_io: bdi 0:15: older=4294861016 age=30000 enqueue=0
      flush-0:15-2214  [005]   228.269515: writeback_queue_io: bdi 0:15: older=4294866016 age=30000 enqueue=0
      flush-0:15-2214  [005]   233.254057: writeback_queue_io: bdi 0:15: older=4294871016 age=30000 enqueue=0
      flush-0:15-2214  [005]   238.238619: writeback_queue_io: bdi 0:15: older=4294876016 age=30000 enqueue=2
      flush-0:15-2214  [005]   243.223157: writeback_queue_io: bdi 0:15: older=4294881016 age=30000 enqueue=1
      flush-0:15-2214  [005]   248.207715: writeback_queue_io: bdi 0:15: older=4294886016 age=30000 enqueue=0
      flush-0:15-2214  [005]   253.192251: writeback_queue_io: bdi 0:15: older=4294891016 age=30000 enqueue=0
      flush-0:15-2214  [005]   258.176824: writeback_queue_io: bdi 0:15: older=4294896016 age=30000 enqueue=0
      flush-0:15-2214  [005]   263.161348: writeback_queue_io: bdi 0:15: older=4294901016 age=30000 enqueue=0
      flush-0:15-2214  [005]   268.145922: writeback_queue_io: bdi 0:15: older=4294906016 age=30000 enqueue=2
      flush-0:15-2214  [005]   273.130463: writeback_queue_io: bdi 0:15: older=4294911016 age=30000 enqueue=1
      flush-0:15-2214  [005]   278.115008: writeback_queue_io: bdi 0:15: older=4294916016 age=30000 enqueue=0
      flush-0:15-2214  [005]   283.099572: writeback_queue_io: bdi 0:15: older=4294921016 age=30000 enqueue=0
      flush-0:15-2214  [005]   288.084050: writeback_queue_io: bdi 0:15: older=4294926016 age=30000 enqueue=0
      flush-0:15-2214  [005]   293.068668: writeback_queue_io: bdi 0:15: older=4294931016 age=30000 enqueue=0
      flush-0:15-2214  [005]   298.053228: writeback_queue_io: bdi 0:15: older=4294936016 age=30000 enqueue=2
      flush-0:15-2214  [005]   303.037756: writeback_queue_io: bdi 0:15: older=4294941016 age=30000 enqueue=1
      flush-0:15-2214  [005]   308.022329: writeback_queue_io: bdi 0:15: older=4294946016 age=30000 enqueue=0
      flush-0:15-2214  [005]   313.006864: writeback_queue_io: bdi 0:15: older=4294951016 age=30000 enqueue=0
      flush-0:15-2214  [005]   317.991431: writeback_queue_io: bdi 0:15: older=4294956016 age=30000 enqueue=0
      flush-0:15-2214  [005]   322.975973: writeback_queue_io: bdi 0:15: older=4294961016 age=30000 enqueue=0
      flush-0:15-2214  [005]   327.960530: writeback_queue_io: bdi 0:15: older=4294966016 age=30000 enqueue=2
      flush-0:15-2214  [005]   332.945074: writeback_queue_io: bdi 0:15: older=4294971016 age=30000 enqueue=1
      flush-0:15-2214  [005]   337.929629: writeback_queue_io: bdi 0:15: older=4294976016 age=30000 enqueue=0
      flush-0:15-2214  [005]   342.914183: writeback_queue_io: bdi 0:15: older=4294981016 age=30000 enqueue=0
      flush-0:15-2214  [005]   347.898736: writeback_queue_io: bdi 0:15: older=4294986016 age=30000 enqueue=0
      flush-0:15-2214  [005]   352.883285: writeback_queue_io: bdi 0:15: older=4294991016 age=30000 enqueue=0
      flush-0:15-2214  [005]   357.867829: writeback_queue_io: bdi 0:15: older=4294996016 age=30000 enqueue=2
      flush-0:15-2214  [005]   362.852371: writeback_queue_io: bdi 0:15: older=4295001016 age=30000 enqueue=1
      flush-0:15-2214  [005]   367.836939: writeback_queue_io: bdi 0:15: older=4295006016 age=30000 enqueue=0
      flush-0:15-2214  [005]   372.821491: writeback_queue_io: bdi 0:15: older=4295011016 age=30000 enqueue=0
      flush-0:15-2214  [005]   377.806047: writeback_queue_io: bdi 0:15: older=4295016016 age=30000 enqueue=0
      flush-0:15-2214  [005]   382.790584: writeback_queue_io: bdi 0:15: older=4295021016 age=30000 enqueue=0
      flush-0:15-2214  [005]   387.775142: writeback_queue_io: bdi 0:15: older=4295026016 age=30000 enqueue=2
      flush-0:15-2214  [005]   392.759681: writeback_queue_io: bdi 0:15: older=4295031016 age=30000 enqueue=1
      flush-0:15-2214  [005]   397.744251: writeback_queue_io: bdi 0:15: older=4295036016 age=30000 enqueue=0
      flush-0:15-2214  [005]   402.728786: writeback_queue_io: bdi 0:15: older=4295041016 age=30000 enqueue=0
      flush-0:15-2214  [005]   407.713347: writeback_queue_io: bdi 0:15: older=4295046016 age=30000 enqueue=0
      flush-0:15-2214  [005]   412.697887: writeback_queue_io: bdi 0:15: older=4295051016 age=30000 enqueue=0
      flush-0:15-2214  [005]   417.682447: writeback_queue_io: bdi 0:15: older=4295056016 age=30000 enqueue=2
      flush-0:15-2214  [005]   422.666987: writeback_queue_io: bdi 0:15: older=4295061016 age=30000 enqueue=1
      flush-0:15-2214  [005]   427.651550: writeback_queue_io: bdi 0:15: older=4295066016 age=30000 enqueue=0
      flush-0:15-2214  [005]   432.636088: writeback_queue_io: bdi 0:15: older=4295071016 age=30000 enqueue=0
      flush-0:15-2214  [005]   437.620649: writeback_queue_io: bdi 0:15: older=4295076016 age=30000 enqueue=0
      flush-0:15-2214  [005]   442.605199: writeback_queue_io: bdi 0:15: older=4295081016 age=30000 enqueue=0
      flush-0:15-2214  [005]   447.589750: writeback_queue_io: bdi 0:15: older=4295086016 age=30000 enqueue=2
      flush-0:15-2214  [005]   452.574296: writeback_queue_io: bdi 0:15: older=4295091016 age=30000 enqueue=1
      flush-0:15-2214  [005]   457.558861: writeback_queue_io: bdi 0:15: older=4295096016 age=30000 enqueue=0
      flush-0:15-2214  [005]   462.543399: writeback_queue_io: bdi 0:15: older=4295101016 age=30000 enqueue=0
      flush-0:15-2214  [005]   467.527961: writeback_queue_io: bdi 0:15: older=4295106016 age=30000 enqueue=0
      flush-0:15-2214  [005]   472.512492: writeback_queue_io: bdi 0:15: older=4295111016 age=30000 enqueue=0
      flush-0:15-2214  [005]   477.497055: writeback_queue_io: bdi 0:15: older=4295116016 age=30000 enqueue=2
      flush-0:15-2214  [005]   482.481630: writeback_queue_io: bdi 0:15: older=4295121016 age=30000 enqueue=1
      flush-0:15-2214  [005]   487.466162: writeback_queue_io: bdi 0:15: older=4295126016 age=30000 enqueue=0
      flush-0:15-2214  [005]   492.450709: writeback_queue_io: bdi 0:15: older=4295131016 age=30000 enqueue=0
      flush-0:15-2214  [005]   497.435268: writeback_queue_io: bdi 0:15: older=4295136016 age=30000 enqueue=0
      flush-0:15-2214  [005]   502.419812: writeback_queue_io: bdi 0:15: older=4295141016 age=30000 enqueue=0
      flush-0:15-2214  [005]   507.404385: writeback_queue_io: bdi 0:15: older=4295146016 age=30000 enqueue=2
      flush-0:15-2214  [005]   512.388906: writeback_queue_io: bdi 0:15: older=4295151016 age=30000 enqueue=1
      flush-0:15-2214  [005]   517.373458: writeback_queue_io: bdi 0:15: older=4295156016 age=30000 enqueue=0
      flush-0:15-2214  [005]   522.358005: writeback_queue_io: bdi 0:15: older=4295161016 age=30000 enqueue=0
      flush-0:15-2214  [005]   527.342587: writeback_queue_io: bdi 0:15: older=4295166016 age=30000 enqueue=0
      flush-0:15-2214  [005]   532.327116: writeback_queue_io: bdi 0:15: older=4295171016 age=30000 enqueue=0

Thanks,
Fengguang

Wu Fengguang

author

2011-05-06 08:42 UTC #26 lore raw

RFC PATCH 1/1 writeback: limit number of moved inodes in queue_io()

Quoted 9 lines

patched trace-tar-dd-ext4-2.6.39-rc3+

       flush-8:0-3048  [004]  1929.981734: writeback_queue_io: bdi 8:0: older=4296600898 age=2 enqueue=13227

vanilla trace-tar-dd-ext4-2.6.39-rc3

       flush-8:0-2911  [004]    77.158312: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=18938

       flush-8:0-2911  [000]    82.461064: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=6957

It looks too much to move 13227 and 18938 inodes at once. So I tried arbitrarily limiting the max move number to 1000 and it helps reduce the lock hold time and contentions a lot.

---
Subject: writeback: limit number of moved inodes in queue_io()
Date: Fri May 06 13:34:08 CST 2011

Only move 1000 inodes from b_dirty to b_io at one time. This reduces lock hold time and lock contentions by many times in a simple dd+tar workload in a 8p test box. This workload was observed to move 10000+ inodes in one shot on ext4 which was obviously too much.

class name con-bounces contentions waittime-min waittime-max waittime-total acq-b
ounces acquisitions holdtime-min holdtime-max holdtime-total
----------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------

                      inode_wb_list_lock:          2063           2065           0.12        2648.66        5948.99
 27475         943778           0.09        2704.76      498340.24
                      ------------------
                      inode_wb_list_lock             89          [<ffffffff8115cf3a>] sync_inode+0x28/0x5f
                      inode_wb_list_lock             38          [<ffffffff8115ccab>] inode_wait_for_writeback+0xa8/0xc6
                      inode_wb_list_lock            629          [<ffffffff8115da35>] __mark_inode_dirty+0x170/0x1d0
                      inode_wb_list_lock            842          [<ffffffff8115d334>] writeback_sb_inodes+0x10f/0x157
                      ------------------
                      inode_wb_list_lock            891          [<ffffffff8115ce3e>] writeback_single_inode+0x175/0x249
                      inode_wb_list_lock             13          [<ffffffff8115dc4e>] writeback_inodes_wb+0x3a/0x143
                      inode_wb_list_lock            499          [<ffffffff8115da35>] __mark_inode_dirty+0x170/0x1d0
                      inode_wb_list_lock            617          [<ffffffff8115d334>] writeback_sb_inodes+0x10f/0x157


                &(&wb->list_lock)->rlock:           842            842           0.14         101.10        1013.34
 20489         970892           0.09         234.11      509829.79
                ------------------------
                &(&wb->list_lock)->rlock            275          [<ffffffff8115db09>] __mark_inode_dirty+0x173/0x1cf
                &(&wb->list_lock)->rlock            114          [<ffffffff8115cdd3>] writeback_single_inode+0x18a/0x27e
                &(&wb->list_lock)->rlock             56          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                &(&wb->list_lock)->rlock            132          [<ffffffff8115cf2a>] sync_inode+0x63/0xa2
                ------------------------
                &(&wb->list_lock)->rlock              2          [<ffffffff8115dfea>] inode_wb_list_del+0x5f/0x85
                &(&wb->list_lock)->rlock             33          [<ffffffff8115cf2a>] sync_inode+0x63/0xa2
                &(&wb->list_lock)->rlock              9          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                &(&wb->list_lock)->rlock            430          [<ffffffff8115cdd3>] writeback_single_inode+0x18a/0x27e

---
 fs/fs-writeback.c |    2 ++
 1 file changed, 2 insertions(+)

fs/fs-writeback.c +2 -0

@@ -279,6 +279,8 @@ static int move_expired_inodes(struct li
 		sb = inode->i_sb;
 		list_move(&inode->i_wb_list, &tmp);
 		moved++;
+		if (unlikely(moved >= 1000))	/* limit spinlock hold time */
+			break;
 	}
 
 	/* just one sb in list, splice to dispatch_queue and we're done */

Wu Fengguang

author

2011-05-06 10:06 UTC #27 lore raw

RFC PATCH v2 1/1 writeback: limit number of moved inodes in queue_io()

Quoted 6 lines · Wu Fengguang

Quoted 5 lines
patched trace-tar-dd-ext4-2.6.39-rc3+
       flush-8:0-3048  [004]  1929.981734: writeback_queue_io: bdi 8:0: older=4296600898 age=2 enqueue=13227
vanilla trace-tar-dd-ext4-2.6.39-rc3

Quoted 7 lines

Quoted 3 lines
       flush-8:0-2911  [004]    77.158312: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=18938

       flush-8:0-2911  [000]    82.461064: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=6957
It looks too much to move 13227 and 18938 inodes at once. So I tried arbitrarily limiting the max move number to 1000 and it helps reduce the lock hold time and contentions a lot.

Oh it seems 1000 is too small at least for this workload, it hurts dd+tar+sync total elapsed time.

no limit:

                avg        167.486 
                stddev       8.996 
limit=1000:
                avg        171.222 
                stddev       5.588 
limit=3000:
                avg        165.335 
                stddev       5.503

So use 3000 as the new limit.

Thanks,
Fengguang
---
Subject: writeback: limit number of moved inodes in queue_io()
Date: Fri May 06 13:34:08 CST 2011

Only move 3000 inodes from b_dirty to b_io at one time. This reduces lock max hold time and lock contentions by many times in a simple dd+tar workload in a 8p test box. This workload was observed to move 10000+ inodes in one shot on ext4 which was obviously too much.

                      inode_wb_list_lock:          2063           2065           0.12        2648.66        5948.99
 27475         943778           0.09        2704.76      498340.24
                      ------------------
                      inode_wb_list_lock             89          [<ffffffff8115cf3a>] sync_inode+0x28/0x5f
                      inode_wb_list_lock             38          [<ffffffff8115ccab>] inode_wait_for_writeback+0xa8/0xc6
                      inode_wb_list_lock            629          [<ffffffff8115da35>] __mark_inode_dirty+0x170/0x1d0
                      inode_wb_list_lock            842          [<ffffffff8115d334>] writeback_sb_inodes+0x10f/0x157
                      ------------------
                      inode_wb_list_lock            891          [<ffffffff8115ce3e>] writeback_single_inode+0x175/0x249
                      inode_wb_list_lock             13          [<ffffffff8115dc4e>] writeback_inodes_wb+0x3a/0x143
                      inode_wb_list_lock            499          [<ffffffff8115da35>] __mark_inode_dirty+0x170/0x1d0
                      inode_wb_list_lock            617          [<ffffffff8115d334>] writeback_sb_inodes+0x10f/0x157

limit=1000:

dd+tar+sync total elapsed time (10 runs):

				avg        171.222 
				stddev       5.588 

                &(&wb->list_lock)->rlock:           842            842           0.14         101.10        1013.34
 20489         970892           0.09         234.11      509829.79
                ------------------------
                &(&wb->list_lock)->rlock            275          [<ffffffff8115db09>] __mark_inode_dirty+0x173/0x1cf
                &(&wb->list_lock)->rlock            114          [<ffffffff8115cdd3>] writeback_single_inode+0x18a/0x27e
                &(&wb->list_lock)->rlock             56          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                &(&wb->list_lock)->rlock            132          [<ffffffff8115cf2a>] sync_inode+0x63/0xa2
                ------------------------
                &(&wb->list_lock)->rlock              2          [<ffffffff8115dfea>] inode_wb_list_del+0x5f/0x85
                &(&wb->list_lock)->rlock             33          [<ffffffff8115cf2a>] sync_inode+0x63/0xa2
                &(&wb->list_lock)->rlock              9          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                &(&wb->list_lock)->rlock            430          [<ffffffff8115cdd3>] writeback_single_inode+0x18a/0x27e

limit=3000:

dd+tar+sync total elapsed time (10 runs):

				avg        165.335
				stddev       5.503

                &(&wb->list_lock)->rlock:          1088           1092           0.11         245.08        3268.75
 21124        1718636           0.09         384.53      849827.20
                ------------------------
                &(&wb->list_lock)->rlock            518          [<ffffffff8115db09>] __mark_inode_dirty+0x173/0x1cf
                &(&wb->list_lock)->rlock              3          [<ffffffff8115dfea>] inode_wb_list_del+0x5f/0x85
                &(&wb->list_lock)->rlock             54          [<ffffffff8115cf2a>] sync_inode+0x63/0xa2
                &(&wb->list_lock)->rlock             10          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                ------------------------
                &(&wb->list_lock)->rlock              4          [<ffffffff8115dfea>] inode_wb_list_del+0x5f/0x85
                &(&wb->list_lock)->rlock            379          [<ffffffff8115db09>] __mark_inode_dirty+0x173/0x1cf
                &(&wb->list_lock)->rlock              4          [<ffffffff8115cc29>] inode_wait_for_writeback+0xac/0xcc
                &(&wb->list_lock)->rlock            446          [<ffffffff8115cdd3>] writeback_single_inode+0x18a/0x27e

---
 fs/fs-writeback.c |    2 ++
 1 file changed, 2 insertions(+)

fs/fs-writeback.c +2 -0

@@ -279,6 +279,8 @@ static int move_expired_inodes(struct li
 		sb = inode->i_sb;
 		list_move(&inode->i_wb_list, &tmp);
 		moved++;
+		if (unlikely(moved >= 3000))	/* limit spinlock hold time */
+			break;
 	}
 
 	/* just one sb in list, splice to dispatch_queue and we're done */

Dave Chinner

2011-05-06 23:06 UTC #28 lore raw

Re: [RFC][PATCH v2] writeback: limit number of moved inodes in queue_io()

Quoted 28 lines · Wu Fengguang

Quoted 6 lines · Wu Fengguang
Quoted 5 lines
patched trace-tar-dd-ext4-2.6.39-rc3+
       flush-8:0-3048  [004]  1929.981734: writeback_queue_io: bdi 8:0: older=4296600898 age=2 enqueue=13227
vanilla trace-tar-dd-ext4-2.6.39-rc3
Quoted 7 lines
Quoted 3 lines
       flush-8:0-2911  [004]    77.158312: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=18938

       flush-8:0-2911  [000]    82.461064: writeback_queue_io: bdi 8:0: older=0 age=-1 enqueue=6957
It looks too much to move 13227 and 18938 inodes at once. So I tried arbitrarily limiting the max move number to 1000 and it helps reduce the lock hold time and contentions a lot.
Oh it seems 1000 is too small at least for this workload, it hurts dd+tar+sync total elapsed time.
no limit:
                avg        167.486 
                stddev       8.996 
limit=1000:
                avg        171.222 
                stddev       5.588 
limit=3000:
                avg        165.335 
                stddev       5.503 

Quoted 1 line

So use 3000 as the new limit.

I don't think that's even enough. The number is going to be workload dependent and while a limit might be a good idea, I don't think it can be chosen just from one simple benchmark. e.g. what does it to do performance of workloads creating tens of thousands of small dirty files a second?

....

Quoted 7 lines

class name con-bounces contentions waittime-min waittime-max waittime-total acq-b
ounces acquisitions holdtime-min holdtime-max holdtime-total
----------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------
vanilla 2.6.39-rc3:
                      inode_wb_list_lock:          2063           2065           0.12        2648.66        5948.99
27475 943778 0.09 2704.76 498340.24

I wouldn't consider this a contended lock at all on this workload.

FWIW, my profiles on sustained 8-way small file creation workloads on ext4 over tens of millions of inodes show a 0.1% contention rate for the inode_wb_list_lock. That compares to a 2% contention rate for the inode_lru_lock, a 4% contention rate on the inode_sb_list_lock and a 6% contention rate on the inode_hash_lock. So really, the inode_wb_list_lock is not the lock we need to spend effort on optimising to the nth degree right now...

......

Quoted 8 lines

limit=1000:
dd+tar+sync total elapsed time (10 runs):
				avg        171.222 
				stddev       5.588 

                &(&wb->list_lock)->rlock:           842            842           0.14         101.10        1013.34
20489 970892 0.09 234.11 509829.79

.....

Quoted 8 lines

limit=3000:
dd+tar+sync total elapsed time (10 runs):
				avg        165.335
				stddev       5.503

                &(&wb->list_lock)->rlock:          1088           1092           0.11         245.08        3268.75
21124 1718636 0.09 384.53 849827.20

So, from this acquisitions are doubled, and the total lock hold time has almost doubled as well. That seems like there's a fair bit of inefficiency introduced. What does it do to the CPU time consumed by queue_io() (perf top is your friend)?

FYI, queue_io() is already a _massive_ CPU hog. See commit dcd79a1 ("xfs: don't use vfs writeback for pure metadata modifications") for how XFS tries to avoid putting dirty inodes on the list if at all possible:

Under heavy multi-way parallel create workloads, the VFS struggles to write back all the inodes that have been changed in age order. The bdi flusher thread becomes CPU bound, spending 85% of it's time in the VFS code, mostly traversing the superblock dirty inode list to separate dirty inodes old enough to flush.

We already keep an index of all metadata changes in age order - in the AIL - and continued log pressure will do age ordered writeback without any extra overhead at all. If there is no pressure on the log, the xfssyncd will periodically write back metadata in ascending disk address offset order so will be very efficient.
.....

We're moving towards only tracking inodes with dirty pages in the b_dirty list for XFS because this time based expiry is so inefficient. So anything that reduces the efficiency of queue_io()....

Cheers,

Dave.

Signature · 9 lines

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jan Kara

maintainer

2011-05-06 14:22 UTC #29 lore raw

Re: [PATCH 6/6] writeback: refill b_io iff empty