Skip to contentSkip to conversation

some writeback experiments

PATCH In review Wu Fengguang 12 participants 117 messages Oct 7, 2009 – Jul 12, 2010

Subsystems (22)
lore
Share & tools
lore
#0 lore raw

cover some writeback experiments

Hi all,

Here is a collection of writeback patches on

- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes

Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..

The lumpy pageout and nr_segments support is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.

I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!

The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..

NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:

# vmmon -d 1 nr_writeback nr_dirty nr_unstable      # (per 1-second samples)
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913

btrfs seems to maintain a private pool of writeback pages, which can go out of control:

     nr_writeback         nr_dirty
           261075              132
           252891              195
           244795              187
           236851              187
           228830              187
           221040              218
           212674              237
           204981              237

XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.

     nr_writeback         nr_dirty
            80781                0
            37117            37703
            37117            43933
            81044                6
            81050                0
            43943            10199
            43930            36355
            43930            36355
            80293                0
            80285                0
            80285                0

Thanks,
Fengguang

#1 lore raw

PATCH 1/45 writeback: reduce calls to global_page_state in balance_dirty_pages()

From: Richard Kennedy <[email protected]>

Reducing the number of times balance_dirty_pages calls global_page_state reduces the cache references and so improves write performance on a variety of workloads.

'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_threshold approx 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviation

		average (s.d.) in millions (10^6)

2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)

Achieving this reduction is by dropping clip_bdi_dirty_limit as it rereads the counters to apply the dirty_threshold and moving this check up into balance_dirty_pages where it has already read the counters.

Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.

In the common case with no throttling it now calls global_page_state 5 fewer times and bdi_stat 2 fewer.

----
Thanks to everybody for the feedback & suggestions. This patch is against 2.6.31-rc8

---
 mm/page-writeback.c |   99 ++++++++++++++----------------------------
 1 file changed, 33 insertions(+), 66 deletions(-)
mm/page-writeback.c +33 -66
@@ -252,32 +252,6 @@ static void bdi_writeout_fraction(struct
 	}
 }
 
-/*
- * Clip the earned share of dirty pages to that which is actually available.
- * This avoids exceeding the total dirty_limit when the floating averages
- * fluctuate too quickly.
- */
-static void clip_bdi_dirty_limit(struct backing_dev_info *bdi,
-		unsigned long dirty, unsigned long *pbdi_dirty)
-{
-	unsigned long avail_dirty;
-
-	avail_dirty = global_page_state(NR_FILE_DIRTY) +
-		 global_page_state(NR_WRITEBACK) +
-		 global_page_state(NR_UNSTABLE_NFS) +
-		 global_page_state(NR_WRITEBACK_TEMP);
-
-	if (avail_dirty < dirty)
-		avail_dirty = dirty - avail_dirty;
-	else
-		avail_dirty = 0;
-
-	avail_dirty += bdi_stat(bdi, BDI_RECLAIMABLE) +
-		bdi_stat(bdi, BDI_WRITEBACK);
-
-	*pbdi_dirty = min(*pbdi_dirty, avail_dirty);
-}
-
 static inline void task_dirties_fraction(struct task_struct *tsk,
 		long *numerator, long *denominator)
 {
@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro
 			bdi_dirty = dirty * bdi->max_ratio / 100;
 
 		*pbdi_dirty = bdi_dirty;
-		clip_bdi_dirty_limit(bdi, dirty, pbdi_dirty);
 		task_dirty_limit(current, pbdi_dirty);
 	}
 }
@@ -490,7 +463,7 @@ static void balance_dirty_pages(struct a
 	unsigned long bdi_thresh;
 	unsigned long pages_written = 0;
 	unsigned long pause = 1;
-
+	int dirty_exceeded;
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
 	for (;;) {
@@ -503,16 +476,36 @@ static void balance_dirty_pages(struct a
 		};
 
 		get_dirty_limits(&background_thresh, &dirty_thresh,
-				&bdi_thresh, bdi);
+				 &bdi_thresh, bdi);
 
 		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
-					global_page_state(NR_UNSTABLE_NFS);
-		nr_writeback = global_page_state(NR_WRITEBACK);
+				 global_page_state(NR_UNSTABLE_NFS);
+		nr_writeback = global_page_state(NR_WRITEBACK) +
+			       global_page_state(NR_WRITEBACK_TEMP);
 
-		bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
-		bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+		/*
+		 * In order to avoid the stacked BDI deadlock we need
+		 * to ensure we accurately count the 'dirty' pages when
+		 * the threshold is low.
+		 *
+		 * Otherwise it would be possible to get thresh+n pages
+		 * reported dirty, even though there are thresh-m pages
+		 * actually dirty; with m+n sitting in the percpu
+		 * deltas.
+		 */
+		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
+			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
+			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
+		} else {
+			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
+			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+		}
+
+		dirty_exceeded =
+			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
+			|| (nr_reclaimable + nr_writeback >= dirty_thresh);
 
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
+		if (!dirty_exceeded)
 			break;
 
 		/*
@@ -521,7 +514,7 @@ static void balance_dirty_pages(struct a
 		 * when the bdi limits are ramping up.
 		 */
 		if (nr_reclaimable + nr_writeback <
-				(background_thresh + dirty_thresh) / 2)
+		    (background_thresh + dirty_thresh) / 2)
 			break;
 
 		if (!bdi->dirty_exceeded)
@@ -539,33 +532,10 @@ static void balance_dirty_pages(struct a
 		if (bdi_nr_reclaimable > bdi_thresh) {
 			writeback_inodes_wbc(&wbc);
 			pages_written += write_chunk - wbc.nr_to_write;
-			get_dirty_limits(&background_thresh, &dirty_thresh,
-				       &bdi_thresh, bdi);
-		}
-
-		/*
-		 * In order to avoid the stacked BDI deadlock we need
-		 * to ensure we accurately count the 'dirty' pages when
-		 * the threshold is low.
-		 *
-		 * Otherwise it would be possible to get thresh+n pages
-		 * reported dirty, even though there are thresh-m pages
-		 * actually dirty; with m+n sitting in the percpu
-		 * deltas.
-		 */
-		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
-			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
-		} else if (bdi_nr_reclaimable) {
-			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;
 
 	if (writeback_in_progress(bdi))
@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
 	if ((laptop_mode && pages_written) ||
-	    (!laptop_mode && ((global_page_state(NR_FILE_DIRTY)
-			       + global_page_state(NR_UNSTABLE_NFS))
-					  > background_thresh)))
+	    (!laptop_mode && (nr_reclaimable > background_thresh)))
 		bdi_start_writeback(bdi, NULL, 0);
 }
 
maintainer
#2 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Hi,

Quoted 22 lines · Wu Fengguang

From: Richard Kennedy <[email protected]>

Reducing the number of times balance_dirty_pages calls global_page_state reduces the cache references and so improves write performance on a variety of workloads.

'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_threshold approx 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviation

		average (s.d.) in millions (10^6)

2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)

Achieving this reduction is by dropping clip_bdi_dirty_limit as it rereads the counters to apply the dirty_threshold and moving this check up into balance_dirty_pages where it has already read the counters.

Quoted 6 lines

Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.

In the common case with no throttling it now calls global_page_state 5 fewer times and bdi_stat 2 fewer.

Hmm, but the patch changes the behavior of balance_dirty_pages() in
several ways:

Quoted 30 lines
mm/page-writeback.c +0 -26
-/*
- * Clip the earned share of dirty pages to that which is actually available.
- * This avoids exceeding the total dirty_limit when the floating averages
- * fluctuate too quickly.
- */
-static void clip_bdi_dirty_limit(struct backing_dev_info *bdi,
-		unsigned long dirty, unsigned long *pbdi_dirty)
-{
-	unsigned long avail_dirty;
-
-	avail_dirty = global_page_state(NR_FILE_DIRTY) +
-		 global_page_state(NR_WRITEBACK) +
-		 global_page_state(NR_UNSTABLE_NFS) +
-		 global_page_state(NR_WRITEBACK_TEMP);
-
-	if (avail_dirty < dirty)
-		avail_dirty = dirty - avail_dirty;
-	else
-		avail_dirty = 0;
-
-	avail_dirty += bdi_stat(bdi, BDI_RECLAIMABLE) +
-		bdi_stat(bdi, BDI_WRITEBACK);
-
-	*pbdi_dirty = min(*pbdi_dirty, avail_dirty);
-}
-
 static inline void task_dirties_fraction(struct task_struct *tsk,
 		long *numerator, long *denominator)
 {

mm/page-writeback.c get_dirty_limits(unsigned long *pbackgro · line 471

Quoted 4 lines
mm/page-writeback.c +0 -1
@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro
 			bdi_dirty = dirty * bdi->max_ratio / 100;
 
 		*pbdi_dirty = bdi_dirty;
-		clip_bdi_dirty_limit(bdi, dirty, pbdi_dirty);

I don't see, what test in balance_dirty_limits() should replace this
clipping... OTOH clipping does not seem to have too much effect on the behavior of balance_dirty_pages - the limit we clip to (at least BDI_WRITEBACK + BDI_RECLAIMABLE) is large enough so that we break from the loop immediately. So just getting rid of the function is fine but I'd update the changelog accordingly.

Quoted 31 lines
mm/page-writeback.c +19 -5
@@ -503,16 +476,36 @@ static void balance_dirty_pages(struct a
 		};
 
 		get_dirty_limits(&background_thresh, &dirty_thresh,
-				&bdi_thresh, bdi);
+				 &bdi_thresh, bdi);
 
 		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
-					global_page_state(NR_UNSTABLE_NFS);
-		nr_writeback = global_page_state(NR_WRITEBACK);
+				 global_page_state(NR_UNSTABLE_NFS);
+		nr_writeback = global_page_state(NR_WRITEBACK) +
+			       global_page_state(NR_WRITEBACK_TEMP);
 
-		bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
-		bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+		/*
+		 * In order to avoid the stacked BDI deadlock we need
+		 * to ensure we accurately count the 'dirty' pages when
+		 * the threshold is low.
+		 *
+		 * Otherwise it would be possible to get thresh+n pages
+		 * reported dirty, even though there are thresh-m pages
+		 * actually dirty; with m+n sitting in the percpu
+		 * deltas.
+		 */
+		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
+			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
+			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
+		} else {
+			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);

mm/page-writeback.c static void balance_dirty_pages(struct a · line 509

Quoted 10 lines
mm/page-writeback.c +7 -1
+			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+		}
+
+		dirty_exceeded =
+			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
+			|| (nr_reclaimable + nr_writeback >= dirty_thresh);
 
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
+		if (!dirty_exceeded)
 			break;

Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.

Quoted 34 lines
mm/page-writeback.c +1 -21
 		/*
@@ -521,7 +514,7 @@ static void balance_dirty_pages(struct a
 		 * when the bdi limits are ramping up.
 		 */
 		if (nr_reclaimable + nr_writeback <
-				(background_thresh + dirty_thresh) / 2)
+		    (background_thresh + dirty_thresh) / 2)
 			break;
 
 		if (!bdi->dirty_exceeded)
@@ -539,33 +532,10 @@ static void balance_dirty_pages(struct a
 		if (bdi_nr_reclaimable > bdi_thresh) {
 			writeback_inodes_wbc(&wbc);
 			pages_written += write_chunk - wbc.nr_to_write;
-			get_dirty_limits(&background_thresh, &dirty_thresh,
-				       &bdi_thresh, bdi);
-		}
-
-		/*
-		 * In order to avoid the stacked BDI deadlock we need
-		 * to ensure we accurately count the 'dirty' pages when
-		 * the threshold is low.
-		 *
-		 * Otherwise it would be possible to get thresh+n pages
-		 * reported dirty, even though there are thresh-m pages
-		 * actually dirty; with m+n sitting in the percpu
-		 * deltas.
-		 */
-		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
-			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
-		} else if (bdi_nr_reclaimable) {
-			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);

mm/page-writeback.c static void balance_dirty_pages(struct a · line 568

Quoted 10 lines
mm/page-writeback.c +3 -6
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-

Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...
I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...

mm/page-writeback.c static void balance_dirty_pages(struct a · line 551

Quoted 11 lines
mm/page-writeback.c +1 -2
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;

Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...

mm/page-writeback.c static void balance_dirty_pages(struct a · line 567

Quoted 10 lines
mm/page-writeback.c +1 -3
@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
 	if ((laptop_mode && pages_written) ||
-	    (!laptop_mode && ((global_page_state(NR_FILE_DIRTY)
-			       + global_page_state(NR_UNSTABLE_NFS))
-					  > background_thresh)))
+	    (!laptop_mode && (nr_reclaimable > background_thresh)))
 		bdi_start_writeback(bdi, NULL, 0);
 }

This might be based on rather old values in case we break from the loop
after calling writeback_inodes_wbc.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#3 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 3 lines · Jan Kara

Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.

It should have, we should throttle everything calling balance_dirty_pages() when we're over the total limit.

maintainer
#4 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Jan Kara

Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.

It should have, we should throttle everything calling balance_dirty_pages() when we're over the total limit.

OK :) I agree it's reasonable. But Wu, please note this in the
changelog because it might be a substantial change for some loads.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#5 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 10 lines · Jan Kara
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Jan Kara

Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.

It should have, we should throttle everything calling balance_dirty_pages() when we're over the total limit.

OK :) I agree it's reasonable. But Wu, please note this in the
changelog because it might be a substantial change for some loads.

Thanks, I added the note by Peter :)

Note that the total limit check itself may not be sufficient. For example, there are no nr_writeback limit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible

nr_writeback => dirty_thresh
nr_dirty => 0

which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).

So the better solution would be to impose a nr_writeback limit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit with congestion_wait, but now we need to do a wait queue for it.

With the nr_writeback wait queue, it can be guaranteed that once balance_dirty_pages() asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit in balance_dirty_pages().

However, there is still one problem - there are no general coordinations between max nr_writeback and the background/dirty limits.

It is possible (and very likely for some small memory systems) that

	nr_writeback > dirty_thresh - background_thresh
	10,000         20,000         15,000

In this case, it is possible that an application to be throttled because of

	nr_reclaimable + nr_writeback > dirty_thresh
	12,000           10,000         20,000

starts a background writeback work to do job for it, however that work quits immediately because

	nr_reclaimable < background_thresh
	12,000           15,000

In the end, the application did not get throttled at all at dirty_thresh. Instead, it will be throttled at (background_thresh + max_nr_writeback).

One solution (aka. the old behavior) is to respect the dirty_thresh, by not quiting background writeback when there are throttled tasks (this patch). It has the drawback of background writeback not doing its job _actively_. Instead, it will frequently be started and quit at times when applications enter and leave balanced_dirty_pages().

In the above scheme, the background_thresh is disregarded. The other ways would be to disregard dirty_thresh (may be undesirable) or to limit max_nr_writeback (not as easy).

It is still very possible to hit nr_dirty all the way down to 0 if max_nr_writeback > background_thresh.

This is a bit twisting. Any ideas?

---
 fs/fs-writeback.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/fs-writeback.c +2 -1
@@ -781,7 +781,8 @@ static long wb_writeback(struct bdi_writ
 		 * For background writeout, stop when we are below the
 		 * background dirty threshold
 		 */
-		if (args->for_background && !over_bground_thresh())
+		if (args->for_background && !over_bground_thresh() &&
+		    !list_empty(&wb->bdi->throttle_list))
 			break;
 
 		wbc.more_io = 0;
#6 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 18 lines · Wu Fengguang

Note that the total limit check itself may not be sufficient. For example, there are no nr_writeback limit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible

nr_writeback => dirty_thresh
nr_dirty => 0

which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).

So the better solution would be to impose a nr_writeback limit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit with congestion_wait, but now we need to do a wait queue for it.

Quoted 5 lines

With the nr_writeback wait queue, it can be guaranteed that once balance_dirty_pages() asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit in balance_dirty_pages().

nr_reclaim = nr_dirty + nr_writeback + nr_unstable, so anything calling into balance_dirty_pages() would still block on seeing such large amounts of nr_writeback.

Having the constraint nr_dirty + nr_writeback + nr_unstable < dirty_thresh should ensure we never have nr_writeback > dirty_thresh, simply because you cannot dirty more, which then cannot be converted to more writeback.

Or am I missing something?

#7 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 25 lines · Peter Zijlstra
Quoted 18 lines · Wu Fengguang

Note that the total limit check itself may not be sufficient. For example, there are no nr_writeback limit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible

nr_writeback => dirty_thresh
nr_dirty => 0

which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).

So the better solution would be to impose a nr_writeback limit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit with congestion_wait, but now we need to do a wait queue for it.

Quoted 5 lines

With the nr_writeback wait queue, it can be guaranteed that once balance_dirty_pages() asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit in balance_dirty_pages().

Quoted 3 lines

nr_reclaim = nr_dirty + nr_writeback + nr_unstable, so anything calling into balance_dirty_pages() would still block on seeing such large amounts of nr_writeback.

Our terms are a bit different. In my previous mail,
nr_reclaim = nr_dirty + nr_unstable
nr_writeback is added separated when comparing with dirty_thresh, just as the code in balance_dirty_pages().

But that's fine. You are right that the application will be blocked and dirty limit be guaranteed, if we do

        while (over dirty limit) {
                bdi_writeback_wait(pages to write);
        }

But it has a problem: as long as the bdi-flush thread for NFS don't limit nr_writeback, its nr_writeback will grow to near (dirty_thresh-nr_unstable), and its nr_dirty will approach 0. That's not desirable.

So I did this:
- while (over dirty limit) {

+       if (over dirty limit) {
                bdi_writeback_wait(pages to write);
        }

_after_ adding the NFS nr_writeback wait queue ([PATCH 20/45] NFS:
introduce writeback wait queue). With that it's safe to remove the loop.

Quoted 6 lines

Having the constraint nr_dirty + nr_writeback + nr_unstable < dirty_thresh should ensure we never have nr_writeback > dirty_thresh, simply because you cannot dirty more, which then cannot be converted to more writeback.

Or am I missing something?

You are right with the assumption that the loop is still there.

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

Thanks,
Fengguang

#8 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 5 lines · Wu Fengguang

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

OK, it seems we are indeed in agreement.

Just making sure ;-)

#9 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

In reply to Wu Fengguang
Quoted 5 lines · Wu Fengguang

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

Doesn't this require the writeback queue to have a limit < dirty_thresh?

Or more specifically, for the bdi case:

bdi_dirty + bdi_writeback + bdi_unstable <= bdi_thresh

we require that the writeback queue be smaller than bdi_thresh, which could be quite difficult, since bdi_thresh can easily be 0.

Without observing the bdi_thresh constraint we can have:

\Sum_(over bdis) writeback_queue_size

dirty pages outstanding, which could be significantly higher than dirty_thresh.

Or am I confused again?

#10 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

Doesn't this require the writeback queue to have a limit < dirty_thresh?

Yes, this is the key (open) issue. For now we have nothing to limit

nr_writeback < dirty_thresh

Quoted 6 lines

Or more specifically, for the bdi case:

bdi_dirty + bdi_writeback + bdi_unstable <= bdi_thresh

we require that the writeback queue be smaller than bdi_thresh, which could be quite difficult, since bdi_thresh can easily be 0.

We could apply a MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, so bdi_thresh must be non-zero to allow some writeback pages in flight :)

Quoted 6 lines

Without observing the bdi_thresh constraint we can have:

\Sum_(over bdis) writeback_queue_size

dirty pages outstanding, which could be significantly higher than dirty_thresh.

Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).

The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth. MIN_BDI_DIRTY_THRESH could be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.

Thanks,
Fengguang

#11 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 16 lines · Wu Fengguang
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

Doesn't this require the writeback queue to have a limit < dirty_thresh?

Yes, this is the key (open) issue. For now we have nothing to limit

nr_writeback < dirty_thresh

Quoted 2 lines

Or more specifically, for the bdi case:

Quoted 8 lines
Quoted 4 lines

bdi_dirty + bdi_writeback + bdi_unstable <= bdi_thresh

we require that the writeback queue be smaller than bdi_thresh, which could be quite difficult, since bdi_thresh can easily be 0.

We could apply a MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, so bdi_thresh must be non-zero to allow some writeback pages in flight :)

Not really, suppose you have 1000 NFS clients, of which you only use a hand full at a time.

Then the bdi_thresh will be 0 for most of them, and only when you switch to one it'll start growing. But it's perfectly reasonable to expect bdi_thresh=0 to work. It just reverts to sync behaviour, we write out everything and block until they're all gone from writeback state.

MIN_BDI_DIRTY_THRESH != 0, will have a side effect of imposing a max number of BDIs on the system, I'm not sure you want to go there.

Quoted 11 lines
Quoted 6 lines

Without observing the bdi_thresh constraint we can have:

\Sum_(over bdis) writeback_queue_size

dirty pages outstanding, which could be significantly higher than dirty_thresh.

Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).

Quoted 8 lines

The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth. MIN_BDI_DIRTY_THRESH could be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.

I'm not seeing why. By simply keeping that loop we're good again, and can have a writeback queue that works well in the saturated case.

#12 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 26 lines · Peter Zijlstra
Quoted 16 lines · Wu Fengguang
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang

Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).

Doesn't this require the writeback queue to have a limit < dirty_thresh?

Yes, this is the key (open) issue. For now we have nothing to limit

nr_writeback < dirty_thresh

Quoted 2 lines

Or more specifically, for the bdi case:

Quoted 8 lines
Quoted 4 lines

bdi_dirty + bdi_writeback + bdi_unstable <= bdi_thresh

we require that the writeback queue be smaller than bdi_thresh, which could be quite difficult, since bdi_thresh can easily be 0.

We could apply a MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, so bdi_thresh must be non-zero to allow some writeback pages in flight :)

Quoted 7 lines

Not really, suppose you have 1000 NFS clients, of which you only use a hand full at a time.

Then the bdi_thresh will be 0 for most of them, and only when you switch to one it'll start growing. But it's perfectly reasonable to expect bdi_thresh=0 to work. It just reverts to sync behaviour, we write out everything and block until they're all gone from writeback state.

Ah I see. We still do writeback when bdi_thresh=0, with any application blocked in balance_dirty_pages().

Quoted 2 lines

MIN_BDI_DIRTY_THRESH != 0, will have a side effect of imposing a max number of BDIs on the system, I'm not sure you want to go there.

OK that's not a good idea.

Quoted 20 lines
Quoted 11 lines
Quoted 6 lines

Without observing the bdi_thresh constraint we can have:

\Sum_(over bdis) writeback_queue_size

dirty pages outstanding, which could be significantly higher than dirty_thresh.

Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).

Quoted 8 lines

The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth. MIN_BDI_DIRTY_THRESH could be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.

Quoted 2 lines

I'm not seeing why. By simply keeping that loop we're good again, and can have a writeback queue that works well in the saturated case.

OK it looks better to keep the loop. The memory tight systems may go into the nr_dirty=0 situation, but it may not be an urgent problem (its nr_dirty will be small anyway).

Thanks,
Fengguang

#13 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

In reply to Jan Kara
Quoted 25 lines · Jan Kara

Hi,

Quoted 22 lines · Wu Fengguang

From: Richard Kennedy <[email protected]>

Reducing the number of times balance_dirty_pages calls global_page_state reduces the cache references and so improves write performance on a variety of workloads.

'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_threshold approx 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviation

		average (s.d.) in millions (10^6)

2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)

Achieving this reduction is by dropping clip_bdi_dirty_limit as it rereads the counters to apply the dirty_threshold and moving this check up into balance_dirty_pages where it has already read the counters.

Quoted 8 lines
Quoted 6 lines

Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.

In the common case with no throttling it now calls global_page_state 5 fewer times and bdi_stat 2 fewer.

Hmm, but the patch changes the behavior of balance_dirty_pages() in
several ways:

Yes, unfortunately the changelog failed to make that clear ..

Quoted 32 lines
Quoted 32 lines
mm/page-writeback.c +0 -26
-/*
- * Clip the earned share of dirty pages to that which is actually available.
- * This avoids exceeding the total dirty_limit when the floating averages
- * fluctuate too quickly.
- */
-static void clip_bdi_dirty_limit(struct backing_dev_info *bdi,
-		unsigned long dirty, unsigned long *pbdi_dirty)
-{
-	unsigned long avail_dirty;
-
-	avail_dirty = global_page_state(NR_FILE_DIRTY) +
-		 global_page_state(NR_WRITEBACK) +
-		 global_page_state(NR_UNSTABLE_NFS) +
-		 global_page_state(NR_WRITEBACK_TEMP);
-
-	if (avail_dirty < dirty)
-		avail_dirty = dirty - avail_dirty;
-	else
-		avail_dirty = 0;
-
-	avail_dirty += bdi_stat(bdi, BDI_RECLAIMABLE) +
-		bdi_stat(bdi, BDI_WRITEBACK);
-
-	*pbdi_dirty = min(*pbdi_dirty, avail_dirty);
-}
-
 static inline void task_dirties_fraction(struct task_struct *tsk,
 		long *numerator, long *denominator)
 {
@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro
 			bdi_dirty = dirty * bdi->max_ratio / 100;
 
Quoted 9 lines

mm/page-writeback.c get_dirty_limits(unsigned long *pbackgro · line 471

Quoted 2 lines
mm/page-writeback.c +0 -1
 		*pbdi_dirty = bdi_dirty;
-		clip_bdi_dirty_limit(bdi, dirty, pbdi_dirty);

I don't see, what test in balance_dirty_limits() should replace this
clipping... OTOH clipping does not seem to have too much effect on the behavior of balance_dirty_pages - the limit we clip to (at least BDI_WRITEBACK + BDI_RECLAIMABLE) is large enough so that we break from the loop immediately. So just getting rid of the function is fine but I'd update the changelog accordingly.

It essentially replace clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.

I added the above text to changelog :)

Quoted 10 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 509

Quoted 7 lines
mm/page-writeback.c +4 -1
+		dirty_exceeded =
+			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
+			|| (nr_reclaimable + nr_writeback >= dirty_thresh);
 
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
+		if (!dirty_exceeded)
 			break;

Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.

This restores the (right) behavior in 2.6.18. And peter have the same goal in mind with clip_bdi_dirty_limit() ;)

Quoted 6 lines
Quoted 6 lines
mm/page-writeback.c +3 -2
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
Quoted 8 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 568

Quoted 4 lines
mm/page-writeback.c +0 -4
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-

Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...

Note that the pages_written check is moved several lines up in the patch :)

Quoted 2 lines

I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...

Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)

Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0 -0
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
Quoted 6 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 551

Quoted 4 lines
mm/page-writeback.c +1 -2
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;

Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...

You must be mistaken: dirty_exceeded = (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).

Quoted 4 lines
Quoted 4 lines
mm/page-writeback.c +0 -0
@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
 	if ((laptop_mode && pages_written) ||
Quoted 8 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 567

Quoted 6 lines
mm/page-writeback.c +1 -3
-	    (!laptop_mode && ((global_page_state(NR_FILE_DIRTY)
-			       + global_page_state(NR_UNSTABLE_NFS))
-					  > background_thresh)))
+	    (!laptop_mode && (nr_reclaimable > background_thresh)))
 		bdi_start_writeback(bdi, NULL, 0);
 }

This might be based on rather old values in case we break from the loop
after calling writeback_inodes_wbc.

Yes that's possible. It's safe because the bdi worker will double check background_thresh. We can call bdi_start_writeback() as long as there are good possibility: the nr_reclaimable is not likely to drop suddenly from during our writeout.

Thanks,
Fengguang

maintainer
#14 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 16 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3 -2
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
Quoted 8 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 568

Quoted 4 lines
mm/page-writeback.c +0 -4
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-

Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...

Quoted 7 lines

Note that the pages_written check is moved several lines up in the patch :)

Quoted 2 lines

I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...

Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)

Yes, since the loop gets removed in the end, this does not matter. You
are right.

Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0 -0
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
Quoted 6 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 551

Quoted 4 lines
mm/page-writeback.c +1 -2
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;

Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...

Quoted 2 lines

You must be mistaken: dirty_exceeded = (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).

Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded = 0;

Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded = 0;

That's clearly not equivalent which is what I was trying to point out.
But looking at where dirty_exceeded is used, your new way is probably more useful. It's just a bit counterintuitive that bdi->dirty_exceeded is set even if the per-bdi limit is not exceeded...

Honza

--
Jan Kara <[email protected]>
SUSE Labs, CR

#15 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 19 lines · Jan Kara
Quoted 18 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3 -2
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
Quoted 8 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 568

Quoted 4 lines
mm/page-writeback.c +0 -4
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-

Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...

Note that the pages_written check is moved several lines up in the patch :)

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...

Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)

Yes, since the loop gets removed in the end, this does not matter. You
are right.

You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:

static void balance_dirty_pages(struct address_space *mapping,
unsigned long write_chunk)

{
	long nr_reclaimable, bdi_nr_reclaimable;
	long nr_writeback, bdi_nr_writeback;
	unsigned long background_thresh;
	unsigned long dirty_thresh;
	unsigned long bdi_thresh;
	int dirty_exceeded;
	struct backing_dev_info *bdi = mapping->backing_dev_info;

	/*
	 * If sync() is in progress, curb the to-be-synced inodes regardless
	 * of dirty limits, so that a fast dirtier won't livelock the sync.
	 */
	if (unlikely(bdi->sync_time &&
		     S_ISREG(mapping->host->i_mode) &&
		     time_after_eq(bdi->sync_time,
				   mapping->host->dirtied_when))) {
		write_chunk *= 2;
		bdi_writeback_wait(bdi, write_chunk);
	}

	for (;;) {
		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
				 global_page_state(NR_UNSTABLE_NFS);
		nr_writeback = global_page_state(NR_WRITEBACK) +
			       global_page_state(NR_WRITEBACK_TEMP);

		global_dirty_thresh(&background_thresh, &dirty_thresh);

		/*
		 * Throttle it only when the background writeback cannot
		 * catch-up. This avoids (excessively) small writeouts
		 * when the bdi limits are ramping up.
		 */
		if (nr_reclaimable + nr_writeback <
		    (background_thresh + dirty_thresh) / 2)
			break;

		bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);

		/*
		 * In order to avoid the stacked BDI deadlock we need
		 * to ensure we accurately count the 'dirty' pages when
		 * the threshold is low.
		 *
		 * Otherwise it would be possible to get thresh+n pages
		 * reported dirty, even though there are thresh-m pages
		 * actually dirty; with m+n sitting in the percpu
		 * deltas.
		 */
		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
		} else {
			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
		}

		/*
		 * The bdi thresh is somehow "soft" limit derived from the
		 * global "hard" limit. The former helps to prevent heavy IO
		 * bdi or process from holding back light ones; The latter is
		 * the last resort safeguard.
		 */
		dirty_exceeded =
			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
			|| (nr_reclaimable + nr_writeback >= dirty_thresh);

		if (!dirty_exceeded)
			break;

		bdi->dirty_exceed_time = jiffies;

		bdi_writeback_wait(bdi, write_chunk);
	}

	/*
	 * In laptop mode, we wait until hitting the higher threshold before
	 * starting background writeout, and then write out all the way down
	 * to the lower threshold.  So slow writers cause minimal disk activity.
	 *
	 * In normal mode, we start background writeout at the lower
	 * background_thresh, to keep the amount of dirty memory low.
	 */
	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
	    can_submit_background_writeback(bdi))
		bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
}
Quoted 20 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0 -0
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
Quoted 6 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 551

Quoted 4 lines
mm/page-writeback.c +1 -2
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;

Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...

Quoted 2 lines

You must be mistaken: dirty_exceeded = (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).

Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded = 0;

Quoted 8 lines

Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded = 0;

That's clearly not equivalent which is what I was trying to point out.
But looking at where dirty_exceeded is used, your new way is probably more useful. It's just a bit counterintuitive that bdi->dirty_exceeded is set even if the per-bdi limit is not exceeded...

Yeah good point. Since the per-bdi limits are more about "soft" limits which are derived from the global "hard" limit, the code makes sense with some comments and updated changelog :)

This patch slightly changes behavior by replacing clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.

The check is necessary because, in principle we should throttle everything calling balance_dirty_pages() when we're over the total limit, as said by Peter.

We now set and clear dirty_exceeded not only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.

We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.

Thanks,
Fengguang

#16 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 104 lines · Wu Fengguang

You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:

This patch slightly changes behavior by replacing clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.

The check is necessary because, in principle we should throttle everything calling balance_dirty_pages() when we're over the total limit, as said by Peter.

We now set and clear dirty_exceeded not only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.

We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.

static void balance_dirty_pages(struct address_space *mapping,
unsigned long write_chunk)

{
	long nr_reclaimable, bdi_nr_reclaimable;
	long nr_writeback, bdi_nr_writeback;
	unsigned long background_thresh;
	unsigned long dirty_thresh;
	unsigned long bdi_thresh;
	int dirty_exceeded;
	struct backing_dev_info *bdi = mapping->backing_dev_info;

	/*
	 * If sync() is in progress, curb the to-be-synced inodes regardless
	 * of dirty limits, so that a fast dirtier won't livelock the sync.
	 */
	if (unlikely(bdi->sync_time &&
		     S_ISREG(mapping->host->i_mode) &&
		     time_after_eq(bdi->sync_time,
				   mapping->host->dirtied_when))) {
		write_chunk *= 2;
		bdi_writeback_wait(bdi, write_chunk);
	}

	for (;;) {
		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
				 global_page_state(NR_UNSTABLE_NFS);
		nr_writeback = global_page_state(NR_WRITEBACK) +
			       global_page_state(NR_WRITEBACK_TEMP);

		global_dirty_thresh(&background_thresh, &dirty_thresh);

		/*
		 * Throttle it only when the background writeback cannot
		 * catch-up. This avoids (excessively) small writeouts
		 * when the bdi limits are ramping up.
		 */
		if (nr_reclaimable + nr_writeback <
		    (background_thresh + dirty_thresh) / 2)
			break;

		bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);

		/*
		 * In order to avoid the stacked BDI deadlock we need
		 * to ensure we accurately count the 'dirty' pages when
		 * the threshold is low.
		 *
		 * Otherwise it would be possible to get thresh+n pages
		 * reported dirty, even though there are thresh-m pages
		 * actually dirty; with m+n sitting in the percpu
		 * deltas.
		 */
		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
		} else {
			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
		}

		/*
		 * The bdi thresh is somehow "soft" limit derived from the
		 * global "hard" limit. The former helps to prevent heavy IO
		 * bdi or process from holding back light ones; The latter is
		 * the last resort safeguard.
		 */
		dirty_exceeded =
			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
			|| (nr_reclaimable + nr_writeback >= dirty_thresh);

		if (!dirty_exceeded)
			break;

		bdi->dirty_exceed_time = jiffies;

		bdi_writeback_wait(bdi, write_chunk);
	}

	/*
	 * In laptop mode, we wait until hitting the higher threshold before
	 * starting background writeout, and then write out all the way down
	 * to the lower threshold.  So slow writers cause minimal disk activity.
Quoted 8 lines
	 *
	 * In normal mode, we start background writeout at the lower
	 * background_thresh, to keep the amount of dirty memory low.
	 */
	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
	    can_submit_background_writeback(bdi))
		bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
}

Looks good, Thanks Wu!

maintainer
#17 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

In reply to Wu Fengguang
Quoted 99 lines · Wu Fengguang
Quoted 19 lines · Jan Kara
Quoted 18 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3 -2
+			/* don't wait if we've done enough */
+			if (pages_written >= write_chunk)
+				break;
 		}
-
-		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
Quoted 8 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 568

Quoted 4 lines
mm/page-writeback.c +0 -4
-			break;
-		if (pages_written >= write_chunk)
-			break;		/* We've done our duty */
-

Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...

Note that the pages_written check is moved several lines up in the patch :)

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...

Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)

Yes, since the loop gets removed in the end, this does not matter. You
are right.

You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:

static void balance_dirty_pages(struct address_space *mapping,
unsigned long write_chunk)

{
	long nr_reclaimable, bdi_nr_reclaimable;
	long nr_writeback, bdi_nr_writeback;
	unsigned long background_thresh;
	unsigned long dirty_thresh;
	unsigned long bdi_thresh;
	int dirty_exceeded;
	struct backing_dev_info *bdi = mapping->backing_dev_info;

	/*
	 * If sync() is in progress, curb the to-be-synced inodes regardless
	 * of dirty limits, so that a fast dirtier won't livelock the sync.
	 */
	if (unlikely(bdi->sync_time &&
		     S_ISREG(mapping->host->i_mode) &&
		     time_after_eq(bdi->sync_time,
				   mapping->host->dirtied_when))) {
		write_chunk *= 2;
		bdi_writeback_wait(bdi, write_chunk);
	}

	for (;;) {
		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
				 global_page_state(NR_UNSTABLE_NFS);
		nr_writeback = global_page_state(NR_WRITEBACK) +
			       global_page_state(NR_WRITEBACK_TEMP);

		global_dirty_thresh(&background_thresh, &dirty_thresh);

		/*
		 * Throttle it only when the background writeback cannot
		 * catch-up. This avoids (excessively) small writeouts
		 * when the bdi limits are ramping up.
		 */
		if (nr_reclaimable + nr_writeback <
		    (background_thresh + dirty_thresh) / 2)
			break;

		bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);

		/*
		 * In order to avoid the stacked BDI deadlock we need
		 * to ensure we accurately count the 'dirty' pages when
		 * the threshold is low.
		 *
		 * Otherwise it would be possible to get thresh+n pages
		 * reported dirty, even though there are thresh-m pages
		 * actually dirty; with m+n sitting in the percpu
		 * deltas.
		 */
		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
		} else {
			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
		}

		/*
		 * The bdi thresh is somehow "soft" limit derived from the
		 * global "hard" limit. The former helps to prevent heavy IO
		 * bdi or process from holding back light ones; The latter is
		 * the last resort safeguard.
		 */
		dirty_exceeded =
			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
Quoted 8 lines
			|| (nr_reclaimable + nr_writeback >= dirty_thresh);

		if (!dirty_exceeded)
			break;

		bdi->dirty_exceed_time = jiffies;

		bdi_writeback_wait(bdi, write_chunk);

Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?

Quoted 58 lines
	}

	/*
	 * In laptop mode, we wait until hitting the higher threshold before
	 * starting background writeout, and then write out all the way down
	 * to the lower threshold.  So slow writers cause minimal disk activity.
	 *
	 * In normal mode, we start background writeout at the lower
	 * background_thresh, to keep the amount of dirty memory low.
	 */
	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
	    can_submit_background_writeback(bdi))
		bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
}
Quoted 20 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0 -0
 		schedule_timeout_interruptible(pause);
 
 		/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
Quoted 6 lines

mm/page-writeback.c static void balance_dirty_pages(struct a · line 551

Quoted 4 lines
mm/page-writeback.c +1 -2
-	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
-			bdi->dirty_exceeded)
+	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;

Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...

Quoted 2 lines

You must be mistaken: dirty_exceeded = (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).

Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded = 0;

Quoted 8 lines

Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded = 0;

That's clearly not equivalent which is what I was trying to point out.
But looking at where dirty_exceeded is used, your new way is probably more useful. It's just a bit counterintuitive that bdi->dirty_exceeded is set even if the per-bdi limit is not exceeded...

Yeah good point. Since the per-bdi limits are more about "soft" limits which are derived from the global "hard" limit, the code makes sense with some comments and updated changelog :)

This patch slightly changes behavior by replacing clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.

The check is necessary because, in principle we should throttle everything calling balance_dirty_pages() when we're over the total limit, as said by Peter.

Quoted 8 lines

We now set and clear dirty_exceeded not only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.

We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.

The new description is good, thanks!

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#18 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 50 lines · Jan Kara
Quoted 50 lines
      for (;;) {
              nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
                               global_page_state(NR_UNSTABLE_NFS);
              nr_writeback = global_page_state(NR_WRITEBACK) +
                             global_page_state(NR_WRITEBACK_TEMP);

              global_dirty_thresh(&background_thresh, &dirty_thresh);

              /*
               * Throttle it only when the background writeback cannot
               * catch-up. This avoids (excessively) small writeouts
               * when the bdi limits are ramping up.
               */
              if (nr_reclaimable + nr_writeback <
                  (background_thresh + dirty_thresh) / 2)
                      break;

              bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);

              /*
               * In order to avoid the stacked BDI deadlock we need
               * to ensure we accurately count the 'dirty' pages when
               * the threshold is low.
               *
               * Otherwise it would be possible to get thresh+n pages
               * reported dirty, even though there are thresh-m pages
               * actually dirty; with m+n sitting in the percpu
               * deltas.
               */
              if (bdi_thresh < 2*bdi_stat_error(bdi)) {
                      bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
                      bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
              } else {
                      bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
                      bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
              }

              /*
               * The bdi thresh is somehow "soft" limit derived from the
               * global "hard" limit. The former helps to prevent heavy IO
               * bdi or process from holding back light ones; The latter is
               * the last resort safeguard.
               */
              dirty_exceeded =
                      (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
                      || (nr_reclaimable + nr_writeback >= dirty_thresh);

              if (!dirty_exceeded)
                      break;
Quoted 7 lines
Quoted 3 lines
              bdi->dirty_exceed_time = jiffies;

              bdi_writeback_wait(bdi, write_chunk);

Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?

Wu was saying that without the loop nr_writeback wasn't limited, but since bdi_writeback_wakeup() is driven from writeout completion, I'm not sure how again that was so.

We can move all of bdi_dirty to bdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.

Possibly unstable is peculiar.

#19 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 51 lines · Peter Zijlstra
Quoted 50 lines · Jan Kara
Quoted 50 lines
      for (;;) {
              nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
                               global_page_state(NR_UNSTABLE_NFS);
              nr_writeback = global_page_state(NR_WRITEBACK) +
                             global_page_state(NR_WRITEBACK_TEMP);

              global_dirty_thresh(&background_thresh, &dirty_thresh);

              /*
               * Throttle it only when the background writeback cannot
               * catch-up. This avoids (excessively) small writeouts
               * when the bdi limits are ramping up.
               */
              if (nr_reclaimable + nr_writeback <
                  (background_thresh + dirty_thresh) / 2)
                      break;

              bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);

              /*
               * In order to avoid the stacked BDI deadlock we need
               * to ensure we accurately count the 'dirty' pages when
               * the threshold is low.
               *
               * Otherwise it would be possible to get thresh+n pages
               * reported dirty, even though there are thresh-m pages
               * actually dirty; with m+n sitting in the percpu
               * deltas.
               */
              if (bdi_thresh < 2*bdi_stat_error(bdi)) {
                      bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
                      bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
              } else {
                      bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
                      bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
              }

              /*
               * The bdi thresh is somehow "soft" limit derived from the
               * global "hard" limit. The former helps to prevent heavy IO
               * bdi or process from holding back light ones; The latter is
               * the last resort safeguard.
               */
              dirty_exceeded =
                      (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
                      || (nr_reclaimable + nr_writeback >= dirty_thresh);

              if (!dirty_exceeded)
                      break;
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
              bdi->dirty_exceed_time = jiffies;

              bdi_writeback_wait(bdi, write_chunk);

Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?

Right, this is a legitimate concern.

Quoted 3 lines

Wu was saying that without the loop nr_writeback wasn't limited, but since bdi_writeback_wakeup() is driven from writeout completion, I'm not sure how again that was so.

Let me summarize the ideas :)

There are two cases:

- there are no bdi or block io queue to limit nr_writeback This must be fixed. It either let nr_writeback grow to dirty_thresh (with loop) and thus squeeze nr_dirty, or grow out of control totally (without loop). Current state is, the nr_writeback wait queue for NFS is there; the one for btrfs is still missing.

- there is a nr_writeback limit, but is larger than dirty_thresh In this case nr_dirty will be close to 0 regardless of the loop. The loop will help to keep
nr_dirty + nr_writeback + nr_unstable < dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by the nr_writeback limit).

Quoted 3 lines

We can move all of bdi_dirty to bdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.

Yes, this explains the second case. It's some trade-off like: the nr_writeback limit can not be trusted in small memory systems, so do the loop to impose the dirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..

We could possibly test (nr_dirty < nr_writeback). If so, the nr_writeback limit could be too large to deserve the loop.

It still don't address the nr_dirty=0 problem for small memory system, that should be acceptable since its nr_dirty will be small anyway.

Quoted 1 line

Possibly unstable is peculiar.

unstable can also go wild. I saw (in current linux-next with the following patch) balance_dirty_pages() sleeping for >30s waiting for the NFS nr_unstable to drop. That is, waiting for the dirty inode to be _expired_ and written to disk on the server.

It's a general uncoordinated double caching problem for NFS (and maybe more).

Thanks,
Fengguang
---

[   45.614799] balance_dirty_pages sleeped 228ms
[   45.954821] balance_dirty_pages sleeped 324ms
[   46.294874] balance_dirty_pages sleeped 324ms
[   46.638810] balance_dirty_pages sleeped 328ms
[   46.670769] balance_dirty_pages sleeped 28ms
[   46.802779] balance_dirty_pages sleeped 128ms
[   46.934788] balance_dirty_pages sleeped 124ms
[   47.066778] balance_dirty_pages sleeped 124ms
[   47.198774] balance_dirty_pages sleeped 128ms
[   47.330763] balance_dirty_pages sleeped 124ms
[   47.462768] balance_dirty_pages sleeped 128ms
[   47.594768] balance_dirty_pages sleeped 124ms
[   47.662763] balance_dirty_pages sleeped 60ms
[   47.798781] balance_dirty_pages sleeped 132ms
[   47.871435] balance_dirty_pages sleeped 64ms
[   48.002749] balance_dirty_pages sleeped 124ms
[   48.138787] balance_dirty_pages sleeped 132ms
[   48.270824] balance_dirty_pages sleeped 124ms
[   48.410762] balance_dirty_pages sleeped 128ms
[   48.542758] balance_dirty_pages sleeped 128ms
[   48.678786] balance_dirty_pages sleeped 132ms
[   48.810781] balance_dirty_pages sleeped 124ms
[   48.946755] balance_dirty_pages sleeped 124ms
[   49.182753] balance_dirty_pages sleeped 228ms
[   49.318773] balance_dirty_pages sleeped 128ms
[   49.666784] balance_dirty_pages sleeped 324ms
[   49.914774] balance_dirty_pages sleeped 228ms
[   79.998354] balance_dirty_pages sleeped 30068ms
[   80.062346] balance_dirty_pages sleeped 60ms
[   80.290414] balance_dirty_pages sleeped 224ms
[   80.542413] balance_dirty_pages sleeped 228ms
[   80.782384] balance_dirty_pages sleeped 228ms
[   81.142379] balance_dirty_pages sleeped 336ms
[  116.005926] balance_dirty_pages sleeped 34852ms
[  141.049584] balance_dirty_pages sleeped 25040ms
---
 mm/page-writeback.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/page-writeback.c +6 -1
@@ -490,6 +490,7 @@ static void balance_dirty_pages(struct a
 	unsigned long bdi_thresh;
 	unsigned long pages_written = 0;
 	unsigned long pause = 1;
+	unsigned long start = jiffies;
 
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
@@ -566,7 +567,8 @@ static void balance_dirty_pages(struct a
 		if (pages_written >= write_chunk)
 			break;		/* We've done our duty */
 
-		schedule_timeout_interruptible(pause);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		io_schedule_timeout(pause);
 
 		/*
 		 * Increase the delay for each loop, up to our previous
@@ -577,6 +579,9 @@ static void balance_dirty_pages(struct a
 			pause = HZ / 10;
 	}
 
+	if (pause > 1)
+		printk("balance_dirty_pages sleeped %lums\n", (jiffies - start) * 1000/HZ);
+
 	if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
 			bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;
#20 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 6 lines · Wu Fengguang
Quoted 4 lines
Quoted 4 lines

Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?

Right, this is a legitimate concern.

Quite.

Quoted 21 lines
Quoted 3 lines

Wu was saying that without the loop nr_writeback wasn't limited, but since bdi_writeback_wakeup() is driven from writeout completion, I'm not sure how again that was so.

Let me summarize the ideas :)

There are two cases:

- there are no bdi or block io queue to limit nr_writeback This must be fixed. It either let nr_writeback grow to dirty_thresh (with loop) and thus squeeze nr_dirty, or grow out of control totally (without loop). Current state is, the nr_writeback wait queue for NFS is there; the one for btrfs is still missing.

- there is a nr_writeback limit, but is larger than dirty_thresh In this case nr_dirty will be close to 0 regardless of the loop. The loop will help to keep
nr_dirty + nr_writeback + nr_unstable < dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by the nr_writeback limit).

Quoted 8 lines
Quoted 3 lines

We can move all of bdi_dirty to bdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.

Yes, this explains the second case. It's some trade-off like: the nr_writeback limit can not be trusted in small memory systems, so do the loop to impose the dirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..

Ok, so I'm still puzzled.

  set_page_dirty()
  balance_dirty_pages_ratelimited()
    balance_dirty_pages_ratelimited_nr(1)
      balance_dirty_pages(nr);

So we call balance_dirty_pages() with an appropriate count for each set_page_dirty() successful invocation, right?

balance_dirty_pages() guarantees that:

nr_dirty + nr_writeback + nr_unstable < dirty_thresh && (nr_dirty + nr_writeback + nr_unstable <

	(dirty_thresh + background_thresh)/2 ||

bdi_dirty + bdi_writeback + bdi_unstable < bdi_thresh)

Now without loop, without writeback limit, I still see no way to actually generate more 'dirty' pages than dirty_thresh.

As soon as we hit dirty_thresh a process will wait for exactly the same amount of pages to get cleaned (writeback completed) as were dirtied (+/- the ratelimit fuzz which should even out over processes).

That should bound things to dirty_thresh -- the wait is on writeback complete, so nr_writeback is bounded too.

[ I forgot the exact semantics of unstable, if we clear writeback before unstable, we need to fix something ]

Now, a nr_writeback queue that limits writeback will still be useful, esp for high speed devices. Once they ramp up and bdi_thresh exceeds the queue size, it'll take effect. So you reap the benefits when needed.

#21 lore raw

Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()

Quoted 35 lines · Peter Zijlstra
Quoted 6 lines · Wu Fengguang
Quoted 4 lines
Quoted 4 lines

Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?

Right, this is a legitimate concern.

Quite.

Quoted 25 lines
Quoted 3 lines

Wu was saying that without the loop nr_writeback wasn't limited, but since bdi_writeback_wakeup() is driven from writeout completion, I'm not sure how again that was so.

Let me summarize the ideas :)

There are two cases:

- there are no bdi or block io queue to limit nr_writeback This must be fixed. It either let nr_writeback grow to dirty_thresh (with loop) and thus squeeze nr_dirty, or grow out of control totally (without loop). Current state is, the nr_writeback wait queue for NFS is there; the one for btrfs is still missing.

- there is a nr_writeback limit, but is larger than dirty_thresh In this case nr_dirty will be close to 0 regardless of the loop. The loop will help to keep
nr_dirty + nr_writeback + nr_unstable < dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by the nr_writeback limit).

Quoted 3 lines

We can move all of bdi_dirty to bdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.

Quoted 6 lines
Quoted 4 lines

Yes, this explains the second case. It's some trade-off like: the nr_writeback limit can not be trusted in small memory systems, so do the loop to impose the dirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..

Ok, so I'm still puzzled.

Big sorry - it's me that was confused (by some buggy tests).

Quoted 7 lines
  set_page_dirty()
  balance_dirty_pages_ratelimited()
    balance_dirty_pages_ratelimited_nr(1)
      balance_dirty_pages(nr);

So we call balance_dirty_pages() with an appropriate count for each set_page_dirty() successful invocation, right?

Right.

Quoted 5 lines

balance_dirty_pages() guarantees that:

nr_dirty + nr_writeback + nr_unstable < dirty_thresh && (nr_dirty + nr_writeback + nr_unstable <

	(dirty_thresh + background_thresh)/2 ||
Quoted 8 lines

bdi_dirty + bdi_writeback + bdi_unstable < bdi_thresh)

Now without loop, without writeback limit, I still see no way to actually generate more 'dirty' pages than dirty_thresh.

As soon as we hit dirty_thresh a process will wait for exactly the same amount of pages to get cleaned (writeback completed) as were dirtied (+/- the ratelimit fuzz which should even out over processes).

Ah yes, we now wait for writeback _completed_ in bdi_writeback_wait(), instead of _start_ writeback in the old fashioned writeback_inodes().

Quoted 2 lines

That should bound things to dirty_thresh -- the wait is on writeback complete, so nr_writeback is bounded too.

Right. It was not bounded in the tests because bdi_writeback_wait() quits _prematurely_, because the background writeback finds it was already under background threshold, and so wakeup the throttled tasks and then quit. Fixed by simply removing the wakeup-all in background writeback and this change:

                if (args->for_background && !over_bground_thresh() &&
+                   list_empty(&wb->bdi->throttle_list))
                        break;

So now
- the throttled tasks are guaranteed to be wakeup
- it will only be wakeup in __bdi_writeout_inc()
- once wakeup, at least write_chunk pages have been written on behalf of it

Quoted 2 lines

[ I forgot the exact semantics of unstable, if we clear writeback before unstable, we need to fix something ]

New tests show that NFS is working fine without loop and without NFS nr_writeback limit:

      $ dd if=/dev/zero of=/mnt/test/zero3 bs=1M count=200 &
      $ vmmon -d 1 nr_writeback nr_dirty nr_unstable
     nr_writeback         nr_dirty      nr_unstable
                0                2                0
                0                2                0
                0            22477               65
                2            20849             1697
                2            19153             3393
                2            17420             5126
            27825                7             5979
            27816                0               41
            26925                0              907
            31064              286              159
            32531                0              213
            32548                0               89
            32405                0              155
            32464                0               98
            32517                0               45
            32560                0              194
            32534                0              220
            32601                0              222
            32490                0               72
            32447                0              115
            32511                0               48
            32535                0              216
            32535                0              216
            32535                0              216
            32535                0              216
            31555                0             1180
            29732                0             3003
            29277                0             3458
            27721                0             5014
            25955                0             6780
            24356                0             8379
            22763                0             9972
            21083                0            11652
            19371                0            13364
            17564                0            15171
            15781                0            16954
            14005                0            18730
            12230                0            20505
            12177                0            20558
            11383                0            21352
             9489                0            23246
             7621                0            25115
             5866                0            26870
             4790                0            27947
             2962                0            29773
             1089                0            31646
                0                0            32735
                0                0            32735
                0                0                0
                0                0                0
Quoted 3 lines

Now, a nr_writeback queue that limits writeback will still be useful, esp for high speed devices. Once they ramp up and bdi_thresh exceeds the queue size, it'll take effect. So you reap the benefits when needed.

Right, the nr_writeback limit avoids

nr_writeback => dirty_thresh
+
nr_dirty + nr_writeback < dirty_thresh
=>

        nr_dirty => 0

Thanks for the clarification, it looks less obscure now :)

Thanks,
Fengguang

#22 lore raw

PATCH 2/45 writeback: reduce calculation of bdi dirty thresholds

In reply to Wu Fengguang

Split get_dirty_limits() into global_dirty_thresh() and bdi_dirty_thresh(), so that the latter can be avoided in balance_dirty_pages() when under global dirty threshold.

CC Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c         |    2 
 include/linux/writeback.h |    5 +-
 mm/backing-dev.c          |    3 -
 mm/page-writeback.c       |   74 ++++++++++++++++++------------------
 4 files changed, 43 insertions(+), 41 deletions(-)
mm/page-writeback.c +37 -37
@@ -266,10 +266,11 @@ static inline void task_dirties_fraction
  *
  *   dirty -= (dirty/8) * p_{t}
  */
-static void task_dirty_limit(struct task_struct *tsk, unsigned long *pdirty)
+static unsigned long task_dirty_thresh(struct task_struct *tsk,
+				       unsigned long bdi_dirty)
 {
 	long numerator, denominator;
-	unsigned long dirty = *pdirty;
+	unsigned long dirty = bdi_dirty;
 	u64 inv = dirty >> 3;
 
 	task_dirties_fraction(tsk, &numerator, &denominator);
@@ -277,10 +278,8 @@ static void task_dirty_limit(struct task
 	do_div(inv, denominator);
 
 	dirty -= inv;
-	if (dirty < *pdirty/2)
-		dirty = *pdirty/2;
 
-	*pdirty = dirty;
+	return max(dirty, bdi_dirty/2);
 }
 
 /*
@@ -390,9 +389,7 @@ unsigned long determine_dirtyable_memory
 	return x + 1;	/* Ensure that we never return 0 */
 }
 
-void
-get_dirty_limits(unsigned long *pbackground, unsigned long *pdirty,
-		 unsigned long *pbdi_dirty, struct backing_dev_info *bdi)
+void global_dirty_thresh(unsigned long *pbackground, unsigned long *pdirty)
 {
 	unsigned long background;
 	unsigned long dirty;
@@ -424,26 +421,28 @@ get_dirty_limits(unsigned long *pbackgro
 	}
 	*pbackground = background;
 	*pdirty = dirty;
+}
 
-	if (bdi) {
-		u64 bdi_dirty;
-		long numerator, denominator;
+unsigned long bdi_dirty_thresh(struct backing_dev_info *bdi,
+			       unsigned long dirty)
+{
+	u64 bdi_dirty;
+	long numerator, denominator;
 
-		/*
-		 * Calculate this BDI's share of the dirty ratio.
-		 */
-		bdi_writeout_fraction(bdi, &numerator, &denominator);
+	/*
+	 * Calculate this BDI's share of the dirty ratio.
+	 */
+	bdi_writeout_fraction(bdi, &numerator, &denominator);
 
-		bdi_dirty = (dirty * (100 - bdi_min_ratio)) / 100;
-		bdi_dirty *= numerator;
-		do_div(bdi_dirty, denominator);
-		bdi_dirty += (dirty * bdi->min_ratio) / 100;
-		if (bdi_dirty > (dirty * bdi->max_ratio) / 100)
-			bdi_dirty = dirty * bdi->max_ratio / 100;
+	bdi_dirty = (dirty * (100 - bdi_min_ratio)) / 100;
+	bdi_dirty *= numerator;
+	do_div(bdi_dirty, denominator);
 
-		*pbdi_dirty = bdi_dirty;
-		task_dirty_limit(current, pbdi_dirty);
-	}
+	bdi_dirty += (dirty * bdi->min_ratio) / 100;
+	if (bdi_dirty > (dirty * bdi->max_ratio) / 100)
+		bdi_dirty = dirty * bdi->max_ratio / 100;
+
+	return task_dirty_thresh(current, bdi_dirty);
 }
 
 /*
@@ -475,14 +474,24 @@ static void balance_dirty_pages(struct a
 			.range_cyclic	= 1,
 		};
 
-		get_dirty_limits(&background_thresh, &dirty_thresh,
-				 &bdi_thresh, bdi);
-
 		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
 				 global_page_state(NR_UNSTABLE_NFS);
 		nr_writeback = global_page_state(NR_WRITEBACK) +
 			       global_page_state(NR_WRITEBACK_TEMP);
 
+		global_dirty_thresh(&background_thresh, &dirty_thresh);
+
+		/*
+		 * Throttle it only when the background writeback cannot
+		 * catch-up. This avoids (excessively) small writeouts
+		 * when the bdi limits are ramping up.
+		 */
+		if (nr_reclaimable + nr_writeback <
+		    (background_thresh + dirty_thresh) / 2)
+			break;
+
+		bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);
+
 		/*
 		 * In order to avoid the stacked BDI deadlock we need
 		 * to ensure we accurately count the 'dirty' pages when
@@ -508,15 +517,6 @@ static void balance_dirty_pages(struct a
 		if (!dirty_exceeded)
 			break;
 
-		/*
-		 * Throttle it only when the background writeback cannot
-		 * catch-up. This avoids (excessively) small writeouts
-		 * when the bdi limits are ramping up.
-		 */
-		if (nr_reclaimable + nr_writeback <
-		    (background_thresh + dirty_thresh) / 2)
-			break;
-
 		if (!bdi->dirty_exceeded)
 			bdi->dirty_exceeded = 1;
 
@@ -626,7 +626,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
 	unsigned long dirty_thresh;
 
         for ( ; ; ) {
-		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+		global_dirty_thresh(&background_thresh, &dirty_thresh);
 
                 /*
                  * Boost the allowable dirty threshold a bit for page
fs/fs-writeback.c +1 -1
@@ -729,7 +729,7 @@ static inline bool over_bground_thresh(v
 {
 	unsigned long background_thresh, dirty_thresh;
 
-	get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+	global_dirty_thresh(&background_thresh, &dirty_thresh);
 
 	return (global_page_state(NR_FILE_DIRTY) +
 		global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
mm/backing-dev.c +2 -1
@@ -83,7 +83,8 @@ static int bdi_debug_stats_show(struct s
 	}
 	spin_unlock(&inode_lock);
 
-	get_dirty_limits(&background_thresh, &dirty_thresh, &bdi_thresh, bdi);
+	global_dirty_thresh(&background_thresh, &dirty_thresh);
+	bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);
 
 #define K(x) ((x) << (PAGE_SHIFT - 10))
 	seq_printf(m,
include/linux/writeback.h +3 -2
@@ -126,8 +126,9 @@ struct ctl_table;
 int dirty_writeback_centisecs_handler(struct ctl_table *, int,
 				      void __user *, size_t *, loff_t *);
 
-void get_dirty_limits(unsigned long *pbackground, unsigned long *pdirty,
-		      unsigned long *pbdi_dirty, struct backing_dev_info *bdi);
+void global_dirty_thresh(unsigned long *pbackground, unsigned long *pdirty);
+unsigned long bdi_dirty_thresh(struct backing_dev_info *bdi,
+			       unsigned long dirty);
 
 void page_writeback_init(void);
 void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
#23 lore raw

PATCH 3/45 ext4: remove unused parameter wbc from __ext4_journalled_writepage()

In reply to Wu Fengguang
CC Theodore Ts'o <[email protected]>
CC Jan Kara <[email protected]>
---
 fs/ext4/inode.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ext4/inode.c +1 -2
@@ -2599,7 +2599,6 @@ static int bput_one(handle_t *handle, st
 }
 
 static int __ext4_journalled_writepage(struct page *page,
-				       struct writeback_control *wbc,
 				       unsigned int len)
 {
 	struct address_space *mapping = page->mapping;
@@ -2757,7 +2756,7 @@ static int ext4_writepage(struct page *p
 		 * doesn't seem much point in redirtying the page here.
 		 */
 		ClearPageChecked(page);
-		return __ext4_journalled_writepage(page, wbc, len);
+		return __ext4_journalled_writepage(page, len);
 	}
 
 	if (test_opt(inode->i_sb, NOBH) && ext4_should_writeback_data(inode))
#24 lore raw

PATCH 4/45 writeback: remove unused nonblocking and congestion checks

In reply to Wu Fengguang

- no one is calling wb_writeback and write_cache_pages with wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion wait

So remove the congestion checks as suggested by Chris.

CC Chris Mason <[email protected]>
CC Jens Axboe <[email protected]>
---
 drivers/staging/pohmelfs/inode.c |    9 ---------
 fs/afs/write.c                   |   16 +---------------
 fs/cifs/file.c                   |   10 ----------
 fs/fs-writeback.c                |    8 --------
 fs/gfs2/aops.c                   |   10 ----------
 fs/xfs/linux-2.6/xfs_aops.c      |    6 +-----
 mm/page-writeback.c              |   12 ------------
 7 files changed, 2 insertions(+), 69 deletions(-)
fs/fs-writeback.c +0 -8
@@ -660,14 +660,6 @@ static void writeback_inodes_wb(struct b
 			continue;
 		}
 
-		if (wbc->nonblocking && bdi_write_congested(wb->bdi)) {
-			wbc->encountered_congestion = 1;
-			if (!is_blkdev_sb)
-				break;		/* Skip a congested fs */
-			requeue_io(inode);
-			continue;		/* Skip a congested blockdev */
-		}
-
 		/*
 		 * Was this inode dirtied after sync_sb_inodes was called?
 		 * This keeps sync from extra jobs and livelock.
mm/page-writeback.c +0 -12
@@ -787,7 +787,6 @@ int write_cache_pages(struct address_spa
 		      struct writeback_control *wbc, writepage_t writepage,
 		      void *data)
 {
-	struct backing_dev_info *bdi = mapping->backing_dev_info;
 	int ret = 0;
 	int done = 0;
 	struct pagevec pvec;
@@ -800,11 +799,6 @@ int write_cache_pages(struct address_spa
 	int range_whole = 0;
 	long nr_to_write = wbc->nr_to_write;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
 		writeback_index = mapping->writeback_index; /* prev offset */
@@ -923,12 +917,6 @@ continue_unlock:
 					break;
 				}
 			}
-
-			if (wbc->nonblocking && bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
-				done = 1;
-				break;
-			}
 		}
 		pagevec_release(&pvec);
 		cond_resched();
drivers/staging/pohmelfs/inode.c +0 -9
@@ -152,11 +152,6 @@ static int pohmelfs_writepages(struct ad
 	int scanned = 0;
 	int range_whole = 0;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index; /* Start from prev offset */
 		end = -1;
@@ -248,10 +243,6 @@ retry:
 
 			if (wbc->nr_to_write <= 0)
 				done = 1;
-			if (wbc->nonblocking && bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
-				done = 1;
-			}
 
 			continue;
 out_continue:
fs/afs/write.c +1 -15
@@ -455,8 +455,6 @@ int afs_writepage(struct page *page, str
 	}
 
 	wbc->nr_to_write -= ret;
-	if (wbc->nonblocking && bdi_write_congested(bdi))
-		wbc->encountered_congestion = 1;
 
 	_leave(" = 0");
 	return 0;
@@ -529,11 +527,6 @@ static int afs_writepages_region(struct 
 
 		wbc->nr_to_write -= ret;
 
-		if (wbc->nonblocking && bdi_write_congested(bdi)) {
-			wbc->encountered_congestion = 1;
-			break;
-		}
-
 		cond_resched();
 	} while (index < end && wbc->nr_to_write > 0);
 
@@ -554,18 +547,11 @@ int afs_writepages(struct address_space 
 
 	_enter("");
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		_leave(" = 0 [congest]");
-		return 0;
-	}
-
 	if (wbc->range_cyclic) {
 		start = mapping->writeback_index;
 		end = -1;
 		ret = afs_writepages_region(mapping, wbc, start, end, &next);
-		if (start > 0 && wbc->nr_to_write > 0 && ret == 0 &&
-		    !(wbc->nonblocking && wbc->encountered_congestion))
+		if (start > 0 && wbc->nr_to_write > 0 && ret == 0)
 			ret = afs_writepages_region(mapping, wbc, 0, start,
 						    &next);
 		mapping->writeback_index = next;
fs/cifs/file.c +0 -10
@@ -1379,16 +1379,6 @@ static int cifs_writepages(struct addres
 		return generic_writepages(mapping, wbc);
 
 
-	/*
-	 * BB: Is this meaningful for a non-block-device file system?
-	 * If it is, we should test it again after we do I/O
-	 */
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		kfree(iov);
-		return 0;
-	}
-
 	xid = GetXid();
 
 	pagevec_init(&pvec, 0);
fs/gfs2/aops.c +0 -10
@@ -313,11 +313,6 @@ static int gfs2_write_jdata_pagevec(stru
 
 		if (ret || (--(wbc->nr_to_write) <= 0))
 			ret = 1;
-		if (wbc->nonblocking && bdi_write_congested(bdi)) {
-			wbc->encountered_congestion = 1;
-			ret = 1;
-		}
-
 	}
 	gfs2_trans_end(sdp);
 	return ret;
@@ -348,11 +343,6 @@ static int gfs2_write_cache_jdata(struct
 	int scanned = 0;
 	int range_whole = 0;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index; /* Start from prev offset */
fs/xfs/linux-2.6/xfs_aops.c +1 -5
@@ -890,12 +890,8 @@ xfs_convert_page(
 
 			bdi = inode->i_mapping->backing_dev_info;
 			wbc->nr_to_write--;
-			if (bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
+			if (wbc->nr_to_write <= 0)
 				done = 1;
-			} else if (wbc->nr_to_write <= 0) {
-				done = 1;
-			}
 		}
 		xfs_start_page_writeback(page, !page_dirty, count);
 	}
maintainer
#25 lore raw

Re: [PATCH 04/45] writeback: remove unused nonblocking and congestion checks

Quoted 6 lines · Wu Fengguang

- no one is calling wb_writeback and write_cache_pages with wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion wait

So remove the congestion checks as suggested by Chris.

Looks good. Since encountered_congestion isn't used, you can delete it as
well... BTW, you might need to split this patch to per-fs chunks for the sake of merging.

Honza

Quoted 197 lines
CC Chris Mason <[email protected]>
CC Jens Axboe <[email protected]>
---
 drivers/staging/pohmelfs/inode.c |    9 ---------
 fs/afs/write.c                   |   16 +---------------
 fs/cifs/file.c                   |   10 ----------
 fs/fs-writeback.c                |    8 --------
 fs/gfs2/aops.c                   |   10 ----------
 fs/xfs/linux-2.6/xfs_aops.c      |    6 +-----
 mm/page-writeback.c              |   12 ------------
 7 files changed, 2 insertions(+), 69 deletions(-)
fs/fs-writeback.c +0 -8
@@ -660,14 +660,6 @@ static void writeback_inodes_wb(struct b
 			continue;
 		}
 
-		if (wbc->nonblocking && bdi_write_congested(wb->bdi)) {
-			wbc->encountered_congestion = 1;
-			if (!is_blkdev_sb)
-				break;		/* Skip a congested fs */
-			requeue_io(inode);
-			continue;		/* Skip a congested blockdev */
-		}
-
 		/*
 		 * Was this inode dirtied after sync_sb_inodes was called?
 		 * This keeps sync from extra jobs and livelock.
mm/page-writeback.c +0 -12
@@ -787,7 +787,6 @@ int write_cache_pages(struct address_spa
 		      struct writeback_control *wbc, writepage_t writepage,
 		      void *data)
 {
-	struct backing_dev_info *bdi = mapping->backing_dev_info;
 	int ret = 0;
 	int done = 0;
 	struct pagevec pvec;
@@ -800,11 +799,6 @@ int write_cache_pages(struct address_spa
 	int range_whole = 0;
 	long nr_to_write = wbc->nr_to_write;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
 		writeback_index = mapping->writeback_index; /* prev offset */
@@ -923,12 +917,6 @@ continue_unlock:
 					break;
 				}
 			}
-
-			if (wbc->nonblocking && bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
-				done = 1;
-				break;
-			}
 		}
 		pagevec_release(&pvec);
 		cond_resched();
drivers/staging/pohmelfs/inode.c +0 -9
@@ -152,11 +152,6 @@ static int pohmelfs_writepages(struct ad
 	int scanned = 0;
 	int range_whole = 0;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index; /* Start from prev offset */
 		end = -1;
@@ -248,10 +243,6 @@ retry:
 
 			if (wbc->nr_to_write <= 0)
 				done = 1;
-			if (wbc->nonblocking && bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
-				done = 1;
-			}
 
 			continue;
 out_continue:
fs/afs/write.c +1 -15
@@ -455,8 +455,6 @@ int afs_writepage(struct page *page, str
 	}
 
 	wbc->nr_to_write -= ret;
-	if (wbc->nonblocking && bdi_write_congested(bdi))
-		wbc->encountered_congestion = 1;
 
 	_leave(" = 0");
 	return 0;
@@ -529,11 +527,6 @@ static int afs_writepages_region(struct 
 
 		wbc->nr_to_write -= ret;
 
-		if (wbc->nonblocking && bdi_write_congested(bdi)) {
-			wbc->encountered_congestion = 1;
-			break;
-		}
-
 		cond_resched();
 	} while (index < end && wbc->nr_to_write > 0);
 
@@ -554,18 +547,11 @@ int afs_writepages(struct address_space 
 
 	_enter("");
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		_leave(" = 0 [congest]");
-		return 0;
-	}
-
 	if (wbc->range_cyclic) {
 		start = mapping->writeback_index;
 		end = -1;
 		ret = afs_writepages_region(mapping, wbc, start, end, &next);
-		if (start > 0 && wbc->nr_to_write > 0 && ret == 0 &&
-		    !(wbc->nonblocking && wbc->encountered_congestion))
+		if (start > 0 && wbc->nr_to_write > 0 && ret == 0)
 			ret = afs_writepages_region(mapping, wbc, 0, start,
 						    &next);
 		mapping->writeback_index = next;
fs/cifs/file.c +0 -10
@@ -1379,16 +1379,6 @@ static int cifs_writepages(struct addres
 		return generic_writepages(mapping, wbc);
 
 
-	/*
-	 * BB: Is this meaningful for a non-block-device file system?
-	 * If it is, we should test it again after we do I/O
-	 */
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		kfree(iov);
-		return 0;
-	}
-
 	xid = GetXid();
 
 	pagevec_init(&pvec, 0);
fs/gfs2/aops.c +0 -10
@@ -313,11 +313,6 @@ static int gfs2_write_jdata_pagevec(stru
 
 		if (ret || (--(wbc->nr_to_write) <= 0))
 			ret = 1;
-		if (wbc->nonblocking && bdi_write_congested(bdi)) {
-			wbc->encountered_congestion = 1;
-			ret = 1;
-		}
-
 	}
 	gfs2_trans_end(sdp);
 	return ret;
@@ -348,11 +343,6 @@ static int gfs2_write_cache_jdata(struct
 	int scanned = 0;
 	int range_whole = 0;
 
-	if (wbc->nonblocking && bdi_write_congested(bdi)) {
-		wbc->encountered_congestion = 1;
-		return 0;
-	}
-
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index; /* Start from prev offset */
fs/xfs/linux-2.6/xfs_aops.c +1 -5
@@ -890,12 +890,8 @@ xfs_convert_page(
 
 			bdi = inode->i_mapping->backing_dev_info;
 			wbc->nr_to_write--;
-			if (bdi_write_congested(bdi)) {
-				wbc->encountered_congestion = 1;
+			if (wbc->nr_to_write <= 0)
 				done = 1;
-			} else if (wbc->nr_to_write <= 0) {
-				done = 1;
-			}
 		}
 		xfs_start_page_writeback(page, !page_dirty, count);
 	}
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#26 lore raw

Re: [PATCH 04/45] writeback: remove unused nonblocking and congestion checks

Quoted 10 lines · Jan Kara
Quoted 6 lines · Wu Fengguang

- no one is calling wb_writeback and write_cache_pages with wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion wait

So remove the congestion checks as suggested by Chris.

Looks good. Since encountered_congestion isn't used, you can delete it as
well... BTW, you might need to split this patch to per-fs chunks for the sake of merging.

OK, good suggestions to follow :)

Thanks,
Fengguang

#27 lore raw

PATCH 5/45 writeback: remove the always false bdi_cap_writeback_dirty() test

In reply to Wu Fengguang

This is dead code because no bdi flush thread will be started for !bdi_cap_writeback_dirty bdis.

---
 fs/fs-writeback.c |   18 ------------------
 1 file changed, 18 deletions(-)
fs/fs-writeback.c +0 -18
@@ -617,7 +617,6 @@ static void writeback_inodes_wb(struct b
 				struct writeback_control *wbc)
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
-	const int is_blkdev_sb = sb_is_blkdev_sb(sb);
 	const unsigned long start = jiffies;	/* livelock avoidance */
 
 	spin_lock(&inode_lock);
@@ -638,23 +637,6 @@ static void writeback_inodes_wb(struct b
 			continue;
 		}
 
-		if (!bdi_cap_writeback_dirty(wb->bdi)) {
-			redirty_tail(inode);
-			if (is_blkdev_sb) {
-				/*
-				 * Dirty memory-backed blockdev: the ramdisk
-				 * driver does this.  Skip just this inode
-				 */
-				continue;
-			}
-			/*
-			 * Dirty memory-backed inode against a filesystem other
-			 * than the kernel-internal bdev filesystem.  Skip the
-			 * entire superblock.
-			 */
-			break;
-		}
-
 		if (inode->i_state & (I_NEW | I_WILL_FREE)) {
 			requeue_io(inode);
 			continue;
#28 lore raw

PATCH 6/45 writeback: use larger ratelimit when dirty_exceeded

In reply to Wu Fengguang

When dirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.

Given the larger ratelimit value, we can safely ignore the low bound check in sync_writeback_pages.

dirty_exceeded is more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.

CC Peter Zijlstra <[email protected]>
CC Richard Kennedy <[email protected]>
---
 mm/page-writeback.c |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
mm/page-writeback.c +5 -7
@@ -39,7 +39,8 @@
  * After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
  * will look to see if it needs to force writeback or throttling.
  */
-static long ratelimit_pages = 32;
+#define MAX_RATELIMIT_PAGES ((4096 * 1024) / PAGE_CACHE_SIZE)
+static long ratelimit_pages = MAX_RATELIMIT_PAGES;
 
 /*
  * When balance_dirty_pages decides that the caller needs to perform some
@@ -49,9 +50,6 @@ static long ratelimit_pages = 32;
  */
 static inline long sync_writeback_pages(unsigned long dirtied)
 {
-	if (dirtied < ratelimit_pages)
-		dirtied = ratelimit_pages;
-
 	return dirtied + dirtied / 2;
 }
 
@@ -600,7 +598,7 @@ void balance_dirty_pages_ratelimited_nr(
 
 	ratelimit = ratelimit_pages;
 	if (mapping->backing_dev_info->dirty_exceeded)
-		ratelimit = 8;
+		ratelimit >>= 3;
 
 	/*
 	 * Check the rate limiting. Also, we do not want to throttle real-time
@@ -722,8 +720,8 @@ void writeback_set_ratelimit(void)
 	ratelimit_pages = vm_total_pages / (num_online_cpus() * 32);
 	if (ratelimit_pages < 16)
 		ratelimit_pages = 16;
-	if (ratelimit_pages * PAGE_CACHE_SIZE > 4096 * 1024)
-		ratelimit_pages = (4096 * 1024) / PAGE_CACHE_SIZE;
+	if (ratelimit_pages > MAX_RATELIMIT_PAGES)
+		ratelimit_pages = MAX_RATELIMIT_PAGES;
 }
 
 static int __cpuinit
#29 lore raw

Re: [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded

Quoted 6 lines · Wu Fengguang

plain text document attachment (writeback-ratelimit-on-dirty-exceeded.patch)
When dirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.

Quoted 6 lines

Given the larger ratelimit value, we can safely ignore the low bound check in sync_writeback_pages.

dirty_exceeded is more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.

Don't forget that ratelimit_pages is a per-cpu limit. So the total error on the dirty limit scales with the number of cpus.

Other than that, I guess this patch needs numbers ;-)

#30 lore raw

Re: [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Wu Fengguang

plain text document attachment (writeback-ratelimit-on-dirty-exceeded.patch)
When dirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.

Given the larger ratelimit value, we can safely ignore the low bound check in sync_writeback_pages.

Quoted 6 lines
Quoted 3 lines

dirty_exceeded is more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.

Don't forget that ratelimit_pages is a per-cpu limit. So the total error on the dirty limit scales with the number of cpus.

Ah yes! Given that a typical NUMA configuration would be to equip each CPU with >1GB memory, 512KB is not a big error :)

Quoted 1 line

Other than that, I guess this patch needs numbers ;-)

Lots of numbers, hehe. Basically I see ~25ms wait time for ext2/3/4 and btrfs (with this patch and the dirty exceed timestamp patch).

One cp process:

[ 3404.554478] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.570407] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 3404.601166] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.632872] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.663633] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.694198] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.724879] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.756655] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.787350] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.817950] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.848640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.880556] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 3404.911372] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.942006] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.972712] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.002672] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.034480] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.065114] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.095870] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.126453] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.158043] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.188801] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.219327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.249922] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.281869] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.312533] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.343194] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.373761] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.410828] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 3405.441993] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.473266] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.505746] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.537103] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.568327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 3405.600730] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.632039] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.663326] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.695674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.727031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.758146] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.790743] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.822129] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.853427] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 3405.885889] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.917116] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.948411] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms

Two:

[ 9393.138529] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9393.171042] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9393.344709] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9393.377045] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9393.555611] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9393.586298] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.747218] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9393.777825] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.808456] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.964086] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.120223] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.273854] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.304842] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.334474] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.480031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.527769] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.557919] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.590690] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9394.621626] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.654424] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9394.678289] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.709837] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.742912] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.774576] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.806149] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.822272] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9394.852609] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.886215] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.915804] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.947044] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.979504] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.010944] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.042221] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.074640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.105938] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.137157] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.169668] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms

Four:

[ 9435.829736] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.702953] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9436.719957] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.753598] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9436.770935] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.804367] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9436.818913] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.854302] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9436.870489] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.903449] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9443.140282] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9443.155674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9443.180418] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9443.697286] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9443.714321] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9443.748455] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9443.763764] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9443.796788] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9443.814309] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9446.493680] write_bandwidth: comm=btrfs-endio-wri pages=193 time=16ms
[ 9446.528486] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9446.888774] write_bandwidth: comm=btrfs-endio-wri pages=192 time=40ms
[ 9446.903518] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9446.933610] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms

Thanks,
Fengguang

#31 lore raw

PATCH 7/45 writeback: dont redirty tail an inode with dirty pages

In reply to Wu Fengguang

This avoids delaying writeback for an expired (XFS) inode with lots of dirty pages, but no active dirtier at the moment.

CC Dave Chinner <[email protected]>
CC Christoph Hellwig <[email protected]>
---
 fs/fs-writeback.c |   20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)
fs/fs-writeback.c +7 -13
@@ -479,18 +479,7 @@ writeback_single_inode(struct inode *ino
 	spin_lock(&inode_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
-		if ((inode->i_state & I_DIRTY_PAGES) && wbc->for_kupdate) {
-			/*
-			 * More pages get dirtied by a fast dirtier.
-			 */
-			goto select_queue;
-		} else if (inode->i_state & I_DIRTY) {
-			/*
-			 * At least XFS will redirty the inode during the
-			 * writeback (delalloc) and on io completion (isize).
-			 */
-			redirty_tail(inode);
-		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
+		if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
 			 * sometimes bales out without doing anything. Redirty
@@ -512,7 +501,6 @@ writeback_single_inode(struct inode *ino
 				 * soon as the queue becomes uncongested.
 				 */
 				inode->i_state |= I_DIRTY_PAGES;
-select_queue:
 				if (wbc->nr_to_write <= 0) {
 					/*
 					 * slice used up: queue for next turn
@@ -535,6 +523,12 @@ select_queue:
 				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
+		} else if (inode->i_state & I_DIRTY) {
+			/*
+			 * At least XFS will redirty the inode during the
+			 * writeback (delalloc) and on io completion (isize).
+			 */
+			redirty_tail(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
maintainer Acked-by on v1 7/45
#32 lore raw

Re: [PATCH 07/45] writeback: dont redirty tail an inode with dirty pages

Quoted 2 lines · Wu Fengguang

This avoids delaying writeback for an expired (XFS) inode with lots of dirty pages, but no active dirtier at the moment.

OK, looks good.

Honza

Quoted 53 lines
CC Dave Chinner <[email protected]>
CC Christoph Hellwig <[email protected]>
---
 fs/fs-writeback.c |   20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)
fs/fs-writeback.c +7 -13
@@ -479,18 +479,7 @@ writeback_single_inode(struct inode *ino
 	spin_lock(&inode_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
-		if ((inode->i_state & I_DIRTY_PAGES) && wbc->for_kupdate) {
-			/*
-			 * More pages get dirtied by a fast dirtier.
-			 */
-			goto select_queue;
-		} else if (inode->i_state & I_DIRTY) {
-			/*
-			 * At least XFS will redirty the inode during the
-			 * writeback (delalloc) and on io completion (isize).
-			 */
-			redirty_tail(inode);
-		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
+		if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
 			 * sometimes bales out without doing anything. Redirty
@@ -512,7 +501,6 @@ writeback_single_inode(struct inode *ino
 				 * soon as the queue becomes uncongested.
 				 */
 				inode->i_state |= I_DIRTY_PAGES;
-select_queue:
 				if (wbc->nr_to_write <= 0) {
 					/*
 					 * slice used up: queue for next turn
@@ -535,6 +523,12 @@ select_queue:
 				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
+		} else if (inode->i_state & I_DIRTY) {
+			/*
+			 * At least XFS will redirty the inode during the
+			 * writeback (delalloc) and on io completion (isize).
+			 */
+			redirty_tail(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#33 lore raw

PATCH 8/45 writeback: quit on wrap for .range_cyclic (write_cache_pages)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort the writeback of the current inode, which instructs writeback_single_inode() to delay it for a while if necessary.

This is the right behavior for
- sync writeback (is already so with range_whole)
we have scanned the inode address space, and don't care any more newly dirtied pages. So shall update its i_dirtied_when and exclude it from the todo list.
- periodic writeback
any more newly dirtied pages may be delayed for a while. This also prevents pointless IO for busy overwriters.
- background writeback
irrelevant because it generally don't care the dirty timestamp.

That should get rid of one inefficient IO pattern of .range_cyclic when writeback_index wraps, in which the submitted pages may be consisted of two distant ranges: submit [10000-10100], (wrap), submit [0-100].

CC Christoph Lameter <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
CC Nick Piggin <[email protected]>
CC Jan Kara <[email protected]>
---
 mm/page-writeback.c |   27 +++++----------------------
 1 file changed, 5 insertions(+), 22 deletions(-)
mm/page-writeback.c +5 -22
@@ -789,31 +789,23 @@ int write_cache_pages(struct address_spa
 	int done = 0;
 	struct pagevec pvec;
 	int nr_pages;
-	pgoff_t uninitialized_var(writeback_index);
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
 	pgoff_t done_index;
-	int cycled;
 	int range_whole = 0;
 	long nr_to_write = wbc->nr_to_write;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
-		writeback_index = mapping->writeback_index; /* prev offset */
-		index = writeback_index;
-		if (index == 0)
-			cycled = 1;
-		else
-			cycled = 0;
+		index = mapping->writeback_index; /* prev offset */
 		end = -1;
 	} else {
 		index = wbc->range_start >> PAGE_CACHE_SHIFT;
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		cycled = 1; /* ignore range_cyclic tests */
 	}
-retry:
+
 	done_index = index;
 	while (!done && (index <= end)) {
 		int i;
@@ -821,8 +813,10 @@ retry:
 		nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 			      PAGECACHE_TAG_DIRTY,
 			      min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
-		if (nr_pages == 0)
+		if (nr_pages == 0) {
+			done_index = 0;
 			break;
+		}
 
 		for (i = 0; i < nr_pages; i++) {
 			struct page *page = pvec.pages[i];
@@ -919,17 +913,6 @@ continue_unlock:
 		pagevec_release(&pvec);
 		cond_resched();
 	}
-	if (!cycled && !done) {
-		/*
-		 * range_cyclic:
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		cycled = 1;
-		index = 0;
-		end = writeback_index - 1;
-		goto retry;
-	}
 	if (!wbc->no_nrwrite_index_update) {
 		if (wbc->range_cyclic || (range_whole && nr_to_write > 0))
 			mapping->writeback_index = done_index;
#34 lore raw

PATCH 9/45 writeback: quit on wrap for .range_cyclic (pohmelfs)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

CC Evgeniy Polyakov <[email protected]>
---
 drivers/staging/pohmelfs/inode.c |   25 ++++++++-----------------
 1 file changed, 8 insertions(+), 17 deletions(-)
drivers/staging/pohmelfs/inode.c +8 -17
@@ -149,7 +149,6 @@ static int pohmelfs_writepages(struct ad
 	int nr_pages;
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
-	int scanned = 0;
 	int range_whole = 0;
 
 	if (wbc->range_cyclic) {
@@ -160,17 +159,18 @@ static int pohmelfs_writepages(struct ad
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && (index <= end)) {
 		unsigned int i = min(end - index, (pgoff_t)psb->trans_max_pages);
 		int path_len;
 		struct netfs_trans *trans;
 
 		err = pohmelfs_inode_has_dirty_pages(mapping, index);
-		if (!err)
+		if (!err) {
+			index = 0;
 			break;
+		}
 
 		err = pohmelfs_path_length(pi);
 		if (err < 0)
@@ -197,15 +197,16 @@ retry:
 		dprintk("%s: t: %p, nr_pages: %u, end: %lu, index: %lu, max: %u.\n",
 				__func__, trans, nr_pages, end, index, trans->page_num);
 
-		if (!nr_pages)
+		if (!nr_pages) {
+			index = 0;
 			goto err_out_reset;
+		}
 
 		err = pohmelfs_write_inode_create(inode, trans);
 		if (err)
 			goto err_out_reset;
 
 		err = 0;
-		scanned = 1;
 
 		for (i = 0; i < trans->page_num; i++) {
 			struct page *page = trans->pages[i];
@@ -215,7 +216,7 @@ retry:
 			if (unlikely(page->mapping != mapping))
 				goto out_continue;
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				goto out_continue;
 			}
@@ -263,16 +264,6 @@ err_out_reset:
 		break;
 	}
 
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
-		index = 0;
-		goto retry;
-	}
-
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;
 
#35 lore raw

Re: [PATCH 09/45] writeback: quit on wrap for .range_cyclic (pohmelfs)

Hi.

Quoted 10 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

I have no objections against this patchset, since I followed the upstream writeback behaviour and did not personally observe such wraps which would be otherwise handled in a single run.

Quoted 2 lines
CC Evgeniy Polyakov <[email protected]>
Signature · 1 line
	Evgeniy Polyakov
#36 lore raw

Re: [PATCH 09/45] writeback: quit on wrap for .range_cyclic (pohmelfs)

Quoted 12 lines · Evgeniy Polyakov

Hi.

Quoted 9 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

Quoted 5 lines
Quoted 1 line

It also prevents submitting pointless IO for busy overwriters.

I have no objections against this patchset, since I followed the upstream writeback behaviour and did not personally observe such wraps which would be otherwise handled in a single run.

OK, thanks!

Fengguang

#37 lore raw

PATCH 10/45 writeback: quit on wrap for .range_cyclic (btrfs)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

---
 fs/btrfs/extent_io.c |   21 ++++++---------------
 1 file changed, 6 insertions(+), 15 deletions(-)
fs/btrfs/extent_io.c +6 -15
@@ -2402,10 +2402,9 @@ static int extent_write_cache_pages(stru
 	int done = 0;
 	int nr_to_write_done = 0;
 	struct pagevec pvec;
-	int nr_pages;
+	int nr_pages = 1;
 	pgoff_t index;
 	pgoff_t end;		/* Inclusive */
-	int scanned = 0;
 	int range_whole = 0;
 
 	pagevec_init(&pvec, 0);
@@ -2417,16 +2416,14 @@ static int extent_write_cache_pages(stru
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && !nr_to_write_done && (index <= end) &&
 	       (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 			      PAGECACHE_TAG_DIRTY, min(end - index,
 				  (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
 		unsigned i;
 
-		scanned = 1;
 		for (i = 0; i < nr_pages; i++) {
 			struct page *page = pvec.pages[i];
 
@@ -2447,7 +2444,7 @@ retry:
 				continue;
 			}
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				unlock_page(page);
 				continue;
@@ -2484,15 +2481,9 @@ retry:
 		pagevec_release(&pvec);
 		cond_resched();
 	}
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
-		index = 0;
-		goto retry;
-	}
+	if (!nr_pages)
+		mapping->writeback_index = 0;
+
 	return ret;
 }
 
#38 lore raw

PATCH 11/45 writeback: quit on wrap for .range_cyclic (cifs)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

---
 fs/cifs/file.c |   18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)
fs/cifs/file.c +4 -14
@@ -1337,7 +1337,6 @@ static int cifs_partialpagewrite(struct 
 static int cifs_writepages(struct address_space *mapping,
 			   struct writeback_control *wbc)
 {
-	struct backing_dev_info *bdi = mapping->backing_dev_info;
 	unsigned int bytes_to_write;
 	unsigned int bytes_written;
 	struct cifs_sb_info *cifs_sb;
@@ -1349,14 +1348,13 @@ static int cifs_writepages(struct addres
 	int len;
 	int n_iov = 0;
 	pgoff_t next;
-	int nr_pages;
+	int nr_pages = 1;
 	__u64 offset = 0;
 	struct cifsFileInfo *open_file;
 	struct cifsInodeInfo *cifsi = CIFS_I(mapping->host);
 	struct page *page;
 	struct pagevec pvec;
 	int rc = 0;
-	int scanned = 0;
 	int xid, long_op;
 
 	cifs_sb = CIFS_SB(mapping->host->i_sb);
@@ -1390,9 +1388,8 @@ static int cifs_writepages(struct addres
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
-retry:
+
 	while (!done && (index <= end) &&
 	       (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 			PAGECACHE_TAG_DIRTY,
@@ -1425,7 +1422,7 @@ retry:
 				break;
 			}
 
-			if (!wbc->range_cyclic && page->index > end) {
+			if (page->index > end) {
 				done = 1;
 				unlock_page(page);
 				break;
@@ -1537,15 +1534,8 @@ retry:
 
 		pagevec_release(&pvec);
 	}
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
+	if (!nr_pages)
 		index = 0;
-		goto retry;
-	}
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;
 
#39 lore raw

PATCH 12/45 writeback: quit on wrap for .range_cyclic (ext4)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

---
 fs/ext4/inode.c |   18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)
fs/ext4/inode.c +4 -14
@@ -2805,7 +2805,7 @@ static int ext4_da_writepages(struct add
 	int pages_written = 0;
 	long pages_skipped;
 	unsigned int max_pages;
-	int range_cyclic, cycled = 1, io_done = 0;
+	int range_cyclic, io_done = 0;
 	int needed_blocks, ret = 0;
 	long desired_nr_to_write, nr_to_writebump = 0;
 	loff_t range_start = wbc->range_start;
@@ -2840,8 +2840,6 @@ static int ext4_da_writepages(struct add
 	range_cyclic = wbc->range_cyclic;
 	if (wbc->range_cyclic) {
 		index = mapping->writeback_index;
-		if (index)
-			cycled = 0;
 		wbc->range_start = index << PAGE_CACHE_SHIFT;
 		wbc->range_end  = LLONG_MAX;
 		wbc->range_cyclic = 0;
@@ -2889,7 +2887,6 @@ static int ext4_da_writepages(struct add
 	wbc->no_nrwrite_index_update = 1;
 	pages_skipped = wbc->pages_skipped;
 
-retry:
 	while (!ret && wbc->nr_to_write > 0) {
 
 		/*
@@ -2963,20 +2960,13 @@ retry:
 			wbc->pages_skipped = pages_skipped;
 			ret = 0;
 			io_done = 1;
-		} else if (wbc->nr_to_write)
+		} else if (wbc->nr_to_write > 0) {
 			/*
 			 * There is no more writeout needed
-			 * or we requested for a noblocking writeout
-			 * and we found the device congested
 			 */
+			index = 0;
 			break;
-	}
-	if (!io_done && !cycled) {
-		cycled = 1;
-		index = 0;
-		wbc->range_start = index << PAGE_CACHE_SHIFT;
-		wbc->range_end  = mapping->writeback_index - 1;
-		goto retry;
+		}
 	}
 	if (pages_skipped != wbc->pages_skipped)
 		ext4_msg(inode->i_sb, KERN_CRIT,
#40 lore raw

PATCH 13/45 writeback: quit on wrap for .range_cyclic (gfs2)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

CC Steven Whitehouse <[email protected]>
---
 fs/gfs2/aops.c |   16 ++--------------
 1 file changed, 2 insertions(+), 14 deletions(-)
fs/gfs2/aops.c +2 -14
@@ -287,7 +287,7 @@ static int gfs2_write_jdata_pagevec(stru
 			continue;
 		}
 
-		if (!wbc->range_cyclic && page->index > end) {
+		if (page->index > end) {
 			ret = 1;
 			unlock_page(page);
 			continue;
@@ -340,7 +340,6 @@ static int gfs2_write_cache_jdata(struct
 	int nr_pages;
 	pgoff_t index;
 	pgoff_t end;
-	int scanned = 0;
 	int range_whole = 0;
 
 	pagevec_init(&pvec, 0);
@@ -352,15 +351,12 @@ static int gfs2_write_cache_jdata(struct
 		end = wbc->range_end >> PAGE_CACHE_SHIFT;
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
-		scanned = 1;
 	}
 
-retry:
 	 while (!done && (index <= end) &&
 		(nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
 					       PAGECACHE_TAG_DIRTY,
 					       min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
-		scanned = 1;
 		ret = gfs2_write_jdata_pagevec(mapping, wbc, &pvec, nr_pages, end);
 		if (ret)
 			done = 1;
@@ -371,16 +367,8 @@ retry:
 		cond_resched();
 	}
 
-	if (!scanned && !done) {
-		/*
-		 * We hit the last page and there is more work to be done: wrap
-		 * back to the start of the file
-		 */
-		scanned = 1;
+	if (!nr_pages)
 		index = 0;
-		goto retry;
-	}
-
 	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
 		mapping->writeback_index = index;
 	return ret;
#41 lore raw

PATCH 14/45 writeback: quit on wrap for .range_cyclic (afs)

In reply to Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

---
 fs/afs/write.c |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
fs/afs/write.c +3 -4
@@ -477,8 +477,10 @@ static int afs_writepages_region(struct 
 	do {
 		n = find_get_pages_tag(mapping, &index, PAGECACHE_TAG_DIRTY,
 				       1, &page);
-		if (!n)
+		if (!n) {
+			index = 0;
 			break;
+		}
 
 		_debug("wback %lx", page->index);
 
@@ -551,9 +553,6 @@ int afs_writepages(struct address_space 
 		start = mapping->writeback_index;
 		end = -1;
 		ret = afs_writepages_region(mapping, wbc, start, end, &next);
-		if (start > 0 && wbc->nr_to_write > 0 && ret == 0)
-			ret = afs_writepages_region(mapping, wbc, 0, start,
-						    &next);
 		mapping->writeback_index = next;
 	} else if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX) {
 		end = (pgoff_t)(LLONG_MAX >> PAGE_CACHE_SHIFT);
maintainer Acked-by on v1 14/45
#42 lore raw

Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)

Quoted 6 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

Quoted 7 lines
	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

#43 lore raw

Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)

Quoted 11 lines · David Howells
Quoted 9 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

Quoted 6 lines
Quoted 4 lines

It also prevents submitting pointless IO for busy overwriters.

I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).

#44 lore raw

Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)

Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).

The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.

If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.

If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.

The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)

Thanks,
Fengguang

#45 lore raw

Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)

Quoted 30 lines · Wu Fengguang
Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).

The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.

If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.

Quoted 7 lines

If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.

The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)

No I mean bandwidth fairness between inodes.

#46 lore raw

Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)

Quoted 36 lines · Nick Piggin
Quoted 35 lines · Wu Fengguang
Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang

Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.

It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:

	submit [10000-10100], (wrap), submit [0-100]

In which the submitted pages may be consisted of two distant ranges.

It also prevents submitting pointless IO for busy overwriters.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).

The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.

If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.

If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.

Quoted 4 lines
Quoted 2 lines

The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)

No I mean bandwidth fairness between inodes.

I guess it's the old semantics that has bandwidth fairness problem :)

Imagine write chunk size is 4MB, and inode A/B with size 6MB/8MB.

The old semantics will have write sequence

        4MB for A; 4MB for B; other inodes;
        4MB for A; 4MB for B; other inodes;
        4MB for A; 4MB for B; other inodes;

while the new sequence would be

        4MB for A; 4MB for B; other inodes;
        2MB for A; 4MB for B; other inodes;
        4MB for A; 4MB for B; other inodes;
        2MB for A; 4MB for B; other inodes;

On average, each page in A used to get more write chance than B's. Now with no-wrap, A and B's pages have the same chance to be writeback.

Thanks,
Fengguang

#47 lore raw

PATCH 15/45 writeback: fix queue_io() ordering

In reply to Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -375,11 +375,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
#48 lore raw

PATCH 16/45 writeback: merge for_kupdate and !for_kupdate cases

In reply to Wu Fengguang

Unify the logic for kupdate and non-kupdate cases.
There won't be starvation because the inodes requeued into b_more_io will later be spliced _after_ the remaining inodes in b_io, hence won't stand in the way of other inodes in the next run.

It avoids unnecessary redirty_tail() calls, hence the update of i_dirtied_when. The timestamp update is undesirable because it could later delay the inode's period writeback, or exclude the inode from the data integrity sync operation (which will check timestamp to avoid extra work and livelock).

CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |   39 ++++++---------------------------------
 1 file changed, 6 insertions(+), 33 deletions(-)
fs/fs-writeback.c +6 -33
@@ -485,45 +485,18 @@ writeback_single_inode(struct inode *ino
 		if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
-			 * sometimes bales out without doing anything. Redirty
-			 * the inode; Move it from b_io onto b_more_io/b_dirty.
+			 * sometimes bales out without doing anything.
 			 */
-			/*
-			 * akpm: if the caller was the kupdate function we put
-			 * this inode at the head of b_dirty so it gets first
-			 * consideration.  Otherwise, move it to the tail, for
-			 * the reasons described there.  I'm not really sure
-			 * how much sense this makes.  Presumably I had a good
-			 * reasons for doing it this way, and I'd rather not
-			 * muck with it at present.
-			 */
-			if (wbc->for_kupdate) {
+			inode->i_state |= I_DIRTY_PAGES;
+			if (wbc->nr_to_write <= 0) {
 				/*
-				 * For the kupdate function we move the inode
-				 * to b_more_io so it will get more writeout as
-				 * soon as the queue becomes uncongested.
+				 * slice used up: queue for next turn
 				 */
-				inode->i_state |= I_DIRTY_PAGES;
-				if (wbc->nr_to_write <= 0) {
-					/*
-					 * slice used up: queue for next turn
-					 */
-					requeue_io(inode);
-				} else {
-					/*
-					 * somehow blocked: retry later
-					 */
-					redirty_tail(inode);
-				}
+				requeue_io(inode);
 			} else {
 				/*
-				 * Otherwise fully redirty the inode so that
-				 * other inodes on this superblock will get some
-				 * writeout.  Otherwise heavy writing to one
-				 * file would indefinitely suspend writeout of
-				 * all the other files.
+				 * somehow blocked: retry later
 				 */
-				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
 		} else if (inode->i_state & I_DIRTY) {
#49 lore raw

PATCH 17/45 writeback: only allow two background writeback works

In reply to Wu Fengguang

balance_dirty_pages() need a reliable way to ensure some background work is scheduled for running. We cannot simply run bdi_start_writeback() because that would queue up huge number of works (which takes memory and time).

---
 fs/fs-writeback.c           |   14 ++------------
 include/linux/backing-dev.h |   26 +++++++++++++++++++++++++-
 2 files changed, 27 insertions(+), 13 deletions(-)
fs/fs-writeback.c +2 -12
@@ -85,18 +85,6 @@ static inline void bdi_work_init(struct 
 int sysctl_dirty_debug __read_mostly;
 
 
-/**
- * writeback_in_progress - determine whether there is writeback in progress
- * @bdi: the device's backing_dev_info structure.
- *
- * Determine whether there is writeback waiting to be handled against a
- * backing device.
- */
-int writeback_in_progress(struct backing_dev_info *bdi)
-{
-	return !list_empty(&bdi->work_list);
-}
-
 static void bdi_work_clear(struct bdi_work *work)
 {
 	clear_bit(WS_USED_B, &work->state);
@@ -147,6 +135,8 @@ static void wb_clear_pending(struct bdi_
 
 		spin_lock(&bdi->wb_lock);
 		list_del_rcu(&work->list);
+		if (work->args.for_background)
+			clear_bit(WB_FLAG_BACKGROUND_WORK, &bdi->wb_mask);
 		spin_unlock(&bdi->wb_lock);
 
 		wb_work_complete(work);
include/linux/backing-dev.h +25 -1
@@ -94,6 +94,11 @@ struct backing_dev_info {
 #endif
 };
 
+/*
+ * background work queued, set to avoid redundant background works
+ */
+#define WB_FLAG_BACKGROUND_WORK		30
+
 int bdi_init(struct backing_dev_info *bdi);
 void bdi_destroy(struct backing_dev_info *bdi);
 
@@ -248,7 +253,26 @@ int bdi_set_max_ratio(struct backing_dev
 extern struct backing_dev_info default_backing_dev_info;
 void default_unplug_io_fn(struct backing_dev_info *bdi, struct page *page);
 
-int writeback_in_progress(struct backing_dev_info *bdi);
+/**
+ * writeback_in_progress - determine whether there is writeback in progress
+ * @bdi: the device's backing_dev_info structure.
+ *
+ * Determine whether there is writeback waiting to be handled against a
+ * backing device.
+ */
+static inline int writeback_in_progress(struct backing_dev_info *bdi)
+{
+	return !list_empty(&bdi->work_list);
+}
+
+/*
+ * Helper to limit # of background writeback works in circulation to 2.
+ * (one running and another queued)
+ */
+static inline int can_submit_background_writeback(struct backing_dev_info *bdi)
+{
+       return !test_and_set_bit(WB_FLAG_BACKGROUND_WORK, &bdi->wb_mask);
+}
 
 static inline int bdi_congested(struct backing_dev_info *bdi, int bdi_bits)
 {
#50 lore raw

PATCH 18/45 writeback: introduce wait queue for balance_dirty_pages()

In reply to Wu Fengguang

As proposed by Chris, Dave and Jan, let balance_dirty_pages() wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the above foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-file nr_to_write
The write_chunk used by balance_dirty_pages() should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait inside balance_dirty_pages() shall be small enough. When it starts its own writeback, it must specify a small nr_to_write. The throttle wait queue removes this dependancy by the way.

CC Chris Mason <[email protected]>
CC Dave Chinner <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 fs/fs-writeback.c           |   71 ++++++++++++++++++++++++++++++++++
 include/linux/backing-dev.h |   15 +++++++
 mm/backing-dev.c            |    4 +
 mm/page-writeback.c         |   53 ++++++-------------------
 4 files changed, 103 insertions(+), 40 deletions(-)
mm/page-writeback.c +13 -40
@@ -218,6 +218,15 @@ static inline void __bdi_writeout_inc(st
 {
 	__prop_inc_percpu_max(&vm_completions, &bdi->completions,
 			      bdi->max_prop_frac);
+
+	/*
+	 * The DIRTY_THROTTLE_PAGES_STOP test is an optional optimization, so
+	 * it's OK to be racy. We set DIRTY_THROTTLE_PAGES_STOP*2 in other
+	 * places to reduce the race possibility.
+	 */
+	if (atomic_read(&bdi->throttle_pages) < DIRTY_THROTTLE_PAGES_STOP &&
+	    atomic_dec_and_test(&bdi->throttle_pages))
+		bdi_writeback_wakeup(bdi);
 }
 
 void bdi_writeout_inc(struct backing_dev_info *bdi)
@@ -458,20 +467,10 @@ static void balance_dirty_pages(struct a
 	unsigned long background_thresh;
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
-	unsigned long pages_written = 0;
-	unsigned long pause = 1;
 	int dirty_exceeded;
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
 	for (;;) {
-		struct writeback_control wbc = {
-			.bdi		= bdi,
-			.sync_mode	= WB_SYNC_NONE,
-			.older_than_this = NULL,
-			.nr_to_write	= write_chunk,
-			.range_cyclic	= 1,
-		};
-
 		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
 				 global_page_state(NR_UNSTABLE_NFS);
 		nr_writeback = global_page_state(NR_WRITEBACK) +
@@ -518,39 +517,13 @@ static void balance_dirty_pages(struct a
 		if (!bdi->dirty_exceeded)
 			bdi->dirty_exceeded = 1;
 
-		/* Note: nr_reclaimable denotes nr_dirty + nr_unstable.
-		 * Unstable writes are a feature of certain networked
-		 * filesystems (i.e. NFS) in which data may have been
-		 * written to the server's write cache, but has not yet
-		 * been flushed to permanent storage.
-		 * Only move pages to writeback if this bdi is over its
-		 * threshold otherwise wait until the disk writes catch
-		 * up.
-		 */
-		if (bdi_nr_reclaimable > bdi_thresh) {
-			writeback_inodes_wbc(&wbc);
-			pages_written += write_chunk - wbc.nr_to_write;
-			/* don't wait if we've done enough */
-			if (pages_written >= write_chunk)
-				break;
-		}
-		schedule_timeout_interruptible(pause);
-
-		/*
-		 * Increase the delay for each loop, up to our previous
-		 * default of taking a 100ms nap.
-		 */
-		pause <<= 1;
-		if (pause > HZ / 10)
-			pause = HZ / 10;
+		bdi_writeback_wait(bdi, write_chunk);
+		break;
 	}
 
 	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;
 
-	if (writeback_in_progress(bdi))
-		return;
-
 	/*
 	 * In laptop mode, we wait until hitting the higher threshold before
 	 * starting background writeout, and then write out all the way down
@@ -559,8 +532,8 @@ static void balance_dirty_pages(struct a
 	 * In normal mode, we start background writeout at the lower
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
-	if ((laptop_mode && pages_written) ||
-	    (!laptop_mode && (nr_reclaimable > background_thresh)))
+	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
+	    can_submit_background_writeback(bdi))
 		bdi_start_writeback(bdi, NULL, 0);
 }
 
include/linux/backing-dev.h +15 -0
@@ -86,6 +86,13 @@ struct backing_dev_info {
 
 	struct list_head work_list;
 
+	/*
+	 * dirtier process throttling
+	 */
+	spinlock_t		throttle_lock;
+	struct list_head	throttle_list;	/* nr to sync for each task */
+	atomic_t		throttle_pages; /* nr to sync for head task */
+
 	struct device *dev;
 
 #ifdef CONFIG_DEBUG_FS
@@ -99,6 +106,12 @@ struct backing_dev_info {
  */
 #define WB_FLAG_BACKGROUND_WORK		30
 
+/*
+ * when no task is throttled, set throttle_pages to larger than this,
+ * to avoid unnecessary atomic decreases.
+ */
+#define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
+
 int bdi_init(struct backing_dev_info *bdi);
 void bdi_destroy(struct backing_dev_info *bdi);
 
@@ -110,6 +123,8 @@ void bdi_start_writeback(struct backing_
 				long nr_pages);
 int bdi_writeback_task(struct bdi_writeback *wb);
 int bdi_has_dirty_io(struct backing_dev_info *bdi);
+int bdi_writeback_wakeup(struct backing_dev_info *bdi);
+void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages);
 
 extern spinlock_t bdi_lock;
 extern struct list_head bdi_list;
fs/fs-writeback.c +71 -0

89 lines not on this page · raw message

mm/backing-dev.c +4 -0
@@ -646,6 +646,10 @@ int bdi_init(struct backing_dev_info *bd
 	bdi->wb_mask = 1;
 	bdi->wb_cnt = 1;
 
+	spin_lock_init(&bdi->throttle_lock);
+	INIT_LIST_HEAD(&bdi->throttle_list);
+	atomic_set(&bdi->throttle_pages, DIRTY_THROTTLE_PAGES_STOP * 2);
+
 	for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
 		err = percpu_counter_init(&bdi->bdi_stat[i], 0);
 		if (err)
#51 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 15 lines · On Wed, 07 Oct 2009 15:38:36 +0800 Wu Fengguang <[email protected]> wrote:

As proposed by Chris, Dave and Jan, let balance_dirty_pages() wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the above

Quoted 9 lines

foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-file nr_to_write
The write_chunk used by balance_dirty_pages() should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait inside balance_dirty_pages() shall be small enough. When it starts its own writeback, it must specify a small nr_to_write. The throttle wait queue removes this dependancy by the way.

May I ask a question ? (maybe not directly related to this patch itself, sorry)

Recent works as "writeback: switch to per-bdi threads for flushing data" removed congestion_wait() from balance_dirty_pages() and added schedule_timeout_interruptible().

And this one replaces it with wake_up+wait_queue.

IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)

{
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
        cputime64_t cputime64 = cputime_to_cputime64(cputime);
        struct rq *rq = this_rq();

        if (atomic_read(&rq->nr_iowait) > 0)
                cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
        else
                cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}

==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().

How this runqueue->nr_iowait is handled now ?

Thanks,
-Kame

Quoted 257 lines
CC Chris Mason <[email protected]>
CC Dave Chinner <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 fs/fs-writeback.c           |   71 ++++++++++++++++++++++++++++++++++
 include/linux/backing-dev.h |   15 +++++++
 mm/backing-dev.c            |    4 +
 mm/page-writeback.c         |   53 ++++++-------------------
 4 files changed, 103 insertions(+), 40 deletions(-)
mm/page-writeback.c +13 -40
@@ -218,6 +218,15 @@ static inline void __bdi_writeout_inc(st
 {
 	__prop_inc_percpu_max(&vm_completions, &bdi->completions,
 			      bdi->max_prop_frac);
+
+	/*
+	 * The DIRTY_THROTTLE_PAGES_STOP test is an optional optimization, so
+	 * it's OK to be racy. We set DIRTY_THROTTLE_PAGES_STOP*2 in other
+	 * places to reduce the race possibility.
+	 */
+	if (atomic_read(&bdi->throttle_pages) < DIRTY_THROTTLE_PAGES_STOP &&
+	    atomic_dec_and_test(&bdi->throttle_pages))
+		bdi_writeback_wakeup(bdi);
 }
 
 void bdi_writeout_inc(struct backing_dev_info *bdi)
@@ -458,20 +467,10 @@ static void balance_dirty_pages(struct a
 	unsigned long background_thresh;
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
-	unsigned long pages_written = 0;
-	unsigned long pause = 1;
 	int dirty_exceeded;
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
 	for (;;) {
-		struct writeback_control wbc = {
-			.bdi		= bdi,
-			.sync_mode	= WB_SYNC_NONE,
-			.older_than_this = NULL,
-			.nr_to_write	= write_chunk,
-			.range_cyclic	= 1,
-		};
-
 		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
 				 global_page_state(NR_UNSTABLE_NFS);
 		nr_writeback = global_page_state(NR_WRITEBACK) +
@@ -518,39 +517,13 @@ static void balance_dirty_pages(struct a
 		if (!bdi->dirty_exceeded)
 			bdi->dirty_exceeded = 1;
 
-		/* Note: nr_reclaimable denotes nr_dirty + nr_unstable.
-		 * Unstable writes are a feature of certain networked
-		 * filesystems (i.e. NFS) in which data may have been
-		 * written to the server's write cache, but has not yet
-		 * been flushed to permanent storage.
-		 * Only move pages to writeback if this bdi is over its
-		 * threshold otherwise wait until the disk writes catch
-		 * up.
-		 */
-		if (bdi_nr_reclaimable > bdi_thresh) {
-			writeback_inodes_wbc(&wbc);
-			pages_written += write_chunk - wbc.nr_to_write;
-			/* don't wait if we've done enough */
-			if (pages_written >= write_chunk)
-				break;
-		}
-		schedule_timeout_interruptible(pause);
-
-		/*
-		 * Increase the delay for each loop, up to our previous
-		 * default of taking a 100ms nap.
-		 */
-		pause <<= 1;
-		if (pause > HZ / 10)
-			pause = HZ / 10;
+		bdi_writeback_wait(bdi, write_chunk);
+		break;
 	}
 
 	if (!dirty_exceeded && bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;
 
-	if (writeback_in_progress(bdi))
-		return;
-
 	/*
 	 * In laptop mode, we wait until hitting the higher threshold before
 	 * starting background writeout, and then write out all the way down
@@ -559,8 +532,8 @@ static void balance_dirty_pages(struct a
 	 * In normal mode, we start background writeout at the lower
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
-	if ((laptop_mode && pages_written) ||
-	    (!laptop_mode && (nr_reclaimable > background_thresh)))
+	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
+	    can_submit_background_writeback(bdi))
 		bdi_start_writeback(bdi, NULL, 0);
 }
 
include/linux/backing-dev.h +15 -0
@@ -86,6 +86,13 @@ struct backing_dev_info {
 
 	struct list_head work_list;
 
+	/*
+	 * dirtier process throttling
+	 */
+	spinlock_t		throttle_lock;
+	struct list_head	throttle_list;	/* nr to sync for each task */
+	atomic_t		throttle_pages; /* nr to sync for head task */
+
 	struct device *dev;
 
 #ifdef CONFIG_DEBUG_FS
@@ -99,6 +106,12 @@ struct backing_dev_info {
  */
 #define WB_FLAG_BACKGROUND_WORK		30
 
+/*
+ * when no task is throttled, set throttle_pages to larger than this,
+ * to avoid unnecessary atomic decreases.
+ */
+#define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
+
 int bdi_init(struct backing_dev_info *bdi);
 void bdi_destroy(struct backing_dev_info *bdi);
 
@@ -110,6 +123,8 @@ void bdi_start_writeback(struct backing_
 				long nr_pages);
 int bdi_writeback_task(struct bdi_writeback *wb);
 int bdi_has_dirty_io(struct backing_dev_info *bdi);
+int bdi_writeback_wakeup(struct backing_dev_info *bdi);
+void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages);
 
 extern spinlock_t bdi_lock;
 extern struct list_head bdi_list;
fs/fs-writeback.c +71 -0
@@ -25,6 +25,7 @@
 #include <linux/blkdev.h>
 #include <linux/backing-dev.h>
 #include <linux/buffer_head.h>
+#include <linux/completion.h>
 #include "internal.h"
 
 #define inode_to_bdi(inode)	((inode)->i_mapping->backing_dev_info)
@@ -265,6 +266,72 @@ void bdi_start_writeback(struct backing_
 	bdi_alloc_queue_work(bdi, &args);
 }
 
+struct dirty_throttle_task {
+	long			nr_pages;
+	struct list_head	list;
+	struct completion	complete;
+};
+
+void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
+{
+	struct dirty_throttle_task tt = {
+		.nr_pages = nr_pages,
+		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
+	};
+	unsigned long flags;
+
+	/*
+	 * register throttle pages
+	 */
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	if (list_empty(&bdi->throttle_list))
+		atomic_set(&bdi->throttle_pages, nr_pages);
+	list_add(&tt.list, &bdi->throttle_list);
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	/*
+	 * make sure we will be woke up by someone
+	 */
+	if (can_submit_background_writeback(bdi))
+		bdi_start_writeback(bdi, NULL, 0);
+
+	wait_for_completion(&tt.complete);
+}
+
+/*
+ * return 1 if there are more waiting tasks.
+ */
+int bdi_writeback_wakeup(struct backing_dev_info *bdi)
+{
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	/*
+	 * remove and wakeup head task
+	 */
+	if (!list_empty(&bdi->throttle_list)) {
+		tt = list_entry(bdi->throttle_list.prev,
+				struct dirty_throttle_task, list);
+		list_del(&tt->list);
+		complete(&tt->complete);
+	}
+	/*
+	 * update throttle pages
+	 */
+	if (!list_empty(&bdi->throttle_list)) {
+		tt = list_entry(bdi->throttle_list.prev,
+				struct dirty_throttle_task, list);
+		atomic_set(&bdi->throttle_pages, tt->nr_pages);
+	} else {
+		tt = NULL;
+		atomic_set(&bdi->throttle_pages, DIRTY_THROTTLE_PAGES_STOP * 2);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return tt != NULL;
+}
+
 /*
  * Redirty an inode: set its when-it-was dirtied timestamp and move it to the
  * furthest end of its superblock's dirty-inode list.
@@ -760,6 +827,10 @@ static long wb_writeback(struct bdi_writ
 		spin_unlock(&inode_lock);
 	}
 
+	if (args->for_background)
+		while (bdi_writeback_wakeup(wb->bdi))
+			;  /* unthrottle all tasks */
+
 	return wrote;
 }
 
mm/backing-dev.c +4 -1
@@ -646,6 +646,10 @@ int bdi_init(struct backing_dev_info *bd
 	bdi->wb_mask = 1;
 	bdi->wb_cnt = 1;
 
+	spin_lock_init(&bdi->throttle_lock);
+	INIT_LIST_HEAD(&bdi->throttle_list);
+	atomic_set(&bdi->throttle_pages, DIRTY_THROTTLE_PAGES_STOP * 2);
+
 	for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
 		err = percpu_counter_init(&bdi->bdi_stat[i], 0);
 		if (err)
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#52 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 32 lines · KAMEZAWA Hiroyuki

tatus: RO
Content-Length: 12481
Lines: 332

Quoted 15 lines · On Wed, 07 Oct 2009 15:38:36 +0800 Wu Fengguang <[email protected]> wrote:

As proposed by Chris, Dave and Jan, let balance_dirty_pages() wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the above

Quoted 9 lines

foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-file nr_to_write
The write_chunk used by balance_dirty_pages() should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait inside balance_dirty_pages() shall be small enough. When it starts its own writeback, it must specify a small nr_to_write. The throttle wait queue removes this dependancy by the way.

Quoted 1 line

May I ask a question ? (maybe not directly related to this patch itself, sorry)

Sure :)

Quoted 5 lines

Recent works as "writeback: switch to per-bdi threads for flushing data" removed congestion_wait() from balance_dirty_pages() and added schedule_timeout_interruptible().

And this one replaces it with wake_up+wait_queue.

Right.

Quoted 13 lines

IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)

{
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
        cputime64_t cputime64 = cputime_to_cputime64(cputime);
        struct rq *rq = this_rq();

        if (atomic_read(&rq->nr_iowait) > 0)
                cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
        else
                cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}
Quoted 6 lines

==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Thanks,
Fengguang

#53 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 23 lines · On Thu, 8 Oct 2009 09:58:22 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 13 lines · KAMEZAWA Hiroyuki

IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)

{
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
        cputime64_t cputime64 = cputime_to_cputime64(cputime);
        struct rq *rq = this_rq();

        if (atomic_read(&rq->nr_iowait) > 0)
                cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
        else
                cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}
Quoted 6 lines

==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

Quoted 8 lines

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Regards,
-Kame

#54 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 30 lines · KAMEZAWA Hiroyuki
Quoted 28 lines · On Thu, 8 Oct 2009 09:58:22 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 13 lines · KAMEZAWA Hiroyuki

IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)

{
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
        cputime64_t cputime64 = cputime_to_cputime64(cputime);
        struct rq *rq = this_rq();

        if (atomic_read(&rq->nr_iowait) > 0)
                cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
        else
                cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}
Quoted 6 lines

==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

Quoted 6 lines
Quoted 3 lines

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)

Quoted 1 line

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Good suggestion. How about this patch?

---
writeback: show per-bdi throttled tasks

All currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.

	# cat /debug/bdi/8:0/throttle_list
	goal=6144kb	waited=32ms
	goal=6144kb     waited=48ms
CC KAMEZAWA Hiroyuki <[email protected]>
---
 fs/fs-writeback.c           |    1 +
 include/linux/backing-dev.h |    2 ++
 mm/backing-dev.c            |   33 +++++++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+)
include/linux/backing-dev.h +2 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 	struct dentry *debug_stats;
+	struct dentry *debug_throttle;
 #endif
 };
 
@@ -116,6 +117,7 @@ struct backing_dev_info {
 #define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
 
 struct dirty_throttle_task {
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
mm/backing-dev.c +33 -0
@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct s
 	return 0;
 }
 
+static int bdi_debug_throttle_show(struct seq_file *m, void *v)
+{
+	struct backing_dev_info *bdi = m->private;
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	list_for_each_entry(tt, &bdi->throttle_list, list) {
+		seq_printf(m, "goal=%lukb\twaited=%lums\n",
+			   tt->nr_pages << (PAGE_SHIFT - 10),
+			   (jiffies - tt->start_time) * 1000 / HZ);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return 0;
+}
+
 static int bdi_debug_stats_open(struct inode *inode, struct file *file)
 {
 	return single_open(file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +144,31 @@ static const struct file_operations bdi_
 	.release	= single_release,
 };
 
+static int bdi_debug_throttle_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, bdi_debug_throttle_show, inode->i_private);
+}
+
+static const struct file_operations bdi_debug_throttle_fops = {
+	.open		= bdi_debug_throttle_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static void bdi_debug_register(struct backing_dev_info *bdi, const char *name)
 {
 	bdi->debug_dir = debugfs_create_dir(name, bdi_debug_root);
 	bdi->debug_stats = debugfs_create_file("stats", 0444, bdi->debug_dir,
 					       bdi, &bdi_debug_stats_fops);
+	bdi->debug_throttle = debugfs_create_file("throttle_list", 0444,
+						  bdi->debug_dir, bdi,
+						  &bdi_debug_throttle_fops);
 }
 
 static void bdi_debug_unregister(struct backing_dev_info *bdi)
 {
+	debugfs_remove(bdi->debug_throttle);
 	debugfs_remove(bdi->debug_stats);
 	debugfs_remove(bdi->debug_dir);
 }
fs/fs-writeback.c +1 -0
@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(str
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
+		.start_time = jiffies,
 		.nr_pages = nr_pages,
 		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
#55 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)

good side effect ;)

Quoted 4 lines
Quoted 1 line

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Good suggestion. How about this patch?

Seems attractive.
Hmm..
==
struct dirty_throttle_task {
+ pid_t owner_pid;
==
and show it ? (too verbose ?

Regards,
-Kame

Quoted 95 lines
CC KAMEZAWA Hiroyuki <[email protected]>
---
 fs/fs-writeback.c           |    1 +
 include/linux/backing-dev.h |    2 ++
 mm/backing-dev.c            |   33 +++++++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+)
include/linux/backing-dev.h +2 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 	struct dentry *debug_stats;
+	struct dentry *debug_throttle;
 #endif
 };
 
@@ -116,6 +117,7 @@ struct backing_dev_info {
 #define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
 
 struct dirty_throttle_task {
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
mm/backing-dev.c +33 -0
@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct s
 	return 0;
 }
 
+static int bdi_debug_throttle_show(struct seq_file *m, void *v)
+{
+	struct backing_dev_info *bdi = m->private;
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	list_for_each_entry(tt, &bdi->throttle_list, list) {
+		seq_printf(m, "goal=%lukb\twaited=%lums\n",
+			   tt->nr_pages << (PAGE_SHIFT - 10),
+			   (jiffies - tt->start_time) * 1000 / HZ);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return 0;
+}
+
 static int bdi_debug_stats_open(struct inode *inode, struct file *file)
 {
 	return single_open(file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +144,31 @@ static const struct file_operations bdi_
 	.release	= single_release,
 };
 
+static int bdi_debug_throttle_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, bdi_debug_throttle_show, inode->i_private);
+}
+
+static const struct file_operations bdi_debug_throttle_fops = {
+	.open		= bdi_debug_throttle_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static void bdi_debug_register(struct backing_dev_info *bdi, const char *name)
 {
 	bdi->debug_dir = debugfs_create_dir(name, bdi_debug_root);
 	bdi->debug_stats = debugfs_create_file("stats", 0444, bdi->debug_dir,
 					       bdi, &bdi_debug_stats_fops);
+	bdi->debug_throttle = debugfs_create_file("throttle_list", 0444,
+						  bdi->debug_dir, bdi,
+						  &bdi_debug_throttle_fops);
 }
 
 static void bdi_debug_unregister(struct backing_dev_info *bdi)
 {
+	debugfs_remove(bdi->debug_throttle);
 	debugfs_remove(bdi->debug_stats);
 	debugfs_remove(bdi->debug_dir);
 }
fs/fs-writeback.c +1 -0
@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(str
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
+		.start_time = jiffies,
 		.nr_pages = nr_pages,
 		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
#56 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 4 lines · KAMEZAWA Hiroyuki
Quoted 1 line · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:

On Thu, Oct 08, 2009 at 10:40:37AM +0800, KAMEZAWA Hiroyuki wrote:

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)

good side effect ;)

Done, hehe.

Quoted 10 lines
Quoted 4 lines
Quoted 1 line

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Good suggestion. How about this patch?

Seems attractive.
Hmm..
==
struct dirty_throttle_task {

(quoted diff) +1 -0
+	pid_t	owner_pid;
Quoted 2 lines

==
and show it ? (too verbose ?

I even thought of adding the task pointer - it won't go away anyway :)

Thanks,
Fengguang

Quoted 99 lines

Regards,
-Kame

Quoted 95 lines
CC KAMEZAWA Hiroyuki <[email protected]>
---
 fs/fs-writeback.c           |    1 +
 include/linux/backing-dev.h |    2 ++
 mm/backing-dev.c            |   33 +++++++++++++++++++++++++++++++++
 3 files changed, 36 insertions(+)
include/linux/backing-dev.h +2 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 	struct dentry *debug_stats;
+	struct dentry *debug_throttle;
 #endif
 };
 
@@ -116,6 +117,7 @@ struct backing_dev_info {
 #define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
 
 struct dirty_throttle_task {
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
mm/backing-dev.c +33 -0
@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct s
 	return 0;
 }
 
+static int bdi_debug_throttle_show(struct seq_file *m, void *v)
+{
+	struct backing_dev_info *bdi = m->private;
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	list_for_each_entry(tt, &bdi->throttle_list, list) {
+		seq_printf(m, "goal=%lukb\twaited=%lums\n",
+			   tt->nr_pages << (PAGE_SHIFT - 10),
+			   (jiffies - tt->start_time) * 1000 / HZ);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return 0;
+}
+
 static int bdi_debug_stats_open(struct inode *inode, struct file *file)
 {
 	return single_open(file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +144,31 @@ static const struct file_operations bdi_
 	.release	= single_release,
 };
 
+static int bdi_debug_throttle_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, bdi_debug_throttle_show, inode->i_private);
+}
+
+static const struct file_operations bdi_debug_throttle_fops = {
+	.open		= bdi_debug_throttle_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static void bdi_debug_register(struct backing_dev_info *bdi, const char *name)
 {
 	bdi->debug_dir = debugfs_create_dir(name, bdi_debug_root);
 	bdi->debug_stats = debugfs_create_file("stats", 0444, bdi->debug_dir,
 					       bdi, &bdi_debug_stats_fops);
+	bdi->debug_throttle = debugfs_create_file("throttle_list", 0444,
+						  bdi->debug_dir, bdi,
+						  &bdi_debug_throttle_fops);
 }
 
 static void bdi_debug_unregister(struct backing_dev_info *bdi)
 {
+	debugfs_remove(bdi->debug_throttle);
 	debugfs_remove(bdi->debug_stats);
 	debugfs_remove(bdi->debug_dir);
 }
fs/fs-writeback.c +1 -0
@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(str
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
+		.start_time = jiffies,
 		.nr_pages = nr_pages,
 		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
#57 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

In reply to KAMEZAWA Hiroyuki
Quoted 23 lines · KAMEZAWA Hiroyuki
Quoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)

good side effect ;)

Quoted 4 lines
Quoted 1 line

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Good suggestion. How about this patch?

Seems attractive.
Hmm..
==
struct dirty_throttle_task {

(quoted diff) +1 -0
+	pid_t	owner_pid;
Quoted 2 lines

==
and show it ? (too verbose ?

Not all all, added comm too :)

---
writeback: show per-bdi throttled tasks

All currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.

	# cat /debug/bdi/0:16/throttle_list
	goal=768kb      waited=36ms     pid=3551        comm=cp
CC KAMEZAWA Hiroyuki <[email protected]>
---
 fs/fs-writeback.c           |    6 +++--
 include/linux/backing-dev.h |    3 ++
 mm/backing-dev.c            |   35 ++++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 2 deletions(-)
include/linux/backing-dev.h +3 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 	struct dentry *debug_stats;
+	struct dentry *debug_throttle;
 #endif
 };
 
@@ -116,6 +117,8 @@ struct backing_dev_info {
 #define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
 
 struct dirty_throttle_task {
+	struct task_struct	*sleeper;
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
mm/backing-dev.c +35 -0
@@ -115,6 +115,25 @@ static int bdi_debug_stats_show(struct s
 	return 0;
 }
 
+static int bdi_debug_throttle_show(struct seq_file *m, void *v)
+{
+	struct backing_dev_info *bdi = m->private;
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	list_for_each_entry(tt, &bdi->throttle_list, list) {
+		seq_printf(m, "goal=%lukb\twaited=%lums\tpid=%d\tcomm=%s\n",
+			   tt->nr_pages << (PAGE_SHIFT - 10),
+			   (jiffies - tt->start_time) * 1000 / HZ,
+			   tt->sleeper->pid,
+			   tt->sleeper->comm);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return 0;
+}
+
 static int bdi_debug_stats_open(struct inode *inode, struct file *file)
 {
 	return single_open(file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +146,31 @@ static const struct file_operations bdi_
 	.release	= single_release,
 };
 
+static int bdi_debug_throttle_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, bdi_debug_throttle_show, inode->i_private);
+}
+
+static const struct file_operations bdi_debug_throttle_fops = {
+	.open		= bdi_debug_throttle_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static void bdi_debug_register(struct backing_dev_info *bdi, const char *name)
 {
 	bdi->debug_dir = debugfs_create_dir(name, bdi_debug_root);
 	bdi->debug_stats = debugfs_create_file("stats", 0444, bdi->debug_dir,
 					       bdi, &bdi_debug_stats_fops);
+	bdi->debug_throttle = debugfs_create_file("throttle_list", 0444,
+						  bdi->debug_dir, bdi,
+						  &bdi_debug_throttle_fops);
 }
 
 static void bdi_debug_unregister(struct backing_dev_info *bdi)
 {
+	debugfs_remove(bdi->debug_throttle);
 	debugfs_remove(bdi->debug_stats);
 	debugfs_remove(bdi->debug_dir);
 }
fs/fs-writeback.c +4 -2
@@ -284,8 +284,10 @@ static void bdi_calc_write_bandwidth(str
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
-		.nr_pages = nr_pages,
-		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
+		.sleeper	= current,
+		.start_time	= jiffies,
+		.nr_pages	= nr_pages,
+		.complete	= COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
 	unsigned long flags;
 
#58 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 19 lines · On Thu, 8 Oct 2009 14:28:09 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki

Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.

Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)

good side effect ;)

Quoted 4 lines
Quoted 1 line

BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.

Good suggestion. How about this patch?

Quoted 10 lines
Quoted 7 lines

Seems attractive.
Hmm..
==
struct dirty_throttle_task {

(quoted diff) +1 -0
+	pid_t	owner_pid;

==
and show it ? (too verbose ?

Not all all, added comm too :)

Wow, nice.

Thank you,
-Kame

Quoted 113 lines

---
writeback: show per-bdi throttled tasks

All currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.

	# cat /debug/bdi/0:16/throttle_list
	goal=768kb      waited=36ms     pid=3551        comm=cp
CC KAMEZAWA Hiroyuki <[email protected]>
---
 fs/fs-writeback.c           |    6 +++--
 include/linux/backing-dev.h |    3 ++
 mm/backing-dev.c            |   35 ++++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 2 deletions(-)
include/linux/backing-dev.h +3 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 	struct dentry *debug_stats;
+	struct dentry *debug_throttle;
 #endif
 };
 
@@ -116,6 +117,8 @@ struct backing_dev_info {
 #define DIRTY_THROTTLE_PAGES_STOP	(1 << 22)
 
 struct dirty_throttle_task {
+	struct task_struct	*sleeper;
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
mm/backing-dev.c +35 -0
@@ -115,6 +115,25 @@ static int bdi_debug_stats_show(struct s
 	return 0;
 }
 
+static int bdi_debug_throttle_show(struct seq_file *m, void *v)
+{
+	struct backing_dev_info *bdi = m->private;
+	struct dirty_throttle_task *tt;
+	unsigned long flags;
+
+	spin_lock_irqsave(&bdi->throttle_lock, flags);
+	list_for_each_entry(tt, &bdi->throttle_list, list) {
+		seq_printf(m, "goal=%lukb\twaited=%lums\tpid=%d\tcomm=%s\n",
+			   tt->nr_pages << (PAGE_SHIFT - 10),
+			   (jiffies - tt->start_time) * 1000 / HZ,
+			   tt->sleeper->pid,
+			   tt->sleeper->comm);
+	}
+	spin_unlock_irqrestore(&bdi->throttle_lock, flags);
+
+	return 0;
+}
+
 static int bdi_debug_stats_open(struct inode *inode, struct file *file)
 {
 	return single_open(file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +146,31 @@ static const struct file_operations bdi_
 	.release	= single_release,
 };
 
+static int bdi_debug_throttle_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, bdi_debug_throttle_show, inode->i_private);
+}
+
+static const struct file_operations bdi_debug_throttle_fops = {
+	.open		= bdi_debug_throttle_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static void bdi_debug_register(struct backing_dev_info *bdi, const char *name)
 {
 	bdi->debug_dir = debugfs_create_dir(name, bdi_debug_root);
 	bdi->debug_stats = debugfs_create_file("stats", 0444, bdi->debug_dir,
 					       bdi, &bdi_debug_stats_fops);
+	bdi->debug_throttle = debugfs_create_file("throttle_list", 0444,
+						  bdi->debug_dir, bdi,
+						  &bdi_debug_throttle_fops);
 }
 
 static void bdi_debug_unregister(struct backing_dev_info *bdi)
 {
+	debugfs_remove(bdi->debug_throttle);
 	debugfs_remove(bdi->debug_stats);
 	debugfs_remove(bdi->debug_dir);
 }
fs/fs-writeback.c +4 -2
@@ -284,8 +284,10 @@ static void bdi_calc_write_bandwidth(str
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
-		.nr_pages = nr_pages,
-		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
+		.sleeper	= current,
+		.start_time	= jiffies,
+		.nr_pages	= nr_pages,
+		.complete	= COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
 	unsigned long flags;
 
#59 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

In reply to Wu Fengguang
Quoted 5 lines · Wu Fengguang
Quoted 2 lines

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

Quoted 7 lines

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..

Hmm?

#60 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

Quoted 11 lines · On Thu, 08 Oct 2009 10:08:36 +0200 Peter Zijlstra <[email protected]>…
Quoted 10 lines · Wu Fengguang
Quoted 2 lines

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

Quoted 8 lines
Quoted 2 lines

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..

Hmm?

Above comment "don't do that if it is a deliberate, throttling IO wait" is really old but ignored.
I pesonally don't like to change the meanig of iowait in /proc/stat. But I'm not sure which is better to change the definitiion (which was ignored) or fix behavior (not correct very long time)...

Hmm?, too ;)

Regards,
-Kame

#61 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Wu Fengguang
Quoted 2 lines

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

Quoted 7 lines
Quoted 2 lines

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..

Hmm?

Yep agree, if we're deliberately waiting on IO, it should count as iowait time.

Signature · 1 line
Jens Axboe
#62 lore raw

PATCH 1/1 writeback: account IO throttling wait as iowait

Quoted 15 lines · Jens Axboe
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Wu Fengguang
Quoted 2 lines

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Quoted 7 lines
Quoted 4 lines

Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..

Hmm?

Yep agree, if we're deliberately waiting on IO, it should count as iowait time.

Then let's revert to the old behavior :)

For one single cp, it increases iowait from 29% to 56%.

Before patch:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   4  64  28   0   3|   0     0 | 272k   10M|   0     0 |1854   863
  0   6  69  23   0   3|   0     0 | 249k   11M|   0     0 |1709   865
  0   6  64  27   0   4|   0     0 | 235k   10M|   0     0 |1807   788
  0   4  61  30   0   4|   0     0 | 271k   12M|   0     0 |1910   898
  0   4  72  21   0   4|   0     0 | 289k   13M|   0     0 |1832   905
  0   6  58  35   0   2|   0     0 | 252k   11M|   0     0 |1713   900
  0   4  54  38   0   4|   0     0 | 257k   11M|   0     0 |1777   841
  0   5  59  30   0   7|   0     0 | 270k   12M|   0     0 |1758   836

After patch:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   5  35  57   0   4|   0     0 | 255k   11M|   0     0 |1705   879
  0   4  38  53   0   4|   0     0 | 326k   14M|   0     0 |1940   980
  0   3  36  59   0   2|   0     0 | 291k   13M|   0     0 |1970   970
  0   4  28  66   0   2|   0     0 | 290k   13M|   0     0 |1805   928
  0   6  38  54   0   3|   0     0 | 230k   10M|   0     0 |1866   842
  0   5  44  49   0   4|   0     0 | 278k   12M|   0     0 |1808   868

Thanks,
Fengguang
---
writeback: account IO throttling wait as iowait

It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.

There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.

If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the io_schedule.

CC KAMEZAWA Hiroyuki <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 kernel/sched.c      |    3 ---
 mm/page-writeback.c |    3 ++-
 2 files changed, 2 insertions(+), 4 deletions(-)
mm/page-writeback.c +2 -1
@@ -566,7 +566,8 @@ static void balance_dirty_pages(struct a
 		if (pages_written >= write_chunk)
 			break;		/* We've done our duty */
 
-		schedule_timeout_interruptible(pause);
+		__set_current_state(TASK_INTERRUPTIBLE);
+		io_schedule_timeout(pause);
 
 		/*
 		 * Increase the delay for each loop, up to our previous
kernel/sched.c +0 -3
@@ -6720,9 +6720,6 @@ EXPORT_SYMBOL(yield);
 /*
  * This task is about to go to sleep on IO. Increment rq->nr_iowait so
  * that process accounting knows that this is a task in IO wait state.
- *
- * But don't do that if it is a deliberate, throttling IO wait (this task
- * has set its backing_dev_info: the queue against which it should throttle)
  */
 void __sched io_schedule(void)
 {
#63 lore raw

Re: [PATCH] writeback: account IO throttling wait as iowait

Quoted 57 lines · Wu Fengguang
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Wu Fengguang
Quoted 2 lines

How this runqueue->nr_iowait is handled now ?

Good question. io_schedule() has an old comment for throttling IO wait:

* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)

So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)

Quoted 7 lines
Quoted 4 lines

Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..

Hmm?

Yep agree, if we're deliberately waiting on IO, it should count as iowait time.

Then let's revert to the old behavior :)

For one single cp, it increases iowait from 29% to 56%.

Before patch:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   4  64  28   0   3|   0     0 | 272k   10M|   0     0 |1854   863
  0   6  69  23   0   3|   0     0 | 249k   11M|   0     0 |1709   865
  0   6  64  27   0   4|   0     0 | 235k   10M|   0     0 |1807   788
  0   4  61  30   0   4|   0     0 | 271k   12M|   0     0 |1910   898
  0   4  72  21   0   4|   0     0 | 289k   13M|   0     0 |1832   905
  0   6  58  35   0   2|   0     0 | 252k   11M|   0     0 |1713   900
  0   4  54  38   0   4|   0     0 | 257k   11M|   0     0 |1777   841
  0   5  59  30   0   7|   0     0 | 270k   12M|   0     0 |1758   836

After patch:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   5  35  57   0   4|   0     0 | 255k   11M|   0     0 |1705   879
  0   4  38  53   0   4|   0     0 | 326k   14M|   0     0 |1940   980
  0   3  36  59   0   2|   0     0 | 291k   13M|   0     0 |1970   970
  0   4  28  66   0   2|   0     0 | 290k   13M|   0     0 |1805   928
  0   6  38  54   0   3|   0     0 | 230k   10M|   0     0 |1866   842
  0   5  44  49   0   4|   0     0 | 278k   12M|   0     0 |1808   868

Thanks,
Fengguang
---
writeback: account IO throttling wait as iowait

Quoted 8 lines

It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.

There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.

If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the io_schedule.

Thanks, queued up.

Signature · 1 line
Jens Axboe
#64 lore raw

Re: [PATCH] writeback: account IO throttling wait as iowait

Quoted 6 lines · Jens Axboe
Quoted 6 lines

---
writeback: account IO throttling wait as iowait

It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.

Quoted 7 lines
Quoted 5 lines

There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.

If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the io_schedule.

Thanks, queued up.

Thank you. Would you also pick up this one if it's OK to you?

Thanks,
Fengguang

---
writeback: kill space in debugfs item name

The space is not script friendly, kill it.

---
 mm/backing-dev.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/backing-dev.c +1 -1
@@ -92,7 +92,7 @@ static int bdi_debug_stats_show(struct s
 		   "BdiDirtyThresh:   %8lu kB\n"
 		   "DirtyThresh:      %8lu kB\n"
 		   "BackgroundThresh: %8lu kB\n"
-		   "WriteBack threads:%8lu\n"
+		   "WritebackThreads: %8lu\n"
 		   "b_dirty:          %8lu\n"
 		   "b_io:             %8lu\n"
 		   "b_more_io:        %8lu\n"
#65 lore raw

Re: [PATCH] writeback: account IO throttling wait as iowait

Quoted 10 lines · Wu Fengguang
Quoted 9 lines · Jens Axboe
Quoted 9 lines

---
writeback: account IO throttling wait as iowait

It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.

There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the io_schedule.

Thanks, queued up.

Thank you. Would you also pick up this one if it's OK to you?

Sure, looks appropriate.

Signature · 1 line
Jens Axboe
#66 lore raw

Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()

In reply to KAMEZAWA Hiroyuki
Quoted 21 lines · KAMEZAWA Hiroyuki

May I ask a question ? (maybe not directly related to this patch itself, sorry)

Recent works as "writeback: switch to per-bdi threads for flushing data" removed congestion_wait() from balance_dirty_pages() and added schedule_timeout_interruptible().

And this one replaces it with wake_up+wait_queue.

IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)

{
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
        cputime64_t cputime64 = cputime_to_cputime64(cputime);
        struct rq *rq = this_rq();

        if (atomic_read(&rq->nr_iowait) > 0)
                cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
        else
                cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}
Quoted 6 lines

==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().

How this runqueue->nr_iowait is handled now ?

Ah, I think you've got a good point, we need a io_schedule() there.

#67 lore raw

PATCH 19/45 writeback: remove the loop in balance_dirty_pages()

In reply to Wu Fengguang

The loop is no longer necessary. Remove it without behavior change.

---
 mm/page-writeback.c |   81 +++++++++++++++++++-----------------------
 1 file changed, 38 insertions(+), 43 deletions(-)
mm/page-writeback.c +38 -43
@@ -470,60 +470,55 @@ static void balance_dirty_pages(struct a
 	int dirty_exceeded;
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
-	for (;;) {
-		nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
-				 global_page_state(NR_UNSTABLE_NFS);
-		nr_writeback = global_page_state(NR_WRITEBACK) +
-			       global_page_state(NR_WRITEBACK_TEMP);
+	nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
+			 global_page_state(NR_UNSTABLE_NFS);
+	nr_writeback = global_page_state(NR_WRITEBACK) +
+		       global_page_state(NR_WRITEBACK_TEMP);
 
-		global_dirty_thresh(&background_thresh, &dirty_thresh);
-
-		/*
-		 * Throttle it only when the background writeback cannot
-		 * catch-up. This avoids (excessively) small writeouts
-		 * when the bdi limits are ramping up.
-		 */
-		if (nr_reclaimable + nr_writeback <
-		    (background_thresh + dirty_thresh) / 2)
-			break;
+	global_dirty_thresh(&background_thresh, &dirty_thresh);
 
-		bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);
+	/*
+	 * Throttle it only when the background writeback cannot
+	 * catch-up. This skips the ramp up phase of bdi limits.
+	 */
+	if (nr_reclaimable + nr_writeback <
+			(background_thresh + dirty_thresh) / 2)
+		goto out;
 
-		/*
-		 * In order to avoid the stacked BDI deadlock we need
-		 * to ensure we accurately count the 'dirty' pages when
-		 * the threshold is low.
-		 *
-		 * Otherwise it would be possible to get thresh+n pages
-		 * reported dirty, even though there are thresh-m pages
-		 * actually dirty; with m+n sitting in the percpu
-		 * deltas.
-		 */
-		if (bdi_thresh < 2*bdi_stat_error(bdi)) {
-			bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
-		} else {
-			bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
-			bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
-		}
+	bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);
+	/*
+	 * In order to avoid the stacked BDI deadlock we need
+	 * to ensure we accurately count the 'dirty' pages when
+	 * the threshold is low.
+	 *
+	 * Otherwise it would be possible to get thresh+n pages
+	 * reported dirty, even though there are thresh-m pages
+	 * actually dirty; with m+n sitting in the percpu
+	 * deltas.
+	 */
+	if (bdi_thresh >= 2 * bdi_stat_error(bdi)) {
+		bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
+		bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
+	} else {
+		bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
+		bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
+	}
 
-		dirty_exceeded =
-			(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
+	dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
 			|| (nr_reclaimable + nr_writeback >= dirty_thresh);
 
-		if (!dirty_exceeded)
-			break;
+	if (!dirty_exceeded)
+		goto out;
 
-		if (!bdi->dirty_exceeded)
-			bdi->dirty_exceeded = 1;
+	if (!bdi->dirty_exceeded)
+		bdi->dirty_exceeded = 1;
 
-		bdi_writeback_wait(bdi, write_chunk);
-		break;
-	}
+	bdi_writeback_wait(bdi, write_chunk);
 
-	if (!dirty_exceeded && bdi->dirty_exceeded)
+	if (bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 0;
 
+out:
 	/*
 	 * In laptop mode, we wait until hitting the higher threshold before
 	 * starting background writeout, and then write out all the way down
#68 lore raw

PATCH 20/45 NFS: introduce writeback wait queue

In reply to Wu Fengguang

The generic writeback routines are departing from congestion_wait() in preferance of get_request_wait(), aka. waiting on the block queues.

Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.

In perticular, balance_dirty_pages() will exit after it pushes write_chunk pages into the PG_writeback page pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number of PG_writeback pages if it is not explicitly limited.

CC Jens Axboe <[email protected]>
CC Chris Mason <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Trond Myklebust <[email protected]>

---

The wait time and network throughput varies a lot! this is a major problem. This means nfs_end_page_writeback() is not called smoothly over time, even when there are plenty of PG_writeback pages on the client side.

[  397.828509] write_bandwidth: comm=nfsiod pages=192 time=16ms
[  397.850976] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  403.065244] write_bandwidth: comm=nfsiod pages=192 time=5212ms
[  403.549134] write_bandwidth: comm=nfsiod pages=1536 time=144ms
[  403.570717] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  403.595749] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  403.622171] write_bandwidth: comm=nfsiod pages=192 time=24ms
[  403.651779] write_bandwidth: comm=nfsiod pages=192 time=28ms
[  403.680543] write_bandwidth: comm=nfsiod pages=192 time=24ms
[  403.712572] write_bandwidth: comm=nfsiod pages=192 time=28ms
[  403.751552] write_bandwidth: comm=nfsiod pages=192 time=36ms
[  403.785979] write_bandwidth: comm=nfsiod pages=192 time=28ms
[  403.823995] write_bandwidth: comm=nfsiod pages=192 time=36ms
[  403.858970] write_bandwidth: comm=nfsiod pages=192 time=32ms
[  403.880786] write_bandwidth: comm=nfsiod pages=192 time=16ms
[  403.902732] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  403.925925] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  403.952044] write_bandwidth: comm=nfsiod pages=258 time=24ms
[  403.974006] write_bandwidth: comm=nfsiod pages=192 time=16ms
[  403.995989] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  405.031049] write_bandwidth: comm=nfsiod pages=192 time=1032ms
[  405.257635] write_bandwidth: comm=nfsiod pages=1536 time=192ms
[  405.279069] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  405.300843] write_bandwidth: comm=nfsiod pages=192 time=16ms
[  405.326031] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  405.350843] write_bandwidth: comm=nfsiod pages=192 time=24ms
[  405.375160] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  409.331015] write_bandwidth: comm=nfsiod pages=192 time=3952ms
[  409.587928] write_bandwidth: comm=nfsiod pages=1536 time=152ms
[  409.610068] write_bandwidth: comm=nfsiod pages=192 time=20ms
[  409.635736] write_bandwidth: comm=nfsiod pages=192 time=24ms
# vmmon -d 1 nr_writeback nr_dirty nr_unstable
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913
            11056            56887             3876
            11062            55298             8155
            11214            54485             9838
            11225            54461             9852
            11225            54461             9852
            11225            54461             4582
            22342            35535             7823

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0   9  92   0   0|   0     0 |  66B  306B|   0     0 |1003   377 
  0   1  39  60   0   1|   0     0 |  90k 1361k|   0     0 |1765  1599 
  0  15  12  43   0  31|   0     0 |2292k   34M|   0     0 |  12k   16k
  0   0  16  84   0   0|   0     0 | 132B  306B|   0     0 |1003   376 
  0   0  43  57   0   0|   0     0 |  66B  306B|   0     0 |1004   376 
  0   7  25  55   0  13|   0     0 |1202k   18M|   0     0 |7331  8921 
  0   8  21  55   0  15|   0     0 |1195k   18M|   0     0 |5382  6579 
  0   0  38  62   0   0|   0     0 |  66B  306B|   0     0 |1002   371 
  0   0  33  67   0   0|   0     0 |  66B  306B|   0     0 |1003   376 
  0  14  20  41   0  24|   0     0 |1621k   24M|   0     0 |8549    10k
  0   5  31  55   0   9|   0     0 | 769k   11M|   0     0 |4444  5180 
  0   0  18  82   0   0|   0     0 |  66B  568B|   0     0 |1004   377 
  0   1  41  54   0   3|   0     0 | 184k 2777k|   0     0 |2609  2619 
  1  13  22  43   0  22|   0     0 |1572k   23M|   0     0 |8138    10k
  0  11   9  59   0  20|   0     0 |1861k   27M|   0     0 |9576    13k
  0   5  23  66   0   5|   0     0 | 540k 8122k|   0     0 |2816  2885 
 fs/nfs/client.c           |    2 
 fs/nfs/write.c            |   81 +++++++++++++++++++++++++++++++-----
 include/linux/nfs_fs_sb.h |    1 
 3 files changed, 73 insertions(+), 11 deletions(-)
fs/nfs/write.c +70 -11
@@ -187,11 +187,54 @@ static int wb_priority(struct writeback_
  * NFS congestion control
  */
 
+#define NFS_WAIT_PAGES	(1024L >> (PAGE_CACHE_SHIFT - 10))
 int nfs_congestion_kb;
 
-#define NFS_CONGESTION_ON_THRESH 	(nfs_congestion_kb >> (PAGE_SHIFT-10))
-#define NFS_CONGESTION_OFF_THRESH	\
-	(NFS_CONGESTION_ON_THRESH - (NFS_CONGESTION_ON_THRESH >> 2))
+/*
+ * SYNC requests will block on (2*limit) and wakeup on (2*limit-NFS_WAIT_PAGES)
+ * ASYNC requests will block on (limit) and wakeup on (limit - NFS_WAIT_PAGES)
+ * In this way SYNC writes will never be blocked by ASYNC ones.
+ */
+
+static void nfs_set_congested(long nr, long limit,
+			      struct backing_dev_info *bdi)
+{
+	if (nr > limit && !test_bit(BDI_async_congested, &bdi->state))
+		set_bdi_congested(bdi, BLK_RW_ASYNC);
+	else if (nr > 2 * limit && !test_bit(BDI_sync_congested, &bdi->state))
+		set_bdi_congested(bdi, BLK_RW_SYNC);
+}
+
+static void nfs_wait_contested(int is_sync,
+			       struct backing_dev_info *bdi,
+			       wait_queue_head_t *wqh)
+{
+	int waitbit = is_sync ? BDI_sync_congested : BDI_async_congested;
+
+	wait_event(wqh[is_sync], !test_bit(waitbit, &bdi->state));
+}
+
+static void nfs_wakeup_congested(long nr, long limit,
+				 struct backing_dev_info *bdi,
+				 wait_queue_head_t *wqh)
+{
+	if (nr < 2*limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_sync_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_SYNC);
+		if (waitqueue_active(&wqh[BLK_RW_SYNC])) {
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_SYNC]);
+		}
+	}
+	if (nr < limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_async_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_ASYNC);
+		if (waitqueue_active(&wqh[BLK_RW_ASYNC])) {
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_ASYNC]);
+		}
+	}
+}
 
 static int nfs_set_page_writeback(struct page *page)
 {
@@ -201,11 +244,9 @@ static int nfs_set_page_writeback(struct
 		struct inode *inode = page->mapping->host;
 		struct nfs_server *nfss = NFS_SERVER(inode);
 
-		if (atomic_long_inc_return(&nfss->writeback) >
-				NFS_CONGESTION_ON_THRESH) {
-			set_bdi_congested(&nfss->backing_dev_info,
-						BLK_RW_ASYNC);
-		}
+		nfs_set_congested(atomic_long_inc_return(&nfss->writeback),
+				  nfs_congestion_kb >> (PAGE_SHIFT-10),
+				  &nfss->backing_dev_info);
 	}
 	return ret;
 }
@@ -216,8 +257,11 @@ static void nfs_end_page_writeback(struc
 	struct nfs_server *nfss = NFS_SERVER(inode);
 
 	end_page_writeback(page);
-	if (atomic_long_dec_return(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH)
-		clear_bdi_congested(&nfss->backing_dev_info, BLK_RW_ASYNC);
+
+	nfs_wakeup_congested(atomic_long_dec_return(&nfss->writeback),
+			     nfs_congestion_kb >> (PAGE_SHIFT-10),
+			     &nfss->backing_dev_info,
+			     nfss->writeback_wait);
 }
 
 static struct nfs_page *nfs_find_and_lock_request(struct page *page)
@@ -309,19 +353,34 @@ static int nfs_writepage_locked(struct p
 
 int nfs_writepage(struct page *page, struct writeback_control *wbc)
 {
+	struct inode *inode = page->mapping->host;
+	struct nfs_server *nfss = NFS_SERVER(inode);
 	int ret;
 
 	ret = nfs_writepage_locked(page, wbc);
 	unlock_page(page);
+
+	nfs_wait_contested(wbc->sync_mode == WB_SYNC_ALL,
+			   &nfss->backing_dev_info,
+			   nfss->writeback_wait);
+
 	return ret;
 }
 
-static int nfs_writepages_callback(struct page *page, struct writeback_control *wbc, void *data)
+static int nfs_writepages_callback(struct page *page,
+				   struct writeback_control *wbc, void *data)
 {
+	struct inode *inode = page->mapping->host;
+	struct nfs_server *nfss = NFS_SERVER(inode);
 	int ret;
 
 	ret = nfs_do_writepage(page, wbc, data);
 	unlock_page(page);
+
+	nfs_wait_contested(wbc->sync_mode == WB_SYNC_ALL,
+			   &nfss->backing_dev_info,
+			   nfss->writeback_wait);
+
 	return ret;
 }
 
include/linux/nfs_fs_sb.h +1 -0
@@ -108,6 +108,7 @@ struct nfs_server {
 	struct nfs_iostats *	io_stats;	/* I/O statistics */
 	struct backing_dev_info	backing_dev_info;
 	atomic_long_t		writeback;	/* number of writeback pages */
+	wait_queue_head_t	writeback_wait[2];
 	int			flags;		/* various flags */
 	unsigned int		caps;		/* server capabilities */
 	unsigned int		rsize;		/* read size */
fs/nfs/client.c +2 -0
@@ -991,6 +991,8 @@ static struct nfs_server *nfs_alloc_serv
 	INIT_LIST_HEAD(&server->master_link);
 
 	atomic_set(&server->active, 0);
+	init_waitqueue_head(&server->writeback_wait[BLK_RW_SYNC]);
+	init_waitqueue_head(&server->writeback_wait[BLK_RW_ASYNC]);
 
 	server->io_stats = nfs_alloc_iostats();
 	if (!server->io_stats) {
#69 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

Quoted 19 lines · Wu Fengguang

plain text document attachment (writeback-nfs-request-queue.patch) The generic writeback routines are departing from congestion_wait() in preferance of get_request_wait(), aka. waiting on the block queues.

Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.

In perticular, balance_dirty_pages() will exit after it pushes write_chunk pages into the PG_writeback page pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number of PG_writeback pages if it is not explicitly limited.

CC Jens Axboe <[email protected]>
CC Chris Mason <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Trond Myklebust <[email protected]>
Quoted 5 lines

---

The wait time and network throughput varies a lot! this is a major problem. This means nfs_end_page_writeback() is not called smoothly over time, even when there are plenty of PG_writeback pages on the client side.

Could that be some ack batching from the nfs protocol/server?

Quoted 12 lines
fs/nfs/write.c +11 -0
+static void nfs_wakeup_congested(long nr, long limit,
+				 struct backing_dev_info *bdi,
+				 wait_queue_head_t *wqh)
+{
+	if (nr < 2*limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_sync_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_SYNC);
+		if (waitqueue_active(&wqh[BLK_RW_SYNC])) {
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_SYNC]);
+		}

fs/nfs/write.c static int wb_priority(struct writeback_ · line 237

Quoted 10 lines
fs/nfs/write.c +10 -0
+	}
+	if (nr < limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_async_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_ASYNC);
+		if (waitqueue_active(&wqh[BLK_RW_ASYNC])) {
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_ASYNC]);
+		}
+	}
+}

wakeup implies a full memory barrier.

#70 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Wu Fengguang

plain text document attachment (writeback-nfs-request-queue.patch) The generic writeback routines are departing from congestion_wait() in preferance of get_request_wait(), aka. waiting on the block queues.

Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.

In perticular, balance_dirty_pages() will exit after it pushes write_chunk pages into the PG_writeback page pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number of PG_writeback pages if it is not explicitly limited.

CC Jens Axboe <[email protected]>
CC Chris Mason <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Trond Myklebust <[email protected]>

---

Quoted 5 lines
Quoted 3 lines

The wait time and network throughput varies a lot! this is a major problem. This means nfs_end_page_writeback() is not called smoothly over time, even when there are plenty of PG_writeback pages on the client side.

Could that be some ack batching from the nfs protocol/server?

Yes possibly. Another possibility is some works in the nfsiod workqueue takes up a long time.

Quoted 16 lines
Quoted 16 lines
fs/nfs/write.c +16 -0
+static void nfs_wakeup_congested(long nr, long limit,
+				 struct backing_dev_info *bdi,
+				 wait_queue_head_t *wqh)
+{
+	if (nr < 2*limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_sync_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_SYNC);
+		if (waitqueue_active(&wqh[BLK_RW_SYNC])) {
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_SYNC]);
+		}
+	}
+	if (nr < limit - min(limit/8, NFS_WAIT_PAGES)) {
+		if (test_bit(BDI_async_congested, &bdi->state))
+			clear_bdi_congested(bdi, BLK_RW_ASYNC);
+		if (waitqueue_active(&wqh[BLK_RW_ASYNC])) {
Quoted 8 lines

fs/nfs/write.c static int wb_priority(struct writeback_ · line 237

Quoted 6 lines
fs/nfs/write.c +5 -0
+			smp_mb__after_clear_bit();
+			wake_up(&wqh[BLK_RW_ASYNC]);
+		}
+	}
+}

wakeup implies a full memory barrier.

If so, this smp_mb__after_clear_bit() line is also not necessary?

void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...

        clear_bit(bit, &bdi->state);
        smp_mb__after_clear_bit();
        if (waitqueue_active(wqh))
                wake_up(wqh);

Thanks,
Fengguang

#71 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

Quoted 5 lines · Wu Fengguang
Quoted 2 lines

wakeup implies a full memory barrier.

If so, this smp_mb__after_clear_bit() line is also not necessary?

Quoted 7 lines

void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...

        clear_bit(bit, &bdi->state);
        smp_mb__after_clear_bit();
        if (waitqueue_active(wqh))
                wake_up(wqh);

Depends on if the barrier is needed even when the wakeup doesn't happen I guess ;-)

#72 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

Quoted 14 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang
Quoted 2 lines

wakeup implies a full memory barrier.

If so, this smp_mb__after_clear_bit() line is also not necessary?

Quoted 7 lines

void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...

        clear_bit(bit, &bdi->state);
        smp_mb__after_clear_bit();
        if (waitqueue_active(wqh))
                wake_up(wqh);
Quoted 2 lines

Depends on if the barrier is needed even when the wakeup doesn't happen I guess ;-)

I see. Thanks!

#73 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

In reply to Wu Fengguang
Quoted 5 lines · Wu Fengguang
Quoted 1 line · Peter Zijlstra

wakeup implies a full memory barrier.

If so, this smp_mb__after_clear_bit() line is also not necessary?

Quoted 7 lines

void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...

        clear_bit(bit, &bdi->state);
        smp_mb__after_clear_bit();
        if (waitqueue_active(wqh))
                wake_up(wqh);

Typically in these patterns you do need a barrier. You need to ensure the load to check the waitqueue is performed after the clear_bit.

The other side does:
add_to_waitqueue
set_current_state /* has mb() in it */
if (test_bit())
schedule()

#74 lore raw

Re: [PATCH 20/45] NFS: introduce writeback wait queue

Quoted 18 lines · Nick Piggin
Quoted 5 lines · Wu Fengguang
Quoted 1 line · Peter Zijlstra

wakeup implies a full memory barrier.

If so, this smp_mb__after_clear_bit() line is also not necessary?

Quoted 7 lines

void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...

        clear_bit(bit, &bdi->state);
        smp_mb__after_clear_bit();
        if (waitqueue_active(wqh))
                wake_up(wqh);

Typically in these patterns you do need a barrier. You need to ensure the load to check the waitqueue is performed after the clear_bit.

Quoted 5 lines

The other side does:
add_to_waitqueue
set_current_state /* has mb() in it */
if (test_bit())
schedule()

Thanks, just moved the mb() up to follow clear_bit():

                if (test_bit(BDI_sync_congested, &bdi->state)) {
                        clear_bdi_congested(bdi, BLK_RW_SYNC);
                        smp_mb__after_clear_bit();
                }
                if (waitqueue_active(&wqh[BLK_RW_SYNC]))
                        wake_up(&wqh[BLK_RW_SYNC]);

Thanks,
Fengguang

#75 lore raw

PATCH 21/45 writeback: estimate bdi write bandwidth

In reply to Wu Fengguang

Estimate bdi write bandwidth in bdi_writeback_wakeup(), at which time the queue is not starved due to the dirtying process and the associated background writeback.

The estimation should be able to reflect the max device capability, unless there are busy reads, in which case we need lower nr_to_write anyway.

CC Theodore Ts'o <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>

---

TODO: the estimated write bandwidth (~30MB/s) is mysteriously only half the real throughput (~60MB/s). Here are some debug printk from bdi_calc_write_bandwidth():

printk("write_bandwidth: comm=%s pages=%lu time=%lums\n",
current->comm, nr_pages, time * 1000 / HZ);

[ 1093.397700] write_bandwidth: comm=swapper pages=1536 time=204ms
[ 1093.594319] write_bandwidth: comm=swapper pages=1536 time=196ms
[ 1093.796642] write_bandwidth: comm=swapper pages=1536 time=200ms
[ 1093.986128] write_bandwidth: comm=swapper pages=1536 time=192ms
[ 1094.179983] write_bandwidth: comm=swapper pages=1536 time=192ms
[ 1094.374021] write_bandwidth: comm=swapper pages=1536 time=196ms
[ 1094.570611] write_bandwidth: comm=swapper pages=1536 time=196ms
[ 1094.771847] write_bandwidth: comm=swapper pages=1536 time=200ms
[ 1094.961981] write_bandwidth: comm=swapper pages=1536 time=192ms

Workload is several concurrent copies.

 fs/fs-writeback.c           |   30 +++++++++++++++++++++---------
 include/linux/backing-dev.h |    2 ++
 include/linux/writeback.h   |   10 ++++++++++
 mm/backing-dev.c            |    2 ++
 4 files changed, 35 insertions(+), 9 deletions(-)
fs/fs-writeback.c +21 -9
@@ -266,7 +266,23 @@ void bdi_start_writeback(struct backing_
 	bdi_alloc_queue_work(bdi, &args);
 }
 
+static int bdi_writeback_chunk(struct backing_dev_info *bdi)
+{
+	return max(MIN_WRITEBACK_PAGES, bdi->write_bandwidth);
+}
+
+static void bdi_calc_write_bandwidth(struct backing_dev_info *bdi,
+				     unsigned long nr_pages,
+				     unsigned long time)
+{
+	unsigned long bw;
+
+	bw = HZ * nr_pages / (time | 1);
+	bdi->write_bandwidth = (bdi->write_bandwidth * 63 + bw) / 64;
+}
+
 struct dirty_throttle_task {
+	unsigned long		start_time;
 	long			nr_pages;
 	struct list_head	list;
 	struct completion	complete;
@@ -275,6 +291,7 @@ struct dirty_throttle_task {
 void bdi_writeback_wait(struct backing_dev_info *bdi, long nr_pages)
 {
 	struct dirty_throttle_task tt = {
+		.start_time = jiffies,
 		.nr_pages = nr_pages,
 		.complete = COMPLETION_INITIALIZER_ONSTACK(tt.complete),
 	};
@@ -314,6 +331,9 @@ int bdi_writeback_wakeup(struct backing_
 		tt = list_entry(bdi->throttle_list.prev,
 				struct dirty_throttle_task, list);
 		list_del(&tt->list);
+		if (atomic_read(&bdi->throttle_pages) == 0)
+			bdi_calc_write_bandwidth(bdi, tt->nr_pages,
+						 jiffies - tt->start_time);
 		complete(&tt->complete);
 	}
 	/*
@@ -323,6 +343,7 @@ int bdi_writeback_wakeup(struct backing_
 		tt = list_entry(bdi->throttle_list.prev,
 				struct dirty_throttle_task, list);
 		atomic_set(&bdi->throttle_pages, tt->nr_pages);
+		tt->start_time = jiffies;
 	} else {
 		tt = NULL;
 		atomic_set(&bdi->throttle_pages, DIRTY_THROTTLE_PAGES_STOP * 2);
@@ -717,15 +738,6 @@ void writeback_inodes_wbc(struct writeba
 	writeback_inodes_wb(&bdi->wb, wbc);
 }
 
-/*
- * The maximum number of pages to writeout in a single bdi flush/kupdate
- * operation.  We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode.  Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES     1024
-
 static inline bool over_bground_thresh(void)
 {
 	unsigned long background_thresh, dirty_thresh;
include/linux/writeback.h +10 -0
@@ -14,6 +14,16 @@ extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
 /*
+ * The max number of pages to writeout for each inode.
+ *
+ * We honor each inode a nr_to_write that will take about 1 second
+ * to finish, based on dynamic estimation of the bdi's write bandwidth.
+ * MAX_ serves as initial bandwidth value; MIN_ serves as low boundary.
+ */
+#define MAX_WRITEBACK_PAGES	(128 << (20 - PAGE_CACHE_SHIFT))
+#define MIN_WRITEBACK_PAGES	( 16 << (20 - PAGE_CACHE_SHIFT))
+
+/*
  * fs/fs-writeback.c
  */
 enum writeback_sync_modes {
include/linux/backing-dev.h +2 -0
@@ -86,6 +86,8 @@ struct backing_dev_info {
 
 	struct list_head work_list;
 
+	int write_bandwidth;	/* pages per second */
+
 	/*
 	 * dirtier process throttling
 	 */
mm/backing-dev.c +2 -0
@@ -646,6 +646,8 @@ int bdi_init(struct backing_dev_info *bd
 	bdi->wb_mask = 1;
 	bdi->wb_cnt = 1;
 
+	bdi->write_bandwidth = MAX_WRITEBACK_PAGES;
+
 	spin_lock_init(&bdi->throttle_lock);
 	INIT_LIST_HEAD(&bdi->throttle_list);
 	atomic_set(&bdi->throttle_pages, DIRTY_THROTTLE_PAGES_STOP * 2);
#76 lore raw

Re: [PATCH 21/45] writeback: estimate bdi write bandwidth

fs/fs-writeback.c void bdi_start_writeback(struct backing_ · line 282

Quoted 9 lines · Wu Fengguang
fs/fs-writeback.c +9 -0
+static void bdi_calc_write_bandwidth(struct backing_dev_info *bdi,
+                                    unsigned long nr_pages,
+                                    unsigned long time)
+{
+       unsigned long bw;
+
+       bw = HZ * nr_pages / (time | 1);
+       bdi->write_bandwidth = (bdi->write_bandwidth * 63 + bw) / 64;
+}

If you have block times < 1 jiffy this all falls apart quite quickly. You could perhaps try to use cpu_clock() for ns resolution timestamps if this is a real issue.

(I could imagine fast arrays with huge throughput causing small sleeps, resulting in underestimates of their bandwidth)

Also, 63/64 seems rather slow progress.. maybe that's good.

#77 lore raw

Re: [PATCH 21/45] writeback: estimate bdi write bandwidth

Quoted 14 lines · Peter Zijlstra

fs/fs-writeback.c void bdi_start_writeback(struct backing_ · line 282

Quoted 9 lines · Wu Fengguang
fs/fs-writeback.c +9 -0
+static void bdi_calc_write_bandwidth(struct backing_dev_info *bdi,
+                                    unsigned long nr_pages,
+                                    unsigned long time)
+{
+       unsigned long bw;
+
+       bw = HZ * nr_pages / (time | 1);
+       bdi->write_bandwidth = (bdi->write_bandwidth * 63 + bw) / 64;
+}

If you have block times < 1 jiffy this all falls apart quite quickly. You could perhaps try to use cpu_clock() for ns resolution timestamps if this is a real issue.

Good point. I'd like to view it in the opposite way: if a sleep takes <= 1 jiffy to wakeup, that would mean high overheads. We could as well lift the ratelimit values :)

Quoted 4 lines

(I could imagine fast arrays with huge throughput causing small sleeps, resulting in underestimates of their bandwidth)

Also, 63/64 seems rather slow progress.. maybe that's good.

Whatever will be fast enough for servers that keep running for years. So the main concern would be how fast it can adapt to slow usb devices.

Here is a quick calculation. Each iteration writes ~800KB:

irb > a=128; 1.upto(100) { |i| a = (a * 63 + 5) / 64; printf "%d\t%d\n", i, a }

[...]

86      12
87      11
88      10
89      9
90      8
91      7
92      6
93      5
94      5

Looks slow. 8 or 16 seems better, which stabilize after writing about 16MB or 32MB data:

a=128; 1.upto(100) { |i| a = (a * 7 + 5) / 8; printf "%d\t%d\n", i, a }

1       112
2       98
3       86
4       75
5       66
6       58
7       51
8       45
9       40
10      35
11      31
12      27
13      24
14      21
15      19
16      17
17      15
18      13
19      12
20      11
21      10
22      9
23      8
24      7
25      6
26      5
27      5
28      5
29      5

16:

34      13
35      12
36      11
37      10
38      9
39      8
40      7
41      6
42      5
43      5

Thanks,
Fengguang

#78 lore raw

PATCH 22/45 writeback: show bdi write bandwidth in debugfs

In reply to Wu Fengguang
CC Theodore Ts'o <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
---
 mm/backing-dev.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
mm/backing-dev.c +4 -2
@@ -93,6 +93,7 @@ static int bdi_debug_stats_show(struct s
 		   "BdiDirtyThresh:   %8lu kB\n"
 		   "DirtyThresh:      %8lu kB\n"
 		   "BackgroundThresh: %8lu kB\n"
+		   "WriteBandwidth:   %8lu kBps\n"
 		   "WriteBack threads:%8lu\n"
 		   "b_dirty:          %8lu\n"
 		   "b_io:             %8lu\n"
@@ -104,8 +105,9 @@ static int bdi_debug_stats_show(struct s
 		   "wb_cnt:           %8u\n",
 		   (unsigned long) K(bdi_stat(bdi, BDI_WRITEBACK)),
 		   (unsigned long) K(bdi_stat(bdi, BDI_RECLAIMABLE)),
-		   K(bdi_thresh), K(dirty_thresh),
-		   K(background_thresh), nr_wb, nr_dirty, nr_io, nr_more_io,
+		   K(bdi_thresh), K(dirty_thresh), K(background_thresh),
+		   (unsigned long) K(bdi->write_bandwidth),
+		   nr_wb, nr_dirty, nr_io, nr_more_io,
 		   !list_empty(&bdi->bdi_list), bdi->state, bdi->wb_mask,
 		   !list_empty(&bdi->wb_list), bdi->wb_cnt);
 #undef K
#79 lore raw

PATCH 23/45 writeback: kill space in debugfs item name

In reply to Wu Fengguang

The space is not script friendly, kill it.

---
 mm/backing-dev.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/backing-dev.c +1 -1
@@ -94,7 +94,7 @@ static int bdi_debug_stats_show(struct s
 		   "DirtyThresh:      %8lu kB\n"
 		   "BackgroundThresh: %8lu kB\n"
 		   "WriteBandwidth:   %8lu kBps\n"
-		   "WriteBack threads:%8lu\n"
+		   "WriteBackThreads: %8lu\n"
 		   "b_dirty:          %8lu\n"
 		   "b_io:             %8lu\n"
 		   "b_more_io:        %8lu\n"
#80 lore raw

PATCH 24/45 writeback: remove global nr_to_write and use timeout instead

In reply to Wu Fengguang

The global wbc.timeout could be useful when we decide to do two sync scans on dirty pages and dirty metadata. XFS could say: please return to sync dirty metadata after 10s. Would need another b_io_metadata queue, but that's possible.

CC Theodore Ts'o <[email protected]>
CC Chris Mason <[email protected]>
CC Dave Chinner <[email protected]>
CC Christoph Hellwig <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 fs/fs-writeback.c         |    6 ++++++
 include/linux/writeback.h |    1 +
 mm/backing-dev.c          |    1 +
 3 files changed, 8 insertions(+)
fs/fs-writeback.c +6 -0
@@ -666,6 +666,10 @@ static void writeback_inodes_wb(struct b
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
 	const unsigned long start = jiffies;	/* livelock avoidance */
+	unsigned long stop_time = 0;
+
+	if (wbc->timeout)
+		stop_time = (start + wbc->timeout) | 1;
 
 	spin_lock(&inode_lock);
 
@@ -723,6 +727,8 @@ static void writeback_inodes_wb(struct b
 		}
 		if (!list_empty(&wb->b_more_io))
 			wbc->more_io = 1;
+		if (stop_time && time_after(jiffies, stop_time))
+			break;
 	}
 
 	unpin_sb_for_writeback(&pin_sb);
mm/backing-dev.c +1 -0
@@ -339,6 +339,7 @@ static void bdi_flush_io(struct backing_
 		.older_than_this	= NULL,
 		.range_cyclic		= 1,
 		.nr_to_write		= 1024,
+		.timeout		= HZ,
 	};
 
 	writeback_inodes_wbc(&wbc);
include/linux/writeback.h +1 -0
@@ -42,6 +42,7 @@ struct writeback_control {
 	struct super_block *sb;		/* if !NULL, only write inodes from
 					   this super_block */
 	enum writeback_sync_modes sync_mode;
+	int timeout;
 	unsigned long *older_than_this;	/* If !NULL, only write back inodes
 					   older than this */
 	long nr_to_write;		/* Write this many pages, and decrement
#81 lore raw

PATCH 25/45 writeback: convert wbc.nr_to_write to per-file parameter

In reply to Wu Fengguang
CC Theodore Ts'o <[email protected]>
CC Chris Mason <[email protected]>
CC Dave Chinner <[email protected]>
CC Christoph Hellwig <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 fs/fs-writeback.c         |   24 ++++++++++++------------
 include/linux/writeback.h |    5 +++--
 mm/backing-dev.c          |    1 -
 3 files changed, 15 insertions(+), 15 deletions(-)
fs/fs-writeback.c +12 -12
@@ -661,12 +661,13 @@ pinned:
 	return 0;
 }
 
-static void writeback_inodes_wb(struct bdi_writeback *wb,
+static long writeback_inodes_wb(struct bdi_writeback *wb,
 				struct writeback_control *wbc)
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
 	const unsigned long start = jiffies;	/* livelock avoidance */
 	unsigned long stop_time = 0;
+	unsigned long wrote = 0;
 
 	if (wbc->timeout)
 		stop_time = (start + wbc->timeout) | 1;
@@ -709,7 +710,10 @@ static void writeback_inodes_wb(struct b
 		BUG_ON(inode->i_state & (I_FREEING | I_CLEAR));
 		__iget(inode);
 		pages_skipped = wbc->pages_skipped;
+		wbc->nr_to_write = bdi_writeback_chunk(wb->bdi);
+		wrote += wbc->nr_to_write;
 		writeback_single_inode(inode, wbc);
+		wrote -= wbc->nr_to_write;
 		if (wbc->pages_skipped != pages_skipped) {
 			/*
 			 * writeback is not making progress due to locked
@@ -735,6 +739,7 @@ static void writeback_inodes_wb(struct b
 
 	spin_unlock(&inode_lock);
 	/* Leave any unwritten inodes on b_io */
+	return wrote;
 }
 
 void writeback_inodes_wbc(struct writeback_control *wbc)
@@ -782,6 +787,7 @@ static long wb_writeback(struct bdi_writ
 	};
 	unsigned long oldest_jif;
 	long wrote = 0;
+	long nr;
 	struct inode *inode;
 
 	if (wbc.for_kupdate) {
@@ -810,26 +816,20 @@ static long wb_writeback(struct bdi_writ
 
 		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		wbc.pages_skipped = 0;
-		writeback_inodes_wb(wb, &wbc);
-		args->nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
-		wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		nr = writeback_inodes_wb(wb, &wbc);
+		args->nr_pages -= nr;
+		wrote += nr;
 
 		/*
-		 * If we consumed everything, see if we have more
-		 */
-		if (wbc.nr_to_write <= 0)
-			continue;
-		/*
-		 * Didn't write everything and we don't have more IO, bail
+		 * Bail if no more IO
 		 */
 		if (!wbc.more_io)
 			break;
 		/*
 		 * Did we write something? Try for more
 		 */
-		if (wbc.nr_to_write < MAX_WRITEBACK_PAGES)
+		if (nr)
 			continue;
 		/*
 		 * Nothing written. Wait for some inode to
include/linux/writeback.h +3 -2
@@ -45,8 +45,9 @@ struct writeback_control {
 	int timeout;
 	unsigned long *older_than_this;	/* If !NULL, only write back inodes
 					   older than this */
-	long nr_to_write;		/* Write this many pages, and decrement
-					   this for each page written */
+	long nr_to_write;		/* Max pages to write per file, and
+					   decrement this for each page written
+					 */
 	long pages_skipped;		/* Pages which were not written */
 
 	/*
mm/backing-dev.c +0 -1
@@ -338,7 +338,6 @@ static void bdi_flush_io(struct backing_
 		.sync_mode		= WB_SYNC_NONE,
 		.older_than_this	= NULL,
 		.range_cyclic		= 1,
-		.nr_to_write		= 1024,
 		.timeout		= HZ,
 	};
 
#82 lore raw

PATCH 26/45 block: pass the non-rotational queue flag to backing_dev_info

In reply to Wu Fengguang

For now only pass the non-rotational bit to the queue's bdi.

To support hierarchical dm/md configurations, blk_set_rotational() shall be expanded to make several iterations of the bdi list and call its bdi->rotational_fn to pass the bit up. But that would imply we don's support mixed SSD/HD arrays.

---
 block/blk-core.c             |   14 ++++++++++++++
 block/blk-sysfs.c            |    7 +------
 drivers/block/nbd.c          |    2 +-
 drivers/block/xen-blkfront.c |    1 +
 drivers/mmc/card/queue.c     |    2 +-
 drivers/scsi/sd.c            |    2 +-
 include/linux/backing-dev.h  |    8 ++++++++
 include/linux/blkdev.h       |    1 +
 8 files changed, 28 insertions(+), 9 deletions(-)
block/blk-sysfs.c +1 -6
@@ -162,12 +162,7 @@ static ssize_t queue_nonrot_store(struct
 	unsigned long nm;
 	ssize_t ret = queue_var_store(&nm, page, count);
 
-	spin_lock_irq(q->queue_lock);
-	if (nm)
-		queue_flag_clear(QUEUE_FLAG_NONROT, q);
-	else
-		queue_flag_set(QUEUE_FLAG_NONROT, q);
-	spin_unlock_irq(q->queue_lock);
+	blk_set_rotational(q, nm);
 
 	return ret;
 }
include/linux/backing-dev.h +8 -0
@@ -29,6 +29,7 @@ enum bdi_state {
 	BDI_wb_alloc,		/* Default embedded wb allocated */
 	BDI_async_congested,	/* The async (write) queue is getting full */
 	BDI_sync_congested,	/* The sync queue is getting full */
+	BDI_non_rotational,	/* Underlying device is SSD or virtual */
 	BDI_registered,		/* bdi_register() was done */
 	BDI_unused,		/* Available bits start here */
 };
@@ -254,6 +255,8 @@ int bdi_set_max_ratio(struct backing_dev
 #define BDI_CAP_NO_ACCT_WB	0x00000080
 #define BDI_CAP_SWAP_BACKED	0x00000100
 
+#define BDI_CAP_NONROT		0x00000200
+
 #define BDI_CAP_VMFLAGS \
 	(BDI_CAP_READ_MAP | BDI_CAP_WRITE_MAP | BDI_CAP_EXEC_MAP)
 
@@ -291,6 +294,11 @@ static inline int can_submit_background_
        return !test_and_set_bit(WB_FLAG_BACKGROUND_WORK, &bdi->wb_mask);
 }
 
+static inline bool bdi_nonrot(struct backing_dev_info *bdi)
+{
+	return bdi->state & (1 << BDI_non_rotational);
+}
+
 static inline int bdi_congested(struct backing_dev_info *bdi, int bdi_bits)
 {
 	if (bdi->congested_fn)
block/blk-core.c +14 -0
@@ -2486,6 +2486,20 @@ free_and_out:
 }
 EXPORT_SYMBOL_GPL(blk_rq_prep_clone);
 
+void blk_set_rotational(struct request_queue *q, int rotational)
+{
+	spin_lock_irq(q->queue_lock);
+	if (rotational) {
+		queue_flag_clear(QUEUE_FLAG_NONROT, q);
+		clear_bit(BDI_non_rotational, &q->backing_dev_info.state);
+	} else {
+		queue_flag_set(QUEUE_FLAG_NONROT, q);
+		set_bit(BDI_non_rotational, &q->backing_dev_info.state);
+	}
+	spin_unlock_irq(q->queue_lock);
+}
+EXPORT_SYMBOL(blk_set_rotational);
+
 int kblockd_schedule_work(struct request_queue *q, struct work_struct *work)
 {
 	return queue_work(kblockd_workqueue, work);
include/linux/blkdev.h +1 -0
@@ -666,6 +666,7 @@ static inline void blk_clear_queue_full(
 		queue_flag_clear(QUEUE_FLAG_ASYNCFULL, q);
 }
 
+void blk_set_rotational(struct request_queue *q, int rotational);
 
 /*
  * mergeable request must not have _NOMERGE or _BARRIER bit set, nor may
drivers/block/nbd.c +1 -1
@@ -772,7 +772,7 @@ static int __init nbd_init(void)
 		/*
 		 * Tell the block layer that we are not a rotational device
 		 */
-		queue_flag_set_unlocked(QUEUE_FLAG_NONROT, disk->queue);
+		blk_set_rotational(disk->queue, 0);
 	}
 
 	if (register_blkdev(NBD_MAJOR, "nbd")) {
drivers/block/xen-blkfront.c +1 -0
@@ -342,6 +342,7 @@ static int xlvbd_init_blk_queue(struct g
 		return -1;
 
 	queue_flag_set_unlocked(QUEUE_FLAG_VIRT, rq);
+	blk_set_rotational(rq, 0);
 
 	/* Hard sector size and max sectors impersonate the equiv. hardware. */
 	blk_queue_logical_block_size(rq, sector_size);
drivers/mmc/card/queue.c +1 -1
@@ -127,7 +127,7 @@ int mmc_init_queue(struct mmc_queue *mq,
 
 	blk_queue_prep_rq(mq->queue, mmc_prep_request);
 	blk_queue_ordered(mq->queue, QUEUE_ORDERED_DRAIN, NULL);
-	queue_flag_set_unlocked(QUEUE_FLAG_NONROT, mq->queue);
+	blk_set_rotational(mq->queue, 0);
 
 #ifdef CONFIG_MMC_BLOCK_BOUNCE
 	if (host->max_hw_segs == 1) {
drivers/scsi/sd.c +1 -1
@@ -1898,7 +1898,7 @@ static void sd_read_block_characteristic
 	rot = get_unaligned_be16(&buffer[4]);
 
 	if (rot == 1)
-		queue_flag_set_unlocked(QUEUE_FLAG_NONROT, sdkp->disk->queue);
+		blk_set_rotational(sdkp->disk->queue, 0);
 
 	kfree(buffer);
 }
#83 lore raw

PATCH 27/45 writeback: introduce wbc.for_background

In reply to Wu Fengguang

It will be tested for setting wbc.nr_segments, lower the flush priority for NFS, and maybe more in future.

---
 fs/fs-writeback.c         |    1 +
 fs/nfs/write.c            |    2 +-
 include/linux/writeback.h |    1 +
 3 files changed, 3 insertions(+), 1 deletion(-)
fs/fs-writeback.c +1 -0
@@ -783,6 +783,7 @@ static long wb_writeback(struct bdi_writ
 		.sync_mode		= args->sync_mode,
 		.older_than_this	= NULL,
 		.for_kupdate		= args->for_kupdate,
+		.for_background		= args->for_background,
 		.range_cyclic		= args->range_cyclic,
 	};
 	unsigned long oldest_jif;
include/linux/writeback.h +1 -0
@@ -61,6 +61,7 @@ struct writeback_control {
 	unsigned nonblocking:1;		/* Don't get stuck on request queues */
 	unsigned encountered_congestion:1; /* An output: a queue is full */
 	unsigned for_kupdate:1;		/* A kupdate writeback */
+	unsigned for_background:1;	/* A background writeback */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned more_io:1;		/* more io to be dispatched */
fs/nfs/write.c +1 -1
@@ -178,7 +178,7 @@ static int wb_priority(struct writeback_
 {
 	if (wbc->for_reclaim)
 		return FLUSH_HIGHPRI | FLUSH_STABLE;
-	if (wbc->for_kupdate)
+	if (wbc->for_kupdate || wbc->for_background)
 		return FLUSH_LOWPRI;
 	return 0;
 }
#84 lore raw

PATCH 28/45 writeback: introduce wbc.nr_segments

In reply to Wu Fengguang

wbc.nr_segments serves two major purposes:

- fairness between two large files, one is continuously dirtied, another is sparsely dirtied. Given the same amount of dirty pages, it could take vastly different time to sync them to the _same_ device. The nr_segments check helps to favor continuous data.
- avoid seeks/fragmentations. To give each file fair chance of writeback, we have to abort a file when some nr_to_write or timeout is reached. However they are both not good abort conditions. The best is for filesystem to abort earlier in seek boundaries, and treat nr_to_write/timeout as large enough bottom lines.

However a low nr_segments would be inefficient if all files are sparsely dirtied. For example, it may be inefficient for the block device inodes, which has lots of sparsely distributed metadata pages.

The wbc.nr_segments here is determined purely by logical page index
distance: if two pages are 1MB apart, it makes a new segment.

Filesystems could do this better with real extent knowledges. One possible scheme is to record the previous page index in wbc.writeback_index, and let ->writepage compare if the current and previous pages lie in the same extent, and decrease wbc.nr_segments accordingly. Care should taken to avoid double decreases in writepage and write_cache_pages.

CC Theodore Ts'o <[email protected]>
CC Chris Mason <[email protected]>
CC Dave Chinner <[email protected]>
CC Christoph Hellwig <[email protected]>
CC Jan Kara <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Jens Axboe <[email protected]>
---
 fs/fs-writeback.c         |    8 +++++++-
 fs/jbd2/commit.c          |    1 +
 include/linux/writeback.h |   10 +++++++++-
 mm/filemap.c              |    1 +
 mm/page-writeback.c       |    7 +++++++
 5 files changed, 25 insertions(+), 2 deletions(-)
fs/fs-writeback.c +7 -1
@@ -542,6 +542,11 @@ writeback_single_inode(struct inode *ino
 
 	spin_unlock(&inode_lock);
 
+	if (wbc->for_kupdate || wbc->for_background)
+		wbc->nr_segments = bdi_nonrot(wbc->bdi) ? 100 : 10;
+	else
+		wbc->nr_segments = LONG_MAX;
+
 	ret = do_writepages(mapping, wbc);
 
 	/* Don't write the inode if only I_DIRTY_PAGES was set */
@@ -566,7 +571,8 @@ writeback_single_inode(struct inode *ino
 			 * sometimes bales out without doing anything.
 			 */
 			inode->i_state |= I_DIRTY_PAGES;
-			if (wbc->nr_to_write <= 0) {
+			if (wbc->nr_to_write <= 0 ||
+			    wbc->nr_segments <= 0) {
 				/*
 				 * slice used up: queue for next turn
 				 */
include/linux/writeback.h +9 -1
@@ -48,6 +48,9 @@ struct writeback_control {
 	long nr_to_write;		/* Max pages to write per file, and
 					   decrement this for each page written
 					 */
+	long nr_segments;		/* Max page segments to write per file,
+					   this is a count down value, too
+					 */
 	long pages_skipped;		/* Pages which were not written */
 
 	/*
@@ -77,8 +80,13 @@ struct writeback_control {
 };
 
 /*
+ * if two page ranges are more than 1MB apart, they are taken as two segments.
+ */
+#define WB_SEGMENT_DIST		(1024 >> (PAGE_CACHE_SHIFT - 10))
+
+/*
  * fs/fs-writeback.c
- */	
+ */
 struct bdi_writeback;
 int inode_wait(void *);
 void writeback_inodes_sb(struct super_block *);
mm/filemap.c +1 -0
@@ -216,6 +216,7 @@ int __filemap_fdatawrite_range(struct ad
 	struct writeback_control wbc = {
 		.sync_mode = sync_mode,
 		.nr_to_write = LONG_MAX,
+		.nr_segments = LONG_MAX,
 		.range_start = start,
 		.range_end = end,
 	};
mm/page-writeback.c +7 -0
@@ -805,6 +805,13 @@ int write_cache_pages(struct address_spa
 				break;
 			}
 
+			if (nr_to_write != wbc->nr_to_write &&
+			    done_index + WB_SEGMENT_DIST < page->index &&
+			    --wbc->nr_segments <= 0) {
+				done = 1;
+				break;
+			}
+
 			done_index = page->index + 1;
 
 			lock_page(page);
fs/jbd2/commit.c +1 -0
@@ -219,6 +219,7 @@ static int journal_submit_inode_data_buf
 	struct writeback_control wbc = {
 		.sync_mode =  WB_SYNC_ALL,
 		.nr_to_write = mapping->nrpages * 2,
+		.nr_segments = LONG_MAX,
 		.range_start = 0,
 		.range_end = i_size_read(mapping->host),
 	};
#85 lore raw

PATCH 29/45 writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case

In reply to Wu Fengguang

When shmem returns AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. So write_cache_pages shall stop writting this inode, and shmem shall increase pages_skipped to instruct VFS not to busy retry.

---

---
 mm/page-writeback.c |   23 +++++++++++------------
 mm/shmem.c          |    1 +
 2 files changed, 12 insertions(+), 12 deletions(-)
mm/page-writeback.c +11 -12
@@ -851,19 +851,18 @@ continue_unlock:
 				if (ret == AOP_WRITEPAGE_ACTIVATE) {
 					unlock_page(page);
 					ret = 0;
-				} else {
-					/*
-					 * done_index is set past this page,
-					 * so media errors will not choke
-					 * background writeout for the entire
-					 * file. This has consequences for
-					 * range_cyclic semantics (ie. it may
-					 * not be suitable for data integrity
-					 * writeout).
-					 */
-					done = 1;
-					break;
 				}
+				/*
+				 * done_index is set past this page,
+				 * so media errors will not choke
+				 * background writeout for the entire
+				 * file. This has consequences for
+				 * range_cyclic semantics (ie. it may
+				 * not be suitable for data integrity
+				 * writeout).
+				 */
+				done = 1;
+				break;
  			}
 
 			if (nr_to_write > 0) {
mm/shmem.c +1 -0
@@ -1103,6 +1103,7 @@ unlock:
 	 */
 	swapcache_free(swap, NULL);
 redirty:
+	wbc->pages_skipped++;
 	set_page_dirty(page);
 	if (wbc->for_reclaim)
 		return AOP_WRITEPAGE_ACTIVATE;	/* Return with page locked */
#86 lore raw

Re: [PATCH 29/45] writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case

Quoted 7 lines · Wu Fengguang

When shmem returns AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. So write_cache_pages shall stop writting this inode, and shmem shall increase pages_skipped to instruct VFS not to busy retry.

Okay, it embarrasses me to see AOP_WRITEPAGE_ACTIVATE (and its horrid "in this one case the page is still locked" semantic) still around - my patch to remove it vanished from mmotm (probably caused a temporary conflict) and I've never chased it up (partly out of guilt that I'd not yet kept my promise to contact the openAFS people about their use of it).

But that's orthogonal to your concern here: for so long as there has been a wbc->pages_skipped, I guess shmem_writepage() should have been incrementing it there - thanks. But I don't believe the VFS will ever have any interest in pages_skipped from shmem_writepage(): do you have evidence that it does? If so, I need to investigate.

And the accompanying change to write_cache_pages() seems irrelevant and misguided. Irrelevant because write_cache_pages() should never be dealing with shmem_writepage() (its bdi should keep it well away), and should never be dealing with reclaim, which is the only case in which shmem_writepage() returns AOP_WRITEPAGE_ACTIVATE - or have your other changes, or the bdi work, changed that?

And misguided because in your change to write_cache_pages() you're taking AOP_WRITEPAGE_ACTIVATE to say that it should now give up, not process more pages. We just don't know that. All it means is that this one page couldn't be written and should be reactivated (if it were under reclaim): it might be the case that every other page tried after would get treated in the same way, or it might be the case that the next page would get written successfully. That info is just not provided.

Hugh

Quoted 48 lines
---
 mm/page-writeback.c |   23 +++++++++++------------
 mm/shmem.c          |    1 +
 2 files changed, 12 insertions(+), 12 deletions(-)
mm/page-writeback.c +11 -12
@@ -851,19 +851,18 @@ continue_unlock:
 				if (ret == AOP_WRITEPAGE_ACTIVATE) {
 					unlock_page(page);
 					ret = 0;
-				} else {
-					/*
-					 * done_index is set past this page,
-					 * so media errors will not choke
-					 * background writeout for the entire
-					 * file. This has consequences for
-					 * range_cyclic semantics (ie. it may
-					 * not be suitable for data integrity
-					 * writeout).
-					 */
-					done = 1;
-					break;
 				}
+				/*
+				 * done_index is set past this page,
+				 * so media errors will not choke
+				 * background writeout for the entire
+				 * file. This has consequences for
+				 * range_cyclic semantics (ie. it may
+				 * not be suitable for data integrity
+				 * writeout).
+				 */
+				done = 1;
+				break;
  			}
 
 			if (nr_to_write > 0) {
mm/shmem.c +1 -0
@@ -1103,6 +1103,7 @@ unlock:
 	 */
 	swapcache_free(swap, NULL);
 redirty:
+	wbc->pages_skipped++;
 	set_page_dirty(page);
 	if (wbc->for_reclaim)
 		return AOP_WRITEPAGE_ACTIVATE;	/* Return with page locked */
#87 lore raw

Re: [PATCH 29/45] writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case

Quoted 7 lines · Hugh Dickins
Quoted 5 lines · Wu Fengguang

When shmem returns AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. So write_cache_pages shall stop writting this inode, and shmem shall increase pages_skipped to instruct VFS not to busy retry.

Quoted 8 lines
Quoted 2 lines

Okay, it embarrasses me to see AOP_WRITEPAGE_ACTIVATE (and its horrid "in this one case the page is still locked" semantic) still around - my patch to remove it vanished from mmotm (probably caused a temporary conflict) and I've never chased it up (partly out of guilt that I'd not yet kept my promise to contact the openAFS people about their use of it).

Googled this one :)

http://markmail.org/thread/fivi4bgylwsy26ws

In fact we could just return from shmem_writepage() with PG_dirty and !PG_writeback. Then the page will be put back to LRU with PG_reclaim cleared. It could well happen in other filesystems who has trouble to writeback the page for the time (those already do pages_skipped++).

Quoted 5 lines

But that's orthogonal to your concern here: for so long as there has been a wbc->pages_skipped, I guess shmem_writepage() should have been incrementing it there - thanks. But I don't believe the VFS will ever have any interest in pages_skipped from shmem_writepage(): do you have evidence that it does? If so, I need to investigate.

Yes :) Except in this new code.

Quoted 6 lines

And the accompanying change to write_cache_pages() seems irrelevant and misguided. Irrelevant because write_cache_pages() should never be dealing with shmem_writepage() (its bdi should keep it well away), and should never be dealing with reclaim, which is the only case in which shmem_writepage() returns AOP_WRITEPAGE_ACTIVATE - or have your other changes, or the bdi work, changed that?

That becomes possible with my another patch (30/45). I've added check to avoid doing lumpy reclaim for shmem. So now everything returns to normal :)

Quoted 8 lines

And misguided because in your change to write_cache_pages() you're taking AOP_WRITEPAGE_ACTIVATE to say that it should now give up, not process more pages. We just don't know that. All it means is that this one page couldn't be written and should be reactivated (if it were under reclaim): it might be the case that every other page tried after would get treated in the same way, or it might be the case that the next page would get written successfully. That info is just not provided.

Yes, it was over-smart indeed. I'll revert that chunk (or AOP_WRITEPAGE_ACTIVATE) totally.

Thanks,
Fengguang

Quoted 48 lines
Quoted 48 lines
---
 mm/page-writeback.c |   23 +++++++++++------------
 mm/shmem.c          |    1 +
 2 files changed, 12 insertions(+), 12 deletions(-)
mm/page-writeback.c +11 -12
@@ -851,19 +851,18 @@ continue_unlock:
 				if (ret == AOP_WRITEPAGE_ACTIVATE) {
 					unlock_page(page);
 					ret = 0;
-				} else {
-					/*
-					 * done_index is set past this page,
-					 * so media errors will not choke
-					 * background writeout for the entire
-					 * file. This has consequences for
-					 * range_cyclic semantics (ie. it may
-					 * not be suitable for data integrity
-					 * writeout).
-					 */
-					done = 1;
-					break;
 				}
+				/*
+				 * done_index is set past this page,
+				 * so media errors will not choke
+				 * background writeout for the entire
+				 * file. This has consequences for
+				 * range_cyclic semantics (ie. it may
+				 * not be suitable for data integrity
+				 * writeout).
+				 */
+				done = 1;
+				break;
  			}
 
 			if (nr_to_write > 0) {
mm/shmem.c +1 -0
@@ -1103,6 +1103,7 @@ unlock:
 	 */
 	swapcache_free(swap, NULL);
 redirty:
+	wbc->pages_skipped++;
 	set_page_dirty(page);
 	if (wbc->for_reclaim)
 		return AOP_WRITEPAGE_ACTIVATE;	/* Return with page locked */
#88 lore raw

PATCH 30/45 vmscan: lumpy pageout

In reply to Wu Fengguang

When pageout a dirty page, try to piggy back more consecutive dirty pages (up to 512KB) to improve IO efficiency.

Only ext3/reiserfs which don't have its own aops->writepages are supported in this initial version.

---
 mm/page-writeback.c |   12 ++++++++++++
 mm/vmscan.c         |   11 +++++++++++
 2 files changed, 23 insertions(+)
mm/vmscan.c +11 -0
@@ -344,6 +344,8 @@ typedef enum {
 	PAGE_CLEAN,
 } pageout_t;
 
+#define LUMPY_PAGEOUT_PAGES	(512 * 1024 / PAGE_CACHE_SIZE)
+
 /*
  * pageout is called by shrink_page_list() for each dirty page.
  * Calls ->writepage().
@@ -409,6 +411,15 @@ static pageout_t pageout(struct page *pa
 		}
 
 		/*
+		 * only write_cache_pages() supports for_reclaim for now
+		 */
+		if (!mapping->a_ops->writepages) {
+			wbc.range_start = (page->index + 1) << PAGE_CACHE_SHIFT;
+			wbc.nr_to_write = LUMPY_PAGEOUT_PAGES - 1;
+			generic_writepages(mapping, &wbc);
+		}
+
+		/*
 		 * Wait on writeback if requested to. This happens when
 		 * direct reclaiming a large contiguous area and the
 		 * first attempt to free a range of pages fails.
mm/page-writeback.c +12 -0
@@ -805,6 +805,11 @@ int write_cache_pages(struct address_spa
 				break;
 			}
 
+			if (wbc->for_reclaim && done_index != page->index) {
+				done = 1;
+				break;
+			}
+
 			if (nr_to_write != wbc->nr_to_write &&
 			    done_index + WB_SEGMENT_DIST < page->index &&
 			    --wbc->nr_segments <= 0) {
@@ -846,6 +851,13 @@ continue_unlock:
 			if (!clear_page_dirty_for_io(page))
 				goto continue_unlock;
 
+			/*
+			 * active and unevictable pages will be checked at
+			 * rotate time
+			 */
+			if (wbc->for_reclaim)
+				SetPageReclaim(page);
+
 			ret = (*writepage)(page, wbc, data);
 			if (unlikely(ret)) {
 				if (ret == AOP_WRITEPAGE_ACTIVATE) {
#89 lore raw

PATCH 31/45 writeback: sync old inodes first in background writeback

In reply to Wu Fengguang

A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.

---
 fs/fs-writeback.c |   21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
fs/fs-writeback.c +17 -4
@@ -680,7 +680,7 @@ static long writeback_inodes_wb(struct b
 
 	spin_lock(&inode_lock);
 
-	if (!wbc->for_kupdate || list_empty(&wb->b_io))
+	if (list_empty(&wb->b_io))
 		queue_io(wb, wbc->older_than_this);
 
 	while (!list_empty(&wb->b_io)) {
@@ -793,14 +793,15 @@ static long wb_writeback(struct bdi_writ
 		.range_cyclic		= args->range_cyclic,
 	};
 	unsigned long oldest_jif;
+	int expire_interval = msecs_to_jiffies(dirty_expire_interval * 10);
+	int fg_rounds = 0;
 	long wrote = 0;
 	long nr;
 	struct inode *inode;
 
-	if (wbc.for_kupdate) {
+	if (wbc.for_kupdate || wbc.for_background) {
 		wbc.older_than_this = &oldest_jif;
-		oldest_jif = jiffies -
-				msecs_to_jiffies(dirty_expire_interval * 10);
+		oldest_jif = jiffies - expire_interval;
 	}
 	if (!wbc.range_cyclic) {
 		wbc.range_start = 0;
@@ -828,6 +829,18 @@ static long wb_writeback(struct bdi_writ
 		args->nr_pages -= nr;
 		wrote += nr;
 
+		if (args->for_background && expire_interval &&
+		    ++fg_rounds && list_empty(&wb->b_io)) {
+			if (fg_rounds < 10)
+				expire_interval >>= 1;
+			if (expire_interval)
+				oldest_jif = jiffies - expire_interval;
+			else
+				wbc.older_than_this = 0;
+			fg_rounds = 0;
+			continue;
+		}
+
 		/*
 		 * Bail if no more IO
 		 */
#90 lore raw

Re: [PATCH 31/45] writeback: sync old inodes first in background writeback

Quoted 2 lines · Wu Fengguang

A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.

I've looked at this a bit again after you pointed to this thread in the direct reclaim thread, and I think we should be even more aggressive in pushing out old inodes.

We basically have two types of I/O done from wb_do_writeback:

- either we want to write all inodes for a given bdi/superblock. That includes all WB_SYNC_ALL callers, but also things like writeback_inodes_sb and the wakeup_flusher_threads call from sys_sync.
- or we have a specific goal, like for the background writeback or the wakeup_flusher_threads from free_more_memory.

For the first case there's obviously no point in doing any older_than_this processing as we write out all inodes anyway.

For the second case we should always do a older_than_this pass _first_.
Rationale: we really should get the old inodes out ASAP, so that we keep the amount of changes lost on a crash in the boundaries. Furthermore the callers only need N pages cleaned, and they don't care where from. So if we can reach our goal with the older_than_this writeback we're fine. If the writeback loop is long enough we can keep doing more of these later on as well.

Doing this should also help cleaning the code up a bit by moving the wb_check_old_data_flush logic into wb_writeback and getting rid of the for_kupdate paramter in struct wb_writeback_work. I'm not even sure it's worth keeping it in struct writeback_control.

#91 lore raw

Re: [PATCH 31/45] writeback: sync old inodes first in background writeback

Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Wu Fengguang

A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.

I've looked at this a bit again after you pointed to this thread in the direct reclaim thread, and I think we should be even more aggressive in pushing out old inodes.

Agreed.

Quoted 11 lines

We basically have two types of I/O done from wb_do_writeback:

- either we want to write all inodes for a given bdi/superblock. That includes all WB_SYNC_ALL callers, but also things like writeback_inodes_sb and the wakeup_flusher_threads call from sys_sync.
- or we have a specific goal, like for the background writeback or the wakeup_flusher_threads from free_more_memory.

For the first case there's obviously no point in doing any older_than_this processing as we write out all inodes anyway.

We may also do older_than_this even for the sync-the-whole-world case, as long as this simplifies wb_writeback() and/or other code. This may make a difference for slow devices.

Quoted 1 line

For the second case we should always do a older_than_this pass _first_.

Agree in general.

Quoted 6 lines

Rationale: we really should get the old inodes out ASAP, so that we keep the amount of changes lost on a crash in the boundaries. Furthermore the callers only need N pages cleaned, and they don't care where from. So if we can reach our goal with the older_than_this writeback we're fine. If the writeback loop is long enough we can keep doing more of these later on as well.

Right.

Quoted 4 lines

Doing this should also help cleaning the code up a bit by moving the wb_check_old_data_flush logic into wb_writeback and getting rid of the for_kupdate paramter in struct wb_writeback_work. I'm not even sure it's worth keeping it in struct writeback_control.

I'd also like to see less for_kupdate tests. Whether or not we can totally get rid of the explicit for_kupdate case, there are always four main writeback goals/semantics:

- periodic      stop when all 30s-old inodes are written
- background    stop when background threshold is reached
- nr_pages      stop when nr_pages written (or when all clean)
- sync          stop when all older-than-sync-time inodes are written

Note that
- the "sync" goal is obviously a superset of the "periodic" goal
- the "background" goal may be expanded to include the "periodic" goal
- the latter three goals may all do some "periodic" goal loops, with a moving "old" criterion.

Thanks,
Fengguang

#92 lore raw

PATCH 32/45 writeback: update kupdate expire timestamp on each scan of b_io

In reply to Wu Fengguang

This prevents it to stuck with some very old but busy inodes, and to give newer expired inodes a fair chance.

---
 fs/fs-writeback.c |    3 +++
 1 file changed, 3 insertions(+)
fs/fs-writeback.c +3 -0
@@ -829,6 +829,9 @@ static long wb_writeback(struct bdi_writ
 		args->nr_pages -= nr;
 		wrote += nr;
 
+		if (args->for_kupdate && list_empty(&wb->b_io))
+			oldest_jif = jiffies - expire_interval;
+
 		if (args->for_background && expire_interval &&
 		    ++fg_rounds && list_empty(&wb->b_io)) {
 			if (fg_rounds < 10)
#93 lore raw

PATCH 33/45 writeback: sync livelock - introduce wbc.for_sync

In reply to Wu Fengguang

The sync() is performed in two stages: the WB_SYNC_NONE sync and the WB_SYNC_ALL sync. It is necessary to tag both stages with wbc.for_sync, so as to prevent either of them being livelocked.

The next patch will utilize this flag to do the livelock prevention.

---
 fs/fs-writeback.c           |   30 +++++++++---------------------
 include/linux/backing-dev.h |    2 +-
 include/linux/writeback.h   |   11 +++++++++++
 mm/page-writeback.c         |    2 +-
 4 files changed, 22 insertions(+), 23 deletions(-)
fs/fs-writeback.c +9 -21
@@ -42,6 +42,7 @@ struct wb_writeback_args {
 	long nr_pages;
 	struct super_block *sb;
 	enum writeback_sync_modes sync_mode;
+	int for_sync:1;
 	int for_kupdate:1;
 	int range_cyclic:1;
 	int for_background:1;
@@ -221,6 +222,7 @@ static void bdi_sync_writeback(struct ba
 	struct wb_writeback_args args = {
 		.sb		= sb,
 		.sync_mode	= WB_SYNC_ALL,
+		.for_sync	= 1,
 		.nr_pages	= LONG_MAX,
 		.range_cyclic	= 0,
 	};
@@ -236,7 +238,6 @@ static void bdi_sync_writeback(struct ba
 /**
  * bdi_start_writeback - start writeback
  * @bdi: the backing device to write from
- * @nr_pages: the number of pages to write
  *
  * Description:
  *   This does WB_SYNC_NONE opportunistic writeback. The IO is only
@@ -245,24 +246,17 @@ static void bdi_sync_writeback(struct ba
  *
  */
 void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
-			 long nr_pages)
+			 long mission)
 {
 	struct wb_writeback_args args = {
 		.sb		= sb,
 		.sync_mode	= WB_SYNC_NONE,
-		.nr_pages	= nr_pages,
+		.nr_pages	= LONG_MAX,
+		.for_background	= mission == WB_FOR_BACKGROUND,
+		.for_sync	= mission == WB_FOR_SYNC,
 		.range_cyclic	= 1,
 	};
 
-	/*
-	 * We treat @nr_pages=0 as the special case to do background writeback,
-	 * ie. to sync pages until the background dirty threshold is reached.
-	 */
-	if (!nr_pages) {
-		args.nr_pages = LONG_MAX;
-		args.for_background = 1;
-	}
-
 	bdi_alloc_queue_work(bdi, &args);
 }
 
@@ -310,7 +304,7 @@ void bdi_writeback_wait(struct backing_d
 	 * make sure we will be woke up by someone
 	 */
 	if (can_submit_background_writeback(bdi))
-		bdi_start_writeback(bdi, NULL, 0);
+		bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
 
 	wait_for_completion(&tt.complete);
 }
@@ -790,6 +784,7 @@ static long wb_writeback(struct bdi_writ
 		.older_than_this	= NULL,
 		.for_kupdate		= args->for_kupdate,
 		.for_background		= args->for_background,
+		.for_sync		= args->for_sync,
 		.range_cyclic		= args->range_cyclic,
 	};
 	unsigned long oldest_jif;
@@ -1250,14 +1245,7 @@ static void wait_sb_inodes(struct super_
  */
 void writeback_inodes_sb(struct super_block *sb)
 {
-	unsigned long nr_dirty = global_page_state(NR_FILE_DIRTY);
-	unsigned long nr_unstable = global_page_state(NR_UNSTABLE_NFS);
-	long nr_to_write;
-
-	nr_to_write = nr_dirty + nr_unstable +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
-
-	bdi_start_writeback(sb->s_bdi, sb, nr_to_write);
+	bdi_start_writeback(sb->s_bdi, sb, WB_FOR_SYNC);
 }
 EXPORT_SYMBOL(writeback_inodes_sb);
 
include/linux/backing-dev.h +1 -1
@@ -123,7 +123,7 @@ int bdi_register(struct backing_dev_info
 int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
 void bdi_unregister(struct backing_dev_info *bdi);
 void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
-				long nr_pages);
+				long mission);
 int bdi_writeback_task(struct bdi_writeback *wb);
 int bdi_has_dirty_io(struct backing_dev_info *bdi);
 int bdi_writeback_wakeup(struct backing_dev_info *bdi);
include/linux/writeback.h +11 -0
@@ -31,6 +31,16 @@ enum writeback_sync_modes {
 	WB_SYNC_ALL,	/* Wait on every mapping */
 };
 
+enum writeback_mission {
+	WB_FOR_KUPDATE,		/* writeback expired dirty inodes */
+	WB_FOR_RECLAIM,
+	WB_FOR_BACKGROUND,	/* stop on hitting background threshold */
+	WB_FOR_SYNC,		/* write all now-dirty inodes/pages,
+				 * but take care not to live lock
+				 */
+	WB_NR_PAGES,		/* writeback # of pages if larger than this */
+};
+
 /*
  * A control structure which tells the writeback code what to do.  These are
  * always on the stack, and hence need no locking.  They are always initialised
@@ -65,6 +75,7 @@ struct writeback_control {
 	unsigned encountered_congestion:1; /* An output: a queue is full */
 	unsigned for_kupdate:1;		/* A kupdate writeback */
 	unsigned for_background:1;	/* A background writeback */
+	unsigned for_sync:1;		/* A writeback for sync */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned more_io:1;		/* more io to be dispatched */
mm/page-writeback.c +1 -1
@@ -529,7 +529,7 @@ out:
 	 */
 	if (!laptop_mode && (nr_reclaimable > background_thresh) &&
 	    can_submit_background_writeback(bdi))
-		bdi_start_writeback(bdi, NULL, 0);
+		bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
 }
 
 void set_page_dirty_balance(struct page *page, int page_mkwrite)
#94 lore raw

PATCH 34/45 writeback: sync livelock - kick background writeback

In reply to Wu Fengguang

The periodic/background writeback can run forever. So when any sync work is enqueued, increase bdi->sync_works to notify the active non-sync works to exit. Non-sync works queued after all sync works won't be affected.

---
 fs/fs-writeback.c           |   11 +++++++++++
 include/linux/backing-dev.h |    5 +++++
 mm/backing-dev.c            |    1 +
 3 files changed, 17 insertions(+)
fs/fs-writeback.c +11 -0
@@ -139,6 +139,8 @@ static void wb_clear_pending(struct bdi_
 		list_del_rcu(&work->list);
 		if (work->args.for_background)
 			clear_bit(WB_FLAG_BACKGROUND_WORK, &bdi->wb_mask);
+		if (work->args.for_sync)
+			bdi->sync_works--;
 		spin_unlock(&bdi->wb_lock);
 
 		wb_work_complete(work);
@@ -159,6 +161,8 @@ static void bdi_queue_work(struct backin
 	 */
 	spin_lock(&bdi->wb_lock);
 	list_add_tail_rcu(&work->list, &bdi->work_list);
+	if (work->args.for_sync)
+		bdi->sync_works++;
 	spin_unlock(&bdi->wb_lock);
 
 	/*
@@ -811,6 +815,13 @@ static long wb_writeback(struct bdi_writ
 			break;
 
 		/*
+		 * background/periodic works can run forever, need to abort
+		 * on seeing any pending sync work, to prevent livelock it.
+		 */
+		if (!args->for_sync && wb->bdi->sync_works > 0)
+			break;
+
+		/*
 		 * For background writeout, stop when we are below the
 		 * background dirty threshold
 		 */
include/linux/backing-dev.h +5 -0
@@ -84,6 +84,11 @@ struct backing_dev_info {
 	struct list_head wb_list; /* the flusher threads hanging off this bdi */
 	unsigned long wb_mask;	  /* bitmask of registered tasks */
 	unsigned int wb_cnt;	  /* number of registered tasks */
+	/*
+	 * sync works queued, background works shall abort on seeing this,
+	 * to prevent livelocking the sync works
+	 */
+	unsigned int sync_works;
 
 	struct list_head work_list;
 
mm/backing-dev.c +1 -0
@@ -647,6 +647,7 @@ int bdi_init(struct backing_dev_info *bd
 	 */
 	bdi->wb_mask = 1;
 	bdi->wb_cnt = 1;
+	bdi->sync_works = 0;
 
 	bdi->write_bandwidth = MAX_WRITEBACK_PAGES;
 
#95 lore raw

PATCH 35/45 writeback: sync livelock - use single timestamp for whole sync work

In reply to Wu Fengguang

The start time in writeback_inodes_wb() is not very useful because it slips at each invocation time. We shall use one _constant_ time at the beginning to cover this whole sync() work.

The timestamp is now grabbed at work start time. It could be better set at the sync work submission time.

---
 fs/fs-writeback.c |   17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
fs/fs-writeback.c +8 -9
@@ -669,12 +669,11 @@ static long writeback_inodes_wb(struct b
 				struct writeback_control *wbc)
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
-	const unsigned long start = jiffies;	/* livelock avoidance */
 	unsigned long stop_time = 0;
 	unsigned long wrote = 0;
 
 	if (wbc->timeout)
-		stop_time = (start + wbc->timeout) | 1;
+		stop_time = (jiffies + wbc->timeout) | 1;
 
 	spin_lock(&inode_lock);
 
@@ -699,13 +698,6 @@ static long writeback_inodes_wb(struct b
 			continue;
 		}
 
-		/*
-		 * Was this inode dirtied after sync_sb_inodes was called?
-		 * This keeps sync from extra jobs and livelock.
-		 */
-		if (inode_dirtied_after(inode, start))
-			break;
-
 		if (pin_sb_for_writeback(wbc, inode, &pin_sb)) {
 			requeue_io(inode);
 			continue;
@@ -798,6 +790,13 @@ static long wb_writeback(struct bdi_writ
 	long nr;
 	struct inode *inode;
 
+	/*
+	 * keep sync from extra jobs and livelock
+	 */
+	if (wbc.for_sync) {
+		wbc.older_than_this = &oldest_jif;
+		oldest_jif = jiffies;
+	}
 	if (wbc.for_kupdate || wbc.for_background) {
 		wbc.older_than_this = &oldest_jif;
 		oldest_jif = jiffies - expire_interval;
#96 lore raw

PATCH 36/45 writeback: sync livelock - curb dirty speed for inodes to be synced

In reply to Wu Fengguang

The inodes to be synced will be dirty throttled regardless of the dirty threshold. This stops sync() livelock by some fast dirtier.

---
 fs/fs-writeback.c           |    4 ++++
 include/linux/backing-dev.h |    1 +
 mm/page-writeback.c         |   13 +++++++++++++
 3 files changed, 18 insertions(+)
mm/page-writeback.c +13 -0
@@ -478,6 +478,18 @@ static void balance_dirty_pages(struct a
 	global_dirty_thresh(&background_thresh, &dirty_thresh);
 
 	/*
+	 * If sync() is in progress, curb the to-be-synced inodes regardless
+	 * of dirty limits, so that a fast dirtier won't livelock the sync.
+	 */
+	if (unlikely(bdi->sync_time &&
+		     S_ISREG(mapping->host->i_mode) &&
+		     time_after_eq(bdi->sync_time,
+				   mapping->host->dirtied_when))) {
+		write_chunk *= 2;
+		goto throttle;
+	}
+
+	/*
 	 * Throttle it only when the background writeback cannot
 	 * catch-up. This skips the ramp up phase of bdi limits.
 	 */
@@ -510,6 +522,7 @@ static void balance_dirty_pages(struct a
 	if (!dirty_exceeded)
 		goto out;
 
+throttle:
 	if (!bdi->dirty_exceeded)
 		bdi->dirty_exceeded = 1;
 
fs/fs-writeback.c +4 -0
@@ -796,6 +796,7 @@ static long wb_writeback(struct bdi_writ
 	if (wbc.for_sync) {
 		wbc.older_than_this = &oldest_jif;
 		oldest_jif = jiffies;
+		wbc.bdi->sync_time = oldest_jif | 1;
 	}
 	if (wbc.for_kupdate || wbc.for_background) {
 		wbc.older_than_this = &oldest_jif;
@@ -873,6 +874,9 @@ static long wb_writeback(struct bdi_writ
 		spin_unlock(&inode_lock);
 	}
 
+	if (args->for_sync)
+		wb->bdi->sync_time = 0;
+
 	if (args->for_background)
 		while (bdi_writeback_wakeup(wb->bdi))
 			;  /* unthrottle all tasks */
include/linux/backing-dev.h +1 -0
@@ -89,6 +89,7 @@ struct backing_dev_info {
 	 * to prevent livelocking the sync works
 	 */
 	unsigned int sync_works;
+	unsigned long sync_time;
 
 	struct list_head work_list;
 
#97 lore raw

PATCH 37/45 writeback: use timestamp to indicate dirty exceeded

In reply to Wu Fengguang

When there are only one (or several) dirtiers, dirty_exceeded is always (or mostly) off. Converting to timestamp avoids this problem. It helps to use smaller write_chunk for smoother throttling.

We'll lower ratelimit if saw dirty exceeded in the last 1 second.

---

Before patch, the wait time in balance_dirty_pages() are ~200ms:

[ 1093.397700] write_bandwidth: comm=swapper pages=1536 time=204ms
[ 1093.594319] write_bandwidth: comm=swapper pages=1536 time=196ms
[ 1093.796642] write_bandwidth: comm=swapper pages=1536 time=200ms

After patch, ~25ms:

[   90.261339] write_bandwidth: comm=swapper pages=192 time=20ms
[   90.293168] write_bandwidth: comm=swapper pages=192 time=24ms
[   90.323853] write_bandwidth: comm=swapper pages=192 time=24ms
[   90.354510] write_bandwidth: comm=swapper pages=192 time=28ms
[   90.389890] write_bandwidth: comm=swapper pages=192 time=28ms
[   90.421787] write_bandwidth: comm=swapper pages=192 time=24ms
 include/linux/backing-dev.h |    2 +-
 mm/backing-dev.c            |    1 -
 mm/page-writeback.c         |    9 +++------
 3 files changed, 4 insertions(+), 8 deletions(-)
mm/page-writeback.c +3 -6
@@ -523,14 +523,10 @@ static void balance_dirty_pages(struct a
 		goto out;
 
 throttle:
-	if (!bdi->dirty_exceeded)
-		bdi->dirty_exceeded = 1;
+	bdi->dirty_exceed_time = jiffies;
 
 	bdi_writeback_wait(bdi, write_chunk);
 
-	if (bdi->dirty_exceeded)
-		bdi->dirty_exceeded = 0;
-
 out:
 	/*
 	 * In laptop mode, we wait until hitting the higher threshold before
@@ -578,7 +574,8 @@ void balance_dirty_pages_ratelimited_nr(
 	unsigned long *p;
 
 	ratelimit = ratelimit_pages;
-	if (mapping->backing_dev_info->dirty_exceeded)
+	if (jiffies - mapping->backing_dev_info->dirty_exceed_time <
+							(unsigned long) HZ)
 		ratelimit >>= 3;
 
 	/*
include/linux/backing-dev.h +1 -1
@@ -74,7 +74,7 @@ struct backing_dev_info {
 	struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS];
 
 	struct prop_local_percpu completions;
-	int dirty_exceeded;
+	unsigned long dirty_exceed_time;
 
 	unsigned int min_ratio;
 	unsigned int max_ratio, max_prop_frac;
mm/backing-dev.c +0 -1
@@ -661,7 +661,6 @@ int bdi_init(struct backing_dev_info *bd
 			goto err;
 	}
 
-	bdi->dirty_exceeded = 0;
 	err = prop_local_init_percpu(&bdi->completions);
 
 	if (err) {
#98 lore raw

PATCH 38/45 writeback: introduce queue b_more_io_wait

In reply to Wu Fengguang

Introduce the b_more_io_wait queue to park inodes that for some reason cannot be synced immediately. They will be revisited either in the next b_io scan time, or after 0.1s sleep for sync, or retried after 5s in the next periodic writeback.

The new data flow after this patchset:

b_dirty --> b_io --> b_more_io/b_more_io_wait --+

             ^                                  |
	     |                                  |
	     +----------------------------------+

The rational is to address two issues:
- the 30s delay of redirty_tail() may be too long
- redirty_tail() may update i_dirtied_when, however we now rely on it remain unchanged for all candidate inodes of sync(). (to avoid extra work and livelock, we now exclude any inodes from being synced if its dirty time is after the sync time)

Cc Jan Kara <[email protected]>
Cc David Chinner <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c           |   27 ++++++++++++++++-----------
 include/linux/backing-dev.h |    8 +++++---
 mm/backing-dev.c            |   14 +++++++++++---
 3 files changed, 32 insertions(+), 17 deletions(-)
fs/fs-writeback.c +16 -11
@@ -384,6 +384,16 @@ static void requeue_io(struct inode *ino
 	list_move(&inode->i_list, &wb->b_more_io);
 }
 
+/*
+ * The inode should be retried in an opportunistic way.
+ */
+static void requeue_io_wait(struct inode *inode)
+{
+	struct bdi_writeback *wb = &inode_to_bdi(inode)->wb;
+
+	list_move(&inode->i_list, &wb->b_more_io_wait);
+}
+
 static void inode_sync_complete(struct inode *inode)
 {
 	/*
@@ -453,12 +463,14 @@ static void move_expired_inodes(struct l
 /*
  * Queue all expired dirty inodes for io, eldest first:
  * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io_wait inodes
  *                 => b_more_io inodes
  *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
 	list_splice_init(&wb->b_more_io, &wb->b_io);
+	list_splice_init(&wb->b_more_io_wait, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
@@ -860,18 +872,11 @@ static long wb_writeback(struct bdi_writ
 		 */
 		if (nr)
 			continue;
-		/*
-		 * Nothing written. Wait for some inode to
-		 * become available for writeback. Otherwise
-		 * we'll just busyloop.
-		 */
-		spin_lock(&inode_lock);
-		if (!list_empty(&wb->b_more_io))  {
-			inode = list_entry(wb->b_more_io.prev,
-						struct inode, i_list);
-			inode_wait_for_writeback(inode);
+		if (wbc.for_sync && !list_empty(&wb->b_more_io_wait)) {
+			schedule_timeout_interruptible(HZ/10);
+			continue;
 		}
-		spin_unlock(&inode_lock);
+		break;
 	}
 
 	if (args->for_sync)
include/linux/backing-dev.h +5 -3
@@ -56,6 +56,7 @@ struct bdi_writeback {
 	struct list_head	b_dirty;	/* dirty inodes */
 	struct list_head	b_io;		/* parked for writeback */
 	struct list_head	b_more_io;	/* parked for more writeback */
+	struct list_head	b_more_io_wait;	/* opportunistic retry io */
 };
 
 struct backing_dev_info {
@@ -140,9 +141,10 @@ extern struct list_head bdi_list;
 
 static inline int wb_has_dirty_io(struct bdi_writeback *wb)
 {
-	return !list_empty(&wb->b_dirty) ||
-	       !list_empty(&wb->b_io) ||
-	       !list_empty(&wb->b_more_io);
+	return !list_empty(&wb->b_dirty)	||
+	       !list_empty(&wb->b_io)		||
+	       !list_empty(&wb->b_more_io)	||
+	       !list_empty(&wb->b_more_io_wait);
 }
 
 static inline void __add_bdi_stat(struct backing_dev_info *bdi,
mm/backing-dev.c +11 -3
@@ -63,14 +63,17 @@ static int bdi_debug_stats_show(struct s
 	unsigned long background_thresh;
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
-	unsigned long nr_dirty, nr_io, nr_more_io, nr_wb;
+	unsigned long nr_dirty		= 0;
+	unsigned long nr_io		= 0;
+	unsigned long nr_more_io	= 0;
+	unsigned long nr_more_io_wait	= 0;
+	unsigned long nr_wb		= 0;
 	struct inode *inode;
 
 	/*
 	 * inode lock is enough here, the bdi->wb_list is protected by
 	 * RCU on the reader side
 	 */
-	nr_wb = nr_dirty = nr_io = nr_more_io = 0;
 	spin_lock(&inode_lock);
 	list_for_each_entry(wb, &bdi->wb_list, list) {
 		nr_wb++;
@@ -80,6 +83,8 @@ static int bdi_debug_stats_show(struct s
 			nr_io++;
 		list_for_each_entry(inode, &wb->b_more_io, i_list)
 			nr_more_io++;
+		list_for_each_entry(inode, &wb->b_more_io_wait, i_list)
+			nr_more_io_wait++;
 	}
 	spin_unlock(&inode_lock);
 
@@ -98,6 +103,7 @@ static int bdi_debug_stats_show(struct s
 		   "b_dirty:          %8lu\n"
 		   "b_io:             %8lu\n"
 		   "b_more_io:        %8lu\n"
+		   "b_more_io_wait:   %8lu\n"
 		   "bdi_list:         %8u\n"
 		   "state:            %8lx\n"
 		   "wb_mask:          %8lx\n"
@@ -107,7 +113,7 @@ static int bdi_debug_stats_show(struct s
 		   (unsigned long) K(bdi_stat(bdi, BDI_RECLAIMABLE)),
 		   K(bdi_thresh), K(dirty_thresh), K(background_thresh),
 		   (unsigned long) K(bdi->write_bandwidth),
-		   nr_wb, nr_dirty, nr_io, nr_more_io,
+		   nr_wb, nr_dirty, nr_io, nr_more_io, nr_more_io_wait,
 		   !list_empty(&bdi->bdi_list), bdi->state, bdi->wb_mask,
 		   !list_empty(&bdi->wb_list), bdi->wb_cnt);
 #undef K
@@ -264,6 +270,7 @@ static void bdi_wb_init(struct bdi_write
 	INIT_LIST_HEAD(&wb->b_dirty);
 	INIT_LIST_HEAD(&wb->b_io);
 	INIT_LIST_HEAD(&wb->b_more_io);
+	INIT_LIST_HEAD(&wb->b_more_io_wait);
 }
 
 static void bdi_task_init(struct backing_dev_info *bdi,
@@ -688,6 +695,7 @@ void bdi_destroy(struct backing_dev_info
 		list_splice(&bdi->wb.b_dirty, &dst->b_dirty);
 		list_splice(&bdi->wb.b_io, &dst->b_io);
 		list_splice(&bdi->wb.b_more_io, &dst->b_more_io);
+		list_splice(&bdi->wb.b_more_io_wait, &dst->b_more_io_wait);
 		spin_unlock(&inode_lock);
 	}
 
#99 lore raw

PATCH 39/45 writeback: remove wbc.more_io

In reply to Wu Fengguang

It seems no longer required. It was introduced mainly to deal with the complexity from _multiple_ superblock queues. Now there are only one queue, so infomation can be directly queried if necessary.

CC Theodore Ts'o <[email protected]>
CC Dave Chinner <[email protected]>
---
 fs/fs-writeback.c           |   13 +------------
 include/linux/writeback.h   |    1 -
 include/trace/events/ext4.h |    6 ++----
 3 files changed, 3 insertions(+), 17 deletions(-)
fs/fs-writeback.c +1 -12
@@ -733,12 +733,8 @@ static long writeback_inodes_wb(struct b
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
-		if (wbc->nr_to_write <= 0) {
-			wbc->more_io = 1;
+		if (wbc->nr_to_write <= 0)
 			break;
-		}
-		if (!list_empty(&wb->b_more_io))
-			wbc->more_io = 1;
 		if (stop_time && time_after(jiffies, stop_time))
 			break;
 	}
@@ -800,7 +796,6 @@ static long wb_writeback(struct bdi_writ
 	int fg_rounds = 0;
 	long wrote = 0;
 	long nr;
-	struct inode *inode;
 
 	/*
 	 * keep sync from extra jobs and livelock
@@ -840,7 +835,6 @@ static long wb_writeback(struct bdi_writ
 		if (args->for_background && !over_bground_thresh())
 			break;
 
-		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
 		wbc.pages_skipped = 0;
 		nr = writeback_inodes_wb(wb, &wbc);
@@ -863,11 +857,6 @@ static long wb_writeback(struct bdi_writ
 		}
 
 		/*
-		 * Bail if no more IO
-		 */
-		if (!wbc.more_io)
-			break;
-		/*
 		 * Did we write something? Try for more
 		 */
 		if (nr)
include/linux/writeback.h +0 -1
@@ -78,7 +78,6 @@ struct writeback_control {
 	unsigned for_sync:1;		/* A writeback for sync */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
-	unsigned more_io:1;		/* more io to be dispatched */
 	/*
 	 * write_cache_pages() won't update wbc->nr_to_write and
 	 * mapping->writeback_index if no_nrwrite_index_update
include/trace/events/ext4.h +2 -4
@@ -311,7 +311,6 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__field(	int,	pages_written		)
 		__field(	long,	pages_skipped		)
 		__field(	char,	encountered_congestion	)
-		__field(	char,	more_io			)	
 		__field(	char,	no_nrwrite_index_update )
 		__field(       pgoff_t,	writeback_index		)
 	),
@@ -323,16 +322,15 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__entry->pages_written	= pages_written;
 		__entry->pages_skipped	= wbc->pages_skipped;
 		__entry->encountered_congestion	= wbc->encountered_congestion;
-		__entry->more_io	= wbc->more_io;
 		__entry->no_nrwrite_index_update = wbc->no_nrwrite_index_update;
 		__entry->writeback_index = inode->i_mapping->writeback_index;
 	),
 
-	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d more_io %d no_nrwrite_index_update %d writeback_index %lu",
+	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d no_nrwrite_index_update %d writeback_index %lu",
 		  jbd2_dev_to_name(__entry->dev),
 		  (unsigned long) __entry->ino, __entry->ret,
 		  __entry->pages_written, __entry->pages_skipped,
-		  __entry->encountered_congestion, __entry->more_io,
+		  __entry->encountered_congestion,
 		  __entry->no_nrwrite_index_update,
 		  (unsigned long) __entry->writeback_index)
 );
#100 lore raw

PATCH 40/45 writeback: requeue_io_wait() on I_SYNC locked inode

In reply to Wu Fengguang

Use requeue_io_wait() if inode is being synced by others. This queue won't be busy retried, so avoids busy loops.

Cc Jan Kara <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
fs/fs-writeback.c +3 -3
@@ -526,14 +526,14 @@ writeback_single_inode(struct inode *ino
 	if (inode->i_state & I_SYNC) {
 		/*
 		 * If this inode is locked for writeback and we are not doing
-		 * writeback-for-data-integrity, move it to b_more_io so that
-		 * writeback can proceed with the other inodes on s_io.
+		 * writeback-for-data-integrity, move it to b_more_io_wait so
+		 * that writeback can proceed with the other inodes on b_io.
 		 *
 		 * We'll have another go at writing back this inode when we
 		 * completed a full scan of b_io.
 		 */
 		if (!wait) {
-			requeue_io(inode);
+			requeue_io_wait(inode);
 			return 0;
 		}
 
#101 lore raw

PATCH 41/45 writeback: requeue_io_wait() on pages_skipped inode

In reply to Wu Fengguang

Use requeue_io_wait() if some pages were skipped due to locked buffers.

Cc Dave Chinner <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -727,7 +727,7 @@ static long writeback_inodes_wb(struct b
 			 * writeback is not making progress due to locked
 			 * buffers.  Skip this inode for now.
 			 */
-			redirty_tail(inode);
+			requeue_io_wait(inode);
 		}
 		spin_unlock(&inode_lock);
 		iput(inode);
#102 lore raw

PATCH 42/45 writeback: requeue_io_wait() on blocked inode

In reply to Wu Fengguang

Use requeue_io_wait() if inode is somehow blocked. This includes the wrapped around range_cyclic case.

CC Jan Kara <[email protected]>
CC Dave Chinner <[email protected]>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -591,7 +591,7 @@ writeback_single_inode(struct inode *ino
 				/*
 				 * somehow blocked: retry later
 				 */
-				redirty_tail(inode);
+				requeue_io_wait(inode);
 			}
 		} else if (inode->i_state & I_DIRTY) {
 			/*
#103 lore raw

PATCH 43/45 writeback: requeue_io_wait() on fs redirtied inode

In reply to Wu Fengguang

When an inodes is redirtied by the filesystem, its dirty time shall not be updated.

Cc Jan Kara <[email protected]>
Cc Dave Chinner <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -598,7 +598,7 @@ writeback_single_inode(struct inode *ino
 			 * At least XFS will redirty the inode during the
 			 * writeback (delalloc) and on io completion (isize).
 			 */
-			redirty_tail(inode);
+			requeue_io_wait(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
#104 lore raw

PATCH 44/45 NFS: remove NFS_INO_FLUSHING lock

In reply to Wu Fengguang

It was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes

CC Peter Zijlstra <[email protected]>
CC Peter Staubach <[email protected]>
CC Trond Myklebust <[email protected]>
---
 fs/nfs/file.c          |    9 ---------
 fs/nfs/write.c         |   11 -----------
 include/linux/nfs_fs.h |    1 -
 3 files changed, 21 deletions(-)
fs/nfs/file.c +0 -9
@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file *
 		mapping->host->i_ino, len, (long long) pos);
 
 start:
-	/*
-	 * Prevent starvation issues if someone is doing a consistency
-	 * sync-to-disk
-	 */
-	ret = wait_on_bit(&NFS_I(mapping->host)->flags, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (ret)
-		return ret;
-
 	page = grab_cache_page_write_begin(mapping, index, flags);
 	if (!page)
 		return -ENOMEM;
fs/nfs/write.c +0 -11
@@ -387,26 +387,15 @@ static int nfs_writepages_callback(struc
 int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
 	struct inode *inode = mapping->host;
-	unsigned long *bitlock = &NFS_I(inode)->flags;
 	struct nfs_pageio_descriptor pgio;
 	int err;
 
-	/* Stop dirtying of new pages while we sync */
-	err = wait_on_bit_lock(bitlock, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (err)
-		goto out_err;
-
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES);
 
 	nfs_pageio_init_write(&pgio, inode, wb_priority(wbc));
 	err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio);
 	nfs_pageio_complete(&pgio);
 
-	clear_bit_unlock(NFS_INO_FLUSHING, bitlock);
-	smp_mb__after_clear_bit();
-	wake_up_bit(bitlock, NFS_INO_FLUSHING);
-
 	if (err < 0)
 		goto out_err;
 	err = pgio.pg_error;
include/linux/nfs_fs.h +0 -1
@@ -208,7 +208,6 @@ struct nfs_inode {
 #define NFS_INO_STALE		(1)		/* possible stale inode */
 #define NFS_INO_ACL_LRU_SET	(2)		/* Inode is on the LRU list */
 #define NFS_INO_MOUNTPOINT	(3)		/* inode is remote mountpoint */
-#define NFS_INO_FLUSHING	(4)		/* inode is flushing out data */
 #define NFS_INO_FSCACHE		(5)		/* inode can be cached by FS-Cache */
 #define NFS_INO_FSCACHE_LOCK	(6)		/* FS-Cache cookie management lock */
 
#105 lore raw

Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock

Quoted 7 lines · Wu Fengguang

It was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes

I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.

If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)

	Thanx...

		ps
Quoted 66 lines
CC Peter Zijlstra <[email protected]>
CC Peter Staubach <[email protected]>
CC Trond Myklebust <[email protected]>
---
 fs/nfs/file.c          |    9 ---------
 fs/nfs/write.c         |   11 -----------
 include/linux/nfs_fs.h |    1 -
 3 files changed, 21 deletions(-)
fs/nfs/file.c +0 -9
@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file *
 		mapping->host->i_ino, len, (long long) pos);
 
 start:
-	/*
-	 * Prevent starvation issues if someone is doing a consistency
-	 * sync-to-disk
-	 */
-	ret = wait_on_bit(&NFS_I(mapping->host)->flags, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (ret)
-		return ret;
-
 	page = grab_cache_page_write_begin(mapping, index, flags);
 	if (!page)
 		return -ENOMEM;
fs/nfs/write.c +0 -11
@@ -387,26 +387,15 @@ static int nfs_writepages_callback(struc
 int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
 	struct inode *inode = mapping->host;
-	unsigned long *bitlock = &NFS_I(inode)->flags;
 	struct nfs_pageio_descriptor pgio;
 	int err;
 
-	/* Stop dirtying of new pages while we sync */
-	err = wait_on_bit_lock(bitlock, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (err)
-		goto out_err;
-
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES);
 
 	nfs_pageio_init_write(&pgio, inode, wb_priority(wbc));
 	err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio);
 	nfs_pageio_complete(&pgio);
 
-	clear_bit_unlock(NFS_INO_FLUSHING, bitlock);
-	smp_mb__after_clear_bit();
-	wake_up_bit(bitlock, NFS_INO_FLUSHING);
-
 	if (err < 0)
 		goto out_err;
 	err = pgio.pg_error;
include/linux/nfs_fs.h +0 -1
@@ -208,7 +208,6 @@ struct nfs_inode {
 #define NFS_INO_STALE		(1)		/* possible stale inode */
 #define NFS_INO_ACL_LRU_SET	(2)		/* Inode is on the LRU list */
 #define NFS_INO_MOUNTPOINT	(3)		/* inode is remote mountpoint */
-#define NFS_INO_FLUSHING	(4)		/* inode is flushing out data */
 #define NFS_INO_FSCACHE		(5)		/* inode can be cached by FS-Cache */
 #define NFS_INO_FSCACHE_LOCK	(6)		/* FS-Cache cookie management lock */
#106 lore raw

Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock

Quoted 9 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang

It was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes

Quoted 3 lines

I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.

Ah OK.

Quoted 3 lines

If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)

I noticed the i_mutex lock in nfs_getattr(). Do you mean that?

Thanks,
Fengguang

Quoted 66 lines
Quoted 66 lines
CC Peter Zijlstra <[email protected]>
CC Peter Staubach <[email protected]>
CC Trond Myklebust <[email protected]>
---
 fs/nfs/file.c          |    9 ---------
 fs/nfs/write.c         |   11 -----------
 include/linux/nfs_fs.h |    1 -
 3 files changed, 21 deletions(-)
fs/nfs/file.c +0 -9
@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file *
 		mapping->host->i_ino, len, (long long) pos);
 
 start:
-	/*
-	 * Prevent starvation issues if someone is doing a consistency
-	 * sync-to-disk
-	 */
-	ret = wait_on_bit(&NFS_I(mapping->host)->flags, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (ret)
-		return ret;
-
 	page = grab_cache_page_write_begin(mapping, index, flags);
 	if (!page)
 		return -ENOMEM;
fs/nfs/write.c +0 -11
@@ -387,26 +387,15 @@ static int nfs_writepages_callback(struc
 int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
 	struct inode *inode = mapping->host;
-	unsigned long *bitlock = &NFS_I(inode)->flags;
 	struct nfs_pageio_descriptor pgio;
 	int err;
 
-	/* Stop dirtying of new pages while we sync */
-	err = wait_on_bit_lock(bitlock, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (err)
-		goto out_err;
-
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES);
 
 	nfs_pageio_init_write(&pgio, inode, wb_priority(wbc));
 	err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio);
 	nfs_pageio_complete(&pgio);
 
-	clear_bit_unlock(NFS_INO_FLUSHING, bitlock);
-	smp_mb__after_clear_bit();
-	wake_up_bit(bitlock, NFS_INO_FLUSHING);
-
 	if (err < 0)
 		goto out_err;
 	err = pgio.pg_error;
include/linux/nfs_fs.h +0 -1
@@ -208,7 +208,6 @@ struct nfs_inode {
 #define NFS_INO_STALE		(1)		/* possible stale inode */
 #define NFS_INO_ACL_LRU_SET	(2)		/* Inode is on the LRU list */
 #define NFS_INO_MOUNTPOINT	(3)		/* inode is remote mountpoint */
-#define NFS_INO_FLUSHING	(4)		/* inode is flushing out data */
 #define NFS_INO_FSCACHE		(5)		/* inode can be cached by FS-Cache */
 #define NFS_INO_FSCACHE_LOCK	(6)		/* FS-Cache cookie management lock */
#107 lore raw

Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock

Quoted 13 lines · Wu Fengguang
Quoted 11 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang

It was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes

I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.

Quoted 8 lines

Ah OK.

Quoted 3 lines

If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)

I noticed the i_mutex lock in nfs_getattr(). Do you mean that?

Well, that's part of that support as well. That keeps a writing application from dirtying more pages while the application doing the stat is attempting to clean them.

Another approach that I suggested was to keep track of the number of pages which are dirty on a per-inode basis. When enough pages are dirty to fill an over the wire transfer, then schedule an asynchronous write to transmit that data to the server. This ties in with support to ensure that the server/network is not completely overwhelmed by the client by flow controlling the writing application to better match the bandwidth and latencies of the network and server. With this support, the NFS client tends not to fill memory with dirty pages and thus, does not depend upon the other parts of the system to flush these pages.

All of these recent pages make this current flushing happen in a much more orderly fashion, which is great. However, this can still lead to the client attempting to flush potentially gigabytes all at once, which is more than most networks and servers can handle reasonably.

ps

Quoted 69 lines

Thanks,
Fengguang

Quoted 66 lines
Quoted 66 lines
CC Peter Zijlstra <[email protected]>
CC Peter Staubach <[email protected]>
CC Trond Myklebust <[email protected]>
---
 fs/nfs/file.c          |    9 ---------
 fs/nfs/write.c         |   11 -----------
 include/linux/nfs_fs.h |    1 -
 3 files changed, 21 deletions(-)
fs/nfs/file.c +0 -9
@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file *
 		mapping->host->i_ino, len, (long long) pos);
 
 start:
-	/*
-	 * Prevent starvation issues if someone is doing a consistency
-	 * sync-to-disk
-	 */
-	ret = wait_on_bit(&NFS_I(mapping->host)->flags, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (ret)
-		return ret;
-
 	page = grab_cache_page_write_begin(mapping, index, flags);
 	if (!page)
 		return -ENOMEM;
fs/nfs/write.c +0 -11
@@ -387,26 +387,15 @@ static int nfs_writepages_callback(struc
 int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
 	struct inode *inode = mapping->host;
-	unsigned long *bitlock = &NFS_I(inode)->flags;
 	struct nfs_pageio_descriptor pgio;
 	int err;
 
-	/* Stop dirtying of new pages while we sync */
-	err = wait_on_bit_lock(bitlock, NFS_INO_FLUSHING,
-			nfs_wait_bit_killable, TASK_KILLABLE);
-	if (err)
-		goto out_err;
-
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES);
 
 	nfs_pageio_init_write(&pgio, inode, wb_priority(wbc));
 	err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio);
 	nfs_pageio_complete(&pgio);
 
-	clear_bit_unlock(NFS_INO_FLUSHING, bitlock);
-	smp_mb__after_clear_bit();
-	wake_up_bit(bitlock, NFS_INO_FLUSHING);
-
 	if (err < 0)
 		goto out_err;
 	err = pgio.pg_error;
include/linux/nfs_fs.h +0 -1
@@ -208,7 +208,6 @@ struct nfs_inode {
 #define NFS_INO_STALE		(1)		/* possible stale inode */
 #define NFS_INO_ACL_LRU_SET	(2)		/* Inode is on the LRU list */
 #define NFS_INO_MOUNTPOINT	(3)		/* inode is remote mountpoint */
-#define NFS_INO_FLUSHING	(4)		/* inode is flushing out data */
 #define NFS_INO_FSCACHE		(5)		/* inode can be cached by FS-Cache */
 #define NFS_INO_FSCACHE_LOCK	(6)		/* FS-Cache cookie management lock */
#108 lore raw

Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock

Quoted 20 lines · Peter Staubach
Quoted 19 lines · Wu Fengguang
Quoted 11 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang

It was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes

I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.

Ah OK.

Quoted 3 lines

If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)

Quoted 6 lines
Quoted 2 lines

I noticed the i_mutex lock in nfs_getattr(). Do you mean that?

Well, that's part of that support as well. That keeps a writing application from dirtying more pages while the application doing the stat is attempting to clean them.

Instead of blocking totally, we could throttle application writes. The following two patches are the more gentle way of doing this, however it does not guarantee to kill the livelock, since a busy bdi-flush thread could writeback many pages to unfreeze the application prematurely. Anyway I attach them as a demo of idea, whether it be good or bad.

Quoted 2 lines

Another approach that I suggested was to keep track of the number of pages which are dirty on a per-inode basis. When

Yes a per-inode dirty count should be trivial and may be good for others.

Quoted 2 lines

enough pages are dirty to fill an over the wire transfer, then schedule an asynchronous write to transmit that data to

This should also be trivial to support if the location ordered writeback infrastructure is ready.

Quoted 4 lines

the server. This ties in with support to ensure that the server/network is not completely overwhelmed by the client by flow controlling the writing application to better match the bandwidth and latencies of the network and server.

I like that feature :)

Quoted 6 lines

With this support, the NFS client tends not to fill memory with dirty pages and thus, does not depend upon the other parts of the system to flush these pages.

All of these recent pages make this current flushing happen in a much more orderly fashion, which is great. However,

Thanks.

Quoted 3 lines

this can still lead to the client attempting to flush potentially gigabytes all at once, which is more than most networks and servers can handle reasonably.

OK, I now see the need to keep mapping->nr_dirty under control: it could make many NFS operations response in a more bounded fashion. The good thing is, it can share infrastructure with the location based writeback (http://lkml.org/lkml/2007/8/27/45 :)

Thanks,
Fengguang

writeback: sync livelock - throttle mapping dirties if it's being synced

The AS_SYNC_WAITER flag will be set to indicate an active sync waiter.

It's not perfect with respect to file ranges and concurrent syncs. And do not guarantee that the application writes on this mapping can be throttled enough to avoid livelock, especially when there are many background writebacks.

Just my 2 cents.

CC Jan Kara <[email protected]>
CC Peter Staubach <[email protected]>
CC Myklebust Trond <[email protected]>
---
 include/linux/pagemap.h |    3 +++
 mm/filemap.c            |    2 ++
 mm/page-writeback.c     |    5 +++++
 3 files changed, 10 insertions(+)
mm/page-writeback.c +5 -0
@@ -480,7 +480,12 @@ static void balance_dirty_pages(struct a
 	/*
 	 * If sync() is in progress, curb the to-be-synced inodes regardless
 	 * of dirty limits, so that a fast dirtier won't livelock the sync.
+	 * In perticular, NFS syncs the mapping before many file operations.
 	 */
+	if (unlikely(test_bit(AS_SYNC_WAITER, &mapping->flags))) {
+		write_chunk *= 8;
+		goto throttle;
+	}
 	if (unlikely(bdi->sync_time &&
 		     S_ISREG(mapping->host->i_mode) &&
 		     time_after_eq(bdi->sync_time,
include/linux/pagemap.h +3 -0
@@ -23,6 +23,9 @@ enum mapping_flags {
 	AS_ENOSPC	= __GFP_BITS_SHIFT + 1,	/* ENOSPC on async write */
 	AS_MM_ALL_LOCKS	= __GFP_BITS_SHIFT + 2,	/* under mm_take_all_locks() */
 	AS_UNEVICTABLE	= __GFP_BITS_SHIFT + 3,	/* e.g., ramdisk, SHM_LOCK */
+	AS_SYNC_WAITER	= __GFP_BITS_SHIFT + 4,	/* sync&wait under way, page
+						 * dirtying will be throttled
+						 */
 };
 
 static inline void mapping_set_error(struct address_space *mapping, int error)
mm/filemap.c +2 -0
@@ -356,6 +356,7 @@ int filemap_write_and_wait(struct addres
 	int err = 0;
 
 	if (mapping->nrpages) {
+		set_bit(AS_SYNC_WAITER, &mapping->flags);
 		err = filemap_fdatawrite(mapping);
 		/*
 		 * Even if the above returned error, the pages may be
@@ -368,6 +369,7 @@ int filemap_write_and_wait(struct addres
 			if (!err)
 				err = err2;
 		}
+		clear_bit(AS_SYNC_WAITER, &mapping->flags);
 	}
 	return err;
 }

nfs: don't take i_mutex on nfs_wb_nocommit()

Set the AS_SYNC_WAITER flag and let VFS throttle application writes.

CC Peter Staubach <[email protected]>
CC Myklebust Trond <[email protected]>
---
 fs/nfs/inode.c |    9 +--------
 fs/nfs/write.c |    6 +++++-
 2 files changed, 6 insertions(+), 9 deletions(-)
fs/nfs/inode.c +1 -8
@@ -513,16 +513,9 @@ int nfs_getattr(struct vfsmount *mnt, st
 
 	/*
 	 * Flush out writes to the server in order to update c/mtime.
-	 *
-	 * Hold the i_mutex to suspend application writes temporarily;
-	 * this prevents long-running writing applications from blocking
-	 * nfs_wb_nocommit.
 	 */
-	if (S_ISREG(inode->i_mode)) {
-		mutex_lock(&inode->i_mutex);
+	if (S_ISREG(inode->i_mode))
 		nfs_wb_nocommit(inode);
-		mutex_unlock(&inode->i_mutex);
-	}
 
 	/*
 	 * We may force a getattr if the user cares about atime.
fs/nfs/write.c +5 -1
@@ -1539,8 +1539,12 @@ static int nfs_write_mapping(struct addr
 		.range_start = 0,
 		.range_end = LLONG_MAX,
 	};
+	int ret;
 
-	return __nfs_write_mapping(mapping, &wbc, how);
+	set_bit(AS_SYNC_WAITER, &mapping->flags);
+	ret = __nfs_write_mapping(mapping, &wbc, how);
+	clear_bit(AS_SYNC_WAITER, &mapping->flags);
+	return ret;
 }
 
 /*
#109 lore raw

PATCH 45/45 btrfs: fix race on syncing the btree inode

In reply to Wu Fengguang

When doing sync, the btree dirty pages refuse to go away for tens of seconds:

# vmmon -d 1 nr_writeback nr_dirty nr_unstable

nr_writeback nr_dirty nr_unstable
46641 23315 0
46641 23380 0
46641 23381 0
26674 43206 0
18963 51006 0
11252 58721 0
3528 66419 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0

Note that the 70024 pages are under the btree inode's 32MB no-write-metadata threshold. This is racy because the sync work has to sleep and retry it forever for data integrity.

The 32MB threshold may also become a problem for background writeback given a memory tight box. So it may be better to replace the threshold with some informed writeback tricks.

---
 fs/btrfs/disk-io.c |   29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)
fs/btrfs/disk-io.c +13 -16
@@ -707,22 +707,19 @@ static int btree_writepage(struct page *
 static int btree_writepages(struct address_space *mapping,
 			    struct writeback_control *wbc)
 {
-	struct extent_io_tree *tree;
-	tree = &BTRFS_I(mapping->host)->io_tree;
-	if (wbc->sync_mode == WB_SYNC_NONE) {
-		struct btrfs_root *root = BTRFS_I(mapping->host)->root;
-		u64 num_dirty;
-		unsigned long thresh = 32 * 1024 * 1024;
-
-		if (wbc->for_kupdate)
-			return 0;
-
-		/* this is a bit racy, but that's ok */
-		num_dirty = root->fs_info->dirty_metadata_bytes;
-		if (num_dirty < thresh)
-			return 0;
-	}
-	return extent_writepages(tree, mapping, btree_get_extent, wbc);
+	struct extent_io_tree *tree = &BTRFS_I(mapping->host)->io_tree;
+	int ret;
+
+	if (!wbc->for_sync)
+		wbc->nr_segments = 1;
+	ret = extent_writepages(tree, mapping, btree_get_extent, wbc);
+	/*
+	 * Fake some some skipped pages, so that VFS won't
+	 * try hard on writing this inode.
+	 */
+	if (!wbc->for_sync)
+		wbc->pages_skipped++;
+	return ret;
 }
 
 static int btree_readpage(struct file *file, struct page *page)
#110 lore raw
In reply to Wu Fengguang
Quoted 11 lines · Wu Fengguang

Hi all,

Here is a collection of writeback patches on

- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes

Quoted 2 lines

Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..

Awesome bits..

#111 lore raw
In reply to Wu Fengguang
Quoted 21 lines · Wu Fengguang

Hi all,

Here is a collection of writeback patches on

- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes

Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..

The lumpy pageout and nr_segments support is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.

I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!

Quoted 7 lines

The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..

NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:

Yes. It appears that the problem is that too many pages get dirtied and the network/server get overwhelmed by the NFS client attempting to write out all of the pages as quickly as it possibly can.

I think that it would be better if we could better match the number of pages which can be dirty at any given point with the bandwidth available through the network and the server file system and storage.

One approach that I have pondered is immediately queuing an asynchronous request when enough pages are dirtied to be able to completely fill an over the wire transfer. This sort of seems like a per-file bdi, which doesn't seem quite like the right approach to me. What would y'all think about that?

ps

Quoted 50 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable      # (per 1-second samples)
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913

btrfs seems to maintain a private pool of writeback pages, which can go out of control:

     nr_writeback         nr_dirty
           261075              132
           252891              195
           244795              187
           236851              187
           228830              187
           221040              218
           212674              237
           204981              237

XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.

     nr_writeback         nr_dirty
            80781                0
            37117            37703
            37117            43933
            81044                6
            81050                0
            43943            10199
            43930            36355
            43930            36355
            80293                0
            80285                0
            80285                0

Thanks,
Fengguang

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#112 lore raw
Quoted 26 lines · Peter Staubach
Quoted 25 lines · Wu Fengguang

Hi all,

Here is a collection of writeback patches on

- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes

Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..

The lumpy pageout and nr_segments support is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.

I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!

The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..

Quoted 7 lines
Quoted 3 lines

NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:

Yes. It appears that the problem is that too many pages get dirtied and the network/server get overwhelmed by the NFS client attempting to write out all of the pages as quickly as it possibly can.

In theory it should push pages as quickly as possible at first, to fill up the server side queue.

Quoted 4 lines

I think that it would be better if we could better match the number of pages which can be dirty at any given point with the bandwidth available through the network and the server file system and storage.

And then go into this steady state of matched network/disk bandwidth.

Quoted 5 lines

One approach that I have pondered is immediately queuing an asynchronous request when enough pages are dirtied to be able to completely fill an over the wire transfer. This sort of seems like a per-file bdi, which doesn't seem quite like the right approach to me. What would y'all think about that?

Hmm, it sounds like unnecessary complexity. Because it is not going to help the busy-dirtier case anyway. And if we can do good on heavy IO, the pre-flushing policy becomes less interesting.

Quoted 7 lines
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable      # (per 1-second samples)
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044

I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both nr_writeback/nr_unstable. And the stuck nr_writeback will freeze nr_dirty as well, because the dirtying process is throttled until it receives enough "PG_writeback cleared" event, however the bdi-flush thread is also blocked when trying to clear more PG_writeback, because the client side nr_writeback limit has been reached. In summary,

server blocked => nr_writeback stuck => nr_writeback limit reached => bdi-flush blocked => no end_page_writeback() => dirtier blocked => nr_dirty stuck

Thanks,
Fengguang

Quoted 44 lines
Quoted 44 lines
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913

btrfs seems to maintain a private pool of writeback pages, which can go out of control:

     nr_writeback         nr_dirty
           261075              132
           252891              195
           244795              187
           236851              187
           228830              187
           221040              218
           212674              237
           204981              237

XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.

     nr_writeback         nr_dirty
            80781                0
            37117            37703
            37117            43933
            81044                6
            81050                0
            43943            10199
            43930            36355
            43930            36355
            80293                0
            80285                0
            80285                0

Thanks,
Fengguang

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#113 lore raw
Quoted 11 lines · Wu Fengguang
Quoted 7 lines · Peter Staubach
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable      # (per 1-second samples)
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044

I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both

Yeah the server side is blocked. The nfsd are mostly blocked in generic_file_aio_write(), in particular, the i_mutex lock! I'm copying one or two big files over NFS, so the i_mutex lock is heavily contented.

I'm using the default wsize=4096 for NFS-root..

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   1  0.0 D<   log_wait_commit          nfsd
 4697  4697 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.0 D<   sync_buffer              nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   1  0.0 D<   generic_file_aio_write   nfsd
 4697  4697 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.1 D<   get_request_wait         nfsd
 4692  4692 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4697  4697 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   1  0.1 D<   get_write_access         nfsd
 4691  4691 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4694  4694 TS       -  -5  24   1  0.1 D<   get_write_access         nfsd
 4695  4695 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4697  4697 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd

Thanks,
Fengguang

Quoted 57 lines

nr_writeback/nr_unstable. And the stuck nr_writeback will freeze nr_dirty as well, because the dirtying process is throttled until it receives enough "PG_writeback cleared" event, however the bdi-flush thread is also blocked when trying to clear more PG_writeback, because the client side nr_writeback limit has been reached. In summary,

server blocked => nr_writeback stuck => nr_writeback limit reached => bdi-flush blocked => no end_page_writeback() => dirtier blocked => nr_dirty stuck

Thanks,
Fengguang

Quoted 44 lines
Quoted 44 lines
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913

btrfs seems to maintain a private pool of writeback pages, which can go out of control:

     nr_writeback         nr_dirty
           261075              132
           252891              195
           244795              187
           236851              187
           228830              187
           221040              218
           212674              237
           204981              237

XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.

     nr_writeback         nr_dirty
            80781                0
            37117            37703
            37117            43933
            81044                6
            81050                0
            43943            10199
            43930            36355
            43930            36355
            80293                0
            80285                0
            80285                0

Thanks,
Fengguang

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#114 lore raw
Quoted 10 lines · Wu Fengguang
Quoted 9 lines · Wu Fengguang
Quoted 7 lines · Peter Staubach
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable      # (per 1-second samples)
     nr_writeback         nr_dirty      nr_unstable
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
            11227            41463            38044
Quoted 8 lines
Quoted 2 lines

I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both

Yeah the server side is blocked. The nfsd are mostly blocked in generic_file_aio_write(), in particular, the i_mutex lock! I'm copying one or two big files over NFS, so the i_mutex lock is heavily contented.

I'm using the default wsize=4096 for NFS-root..

Just switched to 512k wsize, and things improved: in most time the 8 nfsd are not all blocked. However, the bumpiness still remains:

     nr_writeback         nr_dirty      nr_unstable
            11105            58080            15042
            11105            58080            15042
            11233            54583            18626
            11101            51964            22036
            11105            51978            22065
            11233            52362            22577
            10985            58538            13500
            11233            53748            19721
            11047            51999            21778
            11105            50262            23572
            11105            50262            20441
            10985            52772            20721
            10977            52109            21516
            11105            48296            26629
            11105            48296            26629
            10985            52191            21042
            11166            51456            22296
            10980            50681            24466
            11233            45352            30488
            11233            45352            30488
            11105            45475            30616
            11131            45313            20355
            11233            51126            22637
            11233            51126            22637

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   1  0.1 S<   svc_recv                 nfsd
 4691  4691 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4692  4692 TS       -  -5  24   0  0.1 R<   ?                        nfsd
 4693  4693 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4694  4694 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4695  4695 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4696  4696 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4697  4697 TS       -  -5  24   0  0.1 R<   ?                        nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4692  4692 TS       -  -5  24   1  0.1 D<   log_wait_commit          nfsd
 4693  4693 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4694  4694 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4695  4695 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4697  4697 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   1  0.1 S<   svc_recv                 nfsd
 4691  4691 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4692  4692 TS       -  -5  24   1  0.1 R<   ?                        nfsd
 4693  4693 TS       -  -5  24   1  0.1 R<   ?                        nfsd
 4694  4694 TS       -  -5  24   1  0.1 R<   ?                        nfsd
 4695  4695 TS       -  -5  24   1  0.1 S<   svc_recv                 nfsd
 4696  4696 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4697  4697 TS       -  -5  24   1  0.1 S<   svc_recv                 nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4692  4692 TS       -  -5  24   1  0.1 D<   nfsd_sync                nfsd
 4693  4693 TS       -  -5  24   1  0.1 D<   sync_buffer              nfsd
 4694  4694 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   1  0.1 S<   svc_recv                 nfsd
 4696  4696 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4697  4697 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd

Thanks,
Fengguang

Quoted 103 lines

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   1  0.0 D<   log_wait_commit          nfsd
 4697  4697 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.0 D<   sync_buffer              nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   1  0.0 D<   generic_file_aio_write   nfsd
 4697  4697 TS       -  -5  24   0  0.0 D<   generic_file_aio_write   nfsd

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4691  4691 TS       -  -5  24   0  0.1 D<   get_request_wait         nfsd
 4692  4692 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4694  4694 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4695  4695 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   0  0.1 S<   svc_recv                 nfsd
 4697  4697 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd

wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs

  329   329 TS       -  -5  24   1  0.0 S<   worker_thread            nfsiod
 4690  4690 TS       -  -5  24   1  0.1 D<   get_write_access         nfsd
 4691  4691 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4692  4692 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4693  4693 TS       -  -5  24   1  0.1 D<   generic_file_aio_write   nfsd
 4694  4694 TS       -  -5  24   1  0.1 D<   get_write_access         nfsd
 4695  4695 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4696  4696 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd
 4697  4697 TS       -  -5  24   0  0.1 D<   generic_file_aio_write   nfsd

Thanks,
Fengguang

Quoted 57 lines

nr_writeback/nr_unstable. And the stuck nr_writeback will freeze nr_dirty as well, because the dirtying process is throttled until it receives enough "PG_writeback cleared" event, however the bdi-flush thread is also blocked when trying to clear more PG_writeback, because the client side nr_writeback limit has been reached. In summary,

server blocked => nr_writeback stuck => nr_writeback limit reached => bdi-flush blocked => no end_page_writeback() => dirtier blocked => nr_dirty stuck

Thanks,
Fengguang

Quoted 44 lines
Quoted 44 lines
            11045            53987             6490
            11033            53120             8145
            11195            52143            10886
            11211            52144            10913
            11211            52144            10913
            11211            52144            10913

btrfs seems to maintain a private pool of writeback pages, which can go out of control:

     nr_writeback         nr_dirty
           261075              132
           252891              195
           244795              187
           236851              187
           228830              187
           221040              218
           212674              237
           204981              237

XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.

     nr_writeback         nr_dirty
            80781                0
            37117            37703
            37117            43933
            81044                6
            81050                0
            43943            10199
            43930            36355
            43930            36355
            80293                0
            80285                0
            80285                0

Thanks,
Fengguang

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

maintainer
#115 lore raw
In reply to Wu Fengguang
Quoted 4 lines · Wu Fengguang

The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..

Can you expand on what you mean here? Estimated write bandwidth of what? And what are you comparing it against?

I'm having trouble understanding your note (which I'm guessing you write fairly late at night? :-)

Thanks,

- Ted

#116 lore raw
Quoted 8 lines · Theodore Ts'o
Quoted 4 lines · Wu Fengguang

The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..

Can you expand on what you mean here? Estimated write bandwidth of what? And what are you comparing it against?

Please refer to [PATCH 21/45] writeback: estimate bdi write bandwidth and patch 22, I have some numbers there :)

Quoted 2 lines

I'm having trouble understanding your note (which I'm guessing you write fairly late at night? :-)

Sorry - I wrote that when I got tired on debugging ;)

Thanks,
Fengguang