Skip to contentSkip to conversation

some random writeback fixes

RFC PATCH In review Wu Fengguang 5 participants 27 messages Sep 9, 2009 – Sep 10, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover some random writeback fixes

Hi all,

These are some writeback fixes based on the latest linux-next without the vm.max_writeback_mb knob, since I'm not aware of any strong reasons for introducing it. Instead, the MAX_WRITEBACK_PAGES is directly increased to 64MB.

Other patches are straightforward. Needs more tests but it's enough to show the basic ideas. Comments are welcome!

Thanks,
Fengguang

#1 lore raw

RFC PATCH 1/7 writeback: cleanup writeback_single_inode()

Make the if-else straight in writeback_single_inode(). No behavior change.

Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |   15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
fs/fs-writeback.c +7 -8
@@ -417,8 +417,13 @@ writeback_single_inode(struct inode *ino
 	spin_lock(&inode_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
-		if (!(inode->i_state & I_DIRTY) &&
-		    mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
+		if (inode->i_state & I_DIRTY) {
+			/*
+			 * Someone redirtied the inode while were writing back
+			 * the pages.
+			 */
+			redirty_tail(inode);
+		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
 			 * sometimes bales out without doing anything. Redirty
@@ -462,12 +467,6 @@ writeback_single_inode(struct inode *ino
 				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
-		} else if (inode->i_state & I_DIRTY) {
-			/*
-			 * Someone redirtied the inode while were writing back
-			 * the pages.
-			 */
-			redirty_tail(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
#16 lore raw

RFC PATCH 7/7 writeback: balance_dirty_pages() shall write more than dirtied pages

In reply to Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

---
 mm/page-writeback.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
mm/page-writeback.c +7 -6
@@ -44,12 +44,12 @@ static long ratelimit_pages = 32;
 /*
  * When balance_dirty_pages decides that the caller needs to perform some
  * non-background writeback, this is how many pages it will attempt to write.
- * It should be somewhat larger than RATELIMIT_PAGES to ensure that reasonably
+ * It should be somewhat larger than dirtied pages to ensure that reasonably
  * large amounts of I/O are submitted.
  */
-static inline long sync_writeback_pages(void)
+static inline long sync_writeback_pages(unsigned long dirtied)
 {
-	return ratelimit_pages + ratelimit_pages / 2;
+	return dirtied + dirtied / 2;
 }
 
 /* The following parameters are exported via /proc/sys/vm */
@@ -476,7 +476,8 @@ get_dirty_limits(unsigned long *pbackgro
  * If we're over `background_thresh' then pdflush is woken to perform some
  * writeout.
  */
-static void balance_dirty_pages(struct address_space *mapping)
+static void balance_dirty_pages(struct address_space *mapping,
+				unsigned long write_chunk)
 {
 	long nr_reclaimable, bdi_nr_reclaimable;
 	long nr_writeback, bdi_nr_writeback;
@@ -484,7 +485,6 @@ static void balance_dirty_pages(struct a
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
 	unsigned long pages_written = 0;
-	unsigned long write_chunk = sync_writeback_pages();
 
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
@@ -638,9 +638,10 @@ void balance_dirty_pages_ratelimited_nr(
 	p =  &__get_cpu_var(bdp_ratelimits);
 	*p += nr_pages_dirtied;
 	if (unlikely(*p >= ratelimit)) {
+		ratelimit = sync_writeback_pages(*p);
 		*p = 0;
 		preempt_enable();
-		balance_dirty_pages(mapping);
+		balance_dirty_pages(mapping, ratelimit);
 		return;
 	}
 	preempt_enable();
#3 lore raw

RFC PATCH 2/7 writeback: fix queue_io() ordering

In reply to Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -313,11 +313,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
#15 lore raw

RFC PATCH 6/7 writeback: dont abort inode on congestion

In reply to Wu Fengguang

Retry writeback of the current inode when encountered congestion, instead of redirting it, which may have the problems
- slow down writeback pace
- lead to small writeback io

Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Chad Talbott <[email protected]>
---
 fs/fs-writeback.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/fs-writeback.c +4 -2
@@ -460,9 +460,11 @@ writeback_single_inode(struct inode *ino
 			 * sometimes bales out without doing anything.
 			 */
 			inode->i_state |= I_DIRTY_PAGES;
-			if (wbc->nr_to_write <= 0) {
+			if (wbc->encountered_congestion ||
+			    wbc->nr_to_write <= 0) {
 				/*
-				 * slice used up: queue for next turn
+				 * if slice used up, queue for next round;
+				 * otherwise continue this inode after return
 				 */
 				requeue_partial_io(wbc, inode);
 			} else {
#10 lore raw

RFC PATCH 5/7 writeback: use 64MB MAX_WRITEBACK_PAGES

In reply to Wu Fengguang

Theodore has a nice description:

Originally, MAX_WRITEBACK_PAGES was hard-coded to 1024 because of a concern of not holding I_SYNC for too long. (At least, that was the comment previously.) This doesn't make sense now because the only time we wait for I_SYNC is if we are calling sync or fsync, and in that case we need to write out all of the data anyway. Previously there may have been other code paths that waited on I_SYNC, but not any more.

According to Christoph, the current writeback size is way too small, and XFS had a hack that bumped out nr_to_write to four times the value sent by the VM to be able to saturate medium-sized RAID arrays. This value was also problematic for ext4 as well, as it caused large files to be come interleaved on disk by in 8 megabyte chunks (we bumped up the nr_to_write by a factor of two).

So, in this patch, we make the MAX_WRITEBACK_PAGES a tunable, max_writeback_mb, and set it to a default value of 128 megabytes.

http://bugzilla.kernel.org/show_bug.cgi?id=13930

CC Theodore Ts'o <[email protected]>
CC Dave Chinner <[email protected]>
CC Chris Mason <[email protected]>
CC Peter Zijlstra <[email protected]>
---
 include/linux/writeback.h |   13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
include/linux/writeback.h +6 -7
@@ -14,14 +14,13 @@ extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
 /*
- * The maximum number of pages to writeout in a single bdi flush/kupdate
- * operation.  We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode.  Also, the code reevaluates
- * the dirty each time it has written this many pages.
+ * The maximum number of pages to writeout in a single periodic/background
+ * writeback operation. 64MB means I_SYNC may be hold for up to 1 second.
+ * This is not a big problem since we normally do kind of trylock on I_SYNC
+ * for non-data-integrity writes.  Userspace tasks doing throttled writeback
+ * do not use this value.
  */
-#define MAX_WRITEBACK_PAGES     1024
-
+#define MAX_WRITEBACK_PAGES	(64 * 1024 * 1024 / PAGE_SIZE)
 
 /*
  * fs/fs-writeback.c
#8 lore raw

RFC PATCH 3/7 writeback: merge for_kupdate and !for_kupdate requeue io logics

In reply to Wu Fengguang

Unify the logic for kupdate and non-kupdate cases.
There won't be starvation because the inodes requeued into b_more_io or b_more_io_wait will later be spliced _after_ the remaining inodes in b_io, hence won't stand in the way of other inodes in the next run.

CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |   39 ++++++---------------------------------
 1 file changed, 6 insertions(+), 33 deletions(-)
fs/fs-writeback.c +6 -33
@@ -426,45 +426,18 @@ writeback_single_inode(struct inode *ino
 		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
-			 * sometimes bales out without doing anything. Redirty
-			 * the inode; Move it from b_io onto b_more_io/b_dirty.
+			 * sometimes bales out without doing anything.
 			 */
-			/*
-			 * akpm: if the caller was the kupdate function we put
-			 * this inode at the head of b_dirty so it gets first
-			 * consideration.  Otherwise, move it to the tail, for
-			 * the reasons described there.  I'm not really sure
-			 * how much sense this makes.  Presumably I had a good
-			 * reasons for doing it this way, and I'd rather not
-			 * muck with it at present.
-			 */
-			if (wbc->for_kupdate) {
+			inode->i_state |= I_DIRTY_PAGES;
+			if (wbc->nr_to_write <= 0) {
 				/*
-				 * For the kupdate function we move the inode
-				 * to b_more_io so it will get more writeout as
-				 * soon as the queue becomes uncongested.
+				 * slice used up: queue for next turn
 				 */
-				inode->i_state |= I_DIRTY_PAGES;
-				if (wbc->nr_to_write <= 0) {
-					/*
-					 * slice used up: queue for next turn
-					 */
-					requeue_io(inode);
-				} else {
-					/*
-					 * somehow blocked: retry later
-					 */
-					redirty_tail(inode);
-				}
+				requeue_io(inode);
 			} else {
 				/*
-				 * Otherwise fully redirty the inode so that
-				 * other inodes on this superblock will get some
-				 * writeout.  Otherwise heavy writing to one
-				 * file would indefinitely suspend writeout of
-				 * all the other files.
+				 * somehow blocked: retry later
 				 */
-				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
 		} else if (atomic_read(&inode->i_count)) {
#9 lore raw

RFC PATCH 4/7 writeback: ensure large files are written in MAX_WRITEBACK_PAGES chunks

In reply to Wu Fengguang

Remember pages written for the current file between successive writeback_single_inode() invocations, and modify wbc->nr_to_write accordingly to continue write the file until MAX_WRITEBACK_PAGES is reached for this single file.

This ensures large files will be written in large MAX_WRITEBACK_PAGES chunks. It works best for kernel sync threads which repeatedly call into writeback_single_inode() with the same wbc. For balance_dirty_pages() that normally restart with a fresh wbc, it may never collect enough last_file_written to skip the current large file, hence lead to starvation of other (small) files. However/luckily balance_dirty_pages() writeback is normally interleaved with background writeback, which will do the duty of rotating the writeback files. So this is not a bit problem.

CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
CC Chris Mason <[email protected]>
CC Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c         |   41 +++++++++++++++++++++++++++---------
 include/linux/writeback.h |   12 ++++++++++
 2 files changed, 43 insertions(+), 10 deletions(-)
fs/fs-writeback.c +31 -10
@@ -271,6 +271,19 @@ static void requeue_io(struct inode *ino
 	list_move(&inode->i_list, &wb->b_more_io);
 }
 
+/*
+ * continue io on this inode on next writeback if
+ * it has not accumulated large enough writeback io chunk
+ */
+static void requeue_partial_io(struct writeback_control *wbc, struct inode *inode)
+{
+	if (wbc->last_file_written == 0 ||
+	    wbc->last_file_written >= MAX_WRITEBACK_PAGES)
+		return requeue_io(inode);
+
+	list_move_tail(&inode->i_list, &inode_to_bdi(inode)->wb.b_io);
+}
+
 static void inode_sync_complete(struct inode *inode)
 {
 	/*
@@ -365,6 +378,8 @@ writeback_single_inode(struct inode *ino
 {
 	struct address_space *mapping = inode->i_mapping;
 	int wait = wbc->sync_mode == WB_SYNC_ALL;
+	long last_file_written;
+	long nr_to_write;
 	unsigned dirty;
 	int ret;
 
@@ -402,8 +417,21 @@ writeback_single_inode(struct inode *ino
 
 	spin_unlock(&inode_lock);
 
+	if (wbc->last_file != inode->i_ino)
+		last_file_written = 0;
+	else
+		last_file_written = wbc->last_file_written;
+	wbc->nr_to_write -= last_file_written;
+	nr_to_write = wbc->nr_to_write;
+
 	ret = do_writepages(mapping, wbc);
 
+	if (wbc->last_file != inode->i_ino) {
+		wbc->last_file = inode->i_ino;
+		wbc->last_file_written = nr_to_write - wbc->nr_to_write;
+	} else
+		wbc->last_file_written += nr_to_write - wbc->nr_to_write;
+
 	/* Don't write the inode if only I_DIRTY_PAGES was set */
 	if (dirty & (I_DIRTY_SYNC | I_DIRTY_DATASYNC)) {
 		int err = write_inode(inode, wait);
@@ -436,7 +464,7 @@ writeback_single_inode(struct inode *ino
 				/*
 				 * slice used up: queue for next turn
 				 */
-				requeue_io(inode);
+				requeue_partial_io(wbc, inode);
 			} else {
 				/*
 				 * somehow blocked: retry later
@@ -456,6 +484,8 @@ writeback_single_inode(struct inode *ino
 		}
 	}
 	inode_sync_complete(inode);
+	wbc->nr_to_write += last_file_written;
+
 	return ret;
 }
 
@@ -612,15 +642,6 @@ void writeback_inodes_wbc(struct writeba
 	writeback_inodes_wb(&bdi->wb, wbc);
 }
 
-/*
- * The maximum number of pages to writeout in a single bdi flush/kupdate
- * operation.  We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode.  Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES     1024
-
 static inline bool over_bground_thresh(void)
 {
 	unsigned long background_thresh, dirty_thresh;
include/linux/writeback.h +12 -0
@@ -14,6 +14,16 @@ extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
 /*
+ * The maximum number of pages to writeout in a single bdi flush/kupdate
+ * operation.  We do this so we don't hold I_SYNC against an inode for
+ * enormous amounts of time, which would block a userspace task which has
+ * been forced to throttle against that inode.  Also, the code reevaluates
+ * the dirty each time it has written this many pages.
+ */
+#define MAX_WRITEBACK_PAGES     1024
+
+
+/*
  * fs/fs-writeback.c
  */
 enum writeback_sync_modes {
@@ -36,6 +46,8 @@ struct writeback_control {
 					   older than this */
 	long nr_to_write;		/* Write this many pages, and decrement
 					   this for each page written */
+	unsigned long last_file;	/* Inode number of last written file */
+	long last_file_written;		/* Total pages written for last file */
 	long pages_skipped;		/* Pages which were not written */
 
 	/*
maintainer
#17 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

In reply to Wu Fengguang
Quoted 7 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Honza

Quoted 55 lines
---
 mm/page-writeback.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
mm/page-writeback.c +7 -6
@@ -44,12 +44,12 @@ static long ratelimit_pages = 32;
 /*
  * When balance_dirty_pages decides that the caller needs to perform some
  * non-background writeback, this is how many pages it will attempt to write.
- * It should be somewhat larger than RATELIMIT_PAGES to ensure that reasonably
+ * It should be somewhat larger than dirtied pages to ensure that reasonably
  * large amounts of I/O are submitted.
  */
-static inline long sync_writeback_pages(void)
+static inline long sync_writeback_pages(unsigned long dirtied)
 {
-	return ratelimit_pages + ratelimit_pages / 2;
+	return dirtied + dirtied / 2;
 }
 
 /* The following parameters are exported via /proc/sys/vm */
@@ -476,7 +476,8 @@ get_dirty_limits(unsigned long *pbackgro
  * If we're over `background_thresh' then pdflush is woken to perform some
  * writeout.
  */
-static void balance_dirty_pages(struct address_space *mapping)
+static void balance_dirty_pages(struct address_space *mapping,
+				unsigned long write_chunk)
 {
 	long nr_reclaimable, bdi_nr_reclaimable;
 	long nr_writeback, bdi_nr_writeback;
@@ -484,7 +485,6 @@ static void balance_dirty_pages(struct a
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
 	unsigned long pages_written = 0;
-	unsigned long write_chunk = sync_writeback_pages();
 
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
@@ -638,9 +638,10 @@ void balance_dirty_pages_ratelimited_nr(
 	p =  &__get_cpu_var(bdp_ratelimits);
 	*p += nr_pages_dirtied;
 	if (unlikely(*p >= ratelimit)) {
+		ratelimit = sync_writeback_pages(*p);
 		*p = 0;
 		preempt_enable();
-		balance_dirty_pages(mapping);
+		balance_dirty_pages(mapping, ratelimit);
 		return;
 	}
 	preempt_enable();
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer Acked-by on v1 1/7
#2 lore raw

Re: [RFC][PATCH 1/7] writeback: cleanup writeback_single_inode()

In reply to Wu Fengguang
Quoted 6 lines · Wu Fengguang

Make the if-else straight in writeback_single_inode(). No behavior change.

Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>

The patch looks good.

Quoted 38 lines
---
 fs/fs-writeback.c |   15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
fs/fs-writeback.c +7 -8
@@ -417,8 +417,13 @@ writeback_single_inode(struct inode *ino
 	spin_lock(&inode_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
-		if (!(inode->i_state & I_DIRTY) &&
-		    mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
+		if (inode->i_state & I_DIRTY) {
+			/*
+			 * Someone redirtied the inode while were writing back
+			 * the pages.
+			 */
+			redirty_tail(inode);
+		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
 			 * sometimes bales out without doing anything. Redirty
@@ -462,12 +467,6 @@ writeback_single_inode(struct inode *ino
 				inode->i_state |= I_DIRTY_PAGES;
 				redirty_tail(inode);
 			}
-		} else if (inode->i_state & I_DIRTY) {
-			/*
-			 * Someone redirtied the inode while were writing back
-			 * the pages.
-			 */
-			redirty_tail(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#4 lore raw

Re: [RFC][PATCH 2/7] writeback: fix queue_io() ordering

In reply to Wu Fengguang
Quoted 3 lines · Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

But doesn't this patch also need your "anti-starvation" patch?
Looking into the code, we put inode to b_more_io when nr_to_write drops to zero and this way we'd just start writing it again in the next round...

Honza

Quoted 32 lines
CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -313,11 +313,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw

Re: [RFC][PATCH 5/7] writeback: use 64MB MAX_WRITEBACK_PAGES

In reply to Wu Fengguang
Quoted 5 lines · Wu Fengguang

+ * The maximum number of pages to writeout in a single periodic/background
+ * writeback operation. 64MB means I_SYNC may be hold for up to 1 second.
+ * This is not a big problem since we normally do kind of trylock on I_SYNC
+ * for non-data-integrity writes. Userspace tasks doing throttled writeback
+ * do not use this value.

What's your justification for using 64MB? Where are you getting 1 second from? On a fast RAID array 64MB can be written in much less than 1 second.

More generally, I assume your patches conflict with Jens' per-bdi patches?

- Ted

#12 lore raw

Re: [RFC][PATCH 5/7] writeback: use 64MB MAX_WRITEBACK_PAGES

Quoted 10 lines · Theodore Tso
Quoted 5 lines · Wu Fengguang

+ * The maximum number of pages to writeout in a single periodic/background
+ * writeback operation. 64MB means I_SYNC may be hold for up to 1 second.
+ * This is not a big problem since we normally do kind of trylock on I_SYNC
+ * for non-data-integrity writes. Userspace tasks doing throttled writeback
+ * do not use this value.

What's your justification for using 64MB? Where are you getting 1 second from? On a fast RAID array 64MB can be written in much less than 1 second.

Because this value will be used for desktop and server alike, we have to consider the worst case - for a laptop, it takes about 1 second to write 64MB. It's not accurate to say I_SYNC may be hold for up to 1 second, but the same I_SYNC will be taken, dropped because of congestion, retaken, and this loop could go on for 1 second until a large file have been written 64MB. In the mean while, in the normal case of a single kupdate thread per bdi, that file blocks writeout of other files for 1 second.

Quoted 1 line

More generally, I assume your patches conflict with Jens' per-bdi patches?

It is based on latest linux-next tree with the vm.max_writeback_mb patch reverted. The changelog only states the need to increase the size, but not mentioned why it should be made tunable. So I tend to just bump up MAX_WRITEBACK_PAGES. It seems that a large value won't hurt anyone. Or are there demand to further increase it a lot for some server configurations?

Thanks,
Fengguang

#5 lore raw

Re: [RFC][PATCH 2/7] writeback: fix queue_io() ordering

In reply to Jan Kara
Quoted 5 lines · Jan Kara
Quoted 3 lines · Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

But doesn't this patch also need your "anti-starvation" patch?

Honza, can you show me that patch?

Quoted 3 lines

Looking into the code, we put inode to b_more_io when nr_to_write drops to zero and this way we'd just start writing it again in the next round...

I'm confused. It's OK to start it in next round. Starvation can occur if we start it immediately in the next writeback_inodes() invocation. How can that happen with this patch?

Thanks,
Fengguang

Quoted 36 lines

Honza

Quoted 32 lines
CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -313,11 +313,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#18 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

In reply to Jan Kara
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

Thanks,
Fengguang

Quoted 58 lines
Quoted 55 lines
---
 mm/page-writeback.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
mm/page-writeback.c +7 -6
@@ -44,12 +44,12 @@ static long ratelimit_pages = 32;
 /*
  * When balance_dirty_pages decides that the caller needs to perform some
  * non-background writeback, this is how many pages it will attempt to write.
- * It should be somewhat larger than RATELIMIT_PAGES to ensure that reasonably
+ * It should be somewhat larger than dirtied pages to ensure that reasonably
  * large amounts of I/O are submitted.
  */
-static inline long sync_writeback_pages(void)
+static inline long sync_writeback_pages(unsigned long dirtied)
 {
-	return ratelimit_pages + ratelimit_pages / 2;
+	return dirtied + dirtied / 2;
 }
 
 /* The following parameters are exported via /proc/sys/vm */
@@ -476,7 +476,8 @@ get_dirty_limits(unsigned long *pbackgro
  * If we're over `background_thresh' then pdflush is woken to perform some
  * writeout.
  */
-static void balance_dirty_pages(struct address_space *mapping)
+static void balance_dirty_pages(struct address_space *mapping,
+				unsigned long write_chunk)
 {
 	long nr_reclaimable, bdi_nr_reclaimable;
 	long nr_writeback, bdi_nr_writeback;
@@ -484,7 +485,6 @@ static void balance_dirty_pages(struct a
 	unsigned long dirty_thresh;
 	unsigned long bdi_thresh;
 	unsigned long pages_written = 0;
-	unsigned long write_chunk = sync_writeback_pages();
 
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
 
@@ -638,9 +638,10 @@ void balance_dirty_pages_ratelimited_nr(
 	p =  &__get_cpu_var(bdp_ratelimits);
 	*p += nr_pages_dirtied;
 	if (unlikely(*p >= ratelimit)) {
+		ratelimit = sync_writeback_pages(*p);
 		*p = 0;
 		preempt_enable();
-		balance_dirty_pages(mapping);
+		balance_dirty_pages(mapping, ratelimit);
 		return;
 	}
 	preempt_enable();
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#13 lore raw

Re: [RFC][PATCH 5/7] writeback: use 64MB MAX_WRITEBACK_PAGES

In reply to Theodore Tso
Quoted 10 lines · Theodore Tso
Quoted 5 lines · Wu Fengguang

+ * The maximum number of pages to writeout in a single periodic/background
+ * writeback operation. 64MB means I_SYNC may be hold for up to 1 second.
+ * This is not a big problem since we normally do kind of trylock on I_SYNC
+ * for non-data-integrity writes. Userspace tasks doing throttled writeback
+ * do not use this value.

What's your justification for using 64MB? Where are you getting 1 second from? On a fast RAID array 64MB can be written in much less than 1 second.

Worse, on my 5mb/s usb stick writing out 64m will take forever.

#14 lore raw

Re: [RFC][PATCH 5/7] writeback: use 64MB MAX_WRITEBACK_PAGES

Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Theodore Tso
Quoted 5 lines · Wu Fengguang

+ * The maximum number of pages to writeout in a single periodic/background
+ * writeback operation. 64MB means I_SYNC may be hold for up to 1 second.
+ * This is not a big problem since we normally do kind of trylock on I_SYNC
+ * for non-data-integrity writes. Userspace tasks doing throttled writeback
+ * do not use this value.

Quoted 5 lines
Quoted 3 lines

What's your justification for using 64MB? Where are you getting 1 second from? On a fast RAID array 64MB can be written in much less than 1 second.

Worse, on my 5mb/s usb stick writing out 64m will take forever.

cp bigfile1 bigfile2 /mnt/usb/
sync

In that case the user would notice that kernel keeps writing to one file for up to 13 seconds before switching to another file.

A simple fix would look like this. It stops io continuation on one file after 1 second. It will work because when io is congested, it relies on the io continuation logic (based on last_file*) to retry the same file until MAX_WRITEBACK_PAGES is reached. The queue-able requests between congested <=> uncongested states are not very large. For slow devices, the queue-able pages between empty <=> congested states are also not very large. For example, my USB stick has nr_requests=128 and max_sectors_kb=120. It would take less than 12MB to congest this queue.

With this patch and my usb stick, the kernel may first sync 12MB for bigfile1 (which takes 1-3 seconds), then sync bigfile2 for 1 second, and then bigfile1 for 1 second, and so on.

It seems that we could now safely bump MAX_WRITEBACK_PAGES to even larger values beyond 128MB :)

Thanks,
Fengguang
---

fs/fs-writeback.c +3 -1
@@ -277,7 +277,8 @@ static void requeue_io(struct inode *ino
  */
 static void requeue_partial_io(struct writeback_control *wbc, struct inode *inode)
 {
-	if (wbc->last_file_written == 0 ||
+	if (time_before(wbc->last_file_time + HZ, jiffies) ||
+	    wbc->last_file_written == 0 ||
 	    wbc->last_file_written >= MAX_WRITEBACK_PAGES)
 		return requeue_io(inode);
 
@@ -428,6 +429,7 @@ writeback_single_inode(struct inode *ino
 
 	if (wbc->last_file != inode->i_ino) {
 		wbc->last_file = inode->i_ino;
+		wbc->last_file_time = jiffies;
 		wbc->last_file_written = nr_to_write - wbc->nr_to_write;
 	} else
 		wbc->last_file_written += nr_to_write - wbc->nr_to_write;
include/linux/writeback.h +1 -0
@@ -46,6 +46,7 @@ struct writeback_control {
 	long nr_to_write;		/* Write this many pages, and decrement
 					   this for each page written */
 	unsigned long last_file;	/* Inode number of last written file */
+	unsigned long last_file_time;	/* First sync time for last file */
 	long last_file_written;		/* Total pages written for last file */
 	long pages_skipped;		/* Pages which were not written */
 
#19 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

In reply to Wu Fengguang
Quoted 15 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Quoted 3 lines

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

-chris

#20 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 16 lines · Chris Mason
Quoted 15 lines · Wu Fengguang
Quoted 13 lines · Jan Kara
Quoted 7 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Quoted 6 lines
Quoted 3 lines

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

Btrfs is doing OK. 2MB/4MB looks like reasonable chunk sizes. The need-change part is balance_dirty_pages_ratelimited_nr(), hence this patch :)

Thanks,
Fengguang

maintainer Acked-by
#6 lore raw

Re: [RFC][PATCH 2/7] writeback: fix queue_io() ordering

In reply to Wu Fengguang
Quoted 9 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 3 lines · Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

But doesn't this patch also need your "anti-starvation" patch?

Honza, can you show me that patch?

Quoted 7 lines
Quoted 3 lines

Looking into the code, we put inode to b_more_io when nr_to_write drops to zero and this way we'd just start writing it again in the next round...

I'm confused. It's OK to start it in next round. Starvation can occur if we start it immediately in the next writeback_inodes() invocation. How can that happen with this patch?

Sorry, my fault. For kupdate, we splice the list only once s_io is empty
so that's not an issue. So the patch looks good.

Quoted 34 lines
Quoted 34 lines
Quoted 31 lines
CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -313,11 +313,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw

Re: [RFC][PATCH 2/7] writeback: fix queue_io() ordering

Quoted 14 lines · Jan Kara
Quoted 13 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 3 lines · Wu Fengguang

This was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.

But doesn't this patch also need your "anti-starvation" patch?

Honza, can you show me that patch?

Quoted 3 lines

Looking into the code, we put inode to b_more_io when nr_to_write drops to zero and this way we'd just start writing it again in the next round...

Quoted 6 lines
Quoted 3 lines

I'm confused. It's OK to start it in next round. Starvation can occur if we start it immediately in the next writeback_inodes() invocation. How can that happen with this patch?

Sorry, my fault. For kupdate, we splice the list only once s_io is empty
so that's not an issue. So the patch looks good.

Thank you :)

Regards,
Fengguang

Quoted 37 lines
Quoted 34 lines
Quoted 34 lines
Quoted 31 lines
CC Dave Chinner <[email protected]>
Cc Martin Bligh <[email protected]>
Cc Michael Rubin <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -313,11 +313,14 @@ static void move_expired_inodes(struct l
 }
 
 /*
- * Queue all expired dirty inodes for io, eldest first.
+ * Queue all expired dirty inodes for io, eldest first:
+ * (newly dirtied) => b_dirty inodes
+ *                 => b_more_io inodes
+ *                 => remaining inodes in b_io => (dequeue for sync)
  */
 static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
 {
-	list_splice_init(&wb->b_more_io, wb->b_io.prev);
+	list_splice_init(&wb->b_more_io, &wb->b_io);
 	move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
 }
 
Signature · 1 line
-- 
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#21 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

In reply to Wu Fengguang
Quoted 21 lines · Wu Fengguang
Quoted 20 lines · Chris Mason
Quoted 18 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

Quoted 6 lines
Quoted 2 lines

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

Btrfs is doing OK. 2MB/4MB looks like reasonable chunk sizes. The need-change part is balance_dirty_pages_ratelimited_nr(), hence this patch :)

I'm not getting it, it calls set_page_dirty() for each page, right? and then it calls into balance_dirty_pages_ratelimited_nr(), that sounds right. What is the problem with that?

#22 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Wu Fengguang
Quoted 22 lines · Chris Mason
Quoted 15 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Quoted 3 lines

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

Quoted 7 lines
Quoted 3 lines

Btrfs is doing OK. 2MB/4MB looks like reasonable chunk sizes. The need-change part is balance_dirty_pages_ratelimited_nr(), hence this patch :)

I'm not getting it, it calls set_page_dirty() for each page, right? and then it calls into balance_dirty_pages_ratelimited_nr(), that sounds right. What is the problem with that?

It looks like btrfs_file_write() eventually calls __set_page_dirty_buffers() which in turn won't call balance_dirty_pages*(). This is why do_wp_page() calls set_page_dirty_balance() to do balance_dirty_pages*().

So btrfs_file_write() explicitly calls balance_dirty_pages_ratelimited_nr() to get throttled.

Thanks,
Fengguang

#23 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 34 lines · Wu Fengguang
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Wu Fengguang
Quoted 22 lines · Chris Mason
Quoted 15 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Quoted 3 lines

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

Quoted 7 lines
Quoted 3 lines

Btrfs is doing OK. 2MB/4MB looks like reasonable chunk sizes. The need-change part is balance_dirty_pages_ratelimited_nr(), hence this patch :)

I'm not getting it, it calls set_page_dirty() for each page, right? and then it calls into balance_dirty_pages_ratelimited_nr(), that sounds right. What is the problem with that?

Quoted 7 lines

It looks like btrfs_file_write() eventually calls __set_page_dirty_buffers() which in turn won't call balance_dirty_pages*(). This is why do_wp_page() calls set_page_dirty_balance() to do balance_dirty_pages*().

So btrfs_file_write() explicitly calls balance_dirty_pages_ratelimited_nr() to get throttled.

Right, so what is wrong with than, and how does this patch fix that?

[ the only thing you have to be careful with is that you don't excessively grow the error bound on the dirty limit ]

#24 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 40 lines · Peter Zijlstra
Quoted 39 lines · Wu Fengguang
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Wu Fengguang
Quoted 22 lines · Chris Mason
Quoted 15 lines · Wu Fengguang
Quoted 5 lines · Jan Kara
Quoted 4 lines · Wu Fengguang

Some filesystem may choose to write much more than ratelimit_pages before calling balance_dirty_pages_ratelimited_nr(). So it is safer to determine number to write based on real number of dirtied pages.

Quoted 8 lines
Quoted 3 lines

The increased write_chunk may make the dirtier more bumpy. This is filesystem writers' duty not to dirty too much at a time without checking the ratelimit.

I don't get this. balance_dirty_pages_ratelimited_nr() is called when we
dirty the page, not when we write it out. So a problem would only happen if filesystem dirties pages by set_page_dirty() and won't call balance_dirty_pages_ratelimited_nr(). But e.g. generic_perform_write() and do_wp_page() takes care of that. So where's the problem?

Quoted 3 lines

It seems that btrfs_file_write() is writing in chunks of up to 1024-pages (1024 is the computed nrptrs value in a 32bit kernel). And it calls balance_dirty_pages_ratelimited_nr() each time it dirtied such a chunk.

I can easily change this to call more often, but we do always call balance_dirty_pages to reflect how much ram we've really sent down.

Quoted 7 lines
Quoted 3 lines

Btrfs is doing OK. 2MB/4MB looks like reasonable chunk sizes. The need-change part is balance_dirty_pages_ratelimited_nr(), hence this patch :)

I'm not getting it, it calls set_page_dirty() for each page, right? and then it calls into balance_dirty_pages_ratelimited_nr(), that sounds right. What is the problem with that?

It looks like btrfs_file_write() eventually calls __set_page_dirty_buffers() which in turn won't call balance_dirty_pages*(). This is why do_wp_page() calls set_page_dirty_balance() to do balance_dirty_pages*().

Quoted 7 lines
Quoted 2 lines

So btrfs_file_write() explicitly calls balance_dirty_pages_ratelimited_nr() to get throttled.

Right, so what is wrong with than, and how does this patch fix that?

[ the only thing you have to be careful with is that you don't excessively grow the error bound on the dirty limit ]

Then we could form a loop:

btrfs_file_write(): dirty 1024 pages
balance_dirty_pages(): write up to 12 pages (= ratelimit_pages * 1.5)

in which the writeback rate cannot keep up with dirty rate, and the dirty pages go all the way beyond dirty_thresh.

Sorry for writing such a vague changelog!

Thanks,
Fengguang

#25 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 8 lines · Wu Fengguang
Quoted 7 lines
Quoted 2 lines

So btrfs_file_write() explicitly calls balance_dirty_pages_ratelimited_nr() to get throttled.

Right, so what is wrong with than, and how does this patch fix that?

[ the only thing you have to be careful with is that you don't excessively grow the error bound on the dirty limit ]

Quoted 7 lines

Then we could form a loop:

btrfs_file_write(): dirty 1024 pages
balance_dirty_pages(): write up to 12 pages (= ratelimit_pages * 1.5)

in which the writeback rate cannot keep up with dirty rate, and the dirty pages go all the way beyond dirty_thresh.

Ah, ok so this is to keep the error bound on the dirty limit bounded, because we can break out of balance_dirty_pages() early, the /* We've done our duty */ break.

Which unbalances the duty vs the dirty ratio.

I figure that with the task dirty limit stuff we could maybe try to get rid of this break.. worth a try.

Quoted 1 line

Sorry for writing such a vague changelog!

np, as long as we get there :-)

Change makes sense now, thanks!

#26 lore raw

Re: [RFC][PATCH 7/7] writeback: balance_dirty_pages() shall write more than dirtied pages

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Wu Fengguang
Quoted 7 lines
Quoted 2 lines

So btrfs_file_write() explicitly calls balance_dirty_pages_ratelimited_nr() to get throttled.

Right, so what is wrong with than, and how does this patch fix that?

[ the only thing you have to be careful with is that you don't excessively grow the error bound on the dirty limit ]

Then we could form a loop:

btrfs_file_write(): dirty 1024 pages
balance_dirty_pages(): write up to 12 pages (= ratelimit_pages * 1.5)

Quoted 8 lines
Quoted 2 lines

in which the writeback rate cannot keep up with dirty rate, and the dirty pages go all the way beyond dirty_thresh.

Ah, ok so this is to keep the error bound on the dirty limit bounded, because we can break out of balance_dirty_pages() early, the /* We've done our duty */ break.

Which unbalances the duty vs the dirty ratio.

Right!

Quoted 2 lines

I figure that with the task dirty limit stuff we could maybe try to get rid of this break.. worth a try.

Be careful. Without that break, the time a task get throttled in a single trip may go out of control. For example, task B get blocked for 1000 seconds because there is a task A keep dirtying pages, in the mean time task A's dirty thresh going down slowly, but still larger than B's.

Quoted 5 lines
Quoted 1 line

Sorry for writing such a vague changelog!

np, as long as we get there :-)

Change makes sense now, thanks!

May I add you ack?

Thanks,
Fengguang