Writeback fix for concurrent large and small file writes.

Fixing a bug where writing to large files while concurrently writing to smaller ones creates a situation where writeback cannot keep up with the traffic and memory baloons until the we hit the threshold watermark. This can result in surprising latency spikes when syncing. This latency can take minutes on large memory systems. Upon request I can provide a test to reproduce this situation.

The only concern I have is that this makes the wb_kupdate slightly more agressive. I am not sure it is enough to cause any problems. I think there is enough checks to throttle the background activity.

Feng also the one line change that you recommended here
http://marc.info/?l=linux-kernel&m=119629655402153&w=2 had no effect.

---

Index: 2624rc3_feng/fs/fs-writeback.c
===================================================================
@@ -408,8 +408,7 @@ sync_sb_inodes(struct super_block *sb, s
 {
 	const unsigned long start = jiffies;	/* livelock avoidance */
 
-	if (!wbc->for_kupdate || list_empty(&sb->s_io))
-		queue_io(sb, wbc->older_than_this);
+	queue_io(sb, wbc->older_than_this);
 
 	while (!list_empty(&sb->s_io)) {
 		struct inode *inode = list_entry(sb->s_io.prev,

Index: 2624rc3_feng/mm/page-writeback.c
===================================================================
@@ -638,7 +638,7 @@ static void wb_kupdate(unsigned long arg
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_inodes(&wbc);
 		if (wbc.nr_to_write > 0) {
-			if (wbc.encountered_congestion || wbc.more_io)
+			if (wbc.encountered_congestion)
 				congestion_wait(WRITE, HZ/10);
 			else
 				break;	/* All the old data is written */

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The part I miss here is the rationale on _how_ you solve the problem.

The patch itself is simple enough, but I've been staring at this code for a while now, and I'm just not getting it.

Apologies for the lack of rationale. I have been staring at this code for awhile also and it makes my head hurt. I have a patch coming (hopefully today) that proposes using one data structure with a more consistent priority scheme for 2.6.25. To me it's simpler, but I am biased.

The problem we encounter when we append to a large file at a fast rate while also writing to smaller files is that the wb_kupdate thread does not keep up with disk traffic. In this workload often the inodes end up at fs/fs-writeback.c:287 after do_writepages, since do_writepages did not write all the pages. This can be due to congestion but I think there are other causes also since I have observed so.

The first issue is that the inode is put on the s_more_io queue. This ensures that more_io is set at the end of sync_sb_inodes. The result from that is the wb_kupdate routine will perform a sleep at mm/page-writeback.c:642. This slows us down enough that the wb_kupdate cannot keep up with traffic.

The other issue is that the inode that has been placed on the s_more_io queue cannot be processed by sync_sb_inodes until the entire s_io list is empty. With lots of small files that are not being dirtied as quickly as the one large inode on the s_more_io queue the inode with the most pages being dirtied is not given attention and wb_kupdate cannot keep up again.

mrubin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Michael,

// sorry for the delay...

                sync 4MB of A; sync B,C; congestion_wait();
                sync 4MB of A; sync D,E; congestion_wait();
                sync 4MB of A; sync F,G; congestion_wait();
                ...
- new behavior:
                sync 4MB of A;
                sync 4MB of A;
                sync 4MB of A;
                sync 4MB of A;
                sync 4MB of A;
                ...            // repeat until A is clean
                sync B,C,D,E,F,G;