some writeback experiments
Subsystems (22)
cover some writeback experiments
Hi all,
Here is a collection of writeback patches on
- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes
Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..
The lumpy pageout and nr_segments support is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.
I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!
The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..
NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:
# vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples)
nr_writeback nr_dirty nr_unstable
11227 41463 38044
11227 41463 38044
11227 41463 38044
11227 41463 38044
11045 53987 6490
11033 53120 8145
11195 52143 10886
11211 52144 10913
11211 52144 10913
11211 52144 10913btrfs seems to maintain a private pool of writeback pages, which can go out of control:
nr_writeback nr_dirty
261075 132
252891 195
244795 187
236851 187
228830 187
221040 218
212674 237
204981 237XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.
nr_writeback nr_dirty
80781 0
37117 37703
37117 43933
81044 6
81050 0
43943 10199
43930 36355
43930 36355
80293 0
80285 0
80285 0Thanks,
Fengguang
PATCH 4/45 writeback: remove unused nonblocking and congestion checks
- no one is calling wb_writeback and write_cache_pages with wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion wait
So remove the congestion checks as suggested by Chris.
--- drivers/staging/pohmelfs/inode.c | 9 --------- fs/afs/write.c | 16 +--------------- fs/cifs/file.c | 10 ---------- fs/fs-writeback.c | 8 -------- fs/gfs2/aops.c | 10 ---------- fs/xfs/linux-2.6/xfs_aops.c | 6 +----- mm/page-writeback.c | 12 ------------ 7 files changed, 2 insertions(+), 69 deletions(-)
fs/fs-writeback.c +0 -8
@@ -660,14 +660,6 @@ static void writeback_inodes_wb(struct bcontinue ; } -if (wbc->nonblocking &&bdi_write_congested (wb->bdi)) { - wbc->encountered_congestion =1 ; -if (!is_blkdev_sb) -break ;/* Skip a congested fs */ -requeue_io (inode); -continue ;/* Skip a congested blockdev */ - } -/* * Was this inode dirtied after sync_sb_inodes was called? * This keeps sync from extra jobs and livelock.
mm/page-writeback.c +0 -12
@@ -787,7 +787,6 @@ int write_cache_pages(struct address_spa struct writeback_control *wbc, writepage_t writepage, void *data) { - struct backing_dev_info *bdi = mapping->backing_dev_info; int ret = 0;int done =0 ;struct pagevec pvec;
@@ -800,11 +799,6 @@ int write_cache_pages(struct address_spaint range_whole =0 ;long nr_to_write = wbc->nr_to_write; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -pagevec_init (&pvec,0 );if (wbc->range_cyclic) { writeback_index = mapping->writeback_index;/* prev offset */
@@ -923,12 +917,6 @@ continue_unlock:break ; } } - -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - done =1 ; -break ; - } }pagevec_release (&pvec);cond_resched ();
drivers/staging/pohmelfs/inode.c +0 -9
@@ -152,11 +152,6 @@ static int pohmelfs_writepages(struct adint scanned =0 ;int range_whole =0 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -if (wbc->range_cyclic) { index = mapping->writeback_index;/* Start from prev offset */ end =-1 ;
@@ -248,10 +243,6 @@ retry:if (wbc->nr_to_write <=0 ) done =1 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - done =1 ; - }continue ; out_continue:
fs/afs/write.c +1 -15
@@ -455,8 +455,6 @@ int afs_writepage(struct page *page, str } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) - wbc->encountered_congestion =1 ;_leave (" = 0" );return 0 ;
@@ -529,11 +527,6 @@ static int afs_writepages_region(struct wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -break ; - } -cond_resched (); }while (index < end && wbc->nr_to_write >0 );
@@ -554,18 +547,11 @@ int afs_writepages(struct address_space_enter ("" ); -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -_leave (" = 0 [congest]" ); -return 0 ; - } -if (wbc->range_cyclic) { start = mapping->writeback_index; end =-1 ; ret =afs_writepages_region (mapping, wbc, start, end, &next); -if (start >0 && wbc->nr_to_write >0 && ret ==0 && - !(wbc->nonblocking && wbc->encountered_congestion)) +if (start >0 && wbc->nr_to_write >0 && ret ==0 ) ret =afs_writepages_region (mapping, wbc,0 , start, &next); mapping->writeback_index = next;
fs/cifs/file.c +0 -10
@@ -1379,16 +1379,6 @@ static int cifs_writepages(struct addresreturn generic_writepages (mapping, wbc); -/* -* BB: Is this meaningful for a non-block-device file system? -* If it is, we should test it again after we do I/O -*/ -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -kfree (iov); -return 0 ; - } - xid =GetXid ();pagevec_init (&pvec,0 );
fs/gfs2/aops.c +0 -10
@@ -313,11 +313,6 @@ static int gfs2_write_jdata_pagevec(struif (ret || (--(wbc->nr_to_write) <=0 )) ret =1 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - ret =1 ; - } - }gfs2_trans_end (sdp);return ret;
@@ -348,11 +343,6 @@ static int gfs2_write_cache_jdata(structint scanned =0 ;int range_whole =0 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -pagevec_init (&pvec,0 );if (wbc->range_cyclic) { index = mapping->writeback_index;/* Start from prev offset */
fs/xfs/linux-2.6/xfs_aops.c +1 -5
@@ -890,12 +890,8 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; +if (wbc->nr_to_write <=0 ) done =1 ; - }else if (wbc->nr_to_write <=0 ) { - done =1 ; - } }xfs_start_page_writeback (page, !page_dirty, count); }
PATCH 5/45 writeback: remove the always false bdi_cap_writeback_dirty() test
In reply to Wu FengguangThis is dead code because no bdi flush thread will be started for !bdi_cap_writeback_dirty bdis.
--- fs/fs-writeback.c | 18 ------------------ 1 file changed, 18 deletions(-)
fs/fs-writeback.c +0 -18
@@ -617,7 +617,6 @@ static void writeback_inodes_wb(struct b struct writeback_control *wbc) {struct super_block *sb = wbc->sb, *pin_sb =NULL ; -const int is_blkdev_sb =sb_is_blkdev_sb (sb);const unsigned long start = jiffies;/* livelock avoidance */ spin_lock (&inode_lock);
@@ -638,23 +637,6 @@ static void writeback_inodes_wb(struct bcontinue ; } -if (!bdi_cap_writeback_dirty (wb->bdi)) { -redirty_tail (inode); -if (is_blkdev_sb) { -/* -* Dirty memory-backed blockdev: the ramdisk -* driver does this. Skip just this inode -*/ -continue ; - } -/* -* Dirty memory-backed inode against a filesystem other -* than the kernel-internal bdev filesystem. Skip the -* entire superblock. -*/ -break ; - } -if (inode->i_state & (I_NEW |I_WILL_FREE )) {requeue_io (inode);continue ;
PATCH 7/45 writeback: dont redirty tail an inode with dirty pages
In reply to Wu FengguangThis avoids delaying writeback for an expired (XFS) inode with lots of dirty pages, but no active dirtier at the moment.
--- fs/fs-writeback.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-)
fs/fs-writeback.c +7 -13
@@ -479,18 +479,7 @@ writeback_single_inode(struct inode *inospin_lock (&inode_lock); inode->i_state &= ~I_SYNC ;if (!(inode->i_state & (I_FREEING |I_CLEAR ))) { -if ((inode->i_state &I_DIRTY_PAGES ) && wbc->for_kupdate) { -/* -* More pages get dirtied by a fast dirtier. -*/ - goto select_queue; - }else if (inode->i_state &I_DIRTY ) { -/* -* At least XFS will redirty the inode during the -* writeback (delalloc) and on io completion (isize). -*/ -redirty_tail (inode); - }else if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) { +if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/* * We didn't write back all the pages. nfs_writepages() * sometimes bales out without doing anything. Redirty
@@ -512,7 +501,6 @@ writeback_single_inode(struct inode *ino* soon as the queue becomes uncongested. */ inode->i_state |=I_DIRTY_PAGES ; -select_queue:if (wbc->nr_to_write <=0 ) {/* * slice used up: queue for next turn
@@ -535,6 +523,12 @@ select_queue: inode->i_state |=I_DIRTY_PAGES ;redirty_tail (inode); } + }else if (inode->i_state &I_DIRTY ) { +/* +* At least XFS will redirty the inode during the +* writeback (delalloc) and on io completion (isize). +*/ +redirty_tail (inode); }else if (atomic_read (&inode->i_count)) {/* * The inode is clean, inuse
PATCH 9/45 writeback: quit on wrap for .range_cyclic (pohmelfs)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- drivers/staging/pohmelfs/inode.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-)
drivers/staging/pohmelfs/inode.c +8 -17
@@ -149,7 +149,6 @@ static int pohmelfs_writepages(struct adint nr_pages;pgoff_t index;pgoff_t end;/* Inclusive */ -int scanned =0 ;int range_whole =0 ;if (wbc->range_cyclic) {
@@ -160,17 +159,18 @@ static int pohmelfs_writepages(struct ad end = wbc->range_end >>PAGE_CACHE_SHIFT ;if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) range_whole =1 ; - scanned =1 ; } -retry: +while (!done && (index <= end)) {unsigned int i =min (end - index, (pgoff_t )psb->trans_max_pages);int path_len;struct netfs_trans *trans; err =pohmelfs_inode_has_dirty_pages (mapping, index); -if (!err) +if (!err) { + index =0 ;break ; + } err =pohmelfs_path_length (pi);if (err <0 )
@@ -197,15 +197,16 @@ retry:dprintk ("%s: t: %p, nr_pages: %u, end: %lu, index: %lu, max: %u.\n" , __func__, trans, nr_pages, end, index, trans->page_num); -if (!nr_pages) +if (!nr_pages) { + index =0 ; goto err_out_reset; + } err =pohmelfs_write_inode_create (inode, trans);if (err) goto err_out_reset; err =0 ; - scanned =1 ;for (i =0 ; i < trans->page_num; i++) {struct page *page = trans->pages[i];
@@ -215,7 +216,7 @@ retry:if (unlikely (page->mapping != mapping)) goto out_continue; -if (!wbc->range_cyclic && page->index > end) { +if (page->index > end) { done =1 ; goto out_continue; }
@@ -263,16 +264,6 @@ err_out_reset:break ; } -if (!scanned && !done) { -/* -* We hit the last page and there is more work to be done: wrap -* back to the start of the file -*/ - scanned =1 ; - index =0 ; - goto retry; - } -if (wbc->range_cyclic || (range_whole && wbc->nr_to_write >0 )) mapping->writeback_index = index;
PATCH 11/45 writeback: quit on wrap for .range_cyclic (cifs)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- fs/cifs/file.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
fs/cifs/file.c +4 -14
@@ -1337,7 +1337,6 @@ static int cifs_partialpagewrite(structstatic int cifs_writepages (struct address_space *mapping,struct writeback_control *wbc) { -struct backing_dev_info *bdi = mapping->backing_dev_info;unsigned int bytes_to_write;unsigned int bytes_written;struct cifs_sb_info *cifs_sb;
@@ -1349,14 +1348,13 @@ static int cifs_writepages(struct addresint len;int n_iov =0 ;pgoff_t next; -int nr_pages; +int nr_pages =1 ;__u64 offset =0 ;struct cifsFileInfo *open_file;struct cifsInodeInfo *cifsi =CIFS_I (mapping->host);struct page *page;struct pagevec pvec;int rc =0 ; -int scanned =0 ;int xid, long_op; cifs_sb =CIFS_SB (mapping->host->i_sb);
@@ -1390,9 +1388,8 @@ static int cifs_writepages(struct addres end = wbc->range_end >>PAGE_CACHE_SHIFT ;if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) range_whole =1 ; - scanned =1 ; } -retry: + while (!done && (index <= end) && (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, PAGECACHE_TAG_DIRTY,
@@ -1425,7 +1422,7 @@ retry:break ; } -if (!wbc->range_cyclic && page->index > end) { +if (page->index > end) { done =1 ;unlock_page (page);break ;
@@ -1537,15 +1534,8 @@ retry:pagevec_release (&pvec); } -if (!scanned && !done) { -/* -* We hit the last page and there is more work to be done: wrap -* back to the start of the file -*/ - scanned =1 ; +if (!nr_pages) index =0 ; - goto retry; - }if (wbc->range_cyclic || (range_whole && wbc->nr_to_write >0 )) mapping->writeback_index = index;
PATCH 12/45 writeback: quit on wrap for .range_cyclic (ext4)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- fs/ext4/inode.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
fs/ext4/inode.c +4 -14
@@ -2805,7 +2805,7 @@ static int ext4_da_writepages(struct addint pages_written =0 ;long pages_skipped;unsigned int max_pages; -int range_cyclic, cycled =1 , io_done =0 ; +int range_cyclic, io_done =0 ;int needed_blocks, ret =0 ;long desired_nr_to_write, nr_to_writebump =0 ;loff_t range_start = wbc->range_start;
@@ -2840,8 +2840,6 @@ static int ext4_da_writepages(struct add range_cyclic = wbc->range_cyclic;if (wbc->range_cyclic) { index = mapping->writeback_index; -if (index) - cycled =0 ; wbc->range_start = index <<PAGE_CACHE_SHIFT ; wbc->range_end =LLONG_MAX ; wbc->range_cyclic =0 ;
@@ -2889,7 +2887,6 @@ static int ext4_da_writepages(struct add wbc->no_nrwrite_index_update =1 ; pages_skipped = wbc->pages_skipped; -retry:while (!ret && wbc->nr_to_write >0 ) {/*
@@ -2963,20 +2960,13 @@ retry: wbc->pages_skipped = pages_skipped; ret =0 ; io_done =1 ; - }else if (wbc->nr_to_write) + }else if (wbc->nr_to_write >0 ) {/* * There is no more writeout needed -* or we requested for a noblocking writeout -* and we found the device congested */ + index =0 ;break ; - } -if (!io_done && !cycled) { - cycled =1 ; - index =0 ; - wbc->range_start = index <<PAGE_CACHE_SHIFT ; - wbc->range_end = mapping->writeback_index -1 ; - goto retry; + } }if (pages_skipped != wbc->pages_skipped) ext4_msg(inode->i_sb, KERN_CRIT,
PATCH 13/45 writeback: quit on wrap for .range_cyclic (gfs2)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- fs/gfs2/aops.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-)
fs/gfs2/aops.c +2 -14
@@ -287,7 +287,7 @@ static int gfs2_write_jdata_pagevec(strucontinue ; } -if (!wbc->range_cyclic && page->index > end) { +if (page->index > end) { ret =1 ;unlock_page (page);continue ;
@@ -340,7 +340,6 @@ static int gfs2_write_cache_jdata(structint nr_pages;pgoff_t index;pgoff_t end; -int scanned =0 ;int range_whole =0 ;pagevec_init (&pvec,0 );
@@ -352,15 +351,12 @@ static int gfs2_write_cache_jdata(struct end = wbc->range_end >>PAGE_CACHE_SHIFT ;if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) range_whole =1 ; - scanned =1 ; } -retry:while (!done && (index <= end) && (nr_pages =pagevec_lookup_tag (&pvec, mapping, &index,PAGECACHE_TAG_DIRTY ,min (end - index, (pgoff_t )PAGEVEC_SIZE -1 ) +1 ))) { - scanned =1 ; ret =gfs2_write_jdata_pagevec (mapping, wbc, &pvec, nr_pages, end);if (ret) done =1 ;
@@ -371,16 +367,8 @@ retry:cond_resched (); } -if (!scanned && !done) { -/* -* We hit the last page and there is more work to be done: wrap -* back to the start of the file -*/ - scanned =1 ; +if (!nr_pages) index =0 ; - goto retry; - } -if (wbc->range_cyclic || (range_whole && wbc->nr_to_write >0 )) mapping->writeback_index = index;return ret;
PATCH 17/45 writeback: only allow two background writeback works
In reply to Wu Fengguangbalance_dirty_pages() need a reliable way to ensure some background work is scheduled for running. We cannot simply run bdi_start_writeback() because that would queue up huge number of works (which takes memory and time).
--- fs/fs-writeback.c | 14 ++------------ include/linux/backing-dev.h | 26 +++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 13 deletions(-)
fs/fs-writeback.c +2 -12
@@ -85,18 +85,6 @@ static inline void bdi_work_init(structint sysctl_dirty_debug__read_mostly ; -/** -* writeback_in_progress - determine whether there is writeback in progress -* @bdi: the device's backing_dev_info structure. -* -* Determine whether there is writeback waiting to be handled against a -* backing device. -*/ -int writeback_in_progress (struct backing_dev_info *bdi) -{ -return !list_empty (&bdi->work_list); -} -static void bdi_work_clear (struct bdi_work *work) {clear_bit (WS_USED_B , &work->state);
@@ -147,6 +135,8 @@ static void wb_clear_pending(struct bdi_spin_lock (&bdi->wb_lock);list_del_rcu (&work->list); +if (work->args.for_background) +clear_bit (WB_FLAG_BACKGROUND_WORK , &bdi->wb_mask);spin_unlock (&bdi->wb_lock);wb_work_complete (work);
include/linux/backing-dev.h +25 -1
@@ -94,6 +94,11 @@ struct backing_dev_info {#endif }; +/* +* background work queued, set to avoid redundant background works +*/ +#define WB_FLAG_BACKGROUND_WORK 30 +int bdi_init (struct backing_dev_info *bdi);void bdi_destroy (struct backing_dev_info *bdi);
@@ -248,7 +253,26 @@ int bdi_set_max_ratio(struct backing_devextern struct backing_dev_info default_backing_dev_info;void default_unplug_io_fn (struct backing_dev_info *bdi,struct page *page); -int writeback_in_progress (struct backing_dev_info *bdi); +/** +* writeback_in_progress - determine whether there is writeback in progress +* @bdi: the device's backing_dev_info structure. +* +* Determine whether there is writeback waiting to be handled against a +* backing device. +*/ +static inline int writeback_in_progress (struct backing_dev_info *bdi) +{ +return !list_empty (&bdi->work_list); +} + +/* +* Helper to limit # of background writeback works in circulation to 2. +* (one running and another queued) +*/ +static inline int can_submit_background_writeback (struct backing_dev_info *bdi) +{ +return !test_and_set_bit (WB_FLAG_BACKGROUND_WORK , &bdi->wb_mask); +}static inline int bdi_congested (struct backing_dev_info *bdi,int bdi_bits) {
PATCH 18/45 writeback: introduce wait queue for balance_dirty_pages()
In reply to Wu FengguangAs proposed by Chris, Dave and Jan, let balance_dirty_pages() wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the above foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-file nr_to_write
The write_chunk used by balance_dirty_pages() should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait inside balance_dirty_pages() shall be small enough. When it starts its own writeback, it must specify a small nr_to_write. The throttle wait queue removes this dependancy by the way.
--- fs/fs-writeback.c | 71 ++++++++++++++++++++++++++++++++++ include/linux/backing-dev.h | 15 +++++++ mm/backing-dev.c | 4 + mm/page-writeback.c | 53 ++++++------------------- 4 files changed, 103 insertions(+), 40 deletions(-)
mm/page-writeback.c +13 -40
@@ -218,6 +218,15 @@ static inline void __bdi_writeout_inc(st {__prop_inc_percpu_max (&vm_completions, &bdi->completions, bdi->max_prop_frac); + +/* +* The DIRTY_THROTTLE_PAGES_STOP test is an optional optimization, so +* it's OK to be racy. We set DIRTY_THROTTLE_PAGES_STOP*2 in other +* places to reduce the race possibility. +*/ +if (atomic_read (&bdi->throttle_pages) <DIRTY_THROTTLE_PAGES_STOP && +atomic_dec_and_test (&bdi->throttle_pages)) +bdi_writeback_wakeup (bdi); }void bdi_writeout_inc (struct backing_dev_info *bdi)
@@ -458,20 +467,10 @@ static void balance_dirty_pages(struct aunsigned long background_thresh;unsigned long dirty_thresh;unsigned long bdi_thresh; -unsigned long pages_written =0 ; -unsigned long pause =1 ;int dirty_exceeded;struct backing_dev_info *bdi = mapping->backing_dev_info;for (;;) { -struct writeback_control wbc = { - .bdi = bdi, - .sync_mode =WB_SYNC_NONE , - .older_than_this =NULL , - .nr_to_write = write_chunk, - .range_cyclic =1 , - }; - nr_reclaimable =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ); nr_writeback = global_page_state(NR_WRITEBACK) +
@@ -518,39 +517,13 @@ static void balance_dirty_pages(struct aif (!bdi->dirty_exceeded) bdi->dirty_exceeded =1 ; -/* Note: nr_reclaimable denotes nr_dirty + nr_unstable. -* Unstable writes are a feature of certain networked -* filesystems (i.e. NFS) in which data may have been -* written to the server's write cache, but has not yet -* been flushed to permanent storage. -* Only move pages to writeback if this bdi is over its -* threshold otherwise wait until the disk writes catch -* up. -*/ -if (bdi_nr_reclaimable > bdi_thresh) { -writeback_inodes_wbc (&wbc); - pages_written += write_chunk - wbc.nr_to_write; -/* don't wait if we've done enough */ -if (pages_written >= write_chunk) -break ; - } -schedule_timeout_interruptible (pause); - -/* -* Increase the delay for each loop, up to our previous -* default of taking a 100ms nap. -*/ - pause <<=1 ; -if (pause >HZ /10 ) - pause =HZ /10 ; +bdi_writeback_wait (bdi, write_chunk); +break ; }if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ; -if (writeback_in_progress (bdi)) -return ; -/* * In laptop mode, we wait until hitting the higher threshold before * starting background writeout, and then write out all the way down
@@ -559,8 +532,8 @@ static void balance_dirty_pages(struct a* In normal mode, we start background writeout at the lower * background_thresh, to keep the amount of dirty memory low. */ -if ((laptop_mode && pages_written) || - (!laptop_mode && (nr_reclaimable > background_thresh))) +if (!laptop_mode && (nr_reclaimable > background_thresh) && +can_submit_background_writeback (bdi))bdi_start_writeback (bdi,NULL ,0 ); }
include/linux/backing-dev.h +15 -0
@@ -86,6 +86,13 @@ struct backing_dev_info {struct list_head work_list; +/* +* dirtier process throttling +*/ +spinlock_t throttle_lock; +struct list_head throttle_list;/* nr to sync for each task */ +atomic_t throttle_pages;/* nr to sync for head task */ +struct device *dev;#ifdef CONFIG_DEBUG_FS
@@ -99,6 +106,12 @@ struct backing_dev_info {*/ #define WB_FLAG_BACKGROUND_WORK 30 +/* +* when no task is throttled, set throttle_pages to larger than this, +* to avoid unnecessary atomic decreases. +*/ +#define DIRTY_THROTTLE_PAGES_STOP (1 << 22) +int bdi_init (struct backing_dev_info *bdi);void bdi_destroy (struct backing_dev_info *bdi);
@@ -110,6 +123,8 @@ void bdi_start_writeback(struct backing_ long nr_pages);int bdi_writeback_task (struct bdi_writeback *wb);int bdi_has_dirty_io (struct backing_dev_info *bdi); +int bdi_writeback_wakeup (struct backing_dev_info *bdi); +void bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages);extern spinlock_t bdi_lock;extern struct list_head bdi_list;
fs/fs-writeback.c +71 -0
89 lines not on this page · raw message
mm/backing-dev.c +4 -0
@@ -646,6 +646,10 @@ int bdi_init(struct backing_dev_info *bd bdi->wb_mask =1 ; bdi->wb_cnt =1 ; +spin_lock_init (&bdi->throttle_lock); +INIT_LIST_HEAD (&bdi->throttle_list); +atomic_set (&bdi->throttle_pages,DIRTY_THROTTLE_PAGES_STOP *2 ); +for (i =0 ; i <NR_BDI_STAT_ITEMS ; i++) { err =percpu_counter_init (&bdi->bdi_stat[i],0 );if (err)
PATCH 19/45 writeback: remove the loop in balance_dirty_pages()
In reply to Wu FengguangThe loop is no longer necessary. Remove it without behavior change.
--- mm/page-writeback.c | 81 +++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 43 deletions(-)
mm/page-writeback.c +38 -43
@@ -470,60 +470,55 @@ static void balance_dirty_pages(struct aint dirty_exceeded;struct backing_dev_info *bdi = mapping->backing_dev_info; -for (;;) { - nr_reclaimable =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_UNSTABLE_NFS ); - nr_writeback =global_page_state (NR_WRITEBACK ) + -global_page_state (NR_WRITEBACK_TEMP ); + nr_reclaimable =global_page_state (NR_FILE_DIRTY ) + +global_page_state (NR_UNSTABLE_NFS ); + nr_writeback =global_page_state (NR_WRITEBACK ) + +global_page_state (NR_WRITEBACK_TEMP ); -global_dirty_thresh (&background_thresh, &dirty_thresh); - -/* -* Throttle it only when the background writeback cannot -* catch-up. This avoids (excessively) small writeouts -* when the bdi limits are ramping up. -*/ -if (nr_reclaimable + nr_writeback < - (background_thresh + dirty_thresh) /2 ) -break ; +global_dirty_thresh (&background_thresh, &dirty_thresh); - bdi_thresh =bdi_dirty_thresh (bdi, dirty_thresh); +/* +* Throttle it only when the background writeback cannot +* catch-up. This skips the ramp up phase of bdi limits. +*/ +if (nr_reclaimable + nr_writeback < + (background_thresh + dirty_thresh) /2 ) + goto out; -/* -* In order to avoid the stacked BDI deadlock we need -* to ensure we accurately count the 'dirty' pages when -* the threshold is low. -* -* Otherwise it would be possible to get thresh+n pages -* reported dirty, even though there are thresh-m pages -* actually dirty; with m+n sitting in the percpu -* deltas. -*/ -if (bdi_thresh <2 *bdi_stat_error (bdi)) { - bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); - }else { - bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); - } + bdi_thresh =bdi_dirty_thresh (bdi, dirty_thresh); +/* +* In order to avoid the stacked BDI deadlock we need +* to ensure we accurately count the 'dirty' pages when +* the threshold is low. +* +* Otherwise it would be possible to get thresh+n pages +* reported dirty, even though there are thresh-m pages +* actually dirty; with m+n sitting in the percpu +* deltas. +*/ +if (bdi_thresh >=2 *bdi_stat_error (bdi)) { + bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); + bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); + }else { + bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); + bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); + } - dirty_exceeded = - (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) + dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) || (nr_reclaimable + nr_writeback >= dirty_thresh); -if (!dirty_exceeded) -break ; +if (!dirty_exceeded) + goto out; -if (!bdi->dirty_exceeded) - bdi->dirty_exceeded =1 ; +if (!bdi->dirty_exceeded) + bdi->dirty_exceeded =1 ; -bdi_writeback_wait (bdi, write_chunk); -break ; - } +bdi_writeback_wait (bdi, write_chunk); -if (!dirty_exceeded && bdi->dirty_exceeded) +if (bdi->dirty_exceeded) bdi->dirty_exceeded =0 ; +out:/* * In laptop mode, we wait until hitting the higher threshold before * starting background writeout, and then write out all the way down
PATCH 20/45 NFS: introduce writeback wait queue
In reply to Wu FengguangThe generic writeback routines are departing from congestion_wait() in preferance of get_request_wait(), aka. waiting on the block queues.
Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.
In perticular, balance_dirty_pages() will exit after it pushes write_chunk pages into the PG_writeback page pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number of PG_writeback pages if it is not explicitly limited.
---
The wait time and network throughput varies a lot! this is a major problem. This means nfs_end_page_writeback() is not called smoothly over time, even when there are plenty of PG_writeback pages on the client side.
[ 397.828509] write_bandwidth: comm=nfsiod pages=192 time=16ms [ 397.850976] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 403.065244] write_bandwidth: comm=nfsiod pages=192 time=5212ms [ 403.549134] write_bandwidth: comm=nfsiod pages=1536 time=144ms [ 403.570717] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 403.595749] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 403.622171] write_bandwidth: comm=nfsiod pages=192 time=24ms [ 403.651779] write_bandwidth: comm=nfsiod pages=192 time=28ms [ 403.680543] write_bandwidth: comm=nfsiod pages=192 time=24ms [ 403.712572] write_bandwidth: comm=nfsiod pages=192 time=28ms [ 403.751552] write_bandwidth: comm=nfsiod pages=192 time=36ms [ 403.785979] write_bandwidth: comm=nfsiod pages=192 time=28ms [ 403.823995] write_bandwidth: comm=nfsiod pages=192 time=36ms [ 403.858970] write_bandwidth: comm=nfsiod pages=192 time=32ms [ 403.880786] write_bandwidth: comm=nfsiod pages=192 time=16ms [ 403.902732] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 403.925925] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 403.952044] write_bandwidth: comm=nfsiod pages=258 time=24ms [ 403.974006] write_bandwidth: comm=nfsiod pages=192 time=16ms [ 403.995989] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 405.031049] write_bandwidth: comm=nfsiod pages=192 time=1032ms [ 405.257635] write_bandwidth: comm=nfsiod pages=1536 time=192ms [ 405.279069] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 405.300843] write_bandwidth: comm=nfsiod pages=192 time=16ms [ 405.326031] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 405.350843] write_bandwidth: comm=nfsiod pages=192 time=24ms [ 405.375160] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 409.331015] write_bandwidth: comm=nfsiod pages=192 time=3952ms [ 409.587928] write_bandwidth: comm=nfsiod pages=1536 time=152ms [ 409.610068] write_bandwidth: comm=nfsiod pages=192 time=20ms [ 409.635736] write_bandwidth: comm=nfsiod pages=192 time=24ms
# vmmon -d 1 nr_writeback nr_dirty nr_unstable
nr_writeback nr_dirty nr_unstable
11227 41463 38044
11227 41463 38044
11227 41463 38044
11227 41463 38044
11045 53987 6490
11033 53120 8145
11195 52143 10886
11211 52144 10913
11211 52144 10913
11211 52144 10913
11056 56887 3876
11062 55298 8155
11214 54485 9838
11225 54461 9852
11225 54461 9852
11225 54461 4582
22342 35535 7823----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 9 92 0 0| 0 0 | 66B 306B| 0 0 |1003 377 0 1 39 60 0 1| 0 0 | 90k 1361k| 0 0 |1765 1599 0 15 12 43 0 31| 0 0 |2292k 34M| 0 0 | 12k 16k 0 0 16 84 0 0| 0 0 | 132B 306B| 0 0 |1003 376 0 0 43 57 0 0| 0 0 | 66B 306B| 0 0 |1004 376 0 7 25 55 0 13| 0 0 |1202k 18M| 0 0 |7331 8921 0 8 21 55 0 15| 0 0 |1195k 18M| 0 0 |5382 6579 0 0 38 62 0 0| 0 0 | 66B 306B| 0 0 |1002 371 0 0 33 67 0 0| 0 0 | 66B 306B| 0 0 |1003 376 0 14 20 41 0 24| 0 0 |1621k 24M| 0 0 |8549 10k 0 5 31 55 0 9| 0 0 | 769k 11M| 0 0 |4444 5180 0 0 18 82 0 0| 0 0 | 66B 568B| 0 0 |1004 377 0 1 41 54 0 3| 0 0 | 184k 2777k| 0 0 |2609 2619 1 13 22 43 0 22| 0 0 |1572k 23M| 0 0 |8138 10k 0 11 9 59 0 20| 0 0 |1861k 27M| 0 0 |9576 13k 0 5 23 66 0 5| 0 0 | 540k 8122k| 0 0 |2816 2885
fs/nfs/client.c | 2 fs/nfs/write.c | 81 +++++++++++++++++++++++++++++++----- include/linux/nfs_fs_sb.h | 1 3 files changed, 73 insertions(+), 11 deletions(-)
fs/nfs/write.c +70 -11
@@ -187,11 +187,54 @@ static int wb_priority(struct writeback_* NFS congestion control */ +#define NFS_WAIT_PAGES (1024L >> (PAGE_CACHE_SHIFT - 10))int nfs_congestion_kb; -#define NFS_CONGESTION_ON_THRESH (nfs_congestion_kb >> (PAGE_SHIFT-10)) -#define NFS_CONGESTION_OFF_THRESH \ - (NFS_CONGESTION_ON_THRESH - (NFS_CONGESTION_ON_THRESH >> 2)) +/* +* SYNC requests will block on (2*limit) and wakeup on (2*limit-NFS_WAIT_PAGES) +* ASYNC requests will block on (limit) and wakeup on (limit - NFS_WAIT_PAGES) +* In this way SYNC writes will never be blocked by ASYNC ones. +*/ + +static void nfs_set_congested (long nr,long limit, +struct backing_dev_info *bdi) +{ +if (nr > limit && !test_bit (BDI_async_congested, &bdi->state)) +set_bdi_congested (bdi,BLK_RW_ASYNC ); +else if (nr >2 * limit && !test_bit (BDI_sync_congested, &bdi->state)) +set_bdi_congested (bdi,BLK_RW_SYNC ); +} + +static void nfs_wait_contested (int is_sync, +struct backing_dev_info *bdi, +wait_queue_head_t *wqh) +{ +int waitbit = is_sync ? BDI_sync_congested : BDI_async_congested; + +wait_event (wqh[is_sync], !test_bit (waitbit, &bdi->state)); +} + +static void nfs_wakeup_congested (long nr,long limit, +struct backing_dev_info *bdi, +wait_queue_head_t *wqh) +{ +if (nr <2 *limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_sync_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_SYNC ); +if (waitqueue_active (&wqh[BLK_RW_SYNC ])) { +smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_SYNC ]); + } + } +if (nr < limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_async_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_ASYNC ); +if (waitqueue_active (&wqh[BLK_RW_ASYNC ])) { +smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_ASYNC ]); + } + } +}static int nfs_set_page_writeback (struct page *page) {
@@ -201,11 +244,9 @@ static int nfs_set_page_writeback(structstruct inode *inode = page->mapping->host;struct nfs_server *nfss =NFS_SERVER (inode); -if (atomic_long_inc_return (&nfss->writeback) > -NFS_CONGESTION_ON_THRESH ) { -set_bdi_congested (&nfss->backing_dev_info, -BLK_RW_ASYNC ); - } +nfs_set_congested (atomic_long_inc_return (&nfss->writeback), + nfs_congestion_kb >> (PAGE_SHIFT -10 ), + &nfss->backing_dev_info); }return ret; }
@@ -216,8 +257,11 @@ static void nfs_end_page_writeback(strucstruct nfs_server *nfss =NFS_SERVER (inode);end_page_writeback (page); -if (atomic_long_dec_return (&nfss->writeback) <NFS_CONGESTION_OFF_THRESH ) -clear_bdi_congested (&nfss->backing_dev_info,BLK_RW_ASYNC ); + +nfs_wakeup_congested (atomic_long_dec_return (&nfss->writeback), + nfs_congestion_kb >> (PAGE_SHIFT -10 ), + &nfss->backing_dev_info, + nfss->writeback_wait); }static struct nfs_page *nfs_find_and_lock_request (struct page *page)
@@ -309,19 +353,34 @@ static int nfs_writepage_locked(struct pint nfs_writepage (struct page *page,struct writeback_control *wbc) { +struct inode *inode = page->mapping->host; +struct nfs_server *nfss =NFS_SERVER (inode);int ret; ret =nfs_writepage_locked (page, wbc);unlock_page (page); + +nfs_wait_contested (wbc->sync_mode ==WB_SYNC_ALL , + &nfss->backing_dev_info, + nfss->writeback_wait); +return ret; } -static int nfs_writepages_callback (struct page *page,struct writeback_control *wbc,void *data) +static int nfs_writepages_callback (struct page *page, +struct writeback_control *wbc,void *data) { +struct inode *inode = page->mapping->host; +struct nfs_server *nfss =NFS_SERVER (inode);int ret; ret =nfs_do_writepage (page, wbc, data);unlock_page (page); + +nfs_wait_contested (wbc->sync_mode ==WB_SYNC_ALL , + &nfss->backing_dev_info, + nfss->writeback_wait); +return ret; }
include/linux/nfs_fs_sb.h +1 -0
@@ -108,6 +108,7 @@ struct nfs_server {struct nfs_iostats * io_stats;/* I/O statistics */ struct backing_dev_info backing_dev_info;atomic_long_t writeback;/* number of writeback pages */ +wait_queue_head_t writeback_wait[2 ];int flags;/* various flags */ unsigned int caps;/* server capabilities */ unsigned int rsize;/* read size */
fs/nfs/client.c +2 -0
@@ -991,6 +991,8 @@ static struct nfs_server *nfs_alloc_servINIT_LIST_HEAD (&server->master_link);atomic_set (&server->active,0 ); +init_waitqueue_head (&server->writeback_wait[BLK_RW_SYNC ]); +init_waitqueue_head (&server->writeback_wait[BLK_RW_ASYNC ]); server->io_stats =nfs_alloc_iostats ();if (!server->io_stats) {
PATCH 21/45 writeback: estimate bdi write bandwidth
In reply to Wu FengguangEstimate bdi write bandwidth in bdi_writeback_wakeup(), at which time the queue is not starved due to the dirtying process and the associated background writeback.
The estimation should be able to reflect the max device capability, unless there are busy reads, in which case we need lower nr_to_write anyway.
---
TODO: the estimated write bandwidth (~30MB/s) is mysteriously only half the real throughput (~60MB/s). Here are some debug printk from bdi_calc_write_bandwidth():
printk("write_bandwidth: comm=%s pages=%lu time=%lums\n",
current->comm, nr_pages, time * 1000 / HZ);
[ 1093.397700] write_bandwidth: comm=swapper pages=1536 time=204ms [ 1093.594319] write_bandwidth: comm=swapper pages=1536 time=196ms [ 1093.796642] write_bandwidth: comm=swapper pages=1536 time=200ms [ 1093.986128] write_bandwidth: comm=swapper pages=1536 time=192ms [ 1094.179983] write_bandwidth: comm=swapper pages=1536 time=192ms [ 1094.374021] write_bandwidth: comm=swapper pages=1536 time=196ms [ 1094.570611] write_bandwidth: comm=swapper pages=1536 time=196ms [ 1094.771847] write_bandwidth: comm=swapper pages=1536 time=200ms [ 1094.961981] write_bandwidth: comm=swapper pages=1536 time=192ms
Workload is several concurrent copies.
fs/fs-writeback.c | 30 +++++++++++++++++++++--------- include/linux/backing-dev.h | 2 ++ include/linux/writeback.h | 10 ++++++++++ mm/backing-dev.c | 2 ++ 4 files changed, 35 insertions(+), 9 deletions(-)
fs/fs-writeback.c +21 -9
@@ -266,7 +266,23 @@ void bdi_start_writeback(struct backing_bdi_alloc_queue_work (bdi, &args); } +static int bdi_writeback_chunk (struct backing_dev_info *bdi) +{ +return max (MIN_WRITEBACK_PAGES , bdi->write_bandwidth); +} + +static void bdi_calc_write_bandwidth (struct backing_dev_info *bdi, +unsigned long nr_pages, +unsigned long time) +{ +unsigned long bw; + + bw =HZ * nr_pages / (time |1 ); + bdi->write_bandwidth = (bdi->write_bandwidth *63 + bw) /64 ; +} +struct dirty_throttle_task { +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;
@@ -275,6 +291,7 @@ struct dirty_throttle_task {void bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { + .start_time = jiffies, .nr_pages = nr_pages, .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };
@@ -314,6 +331,9 @@ int bdi_writeback_wakeup(struct backing_ tt =list_entry (bdi->throttle_list.prev, struct dirty_throttle_task, list);list_del (&tt->list); +if (atomic_read (&bdi->throttle_pages) ==0 ) +bdi_calc_write_bandwidth (bdi, tt->nr_pages, + jiffies - tt->start_time);complete (&tt->complete); }/*
@@ -323,6 +343,7 @@ int bdi_writeback_wakeup(struct backing_ tt =list_entry (bdi->throttle_list.prev, struct dirty_throttle_task, list);atomic_set (&bdi->throttle_pages, tt->nr_pages); + tt->start_time = jiffies; }else { tt =NULL ;atomic_set (&bdi->throttle_pages,DIRTY_THROTTLE_PAGES_STOP *2 );
@@ -717,15 +738,6 @@ void writeback_inodes_wbc(struct writebawriteback_inodes_wb (&bdi->wb, wbc); } -/* -* The maximum number of pages to writeout in a single bdi flush/kupdate -* operation. We do this so we don't hold I_SYNC against an inode for -* enormous amounts of time, which would block a userspace task which has -* been forced to throttle against that inode. Also, the code reevaluates -* the dirty each time it has written this many pages. -*/ -#define MAX_WRITEBACK_PAGES 1024 -static inline bool over_bground_thresh (void ) {unsigned long background_thresh, dirty_thresh;
include/linux/writeback.h +10 -0
@@ -14,6 +14,16 @@ extern struct list_head inode_in_use;extern struct list_head inode_unused;/* +* The max number of pages to writeout for each inode. +* +* We honor each inode a nr_to_write that will take about 1 second +* to finish, based on dynamic estimation of the bdi's write bandwidth. +* MAX_ serves as initial bandwidth value; MIN_ serves as low boundary. +*/ +#define MAX_WRITEBACK_PAGES (128 << (20 - PAGE_CACHE_SHIFT)) +#define MIN_WRITEBACK_PAGES ( 16 << (20 - PAGE_CACHE_SHIFT)) + +/* * fs/fs-writeback.c */ enum writeback_sync_modes {
include/linux/backing-dev.h +2 -0
@@ -86,6 +86,8 @@ struct backing_dev_info {struct list_head work_list; +int write_bandwidth;/* pages per second */ +/* * dirtier process throttling */
mm/backing-dev.c +2 -0
@@ -646,6 +646,8 @@ int bdi_init(struct backing_dev_info *bd bdi->wb_mask =1 ; bdi->wb_cnt =1 ; + bdi->write_bandwidth =MAX_WRITEBACK_PAGES ; +spin_lock_init (&bdi->throttle_lock);INIT_LIST_HEAD (&bdi->throttle_list);atomic_set (&bdi->throttle_pages,DIRTY_THROTTLE_PAGES_STOP *2 );
PATCH 22/45 writeback: show bdi write bandwidth in debugfs
In reply to Wu Fengguang--- mm/backing-dev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
mm/backing-dev.c +4 -2
@@ -93,6 +93,7 @@ static int bdi_debug_stats_show(struct s"BdiDirtyThresh: %8lu kB\n" "DirtyThresh: %8lu kB\n" "BackgroundThresh: %8lu kB\n" +"WriteBandwidth: %8lu kBps\n" "WriteBack threads:%8lu\n" "b_dirty: %8lu\n" "b_io: %8lu\n"
@@ -104,8 +105,9 @@ static int bdi_debug_stats_show(struct s"wb_cnt: %8u\n" , (unsigned long )K (bdi_stat (bdi,BDI_WRITEBACK )), (unsigned long )K (bdi_stat (bdi,BDI_RECLAIMABLE )), -K (bdi_thresh),K (dirty_thresh), -K (background_thresh), nr_wb, nr_dirty, nr_io, nr_more_io, +K (bdi_thresh),K (dirty_thresh),K (background_thresh), + (unsigned long )K (bdi->write_bandwidth), + nr_wb, nr_dirty, nr_io, nr_more_io, !list_empty (&bdi->bdi_list), bdi->state, bdi->wb_mask, !list_empty(&bdi->wb_list), bdi->wb_cnt);#undef K
PATCH 23/45 writeback: kill space in debugfs item name
In reply to Wu FengguangThe space is not script friendly, kill it.
--- mm/backing-dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/backing-dev.c +1 -1
@@ -94,7 +94,7 @@ static int bdi_debug_stats_show(struct s"DirtyThresh: %8lu kB\n" "BackgroundThresh: %8lu kB\n" "WriteBandwidth: %8lu kBps\n" -"WriteBack threads:%8lu\n" +"WriteBackThreads: %8lu\n" "b_dirty: %8lu\n" "b_io: %8lu\n" "b_more_io: %8lu\n"
PATCH 25/45 writeback: convert wbc.nr_to_write to per-file parameter
In reply to Wu Fengguang--- fs/fs-writeback.c | 24 ++++++++++++------------ include/linux/writeback.h | 5 +++-- mm/backing-dev.c | 1 - 3 files changed, 15 insertions(+), 15 deletions(-)
fs/fs-writeback.c +12 -12
@@ -661,12 +661,13 @@ pinned:return 0 ; } -static void writeback_inodes_wb (struct bdi_writeback *wb, +static long writeback_inodes_wb (struct bdi_writeback *wb,struct writeback_control *wbc) {struct super_block *sb = wbc->sb, *pin_sb =NULL ;const unsigned long start = jiffies;/* livelock avoidance */ unsigned long stop_time =0 ; +unsigned long wrote =0 ;if (wbc->timeout) stop_time = (start + wbc->timeout) |1 ;
@@ -709,7 +710,10 @@ static void writeback_inodes_wb(struct bBUG_ON (inode->i_state & (I_FREEING |I_CLEAR ));__iget (inode); pages_skipped = wbc->pages_skipped; + wbc->nr_to_write =bdi_writeback_chunk (wb->bdi); + wrote += wbc->nr_to_write;writeback_single_inode (inode, wbc); + wrote -= wbc->nr_to_write;if (wbc->pages_skipped != pages_skipped) {/* * writeback is not making progress due to locked
@@ -735,6 +739,7 @@ static void writeback_inodes_wb(struct bspin_unlock (&inode_lock);/* Leave any unwritten inodes on b_io */ +return wrote; }void writeback_inodes_wbc (struct writeback_control *wbc)
@@ -782,6 +787,7 @@ static long wb_writeback(struct bdi_writ };unsigned long oldest_jif;long wrote =0 ; +long nr;struct inode *inode;if (wbc.for_kupdate) {
@@ -810,26 +816,20 @@ static long wb_writeback(struct bdi_writ wbc.more_io =0 ; wbc.encountered_congestion =0 ; - wbc.nr_to_write =MAX_WRITEBACK_PAGES ; wbc.pages_skipped =0 ; -writeback_inodes_wb (wb, &wbc); - args->nr_pages -=MAX_WRITEBACK_PAGES - wbc.nr_to_write; - wrote +=MAX_WRITEBACK_PAGES - wbc.nr_to_write; + nr =writeback_inodes_wb (wb, &wbc); + args->nr_pages -= nr; + wrote += nr;/* -* If we consumed everything, see if we have more -*/ -if (wbc.nr_to_write <=0 ) -continue ; -/* -* Didn't write everything and we don't have more IO, bail +* Bail if no more IO */ if (!wbc.more_io)break ;/* * Did we write something? Try for more */ -if (wbc.nr_to_write <MAX_WRITEBACK_PAGES ) +if (nr)continue ;/* * Nothing written. Wait for some inode to
include/linux/writeback.h +3 -2
@@ -45,8 +45,9 @@ struct writeback_control {int timeout;unsigned long *older_than_this;/* If !NULL, only write back inodes older than this */ -long nr_to_write;/* Write this many pages, and decrement -this for each page written */ +long nr_to_write;/* Max pages to write per file, and +decrement this for each page written +*/ long pages_skipped;/* Pages which were not written */ /*
mm/backing-dev.c +0 -1
@@ -338,7 +338,6 @@ static void bdi_flush_io(struct backing_ .sync_mode = WB_SYNC_NONE, .older_than_this = NULL, .range_cyclic = 1, - .nr_to_write = 1024, .timeout = HZ, };
PATCH 26/45 block: pass the non-rotational queue flag to backing_dev_info
In reply to Wu FengguangFor now only pass the non-rotational bit to the queue's bdi.
To support hierarchical dm/md configurations, blk_set_rotational() shall be expanded to make several iterations of the bdi list and call its bdi->rotational_fn to pass the bit up. But that would imply we don's support mixed SSD/HD arrays.
--- block/blk-core.c | 14 ++++++++++++++ block/blk-sysfs.c | 7 +------ drivers/block/nbd.c | 2 +- drivers/block/xen-blkfront.c | 1 + drivers/mmc/card/queue.c | 2 +- drivers/scsi/sd.c | 2 +- include/linux/backing-dev.h | 8 ++++++++ include/linux/blkdev.h | 1 + 8 files changed, 28 insertions(+), 9 deletions(-)
block/blk-sysfs.c +1 -6
@@ -162,12 +162,7 @@ static ssize_t queue_nonrot_store(structunsigned long nm;ssize_t ret =queue_var_store (&nm, page, count); -spin_lock_irq (q->queue_lock); -if (nm) -queue_flag_clear (QUEUE_FLAG_NONROT , q); -else -queue_flag_set (QUEUE_FLAG_NONROT , q); -spin_unlock_irq (q->queue_lock); +blk_set_rotational (q, nm);return ret; }
include/linux/backing-dev.h +8 -0
@@ -29,6 +29,7 @@ enum bdi_state { BDI_wb_alloc,/* Default embedded wb allocated */ BDI_async_congested,/* The async (write) queue is getting full */ BDI_sync_congested,/* The sync queue is getting full */ + BDI_non_rotational,/* Underlying device is SSD or virtual */ BDI_registered,/* bdi_register() was done */ BDI_unused, /* Available bits start here */ };
@@ -254,6 +255,8 @@ int bdi_set_max_ratio(struct backing_dev#define BDI_CAP_NO_ACCT_WB 0x00000080#define BDI_CAP_SWAP_BACKED 0x00000100 +#define BDI_CAP_NONROT 0x00000200 +#define BDI_CAP_VMFLAGS \ (BDI_CAP_READ_MAP | BDI_CAP_WRITE_MAP | BDI_CAP_EXEC_MAP)
@@ -291,6 +294,11 @@ static inline int can_submit_background_return !test_and_set_bit (WB_FLAG_BACKGROUND_WORK , &bdi->wb_mask); } +static inline bool bdi_nonrot (struct backing_dev_info *bdi) +{ +return bdi->state & (1 << BDI_non_rotational); +} +static inline int bdi_congested (struct backing_dev_info *bdi,int bdi_bits) {if (bdi->congested_fn)
block/blk-core.c +14 -0
@@ -2486,6 +2486,20 @@ free_and_out: }EXPORT_SYMBOL_GPL (blk_rq_prep_clone); +void blk_set_rotational (struct request_queue *q,int rotational) +{ +spin_lock_irq (q->queue_lock); +if (rotational) { +queue_flag_clear (QUEUE_FLAG_NONROT , q); +clear_bit (BDI_non_rotational, &q->backing_dev_info.state); + }else { +queue_flag_set (QUEUE_FLAG_NONROT , q); +set_bit (BDI_non_rotational, &q->backing_dev_info.state); + } +spin_unlock_irq (q->queue_lock); +} +EXPORT_SYMBOL (blk_set_rotational); +int kblockd_schedule_work (struct request_queue *q,struct work_struct *work) {return queue_work (kblockd_workqueue, work);
include/linux/blkdev.h +1 -0
@@ -666,6 +666,7 @@ static inline void blk_clear_queue_full(queue_flag_clear (QUEUE_FLAG_ASYNCFULL , q); } +void blk_set_rotational (struct request_queue *q,int rotational);/* * mergeable request must not have _NOMERGE or _BARRIER bit set, nor may
drivers/block/nbd.c +1 -1
@@ -772,7 +772,7 @@ static int __init nbd_init(void)/* * Tell the block layer that we are not a rotational device */ -queue_flag_set_unlocked (QUEUE_FLAG_NONROT , disk->queue); +blk_set_rotational (disk->queue,0 ); }if (register_blkdev (NBD_MAJOR ,"nbd" )) {
drivers/block/xen-blkfront.c +1 -0
@@ -342,6 +342,7 @@ static int xlvbd_init_blk_queue(struct greturn -1 ;queue_flag_set_unlocked (QUEUE_FLAG_VIRT , rq); +blk_set_rotational (rq,0 );/* Hard sector size and max sectors impersonate the equiv. hardware. */ blk_queue_logical_block_size (rq, sector_size);
drivers/mmc/card/queue.c +1 -1
@@ -127,7 +127,7 @@ int mmc_init_queue(struct mmc_queue *mq,blk_queue_prep_rq (mq->queue, mmc_prep_request);blk_queue_ordered (mq->queue,QUEUE_ORDERED_DRAIN ,NULL ); -queue_flag_set_unlocked (QUEUE_FLAG_NONROT , mq->queue); +blk_set_rotational (mq->queue,0 );#ifdef CONFIG_MMC_BLOCK_BOUNCE if (host->max_hw_segs ==1 ) {
drivers/scsi/sd.c +1 -1
@@ -1898,7 +1898,7 @@ static void sd_read_block_characteristic rot =get_unaligned_be16 (&buffer[4 ]);if (rot ==1 ) -queue_flag_set_unlocked (QUEUE_FLAG_NONROT , sdkp->disk->queue); +blk_set_rotational (sdkp->disk->queue,0 );kfree (buffer); }
PATCH 27/45 writeback: introduce wbc.for_background
In reply to Wu FengguangIt will be tested for setting wbc.nr_segments, lower the flush priority for NFS, and maybe more in future.
--- fs/fs-writeback.c | 1 + fs/nfs/write.c | 2 +- include/linux/writeback.h | 1 + 3 files changed, 3 insertions(+), 1 deletion(-)
fs/fs-writeback.c +1 -0
@@ -783,6 +783,7 @@ static long wb_writeback(struct bdi_writ .sync_mode = args->sync_mode, .older_than_this = NULL, .for_kupdate = args->for_kupdate, + .for_background = args->for_background, .range_cyclic = args->range_cyclic, };unsigned long oldest_jif;
include/linux/writeback.h +1 -0
@@ -61,6 +61,7 @@ struct writeback_control { unsigned nonblocking:1; /* Don't get stuck on request queues */ unsigned encountered_congestion:1; /* An output: a queue is full */ unsigned for_kupdate:1; /* A kupdate writeback */ + unsigned for_background:1; /* A background writeback */ unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned range_cyclic:1; /* range_start is cyclic */ unsigned more_io:1; /* more io to be dispatched */
fs/nfs/write.c +1 -1
@@ -178,7 +178,7 @@ static int wb_priority(struct writeback_ {if (wbc->for_reclaim)return FLUSH_HIGHPRI |FLUSH_STABLE ; -if (wbc->for_kupdate) +if (wbc->for_kupdate || wbc->for_background)return FLUSH_LOWPRI ;return 0 ; }
PATCH 32/45 writeback: update kupdate expire timestamp on each scan of b_io
In reply to Wu FengguangThis prevents it to stuck with some very old but busy inodes, and to give newer expired inodes a fair chance.
--- fs/fs-writeback.c | 3 +++ 1 file changed, 3 insertions(+)
fs/fs-writeback.c +3 -0
@@ -829,6 +829,9 @@ static long wb_writeback(struct bdi_writ args->nr_pages -= nr; wrote += nr; +if (args->for_kupdate &&list_empty (&wb->b_io)) + oldest_jif = jiffies - expire_interval; +if (args->for_background && expire_interval && ++fg_rounds &&list_empty (&wb->b_io)) {if (fg_rounds <10 )
PATCH 34/45 writeback: sync livelock - kick background writeback
In reply to Wu FengguangThe periodic/background writeback can run forever. So when any sync work is enqueued, increase bdi->sync_works to notify the active non-sync works to exit. Non-sync works queued after all sync works won't be affected.
--- fs/fs-writeback.c | 11 +++++++++++ include/linux/backing-dev.h | 5 +++++ mm/backing-dev.c | 1 + 3 files changed, 17 insertions(+)
fs/fs-writeback.c +11 -0
@@ -139,6 +139,8 @@ static void wb_clear_pending(struct bdi_list_del_rcu (&work->list);if (work->args.for_background)clear_bit (WB_FLAG_BACKGROUND_WORK , &bdi->wb_mask); +if (work->args.for_sync) + bdi->sync_works--;spin_unlock (&bdi->wb_lock);wb_work_complete (work);
@@ -159,6 +161,8 @@ static void bdi_queue_work(struct backin*/ spin_lock (&bdi->wb_lock);list_add_tail_rcu (&work->list, &bdi->work_list); +if (work->args.for_sync) + bdi->sync_works++;spin_unlock (&bdi->wb_lock);/*
@@ -811,6 +815,13 @@ static long wb_writeback(struct bdi_writbreak ;/* +* background/periodic works can run forever, need to abort +* on seeing any pending sync work, to prevent livelock it. +*/ +if (!args->for_sync && wb->bdi->sync_works >0 ) +break ; + +/* * For background writeout, stop when we are below the * background dirty threshold */
include/linux/backing-dev.h +5 -0
@@ -84,6 +84,11 @@ struct backing_dev_info {struct list_head wb_list;/* the flusher threads hanging off this bdi */ unsigned long wb_mask;/* bitmask of registered tasks */ unsigned int wb_cnt;/* number of registered tasks */ +/* +* sync works queued, background works shall abort on seeing this, +* to prevent livelocking the sync works +*/ +unsigned int sync_works;struct list_head work_list;
mm/backing-dev.c +1 -0
@@ -647,6 +647,7 @@ int bdi_init(struct backing_dev_info *bd*/ bdi->wb_mask =1 ; bdi->wb_cnt =1 ; + bdi->sync_works =0 ; bdi->write_bandwidth =MAX_WRITEBACK_PAGES ;
PATCH 37/45 writeback: use timestamp to indicate dirty exceeded
In reply to Wu FengguangWhen there are only one (or several) dirtiers, dirty_exceeded is always (or mostly) off. Converting to timestamp avoids this problem. It helps to use smaller write_chunk for smoother throttling.
We'll lower ratelimit if saw dirty exceeded in the last 1 second.
---
Before patch, the wait time in balance_dirty_pages() are ~200ms:
[ 1093.397700] write_bandwidth: comm=swapper pages=1536 time=204ms [ 1093.594319] write_bandwidth: comm=swapper pages=1536 time=196ms [ 1093.796642] write_bandwidth: comm=swapper pages=1536 time=200ms
After patch, ~25ms:
[ 90.261339] write_bandwidth: comm=swapper pages=192 time=20ms [ 90.293168] write_bandwidth: comm=swapper pages=192 time=24ms [ 90.323853] write_bandwidth: comm=swapper pages=192 time=24ms [ 90.354510] write_bandwidth: comm=swapper pages=192 time=28ms [ 90.389890] write_bandwidth: comm=swapper pages=192 time=28ms [ 90.421787] write_bandwidth: comm=swapper pages=192 time=24ms
include/linux/backing-dev.h | 2 +- mm/backing-dev.c | 1 - mm/page-writeback.c | 9 +++------ 3 files changed, 4 insertions(+), 8 deletions(-)
mm/page-writeback.c +3 -6
@@ -523,14 +523,10 @@ static void balance_dirty_pages(struct a goto out; throttle: -if (!bdi->dirty_exceeded) - bdi->dirty_exceeded =1 ; + bdi->dirty_exceed_time = jiffies;bdi_writeback_wait (bdi, write_chunk); -if (bdi->dirty_exceeded) - bdi->dirty_exceeded =0 ; - out:/* * In laptop mode, we wait until hitting the higher threshold before
@@ -578,7 +574,8 @@ void balance_dirty_pages_ratelimited_nr(unsigned long *p; ratelimit = ratelimit_pages; -if (mapping->backing_dev_info->dirty_exceeded) +if (jiffies - mapping->backing_dev_info->dirty_exceed_time < + (unsigned long )HZ ) ratelimit >>=3 ;/*
include/linux/backing-dev.h +1 -1
@@ -74,7 +74,7 @@ struct backing_dev_info {struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS ];struct prop_local_percpu completions; -int dirty_exceeded; +unsigned long dirty_exceed_time;unsigned int min_ratio;unsigned int max_ratio, max_prop_frac;
mm/backing-dev.c +0 -1
@@ -661,7 +661,6 @@ int bdi_init(struct backing_dev_info *bd goto err; } - bdi->dirty_exceeded =0 ; err =prop_local_init_percpu (&bdi->completions);if (err) {
PATCH 38/45 writeback: introduce queue b_more_io_wait
In reply to Wu FengguangIntroduce the b_more_io_wait queue to park inodes that for some reason cannot be synced immediately. They will be revisited either in the next b_io scan time, or after 0.1s sleep for sync, or retried after 5s in the next periodic writeback.
The new data flow after this patchset:
b_dirty --> b_io --> b_more_io/b_more_io_wait --+
^ | | | +----------------------------------+
The rational is to address two issues:
- the 30s delay of redirty_tail() may be too long
- redirty_tail() may update i_dirtied_when, however we now rely on it remain unchanged for all candidate inodes of sync(). (to avoid extra work and livelock, we now exclude any inodes from being synced if its dirty time is after the sync time)
--- fs/fs-writeback.c | 27 ++++++++++++++++----------- include/linux/backing-dev.h | 8 +++++--- mm/backing-dev.c | 14 +++++++++++--- 3 files changed, 32 insertions(+), 17 deletions(-)
fs/fs-writeback.c +16 -11
@@ -384,6 +384,16 @@ static void requeue_io(struct inode *inolist_move (&inode->i_list, &wb->b_more_io); } +/* +* The inode should be retried in an opportunistic way. +*/ +static void requeue_io_wait (struct inode *inode) +{ +struct bdi_writeback *wb = &inode_to_bdi (inode)->wb; + +list_move (&inode->i_list, &wb->b_more_io_wait); +} +static void inode_sync_complete (struct inode *inode) {/*
@@ -453,12 +463,14 @@ static void move_expired_inodes(struct l/* * Queue all expired dirty inodes for io, eldest first: * (newly dirtied) => b_dirty inodes +* => b_more_io_wait inodes * => b_more_io inodes * => remaining inodes in b_io => (dequeue for sync) */ static void queue_io (struct bdi_writeback *wb,unsigned long *older_than_this) {list_splice_init (&wb->b_more_io, &wb->b_io); +list_splice_init (&wb->b_more_io_wait, &wb->b_io);move_expired_inodes (&wb->b_dirty, &wb->b_io, older_than_this); }
@@ -860,18 +872,11 @@ static long wb_writeback(struct bdi_writ*/ if (nr)continue ; -/* -* Nothing written. Wait for some inode to -* become available for writeback. Otherwise -* we'll just busyloop. -*/ -spin_lock (&inode_lock); -if (!list_empty (&wb->b_more_io)) { - inode =list_entry (wb->b_more_io.prev, - struct inode, i_list); -inode_wait_for_writeback (inode); +if (wbc.for_sync && !list_empty (&wb->b_more_io_wait)) { +schedule_timeout_interruptible (HZ /10 ); +continue ; } -spin_unlock (&inode_lock); +break ; }if (args->for_sync)
include/linux/backing-dev.h +5 -3
@@ -56,6 +56,7 @@ struct bdi_writeback {struct list_head b_dirty;/* dirty inodes */ struct list_head b_io;/* parked for writeback */ struct list_head b_more_io;/* parked for more writeback */ +struct list_head b_more_io_wait;/* opportunistic retry io */ };struct backing_dev_info {
@@ -140,9 +141,10 @@ extern struct list_head bdi_list;static inline int wb_has_dirty_io (struct bdi_writeback *wb) { -return !list_empty (&wb->b_dirty) || - !list_empty (&wb->b_io) || - !list_empty (&wb->b_more_io); +return !list_empty (&wb->b_dirty) || + !list_empty (&wb->b_io) || + !list_empty (&wb->b_more_io) || + !list_empty (&wb->b_more_io_wait); } static inline void __add_bdi_stat(struct backing_dev_info *bdi,
mm/backing-dev.c +11 -3
@@ -63,14 +63,17 @@ static int bdi_debug_stats_show(struct sunsigned long background_thresh;unsigned long dirty_thresh;unsigned long bdi_thresh; -unsigned long nr_dirty, nr_io, nr_more_io, nr_wb; +unsigned long nr_dirty =0 ; +unsigned long nr_io =0 ; +unsigned long nr_more_io =0 ; +unsigned long nr_more_io_wait =0 ; +unsigned long nr_wb =0 ;struct inode *inode;/* * inode lock is enough here, the bdi->wb_list is protected by * RCU on the reader side */ - nr_wb = nr_dirty = nr_io = nr_more_io =0 ;spin_lock (&inode_lock);list_for_each_entry (wb, &bdi->wb_list, list) { nr_wb++;
@@ -80,6 +83,8 @@ static int bdi_debug_stats_show(struct s nr_io++;list_for_each_entry (inode, &wb->b_more_io, i_list) nr_more_io++; +list_for_each_entry (inode, &wb->b_more_io_wait, i_list) + nr_more_io_wait++; }spin_unlock (&inode_lock);
@@ -98,6 +103,7 @@ static int bdi_debug_stats_show(struct s"b_dirty: %8lu\n" "b_io: %8lu\n" "b_more_io: %8lu\n" +"b_more_io_wait: %8lu\n" "bdi_list: %8u\n" "state: %8lx\n" "wb_mask: %8lx\n"
@@ -107,7 +113,7 @@ static int bdi_debug_stats_show(struct s (unsigned long )K (bdi_stat (bdi,BDI_RECLAIMABLE )),K (bdi_thresh),K (dirty_thresh),K (background_thresh), (unsigned long )K (bdi->write_bandwidth), - nr_wb, nr_dirty, nr_io, nr_more_io, + nr_wb, nr_dirty, nr_io, nr_more_io, nr_more_io_wait, !list_empty (&bdi->bdi_list), bdi->state, bdi->wb_mask, !list_empty(&bdi->wb_list), bdi->wb_cnt);#undef K
@@ -264,6 +270,7 @@ static void bdi_wb_init(struct bdi_writeINIT_LIST_HEAD (&wb->b_dirty);INIT_LIST_HEAD (&wb->b_io);INIT_LIST_HEAD (&wb->b_more_io); +INIT_LIST_HEAD (&wb->b_more_io_wait); } static void bdi_task_init(struct backing_dev_info *bdi,
@@ -688,6 +695,7 @@ void bdi_destroy(struct backing_dev_infolist_splice (&bdi->wb.b_dirty, &dst->b_dirty);list_splice (&bdi->wb.b_io, &dst->b_io);list_splice (&bdi->wb.b_more_io, &dst->b_more_io); +list_splice (&bdi->wb.b_more_io_wait, &dst->b_more_io_wait);spin_unlock (&inode_lock); }
PATCH 39/45 writeback: remove wbc.more_io
In reply to Wu FengguangIt seems no longer required. It was introduced mainly to deal with the complexity from _multiple_ superblock queues. Now there are only one queue, so infomation can be directly queried if necessary.
--- fs/fs-writeback.c | 13 +------------ include/linux/writeback.h | 1 - include/trace/events/ext4.h | 6 ++---- 3 files changed, 3 insertions(+), 17 deletions(-)
fs/fs-writeback.c +1 -12
@@ -733,12 +733,8 @@ static long writeback_inodes_wb(struct biput (inode);cond_resched ();spin_lock (&inode_lock); -if (wbc->nr_to_write <=0 ) { - wbc->more_io =1 ; +if (wbc->nr_to_write <=0 )break ; - } -if (!list_empty (&wb->b_more_io)) - wbc->more_io =1 ;if (stop_time &&time_after (jiffies, stop_time))break ; }
@@ -800,7 +796,6 @@ static long wb_writeback(struct bdi_writint fg_rounds =0 ;long wrote =0 ;long nr; -struct inode *inode;/* * keep sync from extra jobs and livelock
@@ -840,7 +835,6 @@ static long wb_writeback(struct bdi_writif (args->for_background && !over_bground_thresh ())break ; - wbc.more_io =0 ; wbc.encountered_congestion =0 ; wbc.pages_skipped =0 ; nr =writeback_inodes_wb (wb, &wbc);
@@ -863,11 +857,6 @@ static long wb_writeback(struct bdi_writ }/* -* Bail if no more IO -*/ -if (!wbc.more_io) -break ; -/* * Did we write something? Try for more */ if (nr)
include/linux/writeback.h +0 -1
@@ -78,7 +78,6 @@ struct writeback_control { unsigned for_sync:1; /* A writeback for sync */ unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned range_cyclic:1; /* range_start is cyclic */ - unsigned more_io:1; /* more io to be dispatched *//* * write_cache_pages() won't update wbc->nr_to_write and * mapping->writeback_index if no_nrwrite_index_update
include/trace/events/ext4.h +2 -4
@@ -311,7 +311,6 @@ TRACE_EVENT(ext4_da_writepages_result, __field( int, pages_written ) __field( long, pages_skipped ) __field( char, encountered_congestion ) - __field( char, more_io ) __field( char, no_nrwrite_index_update )__field ( pgoff_t, writeback_index ) ),
@@ -323,16 +322,15 @@ TRACE_EVENT(ext4_da_writepages_result, __entry->pages_written = pages_written; __entry->pages_skipped = wbc->pages_skipped; __entry->encountered_congestion = wbc->encountered_congestion; - __entry->more_io = wbc->more_io; __entry->no_nrwrite_index_update = wbc->no_nrwrite_index_update; __entry->writeback_index = inode->i_mapping->writeback_index; ), -TP_printk ("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d more_io %d no_nrwrite_index_update %d writeback_index %lu" , +TP_printk ("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d no_nrwrite_index_update %d writeback_index %lu" ,jbd2_dev_to_name (__entry->dev), (unsigned long ) __entry->ino, __entry->ret, __entry->pages_written, __entry->pages_skipped, - __entry->encountered_congestion, __entry->more_io, + __entry->encountered_congestion, __entry->no_nrwrite_index_update, (unsigned long ) __entry->writeback_index) );
PATCH 40/45 writeback: requeue_io_wait() on I_SYNC locked inode
In reply to Wu FengguangUse requeue_io_wait() if inode is being synced by others. This queue won't be busy retried, so avoids busy loops.
--- fs/fs-writeback.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/fs-writeback.c +3 -3
@@ -526,14 +526,14 @@ writeback_single_inode(struct inode *inoif (inode->i_state &I_SYNC ) {/* * If this inode is locked for writeback and we are not doing -* writeback-for-data-integrity, move it to b_more_io so that -* writeback can proceed with the other inodes on s_io. +* writeback-for-data-integrity, move it to b_more_io_wait so +* that writeback can proceed with the other inodes on b_io. * * We'll have another go at writing back this inode when we * completed a full scan of b_io. */ if (!wait) { -requeue_io (inode); +requeue_io_wait (inode);return 0 ; }
PATCH 41/45 writeback: requeue_io_wait() on pages_skipped inode
In reply to Wu FengguangUse requeue_io_wait() if some pages were skipped due to locked buffers.
--- fs/fs-writeback.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -727,7 +727,7 @@ static long writeback_inodes_wb(struct b* writeback is not making progress due to locked * buffers. Skip this inode for now. */ -redirty_tail (inode); +requeue_io_wait (inode); }spin_unlock (&inode_lock);iput (inode);
PATCH 43/45 writeback: requeue_io_wait() on fs redirtied inode
In reply to Wu FengguangWhen an inodes is redirtied by the filesystem, its dirty time shall not be updated.
--- fs/fs-writeback.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -598,7 +598,7 @@ writeback_single_inode(struct inode *ino* At least XFS will redirty the inode during the * writeback (delalloc) and on io completion (isize). */ -redirty_tail (inode); +requeue_io_wait (inode); }else if (atomic_read (&inode->i_count)) {/* * The inode is clean, inuse
PATCH 44/45 NFS: remove NFS_INO_FLUSHING lock
In reply to Wu FengguangIt was introduced in 72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue in balance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes
--- fs/nfs/file.c | 9 --------- fs/nfs/write.c | 11 ----------- include/linux/nfs_fs.h | 1 - 3 files changed, 21 deletions(-)
fs/nfs/file.c +0 -9
@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file * mapping->host->i_ino, len, (long long) pos); start: -/* -* Prevent starvation issues if someone is doing a consistency -* sync-to-disk -*/ - ret =wait_on_bit (&NFS_I (mapping->host)->flags,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (ret) -return ret; - page =grab_cache_page_write_begin (mapping, index, flags);if (!page)return -ENOMEM ;
fs/nfs/write.c +0 -11
@@ -387,26 +387,15 @@ static int nfs_writepages_callback(strucint nfs_writepages (struct address_space *mapping,struct writeback_control *wbc) {struct inode *inode = mapping->host; -unsigned long *bitlock = &NFS_I (inode)->flags;struct nfs_pageio_descriptor pgio;int err; -/* Stop dirtying of new pages while we sync */ - err =wait_on_bit_lock (bitlock,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (err) - goto out_err; -nfs_inc_stats (inode,NFSIOS_VFSWRITEPAGES );nfs_pageio_init_write (&pgio, inode,wb_priority (wbc)); err =write_cache_pages (mapping, wbc, nfs_writepages_callback, &pgio);nfs_pageio_complete (&pgio); -clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); -wake_up_bit (bitlock,NFS_INO_FLUSHING ); -if (err <0 ) goto out_err; err = pgio.pg_error;
include/linux/nfs_fs.h +0 -1
@@ -208,7 +208,6 @@ struct nfs_inode {#define NFS_INO_STALE (1)/* possible stale inode */ #define NFS_INO_ACL_LRU_SET (2)/* Inode is on the LRU list */ #define NFS_INO_MOUNTPOINT (3)/* inode is remote mountpoint */ -#define NFS_INO_FLUSHING (4)/* inode is flushing out data */ #define NFS_INO_FSCACHE (5)/* inode can be cached by FS-Cache */ #define NFS_INO_FSCACHE_LOCK (6)/* FS-Cache cookie management lock */
PATCH 3/45 ext4: remove unused parameter wbc from __ext4_journalled_writepage()
In reply to Wu Fengguang--- fs/ext4/inode.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ext4/inode.c +1 -2
@@ -2599,7 +2599,6 @@ static int bput_one(handle_t *handle, st }static int __ext4_journalled_writepage (struct page *page, -struct writeback_control *wbc,unsigned int len) {struct address_space *mapping = page->mapping;
@@ -2757,7 +2756,7 @@ static int ext4_writepage(struct page *p* doesn't seem much point in redirtying the page here. */ ClearPageChecked (page); -return __ext4_journalled_writepage (page, wbc, len); +return __ext4_journalled_writepage (page, len); }if (test_opt (inode->i_sb,NOBH ) &&ext4_should_writeback_data (inode))
PATCH 24/45 writeback: remove global nr_to_write and use timeout instead
In reply to Wu FengguangThe global wbc.timeout could be useful when we decide to do two sync scans on dirty pages and dirty metadata. XFS could say: please return to sync dirty metadata after 10s. Would need another b_io_metadata queue, but that's possible.
--- fs/fs-writeback.c | 6 ++++++ include/linux/writeback.h | 1 + mm/backing-dev.c | 1 + 3 files changed, 8 insertions(+)
fs/fs-writeback.c +6 -0
@@ -666,6 +666,10 @@ static void writeback_inodes_wb(struct b {struct super_block *sb = wbc->sb, *pin_sb =NULL ;const unsigned long start = jiffies;/* livelock avoidance */ +unsigned long stop_time =0 ; + +if (wbc->timeout) + stop_time = (start + wbc->timeout) |1 ;spin_lock (&inode_lock);
@@ -723,6 +727,8 @@ static void writeback_inodes_wb(struct b }if (!list_empty (&wb->b_more_io)) wbc->more_io =1 ; +if (stop_time &&time_after (jiffies, stop_time)) +break ; }unpin_sb_for_writeback (&pin_sb);
mm/backing-dev.c +1 -0
@@ -339,6 +339,7 @@ static void bdi_flush_io(struct backing_ .older_than_this = NULL, .range_cyclic = 1, .nr_to_write = 1024, + .timeout = HZ, };writeback_inodes_wbc (&wbc);
include/linux/writeback.h +1 -0
@@ -42,6 +42,7 @@ struct writeback_control {struct super_block *sb;/* if !NULL, only write inodes from this super_block */ enum writeback_sync_modes sync_mode; +int timeout;unsigned long *older_than_this;/* If !NULL, only write back inodes older than this */ long nr_to_write;/* Write this many pages, and decrement
PATCH 42/45 writeback: requeue_io_wait() on blocked inode
In reply to Wu FengguangUse requeue_io_wait() if inode is somehow blocked. This includes the wrapped around range_cyclic case.
--- fs/fs-writeback.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fs-writeback.c +1 -1
@@ -591,7 +591,7 @@ writeback_single_inode(struct inode *ino/* * somehow blocked: retry later */ -redirty_tail (inode); +requeue_io_wait (inode); } }else if (inode->i_state &I_DIRTY ) {/*
PATCH 30/45 vmscan: lumpy pageout
In reply to Wu FengguangWhen pageout a dirty page, try to piggy back more consecutive dirty pages (up to 512KB) to improve IO efficiency.
Only ext3/reiserfs which don't have its own aops->writepages are supported in this initial version.
--- mm/page-writeback.c | 12 ++++++++++++ mm/vmscan.c | 11 +++++++++++ 2 files changed, 23 insertions(+)
mm/vmscan.c +11 -0
@@ -344,6 +344,8 @@ typedef enum { PAGE_CLEAN, } pageout_t; +#define LUMPY_PAGEOUT_PAGES (512 * 1024 / PAGE_CACHE_SIZE) +/* * pageout is called by shrink_page_list() for each dirty page. * Calls ->writepage().
@@ -409,6 +411,15 @@ static pageout_t pageout(struct page *pa }/* +* only write_cache_pages() supports for_reclaim for now +*/ +if (!mapping->a_ops->writepages) { + wbc.range_start = (page->index +1 ) <<PAGE_CACHE_SHIFT ; + wbc.nr_to_write =LUMPY_PAGEOUT_PAGES -1 ; +generic_writepages (mapping, &wbc); + } + +/* * Wait on writeback if requested to. This happens when * direct reclaiming a large contiguous area and the * first attempt to free a range of pages fails.
mm/page-writeback.c +12 -0
@@ -805,6 +805,11 @@ int write_cache_pages(struct address_spabreak ; } +if (wbc->for_reclaim && done_index != page->index) { + done =1 ; +break ; + } +if (nr_to_write != wbc->nr_to_write && done_index +WB_SEGMENT_DIST < page->index && --wbc->nr_segments <=0 ) {
@@ -846,6 +851,13 @@ continue_unlock:if (!clear_page_dirty_for_io (page)) goto continue_unlock; +/* +* active and unevictable pages will be checked at +* rotate time +*/ +if (wbc->for_reclaim) +SetPageReclaim (page); + ret = (*writepage)(page, wbc, data);if (unlikely (ret)) {if (ret ==AOP_WRITEPAGE_ACTIVATE ) {
PATCH 14/45 writeback: quit on wrap for .range_cyclic (afs)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- fs/afs/write.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
fs/afs/write.c +3 -4
@@ -477,8 +477,10 @@ static int afs_writepages_region(struct do { n = find_get_pages_tag(mapping, &index, PAGECACHE_TAG_DIRTY, 1, &page); - if (!n) + if (!n) { + index = 0; break; + } _debug("wback %lx", page->index);
@@ -551,9 +553,6 @@ int afs_writepages(struct address_space start = mapping->writeback_index; end =-1 ; ret =afs_writepages_region (mapping, wbc, start, end, &next); -if (start >0 && wbc->nr_to_write >0 && ret ==0 ) - ret =afs_writepages_region (mapping, wbc,0 , start, - &next); mapping->writeback_index = next; }else if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) { end = (pgoff_t)(LLONG_MAX >>PAGE_CACHE_SHIFT );
PATCH 16/45 writeback: merge for_kupdate and !for_kupdate cases
In reply to Wu FengguangUnify the logic for kupdate and non-kupdate cases.
There won't be starvation because the inodes requeued into b_more_io will later be spliced _after_ the remaining inodes in b_io, hence won't stand in the way of other inodes in the next run.
It avoids unnecessary redirty_tail() calls, hence the update of i_dirtied_when. The timestamp update is undesirable because it could later delay the inode's period writeback, or exclude the inode from the data integrity sync operation (which will check timestamp to avoid extra work and livelock).
--- fs/fs-writeback.c | 39 ++++++--------------------------------- 1 file changed, 6 insertions(+), 33 deletions(-)
fs/fs-writeback.c +6 -33
@@ -485,45 +485,18 @@ writeback_single_inode(struct inode *inoif (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/* * We didn't write back all the pages. nfs_writepages() -* sometimes bales out without doing anything. Redirty -* the inode; Move it from b_io onto b_more_io/b_dirty. +* sometimes bales out without doing anything. */ -/* -* akpm: if the caller was the kupdate function we put -* this inode at the head of b_dirty so it gets first -* consideration. Otherwise, move it to the tail, for -* the reasons described there. I'm not really sure -* how much sense this makes. Presumably I had a good -* reasons for doing it this way, and I'd rather not -* muck with it at present. -*/ -if (wbc->for_kupdate) { + inode->i_state |=I_DIRTY_PAGES ; +if (wbc->nr_to_write <=0 ) {/* -* For the kupdate function we move the inode -* to b_more_io so it will get more writeout as -* soon as the queue becomes uncongested. +* slice used up: queue for next turn */ - inode->i_state |=I_DIRTY_PAGES ; -if (wbc->nr_to_write <=0 ) { -/* -* slice used up: queue for next turn -*/ -requeue_io (inode); - }else { -/* -* somehow blocked: retry later -*/ -redirty_tail (inode); - } +requeue_io (inode); }else {/* -* Otherwise fully redirty the inode so that -* other inodes on this superblock will get some -* writeout. Otherwise heavy writing to one -* file would indefinitely suspend writeout of -* all the other files. +* somehow blocked: retry later */ - inode->i_state |=I_DIRTY_PAGES ;redirty_tail (inode); } }else if (inode->i_state &I_DIRTY ) {
PATCH 28/45 writeback: introduce wbc.nr_segments
In reply to Wu Fengguangwbc.nr_segments serves two major purposes:
- fairness between two large files, one is continuously dirtied, another is sparsely dirtied. Given the same amount of dirty pages, it could take vastly different time to sync them to the _same_ device. The nr_segments check helps to favor continuous data.
- avoid seeks/fragmentations. To give each file fair chance of writeback, we have to abort a file when some nr_to_write or timeout is reached. However they are both not good abort conditions. The best is for filesystem to abort earlier in seek boundaries, and treat nr_to_write/timeout as large enough bottom lines.
However a low nr_segments would be inefficient if all files are sparsely dirtied. For example, it may be inefficient for the block device inodes, which has lots of sparsely distributed metadata pages.
The wbc.nr_segments here is determined purely by logical page index
distance: if two pages are 1MB apart, it makes a new segment.
Filesystems could do this better with real extent knowledges. One possible scheme is to record the previous page index in wbc.writeback_index, and let ->writepage compare if the current and previous pages lie in the same extent, and decrease wbc.nr_segments accordingly. Care should taken to avoid double decreases in writepage and write_cache_pages.
--- fs/fs-writeback.c | 8 +++++++- fs/jbd2/commit.c | 1 + include/linux/writeback.h | 10 +++++++++- mm/filemap.c | 1 + mm/page-writeback.c | 7 +++++++ 5 files changed, 25 insertions(+), 2 deletions(-)
fs/fs-writeback.c +7 -1
@@ -542,6 +542,11 @@ writeback_single_inode(struct inode *inospin_unlock (&inode_lock); +if (wbc->for_kupdate || wbc->for_background) + wbc->nr_segments =bdi_nonrot (wbc->bdi) ?100 :10 ; +else + wbc->nr_segments =LONG_MAX ; + ret =do_writepages (mapping, wbc);/* Don't write the inode if only I_DIRTY_PAGES was set */
@@ -566,7 +571,8 @@ writeback_single_inode(struct inode *ino* sometimes bales out without doing anything. */ inode->i_state |=I_DIRTY_PAGES ; -if (wbc->nr_to_write <=0 ) { +if (wbc->nr_to_write <=0 || + wbc->nr_segments <=0 ) {/* * slice used up: queue for next turn */
include/linux/writeback.h +9 -1
@@ -48,6 +48,9 @@ struct writeback_control {long nr_to_write;/* Max pages to write per file, and decrement this for each page written */ +long nr_segments;/* Max page segments to write per file, +this is a count down value, too +*/ long pages_skipped;/* Pages which were not written */ /*
@@ -77,8 +80,13 @@ struct writeback_control { };/* +* if two page ranges are more than 1MB apart, they are taken as two segments. +*/ +#define WB_SEGMENT_DIST (1024 >> (PAGE_CACHE_SHIFT - 10)) + +/* * fs/fs-writeback.c -*/ +*/ struct bdi_writeback ;int inode_wait (void *);void writeback_inodes_sb (struct super_block *);
mm/filemap.c +1 -0
@@ -216,6 +216,7 @@ int __filemap_fdatawrite_range(struct adstruct writeback_control wbc = { .sync_mode = sync_mode, .nr_to_write =LONG_MAX , + .nr_segments =LONG_MAX , .range_start = start, .range_end = end, };
mm/page-writeback.c +7 -0
@@ -805,6 +805,13 @@ int write_cache_pages(struct address_spabreak ; } +if (nr_to_write != wbc->nr_to_write && + done_index +WB_SEGMENT_DIST < page->index && + --wbc->nr_segments <=0 ) { + done =1 ; +break ; + } + done_index = page->index +1 ;lock_page (page);
fs/jbd2/commit.c +1 -0
@@ -219,6 +219,7 @@ static int journal_submit_inode_data_bufstruct writeback_control wbc = { .sync_mode =WB_SYNC_ALL , .nr_to_write = mapping->nrpages *2 , + .nr_segments =LONG_MAX , .range_start =0 , .range_end =i_size_read (mapping->host), };
PATCH 29/45 writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case
In reply to Wu FengguangWhen shmem returns AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. So write_cache_pages shall stop writting this inode, and shmem shall increase pages_skipped to instruct VFS not to busy retry.
---
--- mm/page-writeback.c | 23 +++++++++++------------ mm/shmem.c | 1 + 2 files changed, 12 insertions(+), 12 deletions(-)
mm/page-writeback.c +11 -12
@@ -851,19 +851,18 @@ continue_unlock:if (ret ==AOP_WRITEPAGE_ACTIVATE ) {unlock_page (page); ret =0 ; - }else { -/* -* done_index is set past this page, -* so media errors will not choke -* background writeout for the entire -* file. This has consequences for -* range_cyclic semantics (ie. it may -* not be suitable for data integrity -* writeout). -*/ - done =1 ; -break ; } +/* +* done_index is set past this page, +* so media errors will not choke +* background writeout for the entire +* file. This has consequences for +* range_cyclic semantics (ie. it may +* not be suitable for data integrity +* writeout). +*/ + done =1 ; +break ; }if (nr_to_write >0 ) {
mm/shmem.c +1 -0
@@ -1103,6 +1103,7 @@ unlock:*/ swapcache_free (swap,NULL ); redirty: + wbc->pages_skipped++;set_page_dirty (page);if (wbc->for_reclaim)return AOP_WRITEPAGE_ACTIVATE ;/* Return with page locked */
PATCH 2/45 writeback: reduce calculation of bdi dirty thresholds
In reply to Wu FengguangSplit get_dirty_limits() into global_dirty_thresh() and bdi_dirty_thresh(), so that the latter can be avoided in balance_dirty_pages() when under global dirty threshold.
--- fs/fs-writeback.c | 2 include/linux/writeback.h | 5 +- mm/backing-dev.c | 3 - mm/page-writeback.c | 74 ++++++++++++++++++------------------ 4 files changed, 43 insertions(+), 41 deletions(-)
mm/page-writeback.c +37 -37
@@ -266,10 +266,11 @@ static inline void task_dirties_fraction* * dirty -= (dirty/8) * p_{t} */ -static void task_dirty_limit (struct task_struct *tsk,unsigned long *pdirty) +static unsigned long task_dirty_thresh (struct task_struct *tsk, +unsigned long bdi_dirty) {long numerator, denominator; -unsigned long dirty = *pdirty; +unsigned long dirty = bdi_dirty;u64 inv = dirty >>3 ;task_dirties_fraction (tsk, &numerator, &denominator);
@@ -277,10 +278,8 @@ static void task_dirty_limit(struct taskdo_div (inv, denominator); dirty -= inv; -if (dirty < *pdirty/2 ) - dirty = *pdirty/2 ; - *pdirty = dirty; +return max (dirty, bdi_dirty/2 ); }/*
@@ -390,9 +389,7 @@ unsigned long determine_dirtyable_memoryreturn x +1 ;/* Ensure that we never return 0 */ } -void -get_dirty_limits (unsigned long *pbackground,unsigned long *pdirty, -unsigned long *pbdi_dirty,struct backing_dev_info *bdi) +void global_dirty_thresh (unsigned long *pbackground,unsigned long *pdirty) {unsigned long background;unsigned long dirty;
@@ -424,26 +421,28 @@ get_dirty_limits(unsigned long *pbackgro } *pbackground = background; *pdirty = dirty; +} -if (bdi) { -u64 bdi_dirty; -long numerator, denominator; +unsigned long bdi_dirty_thresh (struct backing_dev_info *bdi, +unsigned long dirty) +{ +u64 bdi_dirty; +long numerator, denominator; -/* -* Calculate this BDI's share of the dirty ratio. -*/ -bdi_writeout_fraction (bdi, &numerator, &denominator); +/* +* Calculate this BDI's share of the dirty ratio. +*/ +bdi_writeout_fraction (bdi, &numerator, &denominator); - bdi_dirty = (dirty * (100 - bdi_min_ratio)) /100 ; - bdi_dirty *= numerator; -do_div (bdi_dirty, denominator); - bdi_dirty += (dirty * bdi->min_ratio) /100 ; -if (bdi_dirty > (dirty * bdi->max_ratio) /100 ) - bdi_dirty = dirty * bdi->max_ratio /100 ; + bdi_dirty = (dirty * (100 - bdi_min_ratio)) /100 ; + bdi_dirty *= numerator; +do_div (bdi_dirty, denominator); - *pbdi_dirty = bdi_dirty; -task_dirty_limit (current, pbdi_dirty); - } + bdi_dirty += (dirty * bdi->min_ratio) /100 ; +if (bdi_dirty > (dirty * bdi->max_ratio) /100 ) + bdi_dirty = dirty * bdi->max_ratio /100 ; + +return task_dirty_thresh (current, bdi_dirty); }/*
@@ -475,14 +474,24 @@ static void balance_dirty_pages(struct a .range_cyclic = 1, }; -get_dirty_limits (&background_thresh, &dirty_thresh, - &bdi_thresh, bdi); - nr_reclaimable =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ); nr_writeback =global_page_state (NR_WRITEBACK ) +global_page_state (NR_WRITEBACK_TEMP ); +global_dirty_thresh (&background_thresh, &dirty_thresh); + +/* +* Throttle it only when the background writeback cannot +* catch-up. This avoids (excessively) small writeouts +* when the bdi limits are ramping up. +*/ +if (nr_reclaimable + nr_writeback < + (background_thresh + dirty_thresh) /2 ) +break ; + + bdi_thresh =bdi_dirty_thresh (bdi, dirty_thresh); +/* * In order to avoid the stacked BDI deadlock we need * to ensure we accurately count the 'dirty' pages when
@@ -508,15 +517,6 @@ static void balance_dirty_pages(struct aif (!dirty_exceeded)break ; -/* -* Throttle it only when the background writeback cannot -* catch-up. This avoids (excessively) small writeouts -* when the bdi limits are ramping up. -*/ -if (nr_reclaimable + nr_writeback < - (background_thresh + dirty_thresh) /2 ) -break ; -if (!bdi->dirty_exceeded) bdi->dirty_exceeded =1 ;
@@ -626,7 +626,7 @@ void throttle_vm_writeout(gfp_t gfp_maskunsigned long dirty_thresh;for ( ; ; ) { -get_dirty_limits (&background_thresh, &dirty_thresh,NULL ,NULL ); +global_dirty_thresh (&background_thresh, &dirty_thresh);/* * Boost the allowable dirty threshold a bit for page
fs/fs-writeback.c +1 -1
@@ -729,7 +729,7 @@ static inline bool over_bground_thresh(v {unsigned long background_thresh, dirty_thresh; -get_dirty_limits (&background_thresh, &dirty_thresh,NULL ,NULL ); +global_dirty_thresh (&background_thresh, &dirty_thresh);return (global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) >= background_thresh);
mm/backing-dev.c +2 -1
@@ -83,7 +83,8 @@ static int bdi_debug_stats_show(struct s }spin_unlock (&inode_lock); -get_dirty_limits (&background_thresh, &dirty_thresh, &bdi_thresh, bdi); +global_dirty_thresh (&background_thresh, &dirty_thresh); + bdi_thresh =bdi_dirty_thresh (bdi, dirty_thresh);#define K (x) ((x) << (PAGE_SHIFT - 10)) seq_printf(m,
include/linux/writeback.h +3 -2
@@ -126,8 +126,9 @@ struct ctl_table;int dirty_writeback_centisecs_handler (struct ctl_table *,int ,void __user *,size_t *,loff_t *); -void get_dirty_limits (unsigned long *pbackground,unsigned long *pdirty, -unsigned long *pbdi_dirty,struct backing_dev_info *bdi); +void global_dirty_thresh (unsigned long *pbackground,unsigned long *pdirty); +unsigned long bdi_dirty_thresh (struct backing_dev_info *bdi, +unsigned long dirty);void page_writeback_init (void ); void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
PATCH 10/45 writeback: quit on wrap for .range_cyclic (btrfs)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs writeback_single_inode() to delay it for a while if necessary.
It removes one inefficient .range_cyclic IO pattern when writeback_index wraps:
submit [10000-10100], (wrap), submit [0-100]
In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
--- fs/btrfs/extent_io.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
fs/btrfs/extent_io.c +6 -15
@@ -2402,10 +2402,9 @@ static int extent_write_cache_pages(struint done =0 ;int nr_to_write_done =0 ;struct pagevec pvec; -int nr_pages; +int nr_pages =1 ;pgoff_t index;pgoff_t end;/* Inclusive */ -int scanned =0 ;int range_whole =0 ;pagevec_init (&pvec,0 );
@@ -2417,16 +2416,14 @@ static int extent_write_cache_pages(stru end = wbc->range_end >>PAGE_CACHE_SHIFT ;if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) range_whole =1 ; - scanned =1 ; } -retry: +while (!done && !nr_to_write_done && (index <= end) && (nr_pages =pagevec_lookup_tag (&pvec, mapping, &index,PAGECACHE_TAG_DIRTY ,min (end - index, (pgoff_t )PAGEVEC_SIZE -1 ) +1 ))) {unsigned i; - scanned =1 ;for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
@@ -2447,7 +2444,7 @@ retry:continue ; } -if (!wbc->range_cyclic && page->index > end) { +if (page->index > end) { done =1 ;unlock_page (page);continue ;
@@ -2484,15 +2481,9 @@ retry:pagevec_release (&pvec);cond_resched (); } -if (!scanned && !done) { -/* -* We hit the last page and there is more work to be done: wrap -* back to the start of the file -*/ - scanned =1 ; - index =0 ; - goto retry; - } +if (!nr_pages) + mapping->writeback_index =0 ; +return ret; }
PATCH 33/45 writeback: sync livelock - introduce wbc.for_sync
In reply to Wu FengguangThe sync() is performed in two stages: the WB_SYNC_NONE sync and the WB_SYNC_ALL sync. It is necessary to tag both stages with wbc.for_sync, so as to prevent either of them being livelocked.
The next patch will utilize this flag to do the livelock prevention.
--- fs/fs-writeback.c | 30 +++++++++--------------------- include/linux/backing-dev.h | 2 +- include/linux/writeback.h | 11 +++++++++++ mm/page-writeback.c | 2 +- 4 files changed, 22 insertions(+), 23 deletions(-)
fs/fs-writeback.c +9 -21
@@ -42,6 +42,7 @@ struct wb_writeback_args {long nr_pages;struct super_block *sb;enum writeback_sync_modes sync_mode; + int for_sync:1; int for_kupdate:1; int range_cyclic:1; int for_background:1;
@@ -221,6 +222,7 @@ static void bdi_sync_writeback(struct bastruct wb_writeback_args args = { .sb = sb, .sync_mode =WB_SYNC_ALL , + .for_sync =1 , .nr_pages =LONG_MAX , .range_cyclic =0 , };
@@ -236,7 +238,6 @@ static void bdi_sync_writeback(struct ba/** * bdi_start_writeback - start writeback * @bdi: the backing device to write from -* @nr_pages: the number of pages to write * * Description: * This does WB_SYNC_NONE opportunistic writeback. The IO is only
@@ -245,24 +246,17 @@ static void bdi_sync_writeback(struct ba* */ void bdi_start_writeback (struct backing_dev_info *bdi,struct super_block *sb, -long nr_pages) +long mission) {struct wb_writeback_args args = { .sb = sb, .sync_mode =WB_SYNC_NONE , - .nr_pages = nr_pages, + .nr_pages =LONG_MAX , + .for_background = mission ==WB_FOR_BACKGROUND , + .for_sync = mission ==WB_FOR_SYNC , .range_cyclic =1 , }; -/* -* We treat @nr_pages=0 as the special case to do background writeback, -* ie. to sync pages until the background dirty threshold is reached. -*/ -if (!nr_pages) { - args.nr_pages =LONG_MAX ; - args.for_background =1 ; - } -bdi_alloc_queue_work (bdi, &args); }
@@ -310,7 +304,7 @@ void bdi_writeback_wait(struct backing_d* make sure we will be woke up by someone */ if (can_submit_background_writeback (bdi)) -bdi_start_writeback (bdi,NULL ,0 ); +bdi_start_writeback (bdi,NULL ,WB_FOR_BACKGROUND );wait_for_completion (&tt.complete); }
@@ -790,6 +784,7 @@ static long wb_writeback(struct bdi_writ .older_than_this = NULL, .for_kupdate = args->for_kupdate, .for_background = args->for_background, + .for_sync = args->for_sync, .range_cyclic = args->range_cyclic, };unsigned long oldest_jif;
@@ -1250,14 +1245,7 @@ static void wait_sb_inodes(struct super_*/ void writeback_inodes_sb (struct super_block *sb) { -unsigned long nr_dirty =global_page_state (NR_FILE_DIRTY ); -unsigned long nr_unstable =global_page_state (NR_UNSTABLE_NFS ); -long nr_to_write; - - nr_to_write = nr_dirty + nr_unstable + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); - -bdi_start_writeback (sb->s_bdi, sb, nr_to_write); +bdi_start_writeback (sb->s_bdi, sb,WB_FOR_SYNC ); }EXPORT_SYMBOL (writeback_inodes_sb);
include/linux/backing-dev.h +1 -1
@@ -123,7 +123,7 @@ int bdi_register(struct backing_dev_infoint bdi_register_dev (struct backing_dev_info *bdi,dev_t dev);void bdi_unregister (struct backing_dev_info *bdi);void bdi_start_writeback (struct backing_dev_info *bdi,struct super_block *sb, -long nr_pages); +long mission);int bdi_writeback_task (struct bdi_writeback *wb);int bdi_has_dirty_io (struct backing_dev_info *bdi);int bdi_writeback_wakeup (struct backing_dev_info *bdi);
include/linux/writeback.h +11 -0
@@ -31,6 +31,16 @@ enum writeback_sync_modes { WB_SYNC_ALL, /* Wait on every mapping */ }; +enum writeback_mission { +WB_FOR_KUPDATE ,/* writeback expired dirty inodes */ +WB_FOR_RECLAIM , +WB_FOR_BACKGROUND ,/* stop on hitting background threshold */ +WB_FOR_SYNC ,/* write all now-dirty inodes/pages, +* but take care not to live lock +*/ +WB_NR_PAGES ,/* writeback # of pages if larger than this */ +}; +/* * A control structure which tells the writeback code what to do. These are * always on the stack, and hence need no locking. They are always initialised
@@ -65,6 +75,7 @@ struct writeback_control { unsigned encountered_congestion:1; /* An output: a queue is full */ unsigned for_kupdate:1; /* A kupdate writeback */ unsigned for_background:1; /* A background writeback */ + unsigned for_sync:1; /* A writeback for sync */ unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned range_cyclic:1; /* range_start is cyclic */ unsigned more_io:1; /* more io to be dispatched */
mm/page-writeback.c +1 -1
@@ -529,7 +529,7 @@ out:*/ if (!laptop_mode && (nr_reclaimable > background_thresh) &&can_submit_background_writeback (bdi)) -bdi_start_writeback (bdi,NULL ,0 ); +bdi_start_writeback (bdi,NULL ,WB_FOR_BACKGROUND ); }void set_page_dirty_balance (struct page *page,int page_mkwrite)
PATCH 35/45 writeback: sync livelock - use single timestamp for whole sync work
In reply to Wu FengguangThe start time in writeback_inodes_wb() is not very useful because it slips at each invocation time. We shall use one _constant_ time at the beginning to cover this whole sync() work.
The timestamp is now grabbed at work start time. It could be better set at the sync work submission time.
--- fs/fs-writeback.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
fs/fs-writeback.c +8 -9
@@ -669,12 +669,11 @@ static long writeback_inodes_wb(struct b struct writeback_control *wbc) {struct super_block *sb = wbc->sb, *pin_sb =NULL ; -const unsigned long start = jiffies;/* livelock avoidance */ unsigned long stop_time =0 ;unsigned long wrote =0 ;if (wbc->timeout) - stop_time = (start + wbc->timeout) |1 ; + stop_time = (jiffies + wbc->timeout) |1 ;spin_lock (&inode_lock);
@@ -699,13 +698,6 @@ static long writeback_inodes_wb(struct bcontinue ; } -/* -* Was this inode dirtied after sync_sb_inodes was called? -* This keeps sync from extra jobs and livelock. -*/ -if (inode_dirtied_after (inode, start)) -break ; -if (pin_sb_for_writeback (wbc, inode, &pin_sb)) {requeue_io (inode);continue ;
@@ -798,6 +790,13 @@ static long wb_writeback(struct bdi_writlong nr;struct inode *inode; +/* +* keep sync from extra jobs and livelock +*/ +if (wbc.for_sync) { + wbc.older_than_this = &oldest_jif; + oldest_jif = jiffies; + }if (wbc.for_kupdate || wbc.for_background) { wbc.older_than_this = &oldest_jif; oldest_jif = jiffies - expire_interval;
PATCH 36/45 writeback: sync livelock - curb dirty speed for inodes to be synced
In reply to Wu FengguangThe inodes to be synced will be dirty throttled regardless of the dirty threshold. This stops sync() livelock by some fast dirtier.
--- fs/fs-writeback.c | 4 ++++ include/linux/backing-dev.h | 1 + mm/page-writeback.c | 13 +++++++++++++ 3 files changed, 18 insertions(+)
mm/page-writeback.c +13 -0
@@ -478,6 +478,18 @@ static void balance_dirty_pages(struct aglobal_dirty_thresh (&background_thresh, &dirty_thresh);/* +* If sync() is in progress, curb the to-be-synced inodes regardless +* of dirty limits, so that a fast dirtier won't livelock the sync. +*/ +if (unlikely (bdi->sync_time && +S_ISREG (mapping->host->i_mode) && +time_after_eq (bdi->sync_time, + mapping->host->dirtied_when))) { + write_chunk *=2 ; + goto throttle; + } + +/* * Throttle it only when the background writeback cannot * catch-up. This skips the ramp up phase of bdi limits. */
@@ -510,6 +522,7 @@ static void balance_dirty_pages(struct aif (!dirty_exceeded) goto out; +throttle:if (!bdi->dirty_exceeded) bdi->dirty_exceeded =1 ;
fs/fs-writeback.c +4 -0
@@ -796,6 +796,7 @@ static long wb_writeback(struct bdi_writif (wbc.for_sync) { wbc.older_than_this = &oldest_jif; oldest_jif = jiffies; + wbc.bdi->sync_time = oldest_jif |1 ; }if (wbc.for_kupdate || wbc.for_background) { wbc.older_than_this = &oldest_jif;
@@ -873,6 +874,9 @@ static long wb_writeback(struct bdi_writspin_unlock (&inode_lock); } +if (args->for_sync) + wb->bdi->sync_time =0 ; +if (args->for_background)while (bdi_writeback_wakeup (wb->bdi)) ;/* unthrottle all tasks */
include/linux/backing-dev.h +1 -0
@@ -89,6 +89,7 @@ struct backing_dev_info {* to prevent livelocking the sync works */ unsigned int sync_works; +unsigned long sync_time;struct list_head work_list;
PATCH 1/45 writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Wu FengguangFrom: Richard Kennedy <[email protected]>
Reducing the number of times balance_dirty_pages calls global_page_state reduces the cache references and so improves write performance on a variety of workloads.
'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_threshold approx 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviation
average (s.d.) in millions (10^6)
2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)
Achieving this reduction is by dropping clip_bdi_dirty_limit as it rereads the counters to apply the dirty_threshold and moving this check up into balance_dirty_pages where it has already read the counters.
Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.
In the common case with no throttling it now calls global_page_state 5 fewer times and bdi_stat 2 fewer.
----
Thanks to everybody for the feedback & suggestions. This patch is against 2.6.31-rc8
--- mm/page-writeback.c | 99 ++++++++++++++---------------------------- 1 file changed, 33 insertions(+), 66 deletions(-)
mm/page-writeback.c +33 -66
@@ -252,32 +252,6 @@ static void bdi_writeout_fraction(struct } } -/* -* Clip the earned share of dirty pages to that which is actually available. -* This avoids exceeding the total dirty_limit when the floating averages -* fluctuate too quickly. -*/ -static void clip_bdi_dirty_limit (struct backing_dev_info *bdi, -unsigned long dirty,unsigned long *pbdi_dirty) -{ -unsigned long avail_dirty; - - avail_dirty =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_WRITEBACK ) + -global_page_state (NR_UNSTABLE_NFS ) + -global_page_state (NR_WRITEBACK_TEMP ); - -if (avail_dirty < dirty) - avail_dirty = dirty - avail_dirty; -else - avail_dirty =0 ; - - avail_dirty +=bdi_stat (bdi,BDI_RECLAIMABLE ) + -bdi_stat (bdi,BDI_WRITEBACK ); - - *pbdi_dirty =min (*pbdi_dirty, avail_dirty); -} -static inline void task_dirties_fraction (struct task_struct *tsk,long *numerator,long *denominator) {
@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro bdi_dirty = dirty * bdi->max_ratio /100 ; *pbdi_dirty = bdi_dirty; -clip_bdi_dirty_limit (bdi, dirty, pbdi_dirty);task_dirty_limit (current, pbdi_dirty); } }
@@ -490,7 +463,7 @@ static void balance_dirty_pages(struct aunsigned long bdi_thresh;unsigned long pages_written =0 ;unsigned long pause =1 ; - +int dirty_exceeded;struct backing_dev_info *bdi = mapping->backing_dev_info;for (;;) {
@@ -503,16 +476,36 @@ static void balance_dirty_pages(struct a };get_dirty_limits (&background_thresh, &dirty_thresh, - &bdi_thresh, bdi); + &bdi_thresh, bdi); nr_reclaimable =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_UNSTABLE_NFS ); - nr_writeback =global_page_state (NR_WRITEBACK ); +global_page_state (NR_UNSTABLE_NFS ); + nr_writeback =global_page_state (NR_WRITEBACK ) + +global_page_state (NR_WRITEBACK_TEMP ); - bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); +/* +* In order to avoid the stacked BDI deadlock we need +* to ensure we accurately count the 'dirty' pages when +* the threshold is low. +* +* Otherwise it would be possible to get thresh+n pages +* reported dirty, even though there are thresh-m pages +* actually dirty; with m+n sitting in the percpu +* deltas. +*/ +if (bdi_thresh <2 *bdi_stat_error (bdi)) { + bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); + bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); + }else { + bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); + bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); + } + + dirty_exceeded = + (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) + || (nr_reclaimable + nr_writeback >= dirty_thresh); -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh) +if (!dirty_exceeded)break ;/*
@@ -521,7 +514,7 @@ static void balance_dirty_pages(struct a* when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < - (background_thresh + dirty_thresh) /2 ) + (background_thresh + dirty_thresh) /2 )break ;if (!bdi->dirty_exceeded)
@@ -539,33 +532,10 @@ static void balance_dirty_pages(struct aif (bdi_nr_reclaimable > bdi_thresh) {writeback_inodes_wbc (&wbc); pages_written += write_chunk - wbc.nr_to_write; -get_dirty_limits (&background_thresh, &dirty_thresh, - &bdi_thresh, bdi); - } - -/* -* In order to avoid the stacked BDI deadlock we need -* to ensure we accurately count the 'dirty' pages when -* the threshold is low. -* -* Otherwise it would be possible to get thresh+n pages -* reported dirty, even though there are thresh-m pages -* actually dirty; with m+n sitting in the percpu -* deltas. -*/ -if (bdi_thresh <2 *bdi_stat_error (bdi)) { - bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); - }else if (bdi_nr_reclaimable) { - bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); +/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh) -break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -schedule_timeout_interruptible (pause);/*
@@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; } -if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;if (writeback_in_progress (bdi))
@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a* background_thresh, to keep the amount of dirty memory low. */ if ((laptop_mode && pages_written) || - (!laptop_mode && ((global_page_state (NR_FILE_DIRTY ) - +global_page_state (NR_UNSTABLE_NFS )) - > background_thresh))) + (!laptop_mode && (nr_reclaimable > background_thresh)))bdi_start_writeback (bdi,NULL ,0 ); }
PATCH 8/45 writeback: quit on wrap for .range_cyclic (write_cache_pages)
In reply to Wu FengguangConvert wbc.range_cyclic to new behavior: when past EOF, abort the writeback of the current inode, which instructs writeback_single_inode() to delay it for a while if necessary.
This is the right behavior for
- sync writeback (is already so with range_whole)
we have scanned the inode address space, and don't care any more newly dirtied pages. So shall update its i_dirtied_when and exclude it from the todo list.
- periodic writeback
any more newly dirtied pages may be delayed for a while. This also prevents pointless IO for busy overwriters.
- background writeback
irrelevant because it generally don't care the dirty timestamp.
That should get rid of one inefficient IO pattern of .range_cyclic when writeback_index wraps, in which the submitted pages may be consisted of two distant ranges: submit [10000-10100], (wrap), submit [0-100].
--- mm/page-writeback.c | 27 +++++---------------------- 1 file changed, 5 insertions(+), 22 deletions(-)
mm/page-writeback.c +5 -22
@@ -789,31 +789,23 @@ int write_cache_pages(struct address_spaint done =0 ;struct pagevec pvec;int nr_pages; -pgoff_t uninitialized_var (writeback_index );pgoff_t index;pgoff_t end;/* Inclusive */ pgoff_t done_index; -int cycled;int range_whole =0 ;long nr_to_write = wbc->nr_to_write;pagevec_init (&pvec,0 );if (wbc->range_cyclic) { - writeback_index = mapping->writeback_index;/* prev offset */ - index = writeback_index; -if (index ==0 ) - cycled =1 ; -else - cycled =0 ; + index = mapping->writeback_index;/* prev offset */ end =-1 ; }else { index = wbc->range_start >>PAGE_CACHE_SHIFT ; end = wbc->range_end >>PAGE_CACHE_SHIFT ;if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX ) range_whole =1 ; - cycled =1 ;/* ignore range_cyclic tests */ } -retry: + done_index = index;while (!done && (index <= end)) {int i;
@@ -821,8 +813,10 @@ retry: nr_pages =pagevec_lookup_tag (&pvec, mapping, &index,PAGECACHE_TAG_DIRTY ,min (end - index, (pgoff_t )PAGEVEC_SIZE -1 ) +1 ); -if (nr_pages ==0 ) +if (nr_pages ==0 ) { + done_index =0 ;break ; + }for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
@@ -919,17 +913,6 @@ continue_unlock:pagevec_release (&pvec);cond_resched (); } -if (!cycled && !done) { -/* -* range_cyclic: -* We hit the last page and there is more work to be done: wrap -* back to the start of the file -*/ - cycled =1 ; - index =0 ; - end = writeback_index -1 ; - goto retry; - }if (!wbc->no_nrwrite_index_update) {if (wbc->range_cyclic || (range_whole && nr_to_write >0 )) mapping->writeback_index = done_index;
PATCH 31/45 writeback: sync old inodes first in background writeback
In reply to Wu FengguangA background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.
--- fs/fs-writeback.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
fs/fs-writeback.c +17 -4
@@ -680,7 +680,7 @@ static long writeback_inodes_wb(struct bspin_lock (&inode_lock); -if (!wbc->for_kupdate ||list_empty (&wb->b_io)) +if (list_empty (&wb->b_io))queue_io (wb, wbc->older_than_this);while (!list_empty (&wb->b_io)) {
@@ -793,14 +793,15 @@ static long wb_writeback(struct bdi_writ .range_cyclic = args->range_cyclic, };unsigned long oldest_jif; +int expire_interval =msecs_to_jiffies (dirty_expire_interval *10 ); +int fg_rounds =0 ;long wrote =0 ;long nr;struct inode *inode; -if (wbc.for_kupdate) { +if (wbc.for_kupdate || wbc.for_background) { wbc.older_than_this = &oldest_jif; - oldest_jif = jiffies - -msecs_to_jiffies (dirty_expire_interval *10 ); + oldest_jif = jiffies - expire_interval; }if (!wbc.range_cyclic) { wbc.range_start =0 ;
@@ -828,6 +829,18 @@ static long wb_writeback(struct bdi_writ args->nr_pages -= nr; wrote += nr; +if (args->for_background && expire_interval && + ++fg_rounds &&list_empty (&wb->b_io)) { +if (fg_rounds <10 ) + expire_interval >>=1 ; +if (expire_interval) + oldest_jif = jiffies - expire_interval; +else + wbc.older_than_this =0 ; + fg_rounds =0 ; +continue ; + } +/* * Bail if no more IO */
PATCH 6/45 writeback: use larger ratelimit when dirty_exceeded
In reply to Wu FengguangWhen dirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.
Given the larger ratelimit value, we can safely ignore the low bound check in sync_writeback_pages.
dirty_exceeded is more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.
--- mm/page-writeback.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/page-writeback.c +5 -7
@@ -39,7 +39,8 @@* After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited * will look to see if it needs to force writeback or throttling. */ -static long ratelimit_pages =32 ; +#define MAX_RATELIMIT_PAGES ((4096 * 1024) / PAGE_CACHE_SIZE) +static long ratelimit_pages =MAX_RATELIMIT_PAGES ;/* * When balance_dirty_pages decides that the caller needs to perform some
@@ -49,9 +50,6 @@ static long ratelimit_pages = 32;*/ static inline long sync_writeback_pages (unsigned long dirtied) { -if (dirtied < ratelimit_pages) - dirtied = ratelimit_pages; -return dirtied + dirtied /2 ; }
@@ -600,7 +598,7 @@ void balance_dirty_pages_ratelimited_nr( ratelimit = ratelimit_pages;if (mapping->backing_dev_info->dirty_exceeded) - ratelimit =8 ; + ratelimit >>=3 ;/* * Check the rate limiting. Also, we do not want to throttle real-time
@@ -722,8 +720,8 @@ void writeback_set_ratelimit(void) ratelimit_pages = vm_total_pages / (num_online_cpus () *32 );if (ratelimit_pages <16 ) ratelimit_pages =16 ; -if (ratelimit_pages *PAGE_CACHE_SIZE >4096 *1024 ) - ratelimit_pages = (4096 *1024 ) /PAGE_CACHE_SIZE ; +if (ratelimit_pages >MAX_RATELIMIT_PAGES ) + ratelimit_pages =MAX_RATELIMIT_PAGES ; } static int __cpuinit
PATCH 15/45 writeback: fix queue_io() ordering
In reply to Wu FengguangThis was not a bug, since b_io is empty for kupdate writeback. The next patch will do requeue_io() for non-kupdate writeback, so let's fix it.
--- fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c +5 -2
@@ -375,11 +375,14 @@ static void move_expired_inodes(struct l }/* -* Queue all expired dirty inodes for io, eldest first. +* Queue all expired dirty inodes for io, eldest first: +* (newly dirtied) => b_dirty inodes +* => b_more_io inodes +* => remaining inodes in b_io => (dequeue for sync) */ static void queue_io (struct bdi_writeback *wb,unsigned long *older_than_this) { -list_splice_init (&wb->b_more_io, wb->b_io.prev); +list_splice_init (&wb->b_more_io, &wb->b_io);move_expired_inodes (&wb->b_dirty, &wb->b_io, older_than_this); }
PATCH 45/45 btrfs: fix race on syncing the btree inode
In reply to Wu FengguangWhen doing sync, the btree dirty pages refuse to go away for tens of seconds:
# vmmon -d 1 nr_writeback nr_dirty nr_unstable
nr_writeback nr_dirty nr_unstable
46641 23315 0
46641 23380 0
46641 23381 0
26674 43206 0
18963 51006 0
11252 58721 0
3528 66419 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
0 70024 0
Note that the 70024 pages are under the btree inode's 32MB no-write-metadata threshold. This is racy because the sync work has to sleep and retry it forever for data integrity.
The 32MB threshold may also become a problem for background writeback given a memory tight box. So it may be better to replace the threshold with some informed writeback tricks.
--- fs/btrfs/disk-io.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-)
fs/btrfs/disk-io.c +13 -16
@@ -707,22 +707,19 @@ static int btree_writepage(struct page *static int btree_writepages (struct address_space *mapping,struct writeback_control *wbc) { -struct extent_io_tree *tree; - tree = &BTRFS_I (mapping->host)->io_tree; -if (wbc->sync_mode ==WB_SYNC_NONE ) { -struct btrfs_root *root =BTRFS_I (mapping->host)->root; -u64 num_dirty; -unsigned long thresh =32 *1024 *1024 ; - -if (wbc->for_kupdate) -return 0 ; - -/* this is a bit racy, but that's ok */ - num_dirty = root->fs_info->dirty_metadata_bytes; -if (num_dirty < thresh) -return 0 ; - } -return extent_writepages (tree, mapping, btree_get_extent, wbc); +struct extent_io_tree *tree = &BTRFS_I (mapping->host)->io_tree; +int ret; + +if (!wbc->for_sync) + wbc->nr_segments =1 ; + ret =extent_writepages (tree, mapping, btree_get_extent, wbc); +/* +* Fake some some skipped pages, so that VFS won't +* try hard on writing this inode. +*/ +if (!wbc->for_sync) + wbc->pages_skipped++; +return ret; }static int btree_readpage (struct file *file,struct page *page)
Re: [PATCH 20/45] NFS: introduce writeback wait queue
In reply to Wu FengguangQuoted 19 lines · Wu Fengguang
plain text document attachment (writeback-nfs-request-queue.patch) The generic writeback routines are departing from
congestion_wait()in preferance ofget_request_wait(), aka. waiting on the block queues.Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.
In perticular,
balance_dirty_pages()will exit after it pusheswrite_chunkpages into thePG_writebackpage pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number ofPG_writebackpages if it is not explicitly limited.CC Jens Axboe <[email protected]>CC Chris Mason <[email protected]>CC Peter Zijlstra <[email protected]>CC Trond Myklebust <[email protected]>Signed-off-by Wu Fengguang <[email protected]>
Quoted 5 lines
---
The wait time and network throughput varies a lot! this is a major problem. This means
nfs_end_page_writeback()is not called smoothly over time, even when there are plenty ofPG_writebackpages on the client side.
Could that be some ack batching from the nfs protocol/server?
Quoted 12 lines
fs/nfs/write.c +11
-0+static void nfs_wakeup_congested (long nr,long limit, +struct backing_dev_info *bdi, +wait_queue_head_t *wqh) +{ +if (nr <2 *limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_sync_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_SYNC ); +if (waitqueue_active (&wqh[BLK_RW_SYNC ])) { +smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_SYNC ]); + }
fs/nfs/write.c static int wb_priority(struct writeback_ · line 237
Quoted 10 lines
fs/nfs/write.c +10
-0+ } +if (nr < limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_async_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_ASYNC ); +if (waitqueue_active (&wqh[BLK_RW_ASYNC ])) { +smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_ASYNC ]); + } + } +}
wakeup implies a full memory barrier.
Quoted 11 lines · Wu Fengguang
Hi all,
Here is a collection of writeback patches on
- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixes
Quoted 2 lines
Sorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..
Awesome bits..
Re: [PATCH 21/45] writeback: estimate bdi write bandwidth
In reply to Wu Fengguangfs/fs-writeback.c void bdi_start_writeback(struct backing_ · line 282
Quoted 9 lines · Wu Fengguang
fs/fs-writeback.c +9
-0+static void bdi_calc_write_bandwidth (struct backing_dev_info *bdi, +unsigned long nr_pages, +unsigned long time) +{ +unsigned long bw; + + bw =HZ * nr_pages / (time |1 ); + bdi->write_bandwidth = (bdi->write_bandwidth *63 + bw) /64 ; +}
If you have block times < 1 jiffy this all falls apart quite quickly. You could perhaps try to use cpu_clock() for ns resolution timestamps if this is a real issue.
(I could imagine fast arrays with huge throughput causing small sleeps, resulting in underestimates of their bandwidth)
Also, 63/64 seems rather slow progress.. maybe that's good.
Re: [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded
In reply to Wu FengguangQuoted 6 lines · Wu Fengguang
plain text document attachment (writeback-ratelimit-on-dirty-exceeded.patch)
Whendirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.
Quoted 6 lines
Given the larger ratelimit value, we can safely ignore the low bound check in
sync_writeback_pages.
dirty_exceededis more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.
Don't forget that ratelimit_pages is a per-cpu limit. So the total error on the dirty limit scales with the number of cpus.
Other than that, I guess this patch needs numbers ;-)
Re: [PATCH 20/45] NFS: introduce writeback wait queue
In reply to Peter ZijlstraQuoted 22 lines · Peter Zijlstra
Quoted 21 lines · Wu Fengguang
plain text document attachment (writeback-nfs-request-queue.patch) The generic writeback routines are departing from
congestion_wait()in preferance ofget_request_wait(), aka. waiting on the block queues.Introduce the missing writeback wait queue for NFS, otherwise its writeback pages may grow out of control.
In perticular,
balance_dirty_pages()will exit after it pusheswrite_chunkpages into thePG_writebackpage pool, _OR_ when the background writeback work quits. The latter is new behavior, and could not only quit (normally) after below background threshold, but also quit when it finds _zero_ dirty pages to write. The latter case gives rise to the number ofPG_writebackpages if it is not explicitly limited.CC Jens Axboe <[email protected]>CC Chris Mason <[email protected]>CC Peter Zijlstra <[email protected]>CC Trond Myklebust <[email protected]>Signed-off-by Wu Fengguang <[email protected]>---
Quoted 5 lines
Quoted 3 lines
The wait time and network throughput varies a lot! this is a major problem. This means
nfs_end_page_writeback()is not called smoothly over time, even when there are plenty ofPG_writebackpages on the client side.Could that be some ack batching from the nfs protocol/server?
Yes possibly. Another possibility is some works in the nfsiod workqueue takes up a long time.
Quoted 16 lines
Quoted 16 lines
fs/nfs/write.c +16
-0+static void nfs_wakeup_congested (long nr,long limit, +struct backing_dev_info *bdi, +wait_queue_head_t *wqh) +{ +if (nr <2 *limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_sync_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_SYNC ); +if (waitqueue_active (&wqh[BLK_RW_SYNC ])) { +smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_SYNC ]); + } + } +if (nr < limit -min (limit/8 ,NFS_WAIT_PAGES )) { +if (test_bit (BDI_async_congested, &bdi->state)) +clear_bdi_congested (bdi,BLK_RW_ASYNC ); +if (waitqueue_active (&wqh[BLK_RW_ASYNC ])) {
Quoted 8 lines
fs/nfs/write.c static int wb_priority(struct writeback_ · line 237
Quoted 6 lines
fs/nfs/write.c +5
-0+smp_mb__after_clear_bit (); +wake_up (&wqh[BLK_RW_ASYNC ]); + } + } +}wakeup implies a full memory barrier.
If so, this smp_mb__after_clear_bit() line is also not necessary?
void clear_bdi_congested(struct backing_dev_info *bdi, int sync) {
//...
clear_bit(bit, &bdi->state);
smp_mb__after_clear_bit();
if (waitqueue_active(wqh))
wake_up(wqh);Thanks,
Fengguang
Re: [PATCH 20/45] NFS: introduce writeback wait queue
Quoted 5 lines · Wu Fengguang
Quoted 2 lines
wakeup implies a full memory barrier.
If so, this
smp_mb__after_clear_bit()line is also not necessary?
Quoted 7 lines
void
clear_bdi_congested(struct backing_dev_info *bdi, int sync){
//...clear_bit(bit, &bdi->state); smp_mb__after_clear_bit(); if (waitqueue_active(wqh)) wake_up(wqh);
Depends on if the barrier is needed even when the wakeup doesn't happen I guess ;-)
Re: [PATCH 20/45] NFS: introduce writeback wait queue
In reply to Wu FengguangQuoted 5 lines · Wu Fengguang
Quoted 1 line · Peter Zijlstra
wakeup implies a full memory barrier.
If so, this
smp_mb__after_clear_bit()line is also not necessary?
Quoted 7 lines
void
clear_bdi_congested(struct backing_dev_info *bdi, int sync){
//...clear_bit(bit, &bdi->state); smp_mb__after_clear_bit(); if (waitqueue_active(wqh)) wake_up(wqh);
Typically in these patterns you do need a barrier. You need to ensure the load to check the waitqueue is performed after the clear_bit.
The other side does:add_to_waitqueueset_current_state /* has mb() in it */
if (test_bit())
schedule()
Re: [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded
In reply to Peter ZijlstraQuoted 10 lines · Peter Zijlstra
Quoted 9 lines · Wu Fengguang
plain text document attachment (writeback-ratelimit-on-dirty-exceeded.patch)
Whendirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to scale up to 512KB for memory bounty systems. This is more efficient than the original 8 pages, and won't risk exceeding the dirty limit too much.Given the larger ratelimit value, we can safely ignore the low bound check in
sync_writeback_pages.
Quoted 6 lines
Quoted 3 lines
dirty_exceededis more likely to be seen when there are multiple dirty processes. In which case the lowered ratelimit will help reduce their overall wait time (latency) in the throttled queue.Don't forget that
ratelimit_pagesis a per-cpu limit. So the total error on the dirty limit scales with the number of cpus.
Ah yes! Given that a typical NUMA configuration would be to equip each CPU with >1GB memory, 512KB is not a big error :)
Quoted 1 line
Other than that, I guess this patch needs numbers ;-)
Lots of numbers, hehe. Basically I see ~25ms wait time for ext2/3/4 and btrfs (with this patch and the dirty exceed timestamp patch).
One cp process:
[ 3404.554478] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.570407] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms [ 3404.601166] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.632872] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.663633] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3404.694198] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3404.724879] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.756655] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.787350] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.817950] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.848640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3404.880556] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 3404.911372] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3404.942006] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3404.972712] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.002672] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.034480] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.065114] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.095870] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.126453] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.158043] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.188801] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.219327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.249922] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.281869] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.312533] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.343194] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.373761] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.410828] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 3405.441993] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.473266] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.505746] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.537103] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.568327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 3405.600730] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.632039] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.663326] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.695674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.727031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.758146] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.790743] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 3405.822129] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.853427] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 3405.885889] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.917116] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 3405.948411] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
Two:
[ 9393.138529] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9393.171042] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9393.344709] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9393.377045] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9393.555611] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9393.586298] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9393.747218] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9393.777825] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9393.808456] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9393.964086] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.120223] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.273854] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9394.304842] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.334474] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.480031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.527769] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9394.557919] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.590690] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9394.621626] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.654424] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9394.678289] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9394.709837] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.742912] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.774576] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.806149] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.822272] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms [ 9394.852609] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.886215] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.915804] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9394.947044] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9394.979504] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9395.010944] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9395.042221] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9395.074640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms [ 9395.105938] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9395.137157] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9395.169668] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
Four:
[ 9435.829736] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9436.702953] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9436.719957] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9436.753598] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms [ 9436.770935] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9436.804367] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9436.818913] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9436.854302] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms [ 9436.870489] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9436.903449] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9443.140282] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9443.155674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms [ 9443.180418] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms [ 9443.697286] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms [ 9443.714321] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9443.748455] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9443.763764] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9443.796788] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms [ 9443.814309] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms [ 9446.493680] write_bandwidth: comm=btrfs-endio-wri pages=193 time=16ms [ 9446.528486] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms [ 9446.888774] write_bandwidth: comm=btrfs-endio-wri pages=192 time=40ms [ 9446.903518] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms [ 9446.933610] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
Thanks,
Fengguang
Re: [PATCH 20/45] NFS: introduce writeback wait queue
In reply to Peter ZijlstraQuoted 14 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang
Quoted 2 lines
wakeup implies a full memory barrier.
If so, this
smp_mb__after_clear_bit()line is also not necessary?Quoted 7 lines
void
clear_bdi_congested(struct backing_dev_info *bdi, int sync){
//...clear_bit(bit, &bdi->state); smp_mb__after_clear_bit(); if (waitqueue_active(wqh)) wake_up(wqh);
Quoted 2 lines
Depends on if the barrier is needed even when the wakeup doesn't happen I guess ;-)
I see. Thanks!
Re: [PATCH 21/45] writeback: estimate bdi write bandwidth
In reply to Peter ZijlstraQuoted 14 lines · Peter Zijlstra
fs/fs-writeback.c void bdi_start_writeback(struct backing_ · line 282
Quoted 9 lines · Wu Fengguang
fs/fs-writeback.c +9
-0+static void bdi_calc_write_bandwidth (struct backing_dev_info *bdi, +unsigned long nr_pages, +unsigned long time) +{ +unsigned long bw; + + bw =HZ * nr_pages / (time |1 ); + bdi->write_bandwidth = (bdi->write_bandwidth *63 + bw) /64 ; +}If you have block times < 1 jiffy this all falls apart quite quickly. You could perhaps try to use
cpu_clock()for ns resolution timestamps if this is a real issue.
Good point. I'd like to view it in the opposite way: if a sleep takes <= 1 jiffy to wakeup, that would mean high overheads. We could as well lift the ratelimit values :)
Quoted 4 lines
(I could imagine fast arrays with huge throughput causing small sleeps, resulting in underestimates of their bandwidth)
Also, 63/64 seems rather slow progress.. maybe that's good.
Whatever will be fast enough for servers that keep running for years. So the main concern would be how fast it can adapt to slow usb devices.
Here is a quick calculation. Each iteration writes ~800KB:
irb > a=128; 1.upto(100) { |i| a = (a * 63 + 5) / 64; printf "%d\t%d\n", i, a }
[...]
86 12 87 11 88 10 89 9 90 8 91 7 92 6 93 5 94 5
Looks slow. 8 or 16 seems better, which stabilize after writing about 16MB or 32MB data:
a=128; 1.upto(100) { |i| a = (a * 7 + 5) / 8; printf "%d\t%d\n", i, a }
1 112 2 98 3 86 4 75 5 66 6 58 7 51 8 45 9 40 10 35 11 31 12 27 13 24 14 21 15 19 16 17 17 15 18 13 19 12 20 11 21 10 22 9 23 8 24 7 25 6 26 5 27 5 28 5 29 5
16:
34 13 35 12 36 11 37 10 38 9 39 8 40 7 41 6 42 5 43 5
Thanks,
Fengguang
Re: [PATCH 20/45] NFS: introduce writeback wait queue
In reply to Nick PigginQuoted 18 lines · Nick Piggin
Quoted 5 lines · Wu Fengguang
Quoted 1 line · Peter Zijlstra
wakeup implies a full memory barrier.
If so, this
smp_mb__after_clear_bit()line is also not necessary?Quoted 7 lines
void
clear_bdi_congested(struct backing_dev_info *bdi, int sync){
//...clear_bit(bit, &bdi->state); smp_mb__after_clear_bit(); if (waitqueue_active(wqh)) wake_up(wqh);Typically in these patterns you do need a barrier. You need to ensure the load to check the waitqueue is performed after the
clear_bit.
Quoted 5 lines
The other side does:
add_to_waitqueueset_current_state/* hasmb()in it */
if (test_bit())
schedule()
Thanks, just moved the mb() up to follow clear_bit():
if (test_bit(BDI_sync_congested, &bdi->state)) {
clear_bdi_congested(bdi, BLK_RW_SYNC);
smp_mb__after_clear_bit();
}
if (waitqueue_active(&wqh[BLK_RW_SYNC]))
wake_up(&wqh[BLK_RW_SYNC]);Thanks,
Fengguang
Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)
In reply to Wu FengguangQuoted 6 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:
Quoted 7 lines
submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
CC David Howells <[email protected]>Signed-off-by Wu Fengguang <[email protected]>
Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)
Quoted 11 lines · David Howells
Quoted 9 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
Quoted 6 lines
Quoted 4 lines
It also prevents submitting pointless IO for busy overwriters.
CC David Howells <[email protected]>Signed-off-by Wu Fengguang <[email protected]>Acked-by David Howells <[email protected]>
I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).
Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)
Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
CC David Howells <[email protected]>Signed-off-by Wu Fengguang <[email protected]>Acked-by David Howells <[email protected]>I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).
The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.
If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.
If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.
The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)
Thanks,
Fengguang
Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)
Quoted 30 lines · Wu Fengguang
Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
CC David Howells <[email protected]>Signed-off-by Wu Fengguang <[email protected]>Acked-by David Howells <[email protected]>I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).
The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.
If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.
Quoted 7 lines
If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.
The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)
No I mean bandwidth fairness between inodes.
Re: [PATCH 29/45] writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case
In reply to Wu FengguangQuoted 7 lines · Wu Fengguang
When shmem returns
AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. Sowrite_cache_pagesshall stop writting this inode, and shmem shall increasepages_skippedto instruct VFS not to busy retry.CC Hugh Dickins <[email protected]>Signed-off-by Wu Fengguang <[email protected]>
Okay, it embarrasses me to see AOP_WRITEPAGE_ACTIVATE (and its horrid "in this one case the page is still locked" semantic) still around - my patch to remove it vanished from mmotm (probably caused a temporary conflict) and I've never chased it up (partly out of guilt that I'd not yet kept my promise to contact the openAFS people about their use of it).
But that's orthogonal to your concern here: for so long as there has been a wbc->pages_skipped, I guess shmem_writepage() should have been incrementing it there - thanks. But I don't believe the VFS will ever have any interest in pages_skipped from shmem_writepage(): do you have evidence that it does? If so, I need to investigate.
And the accompanying change to write_cache_pages() seems irrelevant and misguided. Irrelevant because write_cache_pages() should never be dealing with shmem_writepage() (its bdi should keep it well away), and should never be dealing with reclaim, which is the only case in which shmem_writepage() returns AOP_WRITEPAGE_ACTIVATE - or have your other changes, or the bdi work, changed that?
And misguided because in your change to write_cache_pages() you're taking AOP_WRITEPAGE_ACTIVATE to say that it should now give up, not process more pages. We just don't know that. All it means is that this one page couldn't be written and should be reactivated (if it were under reclaim): it might be the case that every other page tried after would get treated in the same way, or it might be the case that the next page would get written successfully. That info is just not provided.
Hugh
Quoted 48 lines
--- mm/page-writeback.c | 23 +++++++++++------------ mm/shmem.c | 1 + 2 files changed, 12 insertions(+), 12 deletions(-)mm/page-writeback.c +11
-12@@ -851,19 +851,18 @@ continue_unlock:if (ret ==AOP_WRITEPAGE_ACTIVATE ) {unlock_page (page); ret =0 ; - }else { -/* -* done_index is set past this page, -* so media errors will not choke -* background writeout for the entire -* file. This has consequences for -* range_cyclic semantics (ie. it may -* not be suitable for data integrity -* writeout). -*/ - done =1 ; -break ; } +/* +* done_index is set past this page, +* so media errors will not choke +* background writeout for the entire +* file. This has consequences for +* range_cyclic semantics (ie. it may +* not be suitable for data integrity +* writeout). +*/ + done =1 ; +break ; }if (nr_to_write >0 ) {mm/shmem.c +1
-0@@ -1103,6 +1103,7 @@ unlock:*/ swapcache_free (swap,NULL ); redirty: + wbc->pages_skipped++;set_page_dirty (page);if (wbc->for_reclaim)return AOP_WRITEPAGE_ACTIVATE ;/* Return with page locked */
Re: [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs)
In reply to Nick PigginQuoted 36 lines · Nick Piggin
Quoted 35 lines · Wu Fengguang
Quoted 14 lines · Nick Piggin
Quoted 13 lines · David Howells
Quoted 11 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
CC David Howells <[email protected]>Signed-off-by Wu Fengguang <[email protected]>Acked-by David Howells <[email protected]>I don't see why. Then the inode is given less write bandwidth than those which don't wrap (or wrap on "nice" boundaries).
The "return on wrapped" behavior itself only offers a natural seek boundary to the upper layer. It's mainly the "whether to delay" policy that will affect (overall) bandwidth.
If we choose to not sleep, and to go on with other inodes and then back to this inode, no bandwidth will be lost.
If we have done work with other inodes (if any), and choose to sleep for a while before restarting this inode, then we could lose bandwidth. The plus side is, we possibly avoid submitting extra IO if this inode is being busy overwritten. So it's a tradeoff.
Quoted 4 lines
Quoted 2 lines
The behavior after this patchset is, to keep busy as long as we can write any pages (in patch 38/45). So we still opt for bandwidth :)
No I mean bandwidth fairness between inodes.
I guess it's the old semantics that has bandwidth fairness problem :)
Imagine write chunk size is 4MB, and inode A/B with size 6MB/8MB.
The old semantics will have write sequence
4MB for A; 4MB for B; other inodes;
4MB for A; 4MB for B; other inodes;
4MB for A; 4MB for B; other inodes;while the new sequence would be
4MB for A; 4MB for B; other inodes;
2MB for A; 4MB for B; other inodes;
4MB for A; 4MB for B; other inodes;
2MB for A; 4MB for B; other inodes;On average, each page in A used to get more write chance than B's. Now with no-wrap, A and B's pages have the same chance to be writeback.
Thanks,
Fengguang
Re: [PATCH 09/45] writeback: quit on wrap for .range_cyclic (pohmelfs)
In reply to Wu FengguangHi.
Quoted 10 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
It also prevents submitting pointless IO for busy overwriters.
I have no objections against this patchset, since I followed the upstream writeback behaviour and did not personally observe such wraps which would be otherwise handled in a single run.
Quoted 2 lines
CC Evgeniy Polyakov <[email protected]>Signed-off-by Wu Fengguang <[email protected]>
Signature · 1 line
Evgeniy Polyakov
Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock
In reply to Wu FengguangQuoted 7 lines · Wu Fengguang
It was introduced in
72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue inbalance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes
I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.
If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)
Thanx... ps
Quoted 66 lines
CC Peter Zijlstra <[email protected]>CC Peter Staubach <[email protected]>CC Trond Myklebust <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/nfs/file.c | 9 --------- fs/nfs/write.c | 11 ----------- include/linux/nfs_fs.h | 1 - 3 files changed, 21 deletions(-)fs/nfs/file.c +0
-9@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file * mapping->host->i_ino, len, (long long) pos); start: -/* -* Prevent starvation issues if someone is doing a consistency -* sync-to-disk -*/ - ret =wait_on_bit (&NFS_I (mapping->host)->flags,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (ret) -return ret; - page =grab_cache_page_write_begin (mapping, index, flags);if (!page)return -ENOMEM ;fs/nfs/write.c +0
-11@@ -387,26 +387,15 @@ static int nfs_writepages_callback(strucint nfs_writepages (struct address_space *mapping,struct writeback_control *wbc) {struct inode *inode = mapping->host; -unsigned long *bitlock = &NFS_I (inode)->flags;struct nfs_pageio_descriptor pgio;int err; -/* Stop dirtying of new pages while we sync */ - err =wait_on_bit_lock (bitlock,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (err) - goto out_err; -nfs_inc_stats (inode,NFSIOS_VFSWRITEPAGES );nfs_pageio_init_write (&pgio, inode,wb_priority (wbc)); err =write_cache_pages (mapping, wbc, nfs_writepages_callback, &pgio);nfs_pageio_complete (&pgio); -clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); -wake_up_bit (bitlock,NFS_INO_FLUSHING ); -if (err <0 ) goto out_err; err = pgio.pg_error;include/linux/nfs_fs.h +0
-1@@ -208,7 +208,6 @@ struct nfs_inode {#define NFS_INO_STALE (1)/* possible stale inode */ #define NFS_INO_ACL_LRU_SET (2)/* Inode is on the LRU list */ #define NFS_INO_MOUNTPOINT (3)/* inode is remote mountpoint */ -#define NFS_INO_FLUSHING (4)/* inode is flushing out data */ #define NFS_INO_FSCACHE (5)/* inode can be cached by FS-Cache */ #define NFS_INO_FSCACHE_LOCK (6)/* FS-Cache cookie management lock */
Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock
Quoted 9 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang
It was introduced in
72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue inbalance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodes
Quoted 3 lines
I don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.
Ah OK.
Quoted 3 lines
If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)
I noticed the i_mutex lock in nfs_getattr(). Do you mean that?
Thanks,
Fengguang
Quoted 66 lines
Quoted 66 lines
CC Peter Zijlstra <[email protected]>CC Peter Staubach <[email protected]>CC Trond Myklebust <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/nfs/file.c | 9 --------- fs/nfs/write.c | 11 ----------- include/linux/nfs_fs.h | 1 - 3 files changed, 21 deletions(-)fs/nfs/file.c +0
-9@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file * mapping->host->i_ino, len, (long long) pos); start: -/* -* Prevent starvation issues if someone is doing a consistency -* sync-to-disk -*/ - ret =wait_on_bit (&NFS_I (mapping->host)->flags,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (ret) -return ret; - page =grab_cache_page_write_begin (mapping, index, flags);if (!page)return -ENOMEM ;fs/nfs/write.c +0
-11@@ -387,26 +387,15 @@ static int nfs_writepages_callback(strucint nfs_writepages (struct address_space *mapping,struct writeback_control *wbc) {struct inode *inode = mapping->host; -unsigned long *bitlock = &NFS_I (inode)->flags;struct nfs_pageio_descriptor pgio;int err; -/* Stop dirtying of new pages while we sync */ - err =wait_on_bit_lock (bitlock,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (err) - goto out_err; -nfs_inc_stats (inode,NFSIOS_VFSWRITEPAGES );nfs_pageio_init_write (&pgio, inode,wb_priority (wbc)); err =write_cache_pages (mapping, wbc, nfs_writepages_callback, &pgio);nfs_pageio_complete (&pgio); -clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); -wake_up_bit (bitlock,NFS_INO_FLUSHING ); -if (err <0 ) goto out_err; err = pgio.pg_error;include/linux/nfs_fs.h +0
-1@@ -208,7 +208,6 @@ struct nfs_inode {#define NFS_INO_STALE (1)/* possible stale inode */ #define NFS_INO_ACL_LRU_SET (2)/* Inode is on the LRU list */ #define NFS_INO_MOUNTPOINT (3)/* inode is remote mountpoint */ -#define NFS_INO_FLUSHING (4)/* inode is flushing out data */ #define NFS_INO_FSCACHE (5)/* inode can be cached by FS-Cache */ #define NFS_INO_FSCACHE_LOCK (6)/* FS-Cache cookie management lock */
Quoted 21 lines · Wu Fengguang
Hi all,
Here is a collection of writeback patches on
- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixesSorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..
The lumpy pageout and
nr_segmentssupport is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!
Quoted 7 lines
The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..
NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:
Yes. It appears that the problem is that too many pages get dirtied and the network/server get overwhelmed by the NFS client attempting to write out all of the pages as quickly as it possibly can.
I think that it would be better if we could better match the number of pages which can be dirty at any given point with the bandwidth available through the network and the server file system and storage.
One approach that I have pondered is immediately queuing an asynchronous request when enough pages are dirtied to be able to completely fill an over the wire transfer. This sort of seems like a per-file bdi, which doesn't seem quite like the right approach to me. What would y'all think about that?
ps
Quoted 50 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples) nr_writeback nr_dirty nr_unstable 11227 41463 38044 11227 41463 38044 11227 41463 38044 11227 41463 38044 11045 53987 6490 11033 53120 8145 11195 52143 10886 11211 52144 10913 11211 52144 10913 11211 52144 10913btrfs seems to maintain a private pool of writeback pages, which can go out of control:
nr_writeback nr_dirty 261075 132 252891 195 244795 187 236851 187 228830 187 221040 218 212674 237 204981 237XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.
nr_writeback nr_dirty 80781 0 37117 37703 37117 43933 81044 6 81050 0 43943 10199 43930 36355 43930 36355 80293 0 80285 0 80285 0Thanks,
Fengguang--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 29/45] writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case
In reply to Hugh DickinsQuoted 7 lines · Hugh Dickins
Quoted 5 lines · Wu Fengguang
When shmem returns
AOP_WRITEPAGE_ACTIVATE, the inode pages cannot be synced in the near future. Sowrite_cache_pagesshall stop writting this inode, and shmem shall increasepages_skippedto instruct VFS not to busy retry.
Quoted 8 lines
Quoted 2 lines
CC Hugh Dickins <[email protected]>Signed-off-by Wu Fengguang <[email protected]>Okay, it embarrasses me to see
AOP_WRITEPAGE_ACTIVATE(and its horrid "in this one case the page is still locked" semantic) still around - my patch to remove it vanished from mmotm (probably caused a temporary conflict) and I've never chased it up (partly out of guilt that I'd not yet kept my promise to contact the openAFS people about their use of it).
Googled this one :)
http://markmail.org/thread/fivi4bgylwsy26ws
In fact we could just return from shmem_writepage() with PG_dirty and !PG_writeback. Then the page will be put back to LRU with PG_reclaim cleared. It could well happen in other filesystems who has trouble to writeback the page for the time (those already do pages_skipped++).
Quoted 5 lines
But that's orthogonal to your concern here: for so long as there has been a
wbc->pages_skipped, I guessshmem_writepage()should have been incrementing it there - thanks. But I don't believe the VFS will ever have any interest inpages_skippedfromshmem_writepage(): do you have evidence that it does? If so, I need to investigate.
Yes :) Except in this new code.
Quoted 6 lines
And the accompanying change to
write_cache_pages()seems irrelevant and misguided. Irrelevant becausewrite_cache_pages()should never be dealing withshmem_writepage()(its bdi should keep it well away), and should never be dealing with reclaim, which is the only case in whichshmem_writepage()returnsAOP_WRITEPAGE_ACTIVATE- or have your other changes, or the bdi work, changed that?
That becomes possible with my another patch (30/45). I've added check to avoid doing lumpy reclaim for shmem. So now everything returns to normal :)
Quoted 8 lines
And misguided because in your change to
write_cache_pages()you're takingAOP_WRITEPAGE_ACTIVATEto say that it should now give up, not process more pages. We just don't know that. All it means is that this one page couldn't be written and should be reactivated (if it were under reclaim): it might be the case that every other page tried after would get treated in the same way, or it might be the case that the next page would get written successfully. That info is just not provided.
Yes, it was over-smart indeed. I'll revert that chunk (or AOP_WRITEPAGE_ACTIVATE) totally.
Thanks,
Fengguang
Quoted 48 lines
Quoted 48 lines
--- mm/page-writeback.c | 23 +++++++++++------------ mm/shmem.c | 1 + 2 files changed, 12 insertions(+), 12 deletions(-)mm/page-writeback.c +11
-12@@ -851,19 +851,18 @@ continue_unlock:if (ret ==AOP_WRITEPAGE_ACTIVATE ) {unlock_page (page); ret =0 ; - }else { -/* -* done_index is set past this page, -* so media errors will not choke -* background writeout for the entire -* file. This has consequences for -* range_cyclic semantics (ie. it may -* not be suitable for data integrity -* writeout). -*/ - done =1 ; -break ; } +/* +* done_index is set past this page, +* so media errors will not choke +* background writeout for the entire +* file. This has consequences for +* range_cyclic semantics (ie. it may +* not be suitable for data integrity +* writeout). +*/ + done =1 ; +break ; }if (nr_to_write >0 ) {mm/shmem.c +1
-0@@ -1103,6 +1103,7 @@ unlock:*/ swapcache_free (swap,NULL ); redirty: + wbc->pages_skipped++;set_page_dirty (page);if (wbc->for_reclaim)return AOP_WRITEPAGE_ACTIVATE ;/* Return with page locked */
Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock
In reply to Wu FengguangQuoted 13 lines · Wu Fengguang
Quoted 11 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang
It was introduced in
72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue inbalance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodesI don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.
Quoted 8 lines
Ah OK.
Quoted 3 lines
If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)
I noticed the
i_mutexlock innfs_getattr(). Do you mean that?
Well, that's part of that support as well. That keeps a writing application from dirtying more pages while the application doing the stat is attempting to clean them.
Another approach that I suggested was to keep track of the number of pages which are dirty on a per-inode basis. When enough pages are dirty to fill an over the wire transfer, then schedule an asynchronous write to transmit that data to the server. This ties in with support to ensure that the server/network is not completely overwhelmed by the client by flow controlling the writing application to better match the bandwidth and latencies of the network and server. With this support, the NFS client tends not to fill memory with dirty pages and thus, does not depend upon the other parts of the system to flush these pages.
All of these recent pages make this current flushing happen in a much more orderly fashion, which is great. However, this can still lead to the client attempting to flush potentially gigabytes all at once, which is more than most networks and servers can handle reasonably.
ps
Quoted 69 lines
Thanks,
FengguangQuoted 66 lines
Quoted 66 lines
CC Peter Zijlstra <[email protected]>CC Peter Staubach <[email protected]>CC Trond Myklebust <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/nfs/file.c | 9 --------- fs/nfs/write.c | 11 ----------- include/linux/nfs_fs.h | 1 - 3 files changed, 21 deletions(-)fs/nfs/file.c +0
-9@@ -386,15 +386,6 @@ static int nfs_write_begin(struct file * mapping->host->i_ino, len, (long long) pos); start: -/* -* Prevent starvation issues if someone is doing a consistency -* sync-to-disk -*/ - ret =wait_on_bit (&NFS_I (mapping->host)->flags,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (ret) -return ret; - page =grab_cache_page_write_begin (mapping, index, flags);if (!page)return -ENOMEM ;fs/nfs/write.c +0
-11@@ -387,26 +387,15 @@ static int nfs_writepages_callback(strucint nfs_writepages (struct address_space *mapping,struct writeback_control *wbc) {struct inode *inode = mapping->host; -unsigned long *bitlock = &NFS_I (inode)->flags;struct nfs_pageio_descriptor pgio;int err; -/* Stop dirtying of new pages while we sync */ - err =wait_on_bit_lock (bitlock,NFS_INO_FLUSHING , - nfs_wait_bit_killable,TASK_KILLABLE ); -if (err) - goto out_err; -nfs_inc_stats (inode,NFSIOS_VFSWRITEPAGES );nfs_pageio_init_write (&pgio, inode,wb_priority (wbc)); err =write_cache_pages (mapping, wbc, nfs_writepages_callback, &pgio);nfs_pageio_complete (&pgio); -clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); -wake_up_bit (bitlock,NFS_INO_FLUSHING ); -if (err <0 ) goto out_err; err = pgio.pg_error;include/linux/nfs_fs.h +0
-1@@ -208,7 +208,6 @@ struct nfs_inode {#define NFS_INO_STALE (1)/* possible stale inode */ #define NFS_INO_ACL_LRU_SET (2)/* Inode is on the LRU list */ #define NFS_INO_MOUNTPOINT (3)/* inode is remote mountpoint */ -#define NFS_INO_FLUSHING (4)/* inode is flushing out data */ #define NFS_INO_FSCACHE (5)/* inode can be cached by FS-Cache */ #define NFS_INO_FSCACHE_LOCK (6)/* FS-Cache cookie management lock */
Re: [PATCH 09/45] writeback: quit on wrap for .range_cyclic (pohmelfs)
In reply to Evgeniy PolyakovQuoted 12 lines · Evgeniy Polyakov
Hi.
Quoted 9 lines · Wu Fengguang
Convert wbc.range_cyclic to new behavior: when past EOF, abort writeback of the inode, which instructs
writeback_single_inode()to delay it for a while if necessary.It removes one inefficient .range_cyclic IO pattern when
writeback_indexwraps:submit [10000-10100], (wrap), submit [0-100]In which the submitted pages may be consisted of two distant ranges.
Quoted 5 lines
Quoted 1 line
It also prevents submitting pointless IO for busy overwriters.
I have no objections against this patchset, since I followed the upstream writeback behaviour and did not personally observe such wraps which would be otherwise handled in a single run.
OK, thanks!
Fengguang
Theodore Tso
maintainerQuoted 4 lines · Wu Fengguang
The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..
Can you expand on what you mean here? Estimated write bandwidth of what? And what are you comparing it against?
I'm having trouble understanding your note (which I'm guessing you write fairly late at night? :-)
Thanks,
- Ted
Quoted 8 lines · Theodore Ts'o
Quoted 4 lines · Wu Fengguang
The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..
Can you expand on what you mean here? Estimated write bandwidth of what? And what are you comparing it against?
Please refer to [PATCH 21/45] writeback: estimate bdi write bandwidth and patch 22, I have some numbers there :)
Quoted 2 lines
I'm having trouble understanding your note (which I'm guessing you write fairly late at night? :-)
Sorry - I wrote that when I got tired on debugging ;)
Thanks,
Fengguang
Quoted 26 lines · Peter Staubach
Quoted 25 lines · Wu Fengguang
Hi all,
Here is a collection of writeback patches on
- larger writeback chunk sizes
- single per-bdi flush thread (killing the foreground throttling writeouts)
- lumpy pageout
- sync livelock prevention
- writeback scheduling
- random fixesSorry for posting a too big series - there are many direct or implicit dependencies, and one patch lead to another before I can stop..
The lumpy pageout and
nr_segmentssupport is not complete and do not cover all filesystems for now. It may be better to first convert some of the ->writepages to the generic routines to avoid duplicate work.I managed to address many issues in past week, however there are still known problems. Hints from filesystem developers are highly appreciated. Thanks!
The estimated writeback bandwidth is about 1/2 the real throughput for ext2/3/4 and btrfs; noticeable bigger than real throughput for NFS; and cannot be estimated at all for XFS. Very interesting..
Quoted 7 lines
Quoted 3 lines
NFS writeback is very bumpy. The page numbers and network throughput "freeze" together from time to time:
Yes. It appears that the problem is that too many pages get dirtied and the network/server get overwhelmed by the NFS client attempting to write out all of the pages as quickly as it possibly can.
In theory it should push pages as quickly as possible at first, to fill up the server side queue.
Quoted 4 lines
I think that it would be better if we could better match the number of pages which can be dirty at any given point with the bandwidth available through the network and the server file system and storage.
And then go into this steady state of matched network/disk bandwidth.
Quoted 5 lines
One approach that I have pondered is immediately queuing an asynchronous request when enough pages are dirtied to be able to completely fill an over the wire transfer. This sort of seems like a per-file bdi, which doesn't seem quite like the right approach to me. What would y'all think about that?
Hmm, it sounds like unnecessary complexity. Because it is not going to help the busy-dirtier case anyway. And if we can do good on heavy IO, the pre-flushing policy becomes less interesting.
Quoted 7 lines
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples) nr_writeback nr_dirty nr_unstable 11227 41463 38044 11227 41463 38044 11227 41463 38044 11227 41463 38044
I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both nr_writeback/nr_unstable. And the stuck nr_writeback will freeze nr_dirty as well, because the dirtying process is throttled until it receives enough "PG_writeback cleared" event, however the bdi-flush thread is also blocked when trying to clear more PG_writeback, because the client side nr_writeback limit has been reached. In summary,
server blocked => nr_writeback stuck => nr_writeback limit reached => bdi-flush blocked => no end_page_writeback() => dirtier blocked => nr_dirty stuck
Thanks,
Fengguang
Quoted 44 lines
Quoted 44 lines
11045 53987 6490 11033 53120 8145 11195 52143 10886 11211 52144 10913 11211 52144 10913 11211 52144 10913btrfs seems to maintain a private pool of writeback pages, which can go out of control:
nr_writeback nr_dirty 261075 132 252891 195 244795 187 236851 187 228830 187 221040 218 212674 237 204981 237XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.
nr_writeback nr_dirty 80781 0 37117 37703 37117 43933 81044 6 81050 0 43943 10199 43930 36355 43930 36355 80293 0 80285 0 80285 0Thanks,
Fengguang--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to Wu FengguangQuoted 15 lines · On Wed, 07 Oct 2009 15:38:36 +0800 Wu Fengguang <[email protected]> wrote:
As proposed by Chris, Dave and Jan, let
balance_dirty_pages()wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the above
Quoted 9 lines
foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-filenr_to_write
Thewrite_chunkused bybalance_dirty_pages()should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait insidebalance_dirty_pages()shall be small enough. When it starts its own writeback, it must specify a smallnr_to_write. The throttle wait queue removes this dependancy by the way.
May I ask a question ? (maybe not directly related to this patch itself, sorry)
Recent works as "writeback: switch to per-bdi threads for flushing data" removed congestion_wait() from balance_dirty_pages() and added schedule_timeout_interruptible().
And this one replaces it with wake_up+wait_queue.
IIUC, "iowait" cpustat data was calculated by runqueue->nr_iowait as == kernel/schec.c
void account_idle_time(cputime_t cputime)
{
struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
cputime64_t cputime64 = cputime_to_cputime64(cputime);
struct rq *rq = this_rq();
if (atomic_read(&rq->nr_iowait) > 0)
cpustat->iowait = cputime64_add(cpustat->iowait, cputime64);
else
cpustat->idle = cputime64_add(cpustat->idle, cputime64);
}==
Then, for showing "cpu is in iowait", runqueue->nr_iowait should be modified at some places. In old kernel, congestion_wait() at el did that by calling io_schedule_timeout().
How this runqueue->nr_iowait is handled now ?
Thanks,
-Kame
Quoted 257 lines
CC Chris Mason <[email protected]>CC Dave Chinner <[email protected]>CC Jan Kara <[email protected]>CC Peter Zijlstra <[email protected]>CC Jens Axboe <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 71 ++++++++++++++++++++++++++++++++++ include/linux/backing-dev.h | 15 +++++++ mm/backing-dev.c | 4 + mm/page-writeback.c | 53 ++++++------------------- 4 files changed, 103 insertions(+), 40 deletions(-)mm/page-writeback.c +13
-40@@ -218,6 +218,15 @@ static inline void __bdi_writeout_inc(st {__prop_inc_percpu_max (&vm_completions, &bdi->completions, bdi->max_prop_frac); + +/* +* The DIRTY_THROTTLE_PAGES_STOP test is an optional optimization, so +* it's OK to be racy. We set DIRTY_THROTTLE_PAGES_STOP*2 in other +* places to reduce the race possibility. +*/ +if (atomic_read (&bdi->throttle_pages) <DIRTY_THROTTLE_PAGES_STOP && +atomic_dec_and_test (&bdi->throttle_pages)) +bdi_writeback_wakeup (bdi); }void bdi_writeout_inc (struct backing_dev_info *bdi)@@ -458,20 +467,10 @@ static void balance_dirty_pages(struct aunsigned long background_thresh;unsigned long dirty_thresh;unsigned long bdi_thresh; -unsigned long pages_written =0 ; -unsigned long pause =1 ;int dirty_exceeded;struct backing_dev_info *bdi = mapping->backing_dev_info;for (;;) { -struct writeback_control wbc = { - .bdi = bdi, - .sync_mode =WB_SYNC_NONE , - .older_than_this =NULL , - .nr_to_write = write_chunk, - .range_cyclic =1 , - }; - nr_reclaimable =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ); nr_writeback = global_page_state(NR_WRITEBACK) +@@ -518,39 +517,13 @@ static void balance_dirty_pages(struct aif (!bdi->dirty_exceeded) bdi->dirty_exceeded =1 ; -/* Note: nr_reclaimable denotes nr_dirty + nr_unstable. -* Unstable writes are a feature of certain networked -* filesystems (i.e. NFS) in which data may have been -* written to the server's write cache, but has not yet -* been flushed to permanent storage. -* Only move pages to writeback if this bdi is over its -* threshold otherwise wait until the disk writes catch -* up. -*/ -if (bdi_nr_reclaimable > bdi_thresh) { -writeback_inodes_wbc (&wbc); - pages_written += write_chunk - wbc.nr_to_write; -/* don't wait if we've done enough */ -if (pages_written >= write_chunk) -break ; - } -schedule_timeout_interruptible (pause); - -/* -* Increase the delay for each loop, up to our previous -* default of taking a 100ms nap. -*/ - pause <<=1 ; -if (pause >HZ /10 ) - pause =HZ /10 ; +bdi_writeback_wait (bdi, write_chunk); +break ; }if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ; -if (writeback_in_progress (bdi)) -return ; -/* * In laptop mode, we wait until hitting the higher threshold before * starting background writeout, and then write out all the way down @@ -559,8 +532,8 @@ static void balance_dirty_pages(struct a* In normal mode, we start background writeout at the lower * background_thresh, to keep the amount of dirty memory low. */ -if ((laptop_mode && pages_written) || - (!laptop_mode && (nr_reclaimable > background_thresh))) +if (!laptop_mode && (nr_reclaimable > background_thresh) && +can_submit_background_writeback (bdi))bdi_start_writeback (bdi,NULL ,0 ); }include/linux/backing-dev.h +15
-0@@ -86,6 +86,13 @@ struct backing_dev_info {struct list_head work_list; +/* +* dirtier process throttling +*/ +spinlock_t throttle_lock; +struct list_head throttle_list;/* nr to sync for each task */ +atomic_t throttle_pages;/* nr to sync for head task */ +struct device *dev;#ifdef CONFIG_DEBUG_FS @@ -99,6 +106,12 @@ struct backing_dev_info {*/ #define WB_FLAG_BACKGROUND_WORK 30 +/* +* when no task is throttled, set throttle_pages to larger than this, +* to avoid unnecessary atomic decreases. +*/ +#define DIRTY_THROTTLE_PAGES_STOP (1 << 22) +int bdi_init (struct backing_dev_info *bdi);void bdi_destroy (struct backing_dev_info *bdi);@@ -110,6 +123,8 @@ void bdi_start_writeback(struct backing_ long nr_pages);int bdi_writeback_task (struct bdi_writeback *wb);int bdi_has_dirty_io (struct backing_dev_info *bdi); +int bdi_writeback_wakeup (struct backing_dev_info *bdi); +void bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages);extern spinlock_t bdi_lock;extern struct list_head bdi_list;fs/fs-writeback.c +71
-0@@ -25,6 +25,7 @@#include <linux/blkdev.h> #include <linux/backing-dev.h> #include <linux/buffer_head.h> +#include <linux/completion.h> #include "internal.h" #define inode_to_bdi (inode) ((inode)->i_mapping->backing_dev_info)@@ -265,6 +266,72 @@ void bdi_start_writeback(struct backing_bdi_alloc_queue_work (bdi, &args); } +struct dirty_throttle_task { +long nr_pages; +struct list_head list; +struct completion complete; +}; + +void bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) +{ +struct dirty_throttle_task tt = { + .nr_pages = nr_pages, + .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), + }; +unsigned long flags; + +/* +* register throttle pages +*/ +spin_lock_irqsave (&bdi->throttle_lock, flags); +if (list_empty (&bdi->throttle_list)) +atomic_set (&bdi->throttle_pages, nr_pages); +list_add (&tt.list, &bdi->throttle_list); +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +/* +* make sure we will be woke up by someone +*/ +if (can_submit_background_writeback (bdi)) +bdi_start_writeback (bdi,NULL ,0 ); + +wait_for_completion (&tt.complete); +} + +/* +* return 1 if there are more waiting tasks. +*/ +int bdi_writeback_wakeup (struct backing_dev_info *bdi) +{ +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +/* +* remove and wakeup head task +*/ +if (!list_empty (&bdi->throttle_list)) { + tt =list_entry (bdi->throttle_list.prev, + struct dirty_throttle_task, list); +list_del (&tt->list); +complete (&tt->complete); + } +/* +* update throttle pages +*/ +if (!list_empty (&bdi->throttle_list)) { + tt =list_entry (bdi->throttle_list.prev, + struct dirty_throttle_task, list); +atomic_set (&bdi->throttle_pages, tt->nr_pages); + }else { + tt =NULL ; +atomic_set (&bdi->throttle_pages,DIRTY_THROTTLE_PAGES_STOP *2 ); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return tt !=NULL ; +} +/* * Redirty an inode: set its when-it-was dirtied timestamp and move it to the * furthest end of its superblock's dirty-inode list. @@ -760,6 +827,10 @@ static long wb_writeback(struct bdi_writspin_unlock (&inode_lock); } +if (args->for_background) +while (bdi_writeback_wakeup (wb->bdi)) + ;/* unthrottle all tasks */ +return wrote; }mm/backing-dev.c +4
-1@@ -646,6 +646,10 @@ int bdi_init(struct backing_dev_info *bd bdi->wb_mask =1 ; bdi->wb_cnt =1 ; +spin_lock_init (&bdi->throttle_lock); +INIT_LIST_HEAD (&bdi->throttle_list); +atomic_set (&bdi->throttle_pages,DIRTY_THROTTLE_PAGES_STOP *2 ); +for (i =0 ; i <NR_BDI_STAT_ITEMS ; i++) { err =percpu_counter_init (&bdi->bdi_stat[i],0 );if (err) --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock
In reply to Peter StaubachQuoted 20 lines · Peter Staubach
Quoted 19 lines · Wu Fengguang
Quoted 11 lines · Peter Staubach
Quoted 7 lines · Wu Fengguang
It was introduced in
72cb77f4a5ac, and the several issues have been addressed in generic writeback:
- out of order writeback (or interleaved concurrent writeback) addressed by the per-bdi writeback and wait queue inbalance_dirty_pages()
- sync livelocked by a fast dirtier
addressed by throttling all to-be-synced dirty inodesI don't think that we can just remove this support. It is designed to reduce the effects from doing a stat(2) on a file which is being actively written to.
Ah OK.
Quoted 3 lines
If we do remove it, then we will need to replace this patch with another. Trond and I hadn't quite finished discussing some aspects of that other patch... :-)
Quoted 6 lines
Quoted 2 lines
I noticed the
i_mutexlock innfs_getattr(). Do you mean that?Well, that's part of that support as well. That keeps a writing application from dirtying more pages while the application doing the stat is attempting to clean them.
Instead of blocking totally, we could throttle application writes. The following two patches are the more gentle way of doing this, however it does not guarantee to kill the livelock, since a busy bdi-flush thread could writeback many pages to unfreeze the application prematurely. Anyway I attach them as a demo of idea, whether it be good or bad.
Quoted 2 lines
Another approach that I suggested was to keep track of the number of pages which are dirty on a per-inode basis. When
Yes a per-inode dirty count should be trivial and may be good for others.
Quoted 2 lines
enough pages are dirty to fill an over the wire transfer, then schedule an asynchronous write to transmit that data to
This should also be trivial to support if the location ordered writeback infrastructure is ready.
Quoted 4 lines
the server. This ties in with support to ensure that the server/network is not completely overwhelmed by the client by flow controlling the writing application to better match the bandwidth and latencies of the network and server.
I like that feature :)
Quoted 6 lines
With this support, the NFS client tends not to fill memory with dirty pages and thus, does not depend upon the other parts of the system to flush these pages.
All of these recent pages make this current flushing happen in a much more orderly fashion, which is great. However,
Thanks.
Quoted 3 lines
this can still lead to the client attempting to flush potentially gigabytes all at once, which is more than most networks and servers can handle reasonably.
OK, I now see the need to keep mapping->nr_dirty under control: it could make many NFS operations response in a more bounded fashion. The good thing is, it can share infrastructure with the location based writeback (http://lkml.org/lkml/2007/8/27/45 :)
Thanks,
Fengguang
writeback: sync livelock - throttle mapping dirties if it's being synced
The AS_SYNC_WAITER flag will be set to indicate an active sync waiter.
It's not perfect with respect to file ranges and concurrent syncs. And do not guarantee that the application writes on this mapping can be throttled enough to avoid livelock, especially when there are many background writebacks.
Just my 2 cents.
--- include/linux/pagemap.h | 3 +++ mm/filemap.c | 2 ++ mm/page-writeback.c | 5 +++++ 3 files changed, 10 insertions(+)
mm/page-writeback.c +5 -0
@@ -480,7 +480,12 @@ static void balance_dirty_pages(struct a/* * If sync() is in progress, curb the to-be-synced inodes regardless * of dirty limits, so that a fast dirtier won't livelock the sync. +* In perticular, NFS syncs the mapping before many file operations. */ +if (unlikely (test_bit (AS_SYNC_WAITER , &mapping->flags))) { + write_chunk *=8 ; + goto throttle; + } if (unlikely(bdi->sync_time && S_ISREG(mapping->host->i_mode) && time_after_eq(bdi->sync_time,
include/linux/pagemap.h +3 -0
@@ -23,6 +23,9 @@ enum mapping_flags {AS_ENOSPC = __GFP_BITS_SHIFT +1 ,/* ENOSPC on async write */ AS_MM_ALL_LOCKS = __GFP_BITS_SHIFT +2 ,/* under mm_take_all_locks() */ AS_UNEVICTABLE = __GFP_BITS_SHIFT +3 ,/* e.g., ramdisk, SHM_LOCK */ + AS_SYNC_WAITER = __GFP_BITS_SHIFT + 4, /* sync&wait under way, page + * dirtying will be throttled + */ };static inline void mapping_set_error (struct address_space *mapping,int error)
mm/filemap.c +2 -0
@@ -356,6 +356,7 @@ int filemap_write_and_wait(struct addresint err =0 ;if (mapping->nrpages) { +set_bit (AS_SYNC_WAITER , &mapping->flags); err =filemap_fdatawrite (mapping);/* * Even if the above returned error, the pages may be
@@ -368,6 +369,7 @@ int filemap_write_and_wait(struct addresif (!err) err = err2; } +clear_bit (AS_SYNC_WAITER , &mapping->flags); }return err; }
nfs: don't take i_mutex on nfs_wb_nocommit()
Set the AS_SYNC_WAITER flag and let VFS throttle application writes.
--- fs/nfs/inode.c | 9 +-------- fs/nfs/write.c | 6 +++++- 2 files changed, 6 insertions(+), 9 deletions(-)
fs/nfs/inode.c +1 -8
@@ -513,16 +513,9 @@ int nfs_getattr(struct vfsmount *mnt, st/* * Flush out writes to the server in order to update c/mtime. -* -* Hold the i_mutex to suspend application writes temporarily; -* this prevents long-running writing applications from blocking -* nfs_wb_nocommit. */ -if (S_ISREG (inode->i_mode)) { -mutex_lock (&inode->i_mutex); +if (S_ISREG (inode->i_mode))nfs_wb_nocommit (inode); -mutex_unlock (&inode->i_mutex); - }/* * We may force a getattr if the user cares about atime.
fs/nfs/write.c +5 -1
@@ -1539,8 +1539,12 @@ static int nfs_write_mapping(struct addr .range_start = 0, .range_end = LLONG_MAX, }; +int ret; -return __nfs_write_mapping (mapping, &wbc, how); +set_bit (AS_SYNC_WAITER , &mapping->flags); + ret =__nfs_write_mapping (mapping, &wbc, how); +clear_bit (AS_SYNC_WAITER , &mapping->flags); +return ret; }/*
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to KAMEZAWA HiroyukiQuoted 32 lines · KAMEZAWA Hiroyuki
tatus: RO
Content-Length: 12481
Lines: 332Quoted 15 lines · On Wed, 07 Oct 2009 15:38:36 +0800 Wu Fengguang <[email protected]> wrote:
As proposed by Chris, Dave and Jan, let
balance_dirty_pages()wait for the per-bdi flusher to writeback enough pages for it, instead of starting foreground writeback by itself. By doing so we harvest two benefits:
- avoid concurrent writeback of multiple inodes (Dave Chinner) If every thread doing writes and being throttled start foreground writeback, it leads to N IO submitters from at least N different inodes at the same time, end up with N different sets of IO being issued with potentially zero locality to each other, resulting in much lower elevator sort/merge efficiency and hence we seek the disk all over the place to service the different sets of IO. OTOH, if there is only one submission thread, it doesn't jump between inodes in the same way when congestion clears - it keeps writing to the same inode, resulting in large related chunks of sequential IOs being issued to the disk. This is more efficient than the aboveQuoted 9 lines
foreground writeback because the elevator works better and the disk seeks less.
- avoid one constraint torwards huge per-filenr_to_write
Thewrite_chunkused bybalance_dirty_pages()should be small enough to prevent user noticeable one-shot latency. Ie. each sleep/wait insidebalance_dirty_pages()shall be small enough. When it starts its own writeback, it must specify a smallnr_to_write. The throttle wait queue removes this dependancy by the way.
Quoted 1 line
May I ask a question ? (maybe not directly related to this patch itself, sorry)
Sure :)
Quoted 5 lines
Recent works as "writeback: switch to per-bdi threads for flushing data" removed
congestion_wait()frombalance_dirty_pages()and addedschedule_timeout_interruptible().And this one replaces it with wake_up+wait_queue.
Right.
Quoted 13 lines
IIUC, "iowait" cpustat data was calculated by
runqueue->nr_iowaitas ==kernel/schec.c
voidaccount_idle_time(cputime_t cputime){ struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat; cputime64_t cputime64 = cputime_to_cputime64(cputime); struct rq *rq = this_rq(); if (atomic_read(&rq->nr_iowait) > 0) cpustat->iowait = cputime64_add(cpustat->iowait, cputime64); else cpustat->idle = cputime64_add(cpustat->idle, cputime64); }
Quoted 6 lines
==
Then, for showing "cpu is in iowait",runqueue->nr_iowaitshould be modified at some places. In old kernel,congestion_wait()at el did that by callingio_schedule_timeout().How this
runqueue->nr_iowaitis handled now ?
Good question. io_schedule() has an old comment for throttling IO wait:
* But don't do that if it is a deliberate, throttling IO wait (this task
* has set its backing_dev_info: the queue against which it should throttle) */
void __sched io_schedule(void)
So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for balance_dirty_pages() :)
Thanks,
Fengguang
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
Quoted 23 lines · On Thu, 8 Oct 2009 09:58:22 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 13 lines · KAMEZAWA Hiroyuki
IIUC, "iowait" cpustat data was calculated by
runqueue->nr_iowaitas ==kernel/schec.c
voidaccount_idle_time(cputime_t cputime){ struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat; cputime64_t cputime64 = cputime_to_cputime64(cputime); struct rq *rq = this_rq(); if (atomic_read(&rq->nr_iowait) > 0) cpustat->iowait = cputime64_add(cpustat->iowait, cputime64); else cpustat->idle = cputime64_add(cpustat->idle, cputime64); }Quoted 6 lines
==
Then, for showing "cpu is in iowait",runqueue->nr_iowaitshould be modified at some places. In old kernel,congestion_wait()at el did that by callingio_schedule_timeout().How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:
Quoted 8 lines
* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)
Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to update throttle_vm_writeout() and some in vmscan.c. Thanks for input.
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Regards,
-Kame
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
Quoted 30 lines · KAMEZAWA Hiroyuki
Quoted 28 lines · On Thu, 8 Oct 2009 09:58:22 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 13 lines · KAMEZAWA Hiroyuki
IIUC, "iowait" cpustat data was calculated by
runqueue->nr_iowaitas ==kernel/schec.c
voidaccount_idle_time(cputime_t cputime){ struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat; cputime64_t cputime64 = cputime_to_cputime64(cputime); struct rq *rq = this_rq(); if (atomic_read(&rq->nr_iowait) > 0) cpustat->iowait = cputime64_add(cpustat->iowait, cputime64); else cpustat->idle = cputime64_add(cpustat->idle, cputime64); }Quoted 6 lines
==
Then, for showing "cpu is in iowait",runqueue->nr_iowaitshould be modified at some places. In old kernel,congestion_wait()at el did that by callingio_schedule_timeout().How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)
Quoted 6 lines
Quoted 3 lines
So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to updatethrottle_vm_writeout()and some invmscan.c. Thanks for input.
Thanks, you also reminds me to do io_schedule() in the nfs writeback wait queue :)
Quoted 1 line
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Good suggestion. How about this patch?
---
writeback: show per-bdi throttled tasks
All currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.
# cat /debug/bdi/8:0/throttle_list goal=6144kb waited=32ms goal=6144kb waited=48ms
--- fs/fs-writeback.c | 1 + include/linux/backing-dev.h | 2 ++ mm/backing-dev.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+)
include/linux/backing-dev.h +2 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {#ifdef CONFIG_DEBUG_FS struct dentry *debug_dir;struct dentry *debug_stats; +struct dentry *debug_throttle;#endif };
@@ -116,6 +117,7 @@ struct backing_dev_info {#define DIRTY_THROTTLE_PAGES_STOP (1 << 22)struct dirty_throttle_task { +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;
mm/backing-dev.c +33 -0
@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct sreturn 0 ; } +static int bdi_debug_throttle_show (struct seq_file *m,void *v) +{ +struct backing_dev_info *bdi = m->private; +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +list_for_each_entry (tt, &bdi->throttle_list, list) { +seq_printf (m,"goal=%lukb\twaited=%lums\n" , + tt->nr_pages << (PAGE_SHIFT -10 ), + (jiffies - tt->start_time) *1000 /HZ ); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return 0 ; +} +static int bdi_debug_stats_open (struct inode *inode,struct file *file) {return single_open (file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +144,31 @@ static const struct file_operations bdi_ .release = single_release, }; +static int bdi_debug_throttle_open (struct inode *inode,struct file *file) +{ +return single_open (file, bdi_debug_throttle_show, inode->i_private); +} + +static const struct file_operations bdi_debug_throttle_fops = { + .open = bdi_debug_throttle_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static void bdi_debug_register (struct backing_dev_info *bdi,const char *name) { bdi->debug_dir =debugfs_create_dir (name, bdi_debug_root); bdi->debug_stats =debugfs_create_file ("stats" ,0444 , bdi->debug_dir, bdi, &bdi_debug_stats_fops); + bdi->debug_throttle =debugfs_create_file ("throttle_list" ,0444 , + bdi->debug_dir, bdi, + &bdi_debug_throttle_fops); }static void bdi_debug_unregister (struct backing_dev_info *bdi) { +debugfs_remove (bdi->debug_throttle);debugfs_remove (bdi->debug_stats);debugfs_remove (bdi->debug_dir); }
fs/fs-writeback.c +1 -0
@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(strvoid bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { + .start_time = jiffies, .nr_pages = nr_pages, .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };
Quoted 11 lines · Wu Fengguang
Quoted 7 lines · Peter Staubach
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples) nr_writeback nr_dirty nr_unstable 11227 41463 38044 11227 41463 38044 11227 41463 38044 11227 41463 38044I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both
Yeah the server side is blocked. The nfsd are mostly blocked in generic_file_aio_write(), in particular, the i_mutex lock! I'm copying one or two big files over NFS, so the i_mutex lock is heavily contented.
I'm using the default wsize=4096 for NFS-root..
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 1 0.0 D< log_wait_commit nfsd 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.0 D< sync_buffer nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 1 0.0 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 D< get_request_wait nfsd 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 D< get_write_access nfsd 4691 4691 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 1 0.1 D< get_write_access nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
Thanks,
Fengguang
Quoted 57 lines
nr_writeback/nr_unstable. And the stuck
nr_writebackwill freezenr_dirtyas well, because the dirtying process is throttled until it receives enough "PG_writebackcleared" event, however the bdi-flush thread is also blocked when trying to clear morePG_writeback, because the client sidenr_writebacklimit has been reached. In summary,server blocked =>
nr_writebackstuck =>nr_writebacklimit reached => bdi-flush blocked => noend_page_writeback()=> dirtier blocked =>nr_dirtystuckThanks,
FengguangQuoted 44 lines
Quoted 44 lines
11045 53987 6490 11033 53120 8145 11195 52143 10886 11211 52144 10913 11211 52144 10913 11211 52144 10913btrfs seems to maintain a private pool of writeback pages, which can go out of control:
nr_writeback nr_dirty 261075 132 252891 195 244795 187 236851 187 228830 187 221040 218 212674 237 204981 237XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.
nr_writeback nr_dirty 80781 0 37117 37703 37117 43933 81044 6 81050 0 43943 10199 43930 36355 43930 36355 80293 0 80285 0 80285 0Thanks,
Fengguang--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Wu Fengguang
Quoted 9 lines · Wu Fengguang
Quoted 7 lines · Peter Staubach
Quoted 7 lines
# vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples) nr_writeback nr_dirty nr_unstable 11227 41463 38044 11227 41463 38044 11227 41463 38044 11227 41463 38044
Quoted 8 lines
Quoted 2 lines
I guess in the above 4 seconds, either client or (more likely) server is blocked. A blocked server cannot send ACKs to knock down both
Yeah the server side is blocked. The nfsd are mostly blocked in
generic_file_aio_write(), in particular, thei_mutexlock! I'm copying one or two big files over NFS, so thei_mutexlock is heavily contented.I'm using the default wsize=4096 for NFS-root..
Just switched to 512k wsize, and things improved: in most time the 8 nfsd are not all blocked. However, the bumpiness still remains:
nr_writeback nr_dirty nr_unstable
11105 58080 15042
11105 58080 15042
11233 54583 18626
11101 51964 22036
11105 51978 22065
11233 52362 22577
10985 58538 13500
11233 53748 19721
11047 51999 21778
11105 50262 23572
11105 50262 20441
10985 52772 20721
10977 52109 21516
11105 48296 26629
11105 48296 26629
10985 52191 21042
11166 51456 22296
10980 50681 24466
11233 45352 30488
11233 45352 30488
11105 45475 30616
11131 45313 20355
11233 51126 22637
11233 51126 22637wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 S< svc_recv nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 0 0.1 R< ? nfsd 4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd 4694 4694 TS - -5 24 0 0.1 S< svc_recv nfsd 4695 4695 TS - -5 24 0 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 0 0.1 R< ? nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 D< log_wait_commit nfsd 4693 4693 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 0 0.1 S< svc_recv nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 S< svc_recv nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 R< ? nfsd 4693 4693 TS - -5 24 1 0.1 R< ? nfsd 4694 4694 TS - -5 24 1 0.1 R< ? nfsd 4695 4695 TS - -5 24 1 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 S< svc_recv nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 S< svc_recv nfsd 4692 4692 TS - -5 24 1 0.1 D< nfsd_sync nfsd 4693 4693 TS - -5 24 1 0.1 D< sync_buffer nfsd 4694 4694 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 S< svc_recv nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
Thanks,
Fengguang
Quoted 103 lines
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 1 0.0 D< log_wait_commit nfsd 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs 329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.0 D< sync_buffer nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 1 0.0 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsdwfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4691 4691 TS - -5 24 0 0.1 D< get_request_wait nfsd 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd 4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd 4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsdwfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod 4690 4690 TS - -5 24 1 0.1 D< get_write_access nfsd 4691 4691 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4693 4693 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd 4694 4694 TS - -5 24 1 0.1 D< get_write_access nfsd 4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd 4697 4697 TS - -5 24 0 0.1 D< generic_file_aio_write nfsdThanks,
FengguangQuoted 57 lines
nr_writeback/nr_unstable. And the stuck
nr_writebackwill freezenr_dirtyas well, because the dirtying process is throttled until it receives enough "PG_writebackcleared" event, however the bdi-flush thread is also blocked when trying to clear morePG_writeback, because the client sidenr_writebacklimit has been reached. In summary,server blocked =>
nr_writebackstuck =>nr_writebacklimit reached => bdi-flush blocked => noend_page_writeback()=> dirtier blocked =>nr_dirtystuckThanks,
FengguangQuoted 44 lines
Quoted 44 lines
11045 53987 6490 11033 53120 8145 11195 52143 10886 11211 52144 10913 11211 52144 10913 11211 52144 10913btrfs seems to maintain a private pool of writeback pages, which can go out of control:
nr_writeback nr_dirty 261075 132 252891 195 244795 187 236851 187 228830 187 221040 218 212674 237 204981 237XFS has very interesting "bumpy writeback" behavior: it tends to wait collect enough pages and then write the whole world.
nr_writeback nr_dirty 80781 0 37117 37703 37117 43933 81044 6 81050 0 43943 10199 43930 36355 43930 36355 80293 0 80285 0 80285 0Thanks,
Fengguang--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to Wu FengguangQuoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki
Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to updatethrottle_vm_writeout()and some invmscan.c. Thanks for input.Thanks, you also reminds me to do
io_schedule()in the nfs writeback wait queue :)
good side effect ;)
Quoted 4 lines
Quoted 1 line
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Good suggestion. How about this patch?
Seems attractive.
Hmm..
==
struct dirty_throttle_task {
+ pid_t owner_pid;
==
and show it ? (too verbose ?
Regards,
-Kame
Quoted 95 lines
CC KAMEZAWA Hiroyuki <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 1 + include/linux/backing-dev.h | 2 ++ mm/backing-dev.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+)include/linux/backing-dev.h +2
-0@@ -101,6 +101,7 @@ struct backing_dev_info {#ifdef CONFIG_DEBUG_FS struct dentry *debug_dir;struct dentry *debug_stats; +struct dentry *debug_throttle;#endif };@@ -116,6 +117,7 @@ struct backing_dev_info {#define DIRTY_THROTTLE_PAGES_STOP (1 << 22)struct dirty_throttle_task { +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;mm/backing-dev.c +33
-0@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct sreturn 0 ; } +static int bdi_debug_throttle_show (struct seq_file *m,void *v) +{ +struct backing_dev_info *bdi = m->private; +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +list_for_each_entry (tt, &bdi->throttle_list, list) { +seq_printf (m,"goal=%lukb\twaited=%lums\n" , + tt->nr_pages << (PAGE_SHIFT -10 ), + (jiffies - tt->start_time) *1000 /HZ ); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return 0 ; +} +static int bdi_debug_stats_open (struct inode *inode,struct file *file) {return single_open (file, bdi_debug_stats_show, inode->i_private);@@ -127,15 +144,31 @@ static const struct file_operations bdi_ .release = single_release, }; +static int bdi_debug_throttle_open (struct inode *inode,struct file *file) +{ +return single_open (file, bdi_debug_throttle_show, inode->i_private); +} + +static const struct file_operations bdi_debug_throttle_fops = { + .open = bdi_debug_throttle_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static void bdi_debug_register (struct backing_dev_info *bdi,const char *name) { bdi->debug_dir =debugfs_create_dir (name, bdi_debug_root); bdi->debug_stats =debugfs_create_file ("stats" ,0444 , bdi->debug_dir, bdi, &bdi_debug_stats_fops); + bdi->debug_throttle =debugfs_create_file ("throttle_list" ,0444 , + bdi->debug_dir, bdi, + &bdi_debug_throttle_fops); }static void bdi_debug_unregister (struct backing_dev_info *bdi) { +debugfs_remove (bdi->debug_throttle);debugfs_remove (bdi->debug_stats);debugfs_remove (bdi->debug_dir); }fs/fs-writeback.c +1
-0@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(strvoid bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { + .start_time = jiffies, .nr_pages = nr_pages, .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
Quoted 4 lines · KAMEZAWA Hiroyuki
Quoted 1 line · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
On Thu, Oct 08, 2009 at 10:40:37AM +0800, KAMEZAWA Hiroyuki wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to updatethrottle_vm_writeout()and some invmscan.c. Thanks for input.Thanks, you also reminds me to do
io_schedule()in the nfs writeback wait queue :)good side effect ;)
Done, hehe.
Quoted 10 lines
Quoted 4 lines
Quoted 1 line
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Good suggestion. How about this patch?
Seems attractive.
Hmm..
==
structdirty_throttle_task{(quoted diff) +1
-0+ pid_t owner_pid;
Quoted 2 lines
==
and show it ? (too verbose ?
I even thought of adding the task pointer - it won't go away anyway :)
Thanks,
Fengguang
Quoted 99 lines
Regards,
-KameQuoted 95 lines
CC KAMEZAWA Hiroyuki <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 1 + include/linux/backing-dev.h | 2 ++ mm/backing-dev.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+)include/linux/backing-dev.h +2
-0@@ -101,6 +101,7 @@ struct backing_dev_info {#ifdef CONFIG_DEBUG_FS struct dentry *debug_dir;struct dentry *debug_stats; +struct dentry *debug_throttle;#endif };@@ -116,6 +117,7 @@ struct backing_dev_info {#define DIRTY_THROTTLE_PAGES_STOP (1 << 22)struct dirty_throttle_task { +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;mm/backing-dev.c +33
-0@@ -115,6 +115,23 @@ static int bdi_debug_stats_show(struct sreturn 0 ; } +static int bdi_debug_throttle_show (struct seq_file *m,void *v) +{ +struct backing_dev_info *bdi = m->private; +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +list_for_each_entry (tt, &bdi->throttle_list, list) { +seq_printf (m,"goal=%lukb\twaited=%lums\n" , + tt->nr_pages << (PAGE_SHIFT -10 ), + (jiffies - tt->start_time) *1000 /HZ ); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return 0 ; +} +static int bdi_debug_stats_open (struct inode *inode,struct file *file) {return single_open (file, bdi_debug_stats_show, inode->i_private);@@ -127,15 +144,31 @@ static const struct file_operations bdi_ .release = single_release, }; +static int bdi_debug_throttle_open (struct inode *inode,struct file *file) +{ +return single_open (file, bdi_debug_throttle_show, inode->i_private); +} + +static const struct file_operations bdi_debug_throttle_fops = { + .open = bdi_debug_throttle_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static void bdi_debug_register (struct backing_dev_info *bdi,const char *name) { bdi->debug_dir =debugfs_create_dir (name, bdi_debug_root); bdi->debug_stats =debugfs_create_file ("stats" ,0444 , bdi->debug_dir, bdi, &bdi_debug_stats_fops); + bdi->debug_throttle =debugfs_create_file ("throttle_list" ,0444 , + bdi->debug_dir, bdi, + &bdi_debug_throttle_fops); }static void bdi_debug_unregister (struct backing_dev_info *bdi) { +debugfs_remove (bdi->debug_throttle);debugfs_remove (bdi->debug_stats);debugfs_remove (bdi->debug_dir); }fs/fs-writeback.c +1
-0@@ -284,6 +284,7 @@ static void bdi_calc_write_bandwidth(strvoid bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { + .start_time = jiffies, .nr_pages = nr_pages, .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to KAMEZAWA HiroyukiQuoted 23 lines · KAMEZAWA Hiroyuki
Quoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki
Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to updatethrottle_vm_writeout()and some invmscan.c. Thanks for input.Thanks, you also reminds me to do
io_schedule()in the nfs writeback wait queue :)good side effect ;)
Quoted 4 lines
Quoted 1 line
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Good suggestion. How about this patch?
Seems attractive.
Hmm..
==
structdirty_throttle_task{(quoted diff) +1
-0+ pid_t owner_pid;
Quoted 2 lines
==
and show it ? (too verbose ?
Not all all, added comm too :)
---
writeback: show per-bdi throttled tasks
All currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.
# cat /debug/bdi/0:16/throttle_list goal=768kb waited=36ms pid=3551 comm=cp
--- fs/fs-writeback.c | 6 +++-- include/linux/backing-dev.h | 3 ++ mm/backing-dev.c | 35 ++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 2 deletions(-)
include/linux/backing-dev.h +3 -0
@@ -101,6 +101,7 @@ struct backing_dev_info {#ifdef CONFIG_DEBUG_FS struct dentry *debug_dir;struct dentry *debug_stats; +struct dentry *debug_throttle;#endif };
@@ -116,6 +117,8 @@ struct backing_dev_info {#define DIRTY_THROTTLE_PAGES_STOP (1 << 22)struct dirty_throttle_task { +struct task_struct *sleeper; +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;
mm/backing-dev.c +35 -0
@@ -115,6 +115,25 @@ static int bdi_debug_stats_show(struct sreturn 0 ; } +static int bdi_debug_throttle_show (struct seq_file *m,void *v) +{ +struct backing_dev_info *bdi = m->private; +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +list_for_each_entry (tt, &bdi->throttle_list, list) { +seq_printf (m,"goal=%lukb\twaited=%lums\tpid=%d\tcomm=%s\n" , + tt->nr_pages << (PAGE_SHIFT -10 ), + (jiffies - tt->start_time) *1000 /HZ , + tt->sleeper->pid, + tt->sleeper->comm); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return 0 ; +} +static int bdi_debug_stats_open (struct inode *inode,struct file *file) {return single_open (file, bdi_debug_stats_show, inode->i_private);
@@ -127,15 +146,31 @@ static const struct file_operations bdi_ .release = single_release, }; +static int bdi_debug_throttle_open (struct inode *inode,struct file *file) +{ +return single_open (file, bdi_debug_throttle_show, inode->i_private); +} + +static const struct file_operations bdi_debug_throttle_fops = { + .open = bdi_debug_throttle_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static void bdi_debug_register (struct backing_dev_info *bdi,const char *name) { bdi->debug_dir =debugfs_create_dir (name, bdi_debug_root); bdi->debug_stats =debugfs_create_file ("stats" ,0444 , bdi->debug_dir, bdi, &bdi_debug_stats_fops); + bdi->debug_throttle =debugfs_create_file ("throttle_list" ,0444 , + bdi->debug_dir, bdi, + &bdi_debug_throttle_fops); }static void bdi_debug_unregister (struct backing_dev_info *bdi) { +debugfs_remove (bdi->debug_throttle);debugfs_remove (bdi->debug_stats);debugfs_remove (bdi->debug_dir); }
fs/fs-writeback.c +4 -2
@@ -284,8 +284,10 @@ static void bdi_calc_write_bandwidth(strvoid bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { - .nr_pages = nr_pages, - .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), + .sleeper = current, + .start_time = jiffies, + .nr_pages = nr_pages, + .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };unsigned long flags;
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
Quoted 19 lines · On Thu, 8 Oct 2009 14:28:09 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 8 lines · On Thu, 8 Oct 2009 12:01:36 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 3 lines · KAMEZAWA Hiroyuki
Thank you for clarification.
Then, hmm, %iotwait (which 'top' shows) didn't work as desgined and we need to updatethrottle_vm_writeout()and some invmscan.c. Thanks for input.Thanks, you also reminds me to do
io_schedule()in the nfs writeback wait queue :)good side effect ;)
Quoted 4 lines
Quoted 1 line
BTW, I'm glad if I can know "how many threads/ios are throttoled now" per bdi.
Good suggestion. How about this patch?
Quoted 10 lines
Quoted 7 lines
Seems attractive.
Hmm..
==
structdirty_throttle_task{(quoted diff) +1
-0+ pid_t owner_pid;==
and show it ? (too verbose ?Not all all, added comm too :)
Wow, nice.
Thank you,
-Kame
Quoted 113 lines
---
writeback: show per-bdi throttled tasksAll currently throttled tasks will be listed, showing the pages to writeback for them, and total wait time since blocked.
# cat /debug/bdi/0:16/throttle_list goal=768kb waited=36ms pid=3551 comm=cpCC KAMEZAWA Hiroyuki <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 6 +++-- include/linux/backing-dev.h | 3 ++ mm/backing-dev.c | 35 ++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 2 deletions(-)include/linux/backing-dev.h +3
-0@@ -101,6 +101,7 @@ struct backing_dev_info {#ifdef CONFIG_DEBUG_FS struct dentry *debug_dir;struct dentry *debug_stats; +struct dentry *debug_throttle;#endif };@@ -116,6 +117,8 @@ struct backing_dev_info {#define DIRTY_THROTTLE_PAGES_STOP (1 << 22)struct dirty_throttle_task { +struct task_struct *sleeper; +unsigned long start_time;long nr_pages;struct list_head list;struct completion complete;mm/backing-dev.c +35
-0@@ -115,6 +115,25 @@ static int bdi_debug_stats_show(struct sreturn 0 ; } +static int bdi_debug_throttle_show (struct seq_file *m,void *v) +{ +struct backing_dev_info *bdi = m->private; +struct dirty_throttle_task *tt; +unsigned long flags; + +spin_lock_irqsave (&bdi->throttle_lock, flags); +list_for_each_entry (tt, &bdi->throttle_list, list) { +seq_printf (m,"goal=%lukb\twaited=%lums\tpid=%d\tcomm=%s\n" , + tt->nr_pages << (PAGE_SHIFT -10 ), + (jiffies - tt->start_time) *1000 /HZ , + tt->sleeper->pid, + tt->sleeper->comm); + } +spin_unlock_irqrestore (&bdi->throttle_lock, flags); + +return 0 ; +} +static int bdi_debug_stats_open (struct inode *inode,struct file *file) {return single_open (file, bdi_debug_stats_show, inode->i_private);@@ -127,15 +146,31 @@ static const struct file_operations bdi_ .release = single_release, }; +static int bdi_debug_throttle_open (struct inode *inode,struct file *file) +{ +return single_open (file, bdi_debug_throttle_show, inode->i_private); +} + +static const struct file_operations bdi_debug_throttle_fops = { + .open = bdi_debug_throttle_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static void bdi_debug_register (struct backing_dev_info *bdi,const char *name) { bdi->debug_dir =debugfs_create_dir (name, bdi_debug_root); bdi->debug_stats =debugfs_create_file ("stats" ,0444 , bdi->debug_dir, bdi, &bdi_debug_stats_fops); + bdi->debug_throttle =debugfs_create_file ("throttle_list" ,0444 , + bdi->debug_dir, bdi, + &bdi_debug_throttle_fops); }static void bdi_debug_unregister (struct backing_dev_info *bdi) { +debugfs_remove (bdi->debug_throttle);debugfs_remove (bdi->debug_stats);debugfs_remove (bdi->debug_dir); }fs/fs-writeback.c +4
-2@@ -284,8 +284,10 @@ static void bdi_calc_write_bandwidth(strvoid bdi_writeback_wait (struct backing_dev_info *bdi,long nr_pages) {struct dirty_throttle_task tt = { - .nr_pages = nr_pages, - .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), + .sleeper = current, + .start_time = jiffies, + .nr_pages = nr_pages, + .complete =COMPLETION_INITIALIZER_ONSTACK (tt.complete), };unsigned long flags;
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to KAMEZAWA HiroyukiQuoted 21 lines · KAMEZAWA Hiroyuki
May I ask a question ? (maybe not directly related to this patch itself, sorry)
Recent works as "writeback: switch to per-bdi threads for flushing data" removed
congestion_wait()frombalance_dirty_pages()and addedschedule_timeout_interruptible().And this one replaces it with wake_up+wait_queue.
IIUC, "iowait" cpustat data was calculated by
runqueue->nr_iowaitas ==kernel/schec.c
voidaccount_idle_time(cputime_t cputime){ struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat; cputime64_t cputime64 = cputime_to_cputime64(cputime); struct rq *rq = this_rq(); if (atomic_read(&rq->nr_iowait) > 0) cpustat->iowait = cputime64_add(cpustat->iowait, cputime64); else cpustat->idle = cputime64_add(cpustat->idle, cputime64); }
Quoted 6 lines
==
Then, for showing "cpu is in iowait",runqueue->nr_iowaitshould be modified at some places. In old kernel,congestion_wait()at el did that by callingio_schedule_timeout().How this
runqueue->nr_iowaitis handled now ?
Ah, I think you've got a good point, we need a io_schedule() there.
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to Wu FengguangQuoted 5 lines · Wu Fengguang
Quoted 2 lines
How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:
Quoted 7 lines
* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)
Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..
Hmm?
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
Quoted 11 lines · On Thu, 08 Oct 2009 10:08:36 +0200 Peter Zijlstra <[email protected]>…
Quoted 10 lines · Wu Fengguang
Quoted 2 lines
How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)
Quoted 8 lines
Quoted 2 lines
So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..
Hmm?
Above comment "don't do that if it is a deliberate, throttling IO wait" is really old but ignored.
I pesonally don't like to change the meanig of iowait in /proc/stat. But I'm not sure which is better to change the definitiion (which was ignored) or fix behavior (not correct very long time)...
Hmm?, too ;)
Regards,
-Kame
Re: [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages()
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
Quoted 10 lines · Wu Fengguang
Quoted 2 lines
How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)
Quoted 7 lines
Quoted 2 lines
So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..
Hmm?
Yep agree, if we're deliberately waiting on IO, it should count as iowait time.
Signature · 1 line
Jens Axboe
PATCH 1/1 writeback: account IO throttling wait as iowait
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Wu Fengguang
Quoted 2 lines
How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)
Quoted 7 lines
Quoted 4 lines
Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..
Hmm?
Yep agree, if we're deliberately waiting on IO, it should count as iowait time.
Then let's revert to the old behavior :)
For one single cp, it increases iowait from 29% to 56%.
Before patch:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 4 64 28 0 3| 0 0 | 272k 10M| 0 0 |1854 863 0 6 69 23 0 3| 0 0 | 249k 11M| 0 0 |1709 865 0 6 64 27 0 4| 0 0 | 235k 10M| 0 0 |1807 788 0 4 61 30 0 4| 0 0 | 271k 12M| 0 0 |1910 898 0 4 72 21 0 4| 0 0 | 289k 13M| 0 0 |1832 905 0 6 58 35 0 2| 0 0 | 252k 11M| 0 0 |1713 900 0 4 54 38 0 4| 0 0 | 257k 11M| 0 0 |1777 841 0 5 59 30 0 7| 0 0 | 270k 12M| 0 0 |1758 836
After patch:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 5 35 57 0 4| 0 0 | 255k 11M| 0 0 |1705 879 0 4 38 53 0 4| 0 0 | 326k 14M| 0 0 |1940 980 0 3 36 59 0 2| 0 0 | 291k 13M| 0 0 |1970 970 0 4 28 66 0 2| 0 0 | 290k 13M| 0 0 |1805 928 0 6 38 54 0 3| 0 0 | 230k 10M| 0 0 |1866 842 0 5 44 49 0 4| 0 0 | 278k 12M| 0 0 |1808 868
Thanks,
Fengguang
---
writeback: account IO throttling wait as iowait
It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.
There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.
If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the io_schedule.
--- kernel/sched.c | 3 --- mm/page-writeback.c | 3 ++- 2 files changed, 2 insertions(+), 4 deletions(-)
mm/page-writeback.c +2 -1
@@ -566,7 +566,8 @@ static void balance_dirty_pages(struct aif (pages_written >= write_chunk)break ;/* We've done our duty */ -schedule_timeout_interruptible (pause); +__set_current_state (TASK_INTERRUPTIBLE ); +io_schedule_timeout (pause);/* * Increase the delay for each loop, up to our previous
kernel/sched.c +0 -3
@@ -6720,9 +6720,6 @@ EXPORT_SYMBOL(yield);/* * This task is about to go to sleep on IO. Increment rq->nr_iowait so * that process accounting knows that this is a task in IO wait state. -* -* But don't do that if it is a deliberate, throttling IO wait (this task -* has set its backing_dev_info: the queue against which it should throttle) */ void __sched io_schedule(void) {
Re: [PATCH] writeback: account IO throttling wait as iowait
Quoted 57 lines · Wu Fengguang
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Wu Fengguang
Quoted 2 lines
How this
runqueue->nr_iowaitis handled now ?Good question.
io_schedule()has an old comment for throttling IO wait:* But don't do that if it is a deliberate, throttling IO wait (this task
* has set itsbacking_dev_info: the queue against which it should throttle) */
void__schedio_schedule(void)So it looks both Jens' and this patch behaves right in ignoring the iowait accounting for
balance_dirty_pages():)Quoted 7 lines
Quoted 4 lines
Well it is a change in behaviour, and I think IOWAIT makes sense when we're blocked due to io throttle..
Hmm?
Yep agree, if we're deliberately waiting on IO, it should count as iowait time.
Then let's revert to the old behavior :)
For one single cp, it increases iowait from 29% to 56%.
Before patch:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 4 64 28 0 3| 0 0 | 272k 10M| 0 0 |1854 863 0 6 69 23 0 3| 0 0 | 249k 11M| 0 0 |1709 865 0 6 64 27 0 4| 0 0 | 235k 10M| 0 0 |1807 788 0 4 61 30 0 4| 0 0 | 271k 12M| 0 0 |1910 898 0 4 72 21 0 4| 0 0 | 289k 13M| 0 0 |1832 905 0 6 58 35 0 2| 0 0 | 252k 11M| 0 0 |1713 900 0 4 54 38 0 4| 0 0 | 257k 11M| 0 0 |1777 841 0 5 59 30 0 7| 0 0 | 270k 12M| 0 0 |1758 836After patch:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 5 35 57 0 4| 0 0 | 255k 11M| 0 0 |1705 879 0 4 38 53 0 4| 0 0 | 326k 14M| 0 0 |1940 980 0 3 36 59 0 2| 0 0 | 291k 13M| 0 0 |1970 970 0 4 28 66 0 2| 0 0 | 290k 13M| 0 0 |1805 928 0 6 38 54 0 3| 0 0 | 230k 10M| 0 0 |1866 842 0 5 44 49 0 4| 0 0 | 278k 12M| 0 0 |1808 868Thanks,
Fengguang
---
writeback: account IO throttling wait as iowait
Quoted 8 lines
It makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.
There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.
If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the
io_schedule.
Thanks, queued up.
Signature · 1 line
Jens Axboe
Re: [PATCH] writeback: account IO throttling wait as iowait
Quoted 6 lines · Jens Axboe
Quoted 6 lines
---
writeback: account IO throttling wait as iowaitIt makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.
Quoted 7 lines
Quoted 5 lines
There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.
If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the
io_schedule.Thanks, queued up.
Thank you. Would you also pick up this one if it's OK to you?
Thanks,
Fengguang
---
writeback: kill space in debugfs item name
The space is not script friendly, kill it.
--- mm/backing-dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/backing-dev.c +1 -1
@@ -92,7 +92,7 @@ static int bdi_debug_stats_show(struct s"BdiDirtyThresh: %8lu kB\n" "DirtyThresh: %8lu kB\n" "BackgroundThresh: %8lu kB\n" -"WriteBack threads:%8lu\n" +"WritebackThreads: %8lu\n" "b_dirty: %8lu\n" "b_io: %8lu\n" "b_more_io: %8lu\n"
Re: [PATCH] writeback: account IO throttling wait as iowait
Quoted 10 lines · Wu Fengguang
Quoted 9 lines · Jens Axboe
Quoted 9 lines
---
writeback: account IO throttling wait as iowaitIt makes sense to do IOWAIT when someone is blocked due to IO throttle, as suggested by Kame and Peter.
There is an old comment for not doing IOWAIT on throttle, however it has been mismatching the code for a long time.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If we stop accounting IOWAIT for 2.6.32, it could be an undesirable behavior change. So restore the
io_schedule.Thanks, queued up.
Thank you. Would you also pick up this one if it's OK to you?
Sure, looks appropriate.
Signature · 1 line
Jens Axboe
Jan Kara
maintainerRe: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Wu FengguangHi,
Quoted 22 lines · Wu Fengguang
From: Richard Kennedy <[email protected]>
Reducing the number of times
balance_dirty_pagescallsglobal_page_statereduces the cache references and so improves write performance on a variety of workloads.'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_thresholdapprox 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviationaverage (s.d.) in millions (10^6)2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)Achieving this reduction is by dropping
clip_bdi_dirty_limitas it rereads the counters to apply thedirty_thresholdand moving this check up intobalance_dirty_pageswhere it has already read the counters.
Quoted 6 lines
Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.
In the common case with no throttling it now calls
global_page_state5 fewer times andbdi_stat2 fewer.
Hmm, but the patch changes the behavior of balance_dirty_pages() in
several ways:
Quoted 30 lines
mm/page-writeback.c +0
-26-/* -* Clip the earned share of dirty pages to that which is actually available. -* This avoids exceeding the total dirty_limit when the floating averages -* fluctuate too quickly. -*/ -static void clip_bdi_dirty_limit (struct backing_dev_info *bdi, -unsigned long dirty,unsigned long *pbdi_dirty) -{ -unsigned long avail_dirty; - - avail_dirty =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_WRITEBACK ) + -global_page_state (NR_UNSTABLE_NFS ) + -global_page_state (NR_WRITEBACK_TEMP ); - -if (avail_dirty < dirty) - avail_dirty = dirty - avail_dirty; -else - avail_dirty =0 ; - - avail_dirty +=bdi_stat (bdi,BDI_RECLAIMABLE ) + -bdi_stat (bdi,BDI_WRITEBACK ); - - *pbdi_dirty =min (*pbdi_dirty, avail_dirty); -} -static inline void task_dirties_fraction (struct task_struct *tsk,long *numerator,long *denominator) {
mm/page-writeback.c get_dirty_limits(unsigned long *pbackgro · line 471
Quoted 4 lines
mm/page-writeback.c +0
-1@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro bdi_dirty = dirty * bdi->max_ratio /100 ; *pbdi_dirty = bdi_dirty; -clip_bdi_dirty_limit (bdi, dirty, pbdi_dirty);
I don't see, what test in balance_dirty_limits() should replace this
clipping... OTOH clipping does not seem to have too much effect on the behavior of balance_dirty_pages - the limit we clip to (at least BDI_WRITEBACK + BDI_RECLAIMABLE) is large enough so that we break from the loop immediately. So just getting rid of the function is fine but I'd update the changelog accordingly.
Quoted 31 lines
mm/page-writeback.c +19
-5@@ -503,16 +476,36 @@ static void balance_dirty_pages(struct a };get_dirty_limits (&background_thresh, &dirty_thresh, - &bdi_thresh, bdi); + &bdi_thresh, bdi); nr_reclaimable =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_UNSTABLE_NFS ); - nr_writeback =global_page_state (NR_WRITEBACK ); +global_page_state (NR_UNSTABLE_NFS ); + nr_writeback =global_page_state (NR_WRITEBACK ) + +global_page_state (NR_WRITEBACK_TEMP ); - bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); +/* +* In order to avoid the stacked BDI deadlock we need +* to ensure we accurately count the 'dirty' pages when +* the threshold is low. +* +* Otherwise it would be possible to get thresh+n pages +* reported dirty, even though there are thresh-m pages +* actually dirty; with m+n sitting in the percpu +* deltas. +*/ +if (bdi_thresh <2 *bdi_stat_error (bdi)) { + bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); + bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); + }else { + bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE );
mm/page-writeback.c static void balance_dirty_pages(struct a · line 509
Quoted 10 lines
mm/page-writeback.c +7
-1+ bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK ); + } + + dirty_exceeded = + (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) + || (nr_reclaimable + nr_writeback >= dirty_thresh); -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh) +if (!dirty_exceeded)break ;
Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.
Quoted 34 lines
mm/page-writeback.c +1
-21/* @@ -521,7 +514,7 @@ static void balance_dirty_pages(struct a* when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < - (background_thresh + dirty_thresh) /2 ) + (background_thresh + dirty_thresh) /2 )break ;if (!bdi->dirty_exceeded)@@ -539,33 +532,10 @@ static void balance_dirty_pages(struct aif (bdi_nr_reclaimable > bdi_thresh) {writeback_inodes_wbc (&wbc); pages_written += write_chunk - wbc.nr_to_write; -get_dirty_limits (&background_thresh, &dirty_thresh, - &bdi_thresh, bdi); - } - -/* -* In order to avoid the stacked BDI deadlock we need -* to ensure we accurately count the 'dirty' pages when -* the threshold is low. -* -* Otherwise it would be possible to get thresh+n pages -* reported dirty, even though there are thresh-m pages -* actually dirty; with m+n sitting in the percpu -* deltas. -*/ -if (bdi_thresh <2 *bdi_stat_error (bdi)) { - bdi_nr_reclaimable =bdi_stat_sum (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat_sum (bdi,BDI_WRITEBACK ); - }else if (bdi_nr_reclaimable) { - bdi_nr_reclaimable =bdi_stat (bdi,BDI_RECLAIMABLE ); - bdi_nr_writeback =bdi_stat (bdi,BDI_WRITEBACK );
mm/page-writeback.c static void balance_dirty_pages(struct a · line 568
Quoted 10 lines
mm/page-writeback.c +3
-6+/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh) -break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -
Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...
I'd think that if we did all that work in writeback_inodes_wbc we could
spend the effort on regetting and rechecking the stats...
mm/page-writeback.c static void balance_dirty_pages(struct a · line 551
Quoted 11 lines
mm/page-writeback.c +1
-2schedule_timeout_interruptible (pause);/* @@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; } -if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;
Here we fail to clear dirty_exceeded if we are over global dirty limit
but not over per-bdi dirty limit...
mm/page-writeback.c static void balance_dirty_pages(struct a · line 567
Quoted 10 lines
mm/page-writeback.c +1
-3@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a* background_thresh, to keep the amount of dirty memory low. */ if ((laptop_mode && pages_written) || - (!laptop_mode && ((global_page_state (NR_FILE_DIRTY ) - +global_page_state (NR_UNSTABLE_NFS )) - > background_thresh))) + (!laptop_mode && (nr_reclaimable > background_thresh)))bdi_start_writeback (bdi,NULL ,0 ); }
This might be based on rather old values in case we break from the loop
after calling writeback_inodes_wbc.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 3 lines · Jan Kara
Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.
It should have, we should throttle everything calling balance_dirty_pages() when we're over the total limit.
Jan Kara
maintainerRe: [PATCH 04/45] writeback: remove unused nonblocking and congestion checks
In reply to Wu FengguangQuoted 6 lines · Wu Fengguang
- no one is calling
wb_writebackandwrite_cache_pageswith wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion waitSo remove the congestion checks as suggested by Chris.
Looks good. Since encountered_congestion isn't used, you can delete it as
well... BTW, you might need to split this patch to per-fs chunks for the sake of merging.
Honza
Quoted 197 lines
CC Chris Mason <[email protected]>CC Jens Axboe <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- drivers/staging/pohmelfs/inode.c | 9 --------- fs/afs/write.c | 16 +--------------- fs/cifs/file.c | 10 ---------- fs/fs-writeback.c | 8 -------- fs/gfs2/aops.c | 10 ---------- fs/xfs/linux-2.6/xfs_aops.c | 6 +----- mm/page-writeback.c | 12 ------------ 7 files changed, 2 insertions(+), 69 deletions(-)fs/fs-writeback.c +0
-8@@ -660,14 +660,6 @@ static void writeback_inodes_wb(struct bcontinue ; } -if (wbc->nonblocking &&bdi_write_congested (wb->bdi)) { - wbc->encountered_congestion =1 ; -if (!is_blkdev_sb) -break ;/* Skip a congested fs */ -requeue_io (inode); -continue ;/* Skip a congested blockdev */ - } -/* * Was this inode dirtied after sync_sb_inodes was called? * This keeps sync from extra jobs and livelock. mm/page-writeback.c +0
-12@@ -787,7 +787,6 @@ int write_cache_pages(struct address_spa struct writeback_control *wbc, writepage_t writepage, void *data) { - struct backing_dev_info *bdi = mapping->backing_dev_info; int ret = 0;int done =0 ;struct pagevec pvec;@@ -800,11 +799,6 @@ int write_cache_pages(struct address_spaint range_whole =0 ;long nr_to_write = wbc->nr_to_write; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -pagevec_init (&pvec,0 );if (wbc->range_cyclic) { writeback_index = mapping->writeback_index;/* prev offset */ @@ -923,12 +917,6 @@ continue_unlock:break ; } } - -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - done =1 ; -break ; - } }pagevec_release (&pvec);cond_resched ();drivers/staging/pohmelfs/inode.c +0
-9@@ -152,11 +152,6 @@ static int pohmelfs_writepages(struct adint scanned =0 ;int range_whole =0 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -if (wbc->range_cyclic) { index = mapping->writeback_index;/* Start from prev offset */ end =-1 ;@@ -248,10 +243,6 @@ retry:if (wbc->nr_to_write <=0 ) done =1 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - done =1 ; - }continue ; out_continue:fs/afs/write.c +1
-15@@ -455,8 +455,6 @@ int afs_writepage(struct page *page, str } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) - wbc->encountered_congestion =1 ;_leave (" = 0" );return 0 ;@@ -529,11 +527,6 @@ static int afs_writepages_region(struct wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -break ; - } -cond_resched (); }while (index < end && wbc->nr_to_write >0 );@@ -554,18 +547,11 @@ int afs_writepages(struct address_space_enter ("" ); -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -_leave (" = 0 [congest]" ); -return 0 ; - } -if (wbc->range_cyclic) { start = mapping->writeback_index; end =-1 ; ret =afs_writepages_region (mapping, wbc, start, end, &next); -if (start >0 && wbc->nr_to_write >0 && ret ==0 && - !(wbc->nonblocking && wbc->encountered_congestion)) +if (start >0 && wbc->nr_to_write >0 && ret ==0 ) ret =afs_writepages_region (mapping, wbc,0 , start, &next); mapping->writeback_index = next;fs/cifs/file.c +0
-10@@ -1379,16 +1379,6 @@ static int cifs_writepages(struct addresreturn generic_writepages (mapping, wbc); -/* -* BB: Is this meaningful for a non-block-device file system? -* If it is, we should test it again after we do I/O -*/ -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -kfree (iov); -return 0 ; - } - xid =GetXid ();pagevec_init (&pvec,0 );fs/gfs2/aops.c +0
-10@@ -313,11 +313,6 @@ static int gfs2_write_jdata_pagevec(struif (ret || (--(wbc->nr_to_write) <=0 )) ret =1 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; - ret =1 ; - } - }gfs2_trans_end (sdp);return ret;@@ -348,11 +343,6 @@ static int gfs2_write_cache_jdata(structint scanned =0 ;int range_whole =0 ; -if (wbc->nonblocking &&bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; -return 0 ; - } -pagevec_init (&pvec,0 );if (wbc->range_cyclic) { index = mapping->writeback_index;/* Start from prev offset */ fs/xfs/linux-2.6/xfs_aops.c +1
-5@@ -890,12 +890,8 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { - wbc->encountered_congestion =1 ; +if (wbc->nr_to_write <=0 ) done =1 ; - }else if (wbc->nr_to_write <=0 ) { - done =1 ; - } }xfs_start_page_writeback (page, !page_dirty, count); }
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 07/45] writeback: dont redirty tail an inode with dirty pages
In reply to Wu FengguangQuoted 2 lines · Wu Fengguang
This avoids delaying writeback for an expired (XFS) inode with lots of dirty pages, but no active dirtier at the moment.
OK, looks good.
Honza
Quoted 53 lines
CC Dave Chinner <[email protected]>CC Christoph Hellwig <[email protected]>Signed-off-by Wu Fengguang <[email protected]>--- fs/fs-writeback.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-)fs/fs-writeback.c +7
-13@@ -479,18 +479,7 @@ writeback_single_inode(struct inode *inospin_lock (&inode_lock); inode->i_state &= ~I_SYNC ;if (!(inode->i_state & (I_FREEING |I_CLEAR ))) { -if ((inode->i_state &I_DIRTY_PAGES ) && wbc->for_kupdate) { -/* -* More pages get dirtied by a fast dirtier. -*/ - goto select_queue; - }else if (inode->i_state &I_DIRTY ) { -/* -* At least XFS will redirty the inode during the -* writeback (delalloc) and on io completion (isize). -*/ -redirty_tail (inode); - }else if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) { +if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {/* * We didn't write back all the pages. nfs_writepages() * sometimes bales out without doing anything. Redirty @@ -512,7 +501,6 @@ writeback_single_inode(struct inode *ino* soon as the queue becomes uncongested. */ inode->i_state |=I_DIRTY_PAGES ; -select_queue:if (wbc->nr_to_write <=0 ) {/* * slice used up: queue for next turn @@ -535,6 +523,12 @@ select_queue: inode->i_state |=I_DIRTY_PAGES ;redirty_tail (inode); } + }else if (inode->i_state &I_DIRTY ) { +/* +* At least XFS will redirty the inode during the +* writeback (delalloc) and on io completion (isize). +*/ +redirty_tail (inode); }else if (atomic_read (&inode->i_count)) {/* * The inode is clean, inuse
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerRe: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 3 lines · Jan Kara
Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.It should have, we should throttle everything calling
balance_dirty_pages()when we're over the total limit.
OK :) I agree it's reasonable. But Wu, please note this in the
changelog because it might be a substantial change for some loads.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 04/45] writeback: remove unused nonblocking and congestion checks
In reply to Jan KaraQuoted 10 lines · Jan Kara
Quoted 6 lines · Wu Fengguang
- no one is calling
wb_writebackandwrite_cache_pageswith wbc.nonblocking=1 any more
- lumpy pageout will want to do nonblocking writeback without the congestion waitSo remove the congestion checks as suggested by Chris.
Looks good. Since
encountered_congestionisn't used, you can delete it as
well... BTW, you might need to split this patch to per-fs chunks for the sake of merging.
OK, good suggestions to follow :)
Thanks,
Fengguang
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Jan KaraQuoted 25 lines · Jan Kara
Hi,
Quoted 22 lines · Wu Fengguang
From: Richard Kennedy <[email protected]>
Reducing the number of times
balance_dirty_pagescallsglobal_page_statereduces the cache references and so improves write performance on a variety of workloads.'perf stats' of simple fio write tests shows the reduction in cache access.
Where the test is fio 'write,mmap,600Mb,pre_read' on AMD AthlonX2 with 3Gb memory (dirty_thresholdapprox 600 Mb)
running each test 10 times, dropping the fasted & slowest values then taking
the average & standard deviationaverage (s.d.) in millions (10^6)2.6.31-rc8 648.6 (14.6)
+patch 620.1 (16.5)Achieving this reduction is by dropping
clip_bdi_dirty_limitas it rereads the counters to apply thedirty_thresholdand moving this check up intobalance_dirty_pageswhere it has already read the counters.
Quoted 8 lines
Quoted 6 lines
Also by rearrange the for loop to only contain one copy of the limit tests allows the pdflush test after the loop to use the local copies of the counters rather than rereading them.
In the common case with no throttling it now calls
global_page_state5 fewer times andbdi_stat2 fewer.Hmm, but the patch changes the behavior of
balance_dirty_pages()in
several ways:
Yes, unfortunately the changelog failed to make that clear ..
Quoted 32 lines
Quoted 32 lines
mm/page-writeback.c +0
-26-/* -* Clip the earned share of dirty pages to that which is actually available. -* This avoids exceeding the total dirty_limit when the floating averages -* fluctuate too quickly. -*/ -static void clip_bdi_dirty_limit (struct backing_dev_info *bdi, -unsigned long dirty,unsigned long *pbdi_dirty) -{ -unsigned long avail_dirty; - - avail_dirty =global_page_state (NR_FILE_DIRTY ) + -global_page_state (NR_WRITEBACK ) + -global_page_state (NR_UNSTABLE_NFS ) + -global_page_state (NR_WRITEBACK_TEMP ); - -if (avail_dirty < dirty) - avail_dirty = dirty - avail_dirty; -else - avail_dirty =0 ; - - avail_dirty +=bdi_stat (bdi,BDI_RECLAIMABLE ) + -bdi_stat (bdi,BDI_WRITEBACK ); - - *pbdi_dirty =min (*pbdi_dirty, avail_dirty); -} -static inline void task_dirties_fraction (struct task_struct *tsk,long *numerator,long *denominator) {@@ -468,7 +442,6 @@ get_dirty_limits(unsigned long *pbackgro bdi_dirty = dirty * bdi->max_ratio /100 ;
Quoted 9 lines
mm/page-writeback.c get_dirty_limits(unsigned long *pbackgro · line 471
Quoted 2 lines
mm/page-writeback.c +0
-1*pbdi_dirty = bdi_dirty; -clip_bdi_dirty_limit (bdi, dirty, pbdi_dirty);I don't see, what test in
balance_dirty_limits()should replace this
clipping... OTOH clipping does not seem to have too much effect on the behavior ofbalance_dirty_pages- the limit we clip to (at leastBDI_WRITEBACK+BDI_RECLAIMABLE) is large enough so that we break from the loop immediately. So just getting rid of the function is fine but I'd update the changelog accordingly.
It essentially replace clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.
I added the above text to changelog :)
Quoted 10 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 509
Quoted 7 lines
mm/page-writeback.c +4
-1+ dirty_exceeded = + (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) + || (nr_reclaimable + nr_writeback >= dirty_thresh); -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh) +if (!dirty_exceeded)break ;Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.
This restores the (right) behavior in 2.6.18. And peter have the same goal in mind with clip_bdi_dirty_limit() ;)
Quoted 6 lines
Quoted 6 lines
mm/page-writeback.c +3
-2+/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
Quoted 8 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 568
Quoted 4 lines
mm/page-writeback.c +0
-4-break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...
Note that the pages_written check is moved several lines up in the patch :)
Quoted 2 lines
I'd think that if we did all that work in
writeback_inodes_wbcwe could
spend the effort on regetting and rechecking the stats...
Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0
-0schedule_timeout_interruptible (pause);/* @@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; }
Quoted 6 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 551
Quoted 4 lines
mm/page-writeback.c +1
-2-if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;Here we fail to clear
dirty_exceededif we are over global dirty limit
but not over per-bdi dirty limit...
You must be mistaken: dirty_exceeded = (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).
Quoted 4 lines
Quoted 4 lines
mm/page-writeback.c +0
-0@@ -593,9 +562,7 @@ static void balance_dirty_pages(struct a* background_thresh, to keep the amount of dirty memory low. */ if ((laptop_mode && pages_written) ||
Quoted 8 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 567
Quoted 6 lines
mm/page-writeback.c +1
-3- (!laptop_mode && ((global_page_state(NR_FILE_DIRTY) - + global_page_state(NR_UNSTABLE_NFS)) - > background_thresh))) + (!laptop_mode && (nr_reclaimable > background_thresh)))bdi_start_writeback (bdi,NULL ,0 ); }This might be based on rather old values in case we break from the loop
after callingwriteback_inodes_wbc.
Yes that's possible. It's safe because the bdi worker will double check background_thresh. We can call bdi_start_writeback() as long as there are good possibility: the nr_reclaimable is not likely to drop suddenly from during our writeout.
Thanks,
Fengguang
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Jan KaraQuoted 10 lines · Jan Kara
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Jan Kara
Ugh, but this is not equivalent! We would block the writer on some BDI
without any dirty data if we are over global dirty limit. That didn't happen before.It should have, we should throttle everything calling
balance_dirty_pages()when we're over the total limit.OK :) I agree it's reasonable. But Wu, please note this in the
changelog because it might be a substantial change for some loads.
Thanks, I added the note by Peter :)
Note that the total limit check itself may not be sufficient. For example, there are no nr_writeback limit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible
nr_writeback => dirty_thresh
nr_dirty => 0
which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).
So the better solution would be to impose a nr_writeback limit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit with congestion_wait, but now we need to do a wait queue for it.
With the nr_writeback wait queue, it can be guaranteed that once balance_dirty_pages() asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit in balance_dirty_pages().
However, there is still one problem - there are no general coordinations between max nr_writeback and the background/dirty limits.
It is possible (and very likely for some small memory systems) that
nr_writeback > dirty_thresh - background_thresh 10,000 20,000 15,000
In this case, it is possible that an application to be throttled because of
nr_reclaimable + nr_writeback > dirty_thresh 12,000 10,000 20,000
starts a background writeback work to do job for it, however that work quits immediately because
nr_reclaimable < background_thresh 12,000 15,000
In the end, the application did not get throttled at all at dirty_thresh. Instead, it will be throttled at (background_thresh + max_nr_writeback).
One solution (aka. the old behavior) is to respect the dirty_thresh, by not quiting background writeback when there are throttled tasks (this patch). It has the drawback of background writeback not doing its job _actively_. Instead, it will frequently be started and quit at times when applications enter and leave balanced_dirty_pages().
In the above scheme, the background_thresh is disregarded. The other ways would be to disregard dirty_thresh (may be undesirable) or to limit max_nr_writeback (not as easy).
It is still very possible to hit nr_dirty all the way down to 0 if max_nr_writeback > background_thresh.
This is a bit twisting. Any ideas?
--- fs/fs-writeback.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/fs-writeback.c +2 -1
@@ -781,7 +781,8 @@ static long wb_writeback(struct bdi_writ* For background writeout, stop when we are below the * background dirty threshold */ -if (args->for_background && !over_bground_thresh ()) +if (args->for_background && !over_bground_thresh () && + !list_empty (&wb->bdi->throttle_list))break ; wbc.more_io =0 ;
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 18 lines · Wu Fengguang
Note that the total limit check itself may not be sufficient. For example, there are no
nr_writebacklimit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible
nr_writeback=>dirty_thresh
nr_dirty=> 0which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).
So the better solution would be to impose a
nr_writebacklimit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit withcongestion_wait, but now we need to do a wait queue for it.
Quoted 5 lines
With the
nr_writebackwait queue, it can be guaranteed that oncebalance_dirty_pages()asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit inbalance_dirty_pages().
nr_reclaim = nr_dirty + nr_writeback + nr_unstable, so anything calling into balance_dirty_pages() would still block on seeing such large amounts of nr_writeback.
Having the constraint nr_dirty + nr_writeback + nr_unstable < dirty_thresh should ensure we never have nr_writeback > dirty_thresh, simply because you cannot dirty more, which then cannot be converted to more writeback.
Or am I missing something?
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 25 lines · Peter Zijlstra
Quoted 18 lines · Wu Fengguang
Note that the total limit check itself may not be sufficient. For example, there are no
nr_writebacklimit for NFS (and maybe btrfs) after removing the congestion waits. Therefore it is very possible
nr_writeback=>dirty_thresh
nr_dirty=> 0which is obviously undesirable: everything newly dirtied are soon put to writeback. It violates the 30s expire time and the background threshold rules, and will hurt write-and-truncate operations (ie. temp files).
So the better solution would be to impose a
nr_writebacklimit for every filesystem that didn't already have one (the block io queue). NFS used to have that limit withcongestion_wait, but now we need to do a wait queue for it.Quoted 5 lines
With the
nr_writebackwait queue, it can be guaranteed that oncebalance_dirty_pages()asks for writing 1500 pages, it will be done with necessary sleeping in the bdi flush thread. So we can safely remove the loop and double checking of global dirty limit inbalance_dirty_pages().
Quoted 3 lines
nr_reclaim=nr_dirty+nr_writeback+nr_unstable, so anything calling intobalance_dirty_pages()would still block on seeing such large amounts ofnr_writeback.
Our terms are a bit different. In my previous mail,
nr_reclaim = nr_dirty + nr_unstablenr_writeback is added separated when comparing with dirty_thresh, just as the code in balance_dirty_pages().
But that's fine. You are right that the application will be blocked and dirty limit be guaranteed, if we do
while (over dirty limit) {
bdi_writeback_wait(pages to write);
}But it has a problem: as long as the bdi-flush thread for NFS don't limit nr_writeback, its nr_writeback will grow to near (dirty_thresh-nr_unstable), and its nr_dirty will approach 0. That's not desirable.
So I did this:
- while (over dirty limit) {
+ if (over dirty limit) {
bdi_writeback_wait(pages to write);
}_after_ adding the NFS nr_writeback wait queue ([PATCH 20/45] NFS:
introduce writeback wait queue). With that it's safe to remove the loop.
Quoted 6 lines
Having the constraint
nr_dirty+nr_writeback+nr_unstable<dirty_threshshould ensure we never havenr_writeback>dirty_thresh, simply because you cannot dirty more, which then cannot be converted to more writeback.Or am I missing something?
You are right with the assumption that the loop is still there.
Sorry for the confusion, but I mean, filesystems have to limit nr_writeback (directly or indirectly via the block io queue), otherwise it either hit nr_dirty to 0 (with the loop), or let nr_writeback grow out of control (without the loop).
Thanks,
Fengguang
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 5 lines · Wu Fengguang
Sorry for the confusion, but I mean, filesystems have to limit
nr_writeback(directly or indirectly via the block io queue), otherwise it either hitnr_dirtyto 0 (with the loop), or letnr_writebackgrow out of control (without the loop).
OK, it seems we are indeed in agreement.
Just making sure ;-)
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Wu FengguangQuoted 5 lines · Wu Fengguang
Sorry for the confusion, but I mean, filesystems have to limit
nr_writeback(directly or indirectly via the block io queue), otherwise it either hitnr_dirtyto 0 (with the loop), or letnr_writebackgrow out of control (without the loop).
Doesn't this require the writeback queue to have a limit < dirty_thresh?
Or more specifically, for the bdi case:
bdi_dirty + bdi_writeback + bdi_unstable <= bdi_thresh
we require that the writeback queue be smaller than bdi_thresh, which could be quite difficult, since bdi_thresh can easily be 0.
Without observing the bdi_thresh constraint we can have:
\Sum_(over bdis) writeback_queue_size
dirty pages outstanding, which could be significantly higher than dirty_thresh.
Or am I confused again?
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang
Sorry for the confusion, but I mean, filesystems have to limit
nr_writeback(directly or indirectly via the block io queue), otherwise it either hitnr_dirtyto 0 (with the loop), or letnr_writebackgrow out of control (without the loop).Doesn't this require the writeback queue to have a limit <
dirty_thresh?
Yes, this is the key (open) issue. For now we have nothing to limit
nr_writeback < dirty_thresh
Quoted 6 lines
Or more specifically, for the bdi case:
bdi_dirty+bdi_writeback+bdi_unstable<=bdi_threshwe require that the writeback queue be smaller than
bdi_thresh, which could be quite difficult, sincebdi_threshcan easily be 0.
We could apply a MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, so bdi_thresh must be non-zero to allow some writeback pages in flight :)
Quoted 6 lines
Without observing the
bdi_threshconstraint we can have:\Sum_(over bdis)
writeback_queue_sizedirty pages outstanding, which could be significantly higher than
dirty_thresh.
Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).
The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth. MIN_BDI_DIRTY_THRESH could be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.
Thanks,
Fengguang
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 16 lines · Wu Fengguang
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang
Sorry for the confusion, but I mean, filesystems have to limit
nr_writeback(directly or indirectly via the block io queue), otherwise it either hitnr_dirtyto 0 (with the loop), or letnr_writebackgrow out of control (without the loop).Doesn't this require the writeback queue to have a limit <
dirty_thresh?Yes, this is the key (open) issue. For now we have nothing to limit
nr_writeback<dirty_threshQuoted 2 lines
Or more specifically, for the bdi case:
Quoted 8 lines
Quoted 4 lines
bdi_dirty+bdi_writeback+bdi_unstable<=bdi_threshwe require that the writeback queue be smaller than
bdi_thresh, which could be quite difficult, sincebdi_threshcan easily be 0.We could apply a
MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, sobdi_threshmust be non-zero to allow some writeback pages in flight :)
Not really, suppose you have 1000 NFS clients, of which you only use a hand full at a time.
Then the bdi_thresh will be 0 for most of them, and only when you switch to one it'll start growing. But it's perfectly reasonable to expect bdi_thresh=0 to work. It just reverts to sync behaviour, we write out everything and block until they're all gone from writeback state.
MIN_BDI_DIRTY_THRESH != 0, will have a side effect of imposing a max number of BDIs on the system, I'm not sure you want to go there.
Quoted 11 lines
Quoted 6 lines
Without observing the
bdi_threshconstraint we can have:\Sum_(over bdis)
writeback_queue_sizedirty pages outstanding, which could be significantly higher than
dirty_thresh.Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).
Quoted 8 lines
The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth.
MIN_BDI_DIRTY_THRESHcould be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.
I'm not seeing why. By simply keeping that loop we're good again, and can have a writeback queue that works well in the saturated case.
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 26 lines · Peter Zijlstra
Quoted 16 lines · Wu Fengguang
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Wu Fengguang
Sorry for the confusion, but I mean, filesystems have to limit
nr_writeback(directly or indirectly via the block io queue), otherwise it either hitnr_dirtyto 0 (with the loop), or letnr_writebackgrow out of control (without the loop).Doesn't this require the writeback queue to have a limit <
dirty_thresh?Yes, this is the key (open) issue. For now we have nothing to limit
nr_writeback<dirty_threshQuoted 2 lines
Or more specifically, for the bdi case:
Quoted 8 lines
Quoted 4 lines
bdi_dirty+bdi_writeback+bdi_unstable<=bdi_threshwe require that the writeback queue be smaller than
bdi_thresh, which could be quite difficult, sincebdi_threshcan easily be 0.We could apply a
MIN_BDI_DIRTY_THRESH. Because the bdi threshold is estimated from writeback events, sobdi_threshmust be non-zero to allow some writeback pages in flight :)
Quoted 7 lines
Not really, suppose you have 1000 NFS clients, of which you only use a hand full at a time.
Then the
bdi_threshwill be 0 for most of them, and only when you switch to one it'll start growing. But it's perfectly reasonable to expect bdi_thresh=0 to work. It just reverts to sync behaviour, we write out everything and block until they're all gone from writeback state.
Ah I see. We still do writeback when bdi_thresh=0, with any application blocked in balance_dirty_pages().
Quoted 2 lines
MIN_BDI_DIRTY_THRESH!= 0, will have a side effect of imposing a max number of BDIs on the system, I'm not sure you want to go there.
OK that's not a good idea.
Quoted 20 lines
Quoted 11 lines
Quoted 6 lines
Without observing the
bdi_threshconstraint we can have:\Sum_(over bdis)
writeback_queue_sizedirty pages outstanding, which could be significantly higher than
dirty_thresh.Yes. Maybe we could do some per-bdi and/or global writeback wait queue (ie. some generalized version of the patch 20: NFS: introduce writeback wait queue).
Quoted 8 lines
The per-bdi writeback queue size should ideally be proportional to its available writeback bandwidth.
MIN_BDI_DIRTY_THRESHcould be defined to (2*bdi_writeback_bandwidth) or something close. And if the resulted bdi limits turn out to be too large for a small memory system, we just let the global limit kick in. For such small memory systems, it is very likely there are only one bdi. So it is not likely to lose fairness to base its limits on available memory instead of device capability.
Quoted 2 lines
I'm not seeing why. By simply keeping that loop we're good again, and can have a writeback queue that works well in the saturated case.
OK it looks better to keep the loop. The memory tight systems may go into the nr_dirty=0 situation, but it may not be an urgent problem (its nr_dirty will be small anyway).
Thanks,
Fengguang
Jan Kara
maintainerRe: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Wu FengguangQuoted 16 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3
-2+/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)Quoted 8 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 568
Quoted 4 lines
mm/page-writeback.c +0
-4-break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...
Quoted 7 lines
Note that the
pages_writtencheck is moved several lines up in the patch :)Quoted 2 lines
I'd think that if we did all that work in
writeback_inodes_wbcwe could
spend the effort on regetting and rechecking the stats...Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)
Yes, since the loop gets removed in the end, this does not matter. You
are right.
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0
-0schedule_timeout_interruptible (pause);/* @@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; }Quoted 6 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 551
Quoted 4 lines
mm/page-writeback.c +1
-2-if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;Here we fail to clear
dirty_exceededif we are over global dirty limit
but not over per-bdi dirty limit...
Quoted 2 lines
You must be mistaken:
dirty_exceeded= (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).
Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded = 0;
Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded = 0;
That's clearly not equivalent which is what I was trying to point out.
But looking at where dirty_exceeded is used, your new way is probably more useful. It's just a bit counterintuitive that bdi->dirty_exceeded is set even if the per-bdi limit is not exceeded...
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 19 lines · Jan Kara
Quoted 18 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3
-2+/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)Quoted 8 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 568
Quoted 4 lines
mm/page-writeback.c +0
-4-break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...Note that the
pages_writtencheck is moved several lines up in the patch :)
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I'd think that if we did all that work in
writeback_inodes_wbcwe could
spend the effort on regetting and rechecking the stats...Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)
Yes, since the loop gets removed in the end, this does not matter. You
are right.
You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:
static void balance_dirty_pages(struct address_space *mapping,
unsigned long write_chunk)
{
long nr_reclaimable, bdi_nr_reclaimable;
long nr_writeback, bdi_nr_writeback;
unsigned long background_thresh;
unsigned long dirty_thresh;
unsigned long bdi_thresh;
int dirty_exceeded;
struct backing_dev_info *bdi = mapping->backing_dev_info;
/*
* If sync() is in progress, curb the to-be-synced inodes regardless
* of dirty limits, so that a fast dirtier won't livelock the sync.
*/
if (unlikely(bdi->sync_time &&
S_ISREG(mapping->host->i_mode) &&
time_after_eq(bdi->sync_time,
mapping->host->dirtied_when))) {
write_chunk *= 2;
bdi_writeback_wait(bdi, write_chunk);
}
for (;;) {
nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
nr_writeback = global_page_state(NR_WRITEBACK) +
global_page_state(NR_WRITEBACK_TEMP);
global_dirty_thresh(&background_thresh, &dirty_thresh);
/*
* Throttle it only when the background writeback cannot
* catch-up. This avoids (excessively) small writeouts
* when the bdi limits are ramping up.
*/
if (nr_reclaimable + nr_writeback <
(background_thresh + dirty_thresh) / 2)
break;
bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh);
/*
* In order to avoid the stacked BDI deadlock we need
* to ensure we accurately count the 'dirty' pages when
* the threshold is low.
*
* Otherwise it would be possible to get thresh+n pages
* reported dirty, even though there are thresh-m pages
* actually dirty; with m+n sitting in the percpu
* deltas.
*/
if (bdi_thresh < 2*bdi_stat_error(bdi)) {
bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE);
bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK);
} else {
bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
}
/*
* The bdi thresh is somehow "soft" limit derived from the
* global "hard" limit. The former helps to prevent heavy IO
* bdi or process from holding back light ones; The latter is
* the last resort safeguard.
*/
dirty_exceeded =
(bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
|| (nr_reclaimable + nr_writeback >= dirty_thresh);
if (!dirty_exceeded)
break;
bdi->dirty_exceed_time = jiffies;
bdi_writeback_wait(bdi, write_chunk);
}
/*
* In laptop mode, we wait until hitting the higher threshold before
* starting background writeout, and then write out all the way down
* to the lower threshold. So slow writers cause minimal disk activity.
*
* In normal mode, we start background writeout at the lower
* background_thresh, to keep the amount of dirty memory low.
*/
if (!laptop_mode && (nr_reclaimable > background_thresh) &&
can_submit_background_writeback(bdi))
bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND);
}Quoted 20 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0
-0schedule_timeout_interruptible (pause);/* @@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; }Quoted 6 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 551
Quoted 4 lines
mm/page-writeback.c +1
-2-if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;Here we fail to clear
dirty_exceededif we are over global dirty limit
but not over per-bdi dirty limit...Quoted 2 lines
You must be mistaken:
dirty_exceeded= (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded= 0;
Quoted 8 lines
Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded= 0;That's clearly not equivalent which is what I was trying to point out.
But looking at wheredirty_exceededis used, your new way is probably more useful. It's just a bit counterintuitive thatbdi->dirty_exceededis set even if the per-bdi limit is not exceeded...
Yeah good point. Since the per-bdi limits are more about "soft" limits which are derived from the global "hard" limit, the code makes sense with some comments and updated changelog :)
This patch slightly changes behavior by replacing clip_bdi_dirty_limit() with the explicit check (nr_reclaimable + nr_writeback >= dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.
The check is necessary because, in principle we should throttle everything calling balance_dirty_pages() when we're over the total limit, as said by Peter.
We now set and clear dirty_exceeded not only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.
We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.
Thanks,
Fengguang
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 104 lines · Wu Fengguang
You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:
This patch slightly changes behavior by replacing
clip_bdi_dirty_limit()with the explicit check (nr_reclaimable+nr_writeback>=dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.The check is necessary because, in principle we should throttle everything calling
balance_dirty_pages()when we're over the total limit, as said by Peter.We now set and clear
dirty_exceedednot only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.
static void balance_dirty_pages(struct
address_space*mapping,
unsigned longwrite_chunk){ long nr_reclaimable, bdi_nr_reclaimable; long nr_writeback, bdi_nr_writeback; unsigned long background_thresh; unsigned long dirty_thresh; unsigned long bdi_thresh; int dirty_exceeded; struct backing_dev_info *bdi = mapping->backing_dev_info; /* * If sync() is in progress, curb the to-be-synced inodes regardless * of dirty limits, so that a fast dirtier won't livelock the sync. */ if (unlikely(bdi->sync_time && S_ISREG(mapping->host->i_mode) && time_after_eq(bdi->sync_time, mapping->host->dirtied_when))) { write_chunk *= 2; bdi_writeback_wait(bdi, write_chunk); } for (;;) { nr_reclaimable = global_page_state(NR_FILE_DIRTY) + global_page_state(NR_UNSTABLE_NFS); nr_writeback = global_page_state(NR_WRITEBACK) + global_page_state(NR_WRITEBACK_TEMP); global_dirty_thresh(&background_thresh, &dirty_thresh); /* * Throttle it only when the background writeback cannot * catch-up. This avoids (excessively) small writeouts * when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < (background_thresh + dirty_thresh) / 2) break; bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh); /* * In order to avoid the stacked BDI deadlock we need * to ensure we accurately count the 'dirty' pages when * the threshold is low. * * Otherwise it would be possible to get thresh+n pages * reported dirty, even though there are thresh-m pages * actually dirty; with m+n sitting in the percpu * deltas. */ if (bdi_thresh < 2*bdi_stat_error(bdi)) { bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK); } else { bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK); } /* * The bdi thresh is somehow "soft" limit derived from the * global "hard" limit. The former helps to prevent heavy IO * bdi or process from holding back light ones; The latter is * the last resort safeguard. */ dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) || (nr_reclaimable + nr_writeback >= dirty_thresh); if (!dirty_exceeded) break; bdi->dirty_exceed_time = jiffies; bdi_writeback_wait(bdi, write_chunk); } /* * In laptop mode, we wait until hitting the higher threshold before * starting background writeout, and then write out all the way down * to the lower threshold. So slow writers cause minimal disk activity.
Quoted 8 lines
* * In normal mode, we start background writeout at the lower * background_thresh, to keep the amount of dirty memory low. */ if (!laptop_mode && (nr_reclaimable > background_thresh) && can_submit_background_writeback(bdi)) bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND); }
Looks good, Thanks Wu!
Jan Kara
maintainerRe: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
In reply to Wu FengguangQuoted 99 lines · Wu Fengguang
Quoted 19 lines · Jan Kara
Quoted 18 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
Quoted 6 lines
mm/page-writeback.c +3
-2+/* don't wait if we've done enough */ +if (pages_written >= write_chunk) +break ; } - -if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)Quoted 8 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 568
Quoted 4 lines
mm/page-writeback.c +0
-4-break ; -if (pages_written >= write_chunk) -break ;/* We've done our duty */ -Here, we had an opportunity to break from the loop even if we didn't
manage to write everything (for example because per-bdi thread managed to write enough or because enough IO has completed while we were trying to write). After the patch, we will sleep. IMHO that's not good...Note that the
pages_writtencheck is moved several lines up in the patch :)Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I'd think that if we did all that work in
writeback_inodes_wbcwe could
spend the effort on regetting and rechecking the stats...Yes maybe. I didn't care it because the later throttle queue patch totally removed the loop and hence to need to reget the stats :)
Yes, since the loop gets removed in the end, this does not matter. You
are right.You are right too :) I followed you and Peter's advice to do the loop and the recheck of stats as follows:
static void balance_dirty_pages(struct
address_space*mapping,
unsigned longwrite_chunk){ long nr_reclaimable, bdi_nr_reclaimable; long nr_writeback, bdi_nr_writeback; unsigned long background_thresh; unsigned long dirty_thresh; unsigned long bdi_thresh; int dirty_exceeded; struct backing_dev_info *bdi = mapping->backing_dev_info; /* * If sync() is in progress, curb the to-be-synced inodes regardless * of dirty limits, so that a fast dirtier won't livelock the sync. */ if (unlikely(bdi->sync_time && S_ISREG(mapping->host->i_mode) && time_after_eq(bdi->sync_time, mapping->host->dirtied_when))) { write_chunk *= 2; bdi_writeback_wait(bdi, write_chunk); } for (;;) { nr_reclaimable = global_page_state(NR_FILE_DIRTY) + global_page_state(NR_UNSTABLE_NFS); nr_writeback = global_page_state(NR_WRITEBACK) + global_page_state(NR_WRITEBACK_TEMP); global_dirty_thresh(&background_thresh, &dirty_thresh); /* * Throttle it only when the background writeback cannot * catch-up. This avoids (excessively) small writeouts * when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < (background_thresh + dirty_thresh) / 2) break; bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh); /* * In order to avoid the stacked BDI deadlock we need * to ensure we accurately count the 'dirty' pages when * the threshold is low. * * Otherwise it would be possible to get thresh+n pages * reported dirty, even though there are thresh-m pages * actually dirty; with m+n sitting in the percpu * deltas. */ if (bdi_thresh < 2*bdi_stat_error(bdi)) { bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK); } else { bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK); } /* * The bdi thresh is somehow "soft" limit derived from the * global "hard" limit. The former helps to prevent heavy IO * bdi or process from holding back light ones; The latter is * the last resort safeguard. */ dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh)
Quoted 8 lines
|| (nr_reclaimable + nr_writeback >= dirty_thresh); if (!dirty_exceeded) break; bdi->dirty_exceed_time = jiffies; bdi_writeback_wait(bdi, write_chunk);
Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writing write_chunk... So the time we spend in balance_dirty_pages() is no longer limited, right?
Quoted 58 lines
} /* * In laptop mode, we wait until hitting the higher threshold before * starting background writeout, and then write out all the way down * to the lower threshold. So slow writers cause minimal disk activity. * * In normal mode, we start background writeout at the lower * background_thresh, to keep the amount of dirty memory low. */ if (!laptop_mode && (nr_reclaimable > background_thresh) && can_submit_background_writeback(bdi)) bdi_start_writeback(bdi, NULL, WB_FOR_BACKGROUND); }Quoted 20 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
mm/page-writeback.c +0
-0schedule_timeout_interruptible (pause);/* @@ -577,8 +547,7 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; }Quoted 6 lines
mm/page-writeback.c static void balance_dirty_pages(struct a · line 551
Quoted 4 lines
mm/page-writeback.c +1
-2-if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && - bdi->dirty_exceeded) +if (!dirty_exceeded && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;Here we fail to clear
dirty_exceededif we are over global dirty limit
but not over per-bdi dirty limit...Quoted 2 lines
You must be mistaken:
dirty_exceeded= (over bdi limit || over global limit), so !dirty_exceeded = (!over bdi limit && !over global limit).Exactly. Previously, the check was:
if (!over bdi limit)
bdi->dirty_exceeded= 0;Quoted 8 lines
Now it is
if (!over bdi limit && !over global limit)
bdi->dirty_exceeded= 0;That's clearly not equivalent which is what I was trying to point out.
But looking at wheredirty_exceededis used, your new way is probably more useful. It's just a bit counterintuitive thatbdi->dirty_exceededis set even if the per-bdi limit is not exceeded...Yeah good point. Since the per-bdi limits are more about "soft" limits which are derived from the global "hard" limit, the code makes sense with some comments and updated changelog :)
This patch slightly changes behavior by replacing
clip_bdi_dirty_limit()with the explicit check (nr_reclaimable+nr_writeback>=dirty_thresh) to avoid exceeding the dirty limit. Since the bdi dirty limit is mostly accurate we don't need to do routinely clip. A simple dirty limit check would be enough.The check is necessary because, in principle we should throttle everything calling
balance_dirty_pages()when we're over the total limit, as said by Peter.
Quoted 8 lines
We now set and clear
dirty_exceedednot only based on bdi dirty limits, but also on the global dirty limits. This is a bit counterintuitive, but the global limits are the ultimate goal and shall be always imposed.We may now start background writeback work based on outdated conditions. That's safe because the bdi flush thread will (and have to) double check the states. It reduces overall overheads because the test based on old states still have good chance to be right.
The new description is good, thanks!
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 50 lines · Jan Kara
Quoted 50 lines
for (;;) { nr_reclaimable = global_page_state(NR_FILE_DIRTY) + global_page_state(NR_UNSTABLE_NFS); nr_writeback = global_page_state(NR_WRITEBACK) + global_page_state(NR_WRITEBACK_TEMP); global_dirty_thresh(&background_thresh, &dirty_thresh); /* * Throttle it only when the background writeback cannot * catch-up. This avoids (excessively) small writeouts * when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < (background_thresh + dirty_thresh) / 2) break; bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh); /* * In order to avoid the stacked BDI deadlock we need * to ensure we accurately count the 'dirty' pages when * the threshold is low. * * Otherwise it would be possible to get thresh+n pages * reported dirty, even though there are thresh-m pages * actually dirty; with m+n sitting in the percpu * deltas. */ if (bdi_thresh < 2*bdi_stat_error(bdi)) { bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK); } else { bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK); } /* * The bdi thresh is somehow "soft" limit derived from the * global "hard" limit. The former helps to prevent heavy IO * bdi or process from holding back light ones; The latter is * the last resort safeguard. */ dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) || (nr_reclaimable + nr_writeback >= dirty_thresh); if (!dirty_exceeded) break;
Quoted 7 lines
Quoted 3 lines
bdi->dirty_exceed_time = jiffies; bdi_writeback_wait(bdi, write_chunk);Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writingwrite_chunk... So the time we spend inbalance_dirty_pages()is no longer limited, right?
Wu was saying that without the loop nr_writeback wasn't limited, but since bdi_writeback_wakeup() is driven from writeout completion, I'm not sure how again that was so.
We can move all of bdi_dirty to bdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.
Possibly unstable is peculiar.
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 51 lines · Peter Zijlstra
Quoted 50 lines · Jan Kara
Quoted 50 lines
for (;;) { nr_reclaimable = global_page_state(NR_FILE_DIRTY) + global_page_state(NR_UNSTABLE_NFS); nr_writeback = global_page_state(NR_WRITEBACK) + global_page_state(NR_WRITEBACK_TEMP); global_dirty_thresh(&background_thresh, &dirty_thresh); /* * Throttle it only when the background writeback cannot * catch-up. This avoids (excessively) small writeouts * when the bdi limits are ramping up. */ if (nr_reclaimable + nr_writeback < (background_thresh + dirty_thresh) / 2) break; bdi_thresh = bdi_dirty_thresh(bdi, dirty_thresh); /* * In order to avoid the stacked BDI deadlock we need * to ensure we accurately count the 'dirty' pages when * the threshold is low. * * Otherwise it would be possible to get thresh+n pages * reported dirty, even though there are thresh-m pages * actually dirty; with m+n sitting in the percpu * deltas. */ if (bdi_thresh < 2*bdi_stat_error(bdi)) { bdi_nr_reclaimable = bdi_stat_sum(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat_sum(bdi, BDI_WRITEBACK); } else { bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE); bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK); } /* * The bdi thresh is somehow "soft" limit derived from the * global "hard" limit. The former helps to prevent heavy IO * bdi or process from holding back light ones; The latter is * the last resort safeguard. */ dirty_exceeded = (bdi_nr_reclaimable + bdi_nr_writeback >= bdi_thresh) || (nr_reclaimable + nr_writeback >= dirty_thresh); if (!dirty_exceeded) break;
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
bdi->dirty_exceed_time = jiffies; bdi_writeback_wait(bdi, write_chunk);Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writingwrite_chunk... So the time we spend inbalance_dirty_pages()is no longer limited, right?
Right, this is a legitimate concern.
Quoted 3 lines
Wu was saying that without the loop
nr_writebackwasn't limited, but sincebdi_writeback_wakeup()is driven from writeout completion, I'm not sure how again that was so.
Let me summarize the ideas :)
There are two cases:
- there are no bdi or block io queue to limit nr_writeback This must be fixed. It either let nr_writeback grow to dirty_thresh (with loop) and thus squeeze nr_dirty, or grow out of control totally (without loop). Current state is, the nr_writeback wait queue for NFS is there; the one for btrfs is still missing.
- there is a nr_writeback limit, but is larger than dirty_thresh In this case nr_dirty will be close to 0 regardless of the loop. The loop will help to keep
nr_dirty + nr_writeback + nr_unstable < dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by the nr_writeback limit).
Quoted 3 lines
We can move all of
bdi_dirtytobdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.
Yes, this explains the second case. It's some trade-off like: the nr_writeback limit can not be trusted in small memory systems, so do the loop to impose the dirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..
We could possibly test (nr_dirty < nr_writeback). If so, the nr_writeback limit could be too large to deserve the loop.
It still don't address the nr_dirty=0 problem for small memory system, that should be acceptable since its nr_dirty will be small anyway.
Quoted 1 line
Possibly unstable is peculiar.
unstable can also go wild. I saw (in current linux-next with the following patch) balance_dirty_pages() sleeping for >30s waiting for the NFS nr_unstable to drop. That is, waiting for the dirty inode to be _expired_ and written to disk on the server.
It's a general uncoordinated double caching problem for NFS (and maybe more).
Thanks,
Fengguang
---
[ 45.614799] balance_dirty_pages sleeped 228ms [ 45.954821] balance_dirty_pages sleeped 324ms [ 46.294874] balance_dirty_pages sleeped 324ms [ 46.638810] balance_dirty_pages sleeped 328ms [ 46.670769] balance_dirty_pages sleeped 28ms [ 46.802779] balance_dirty_pages sleeped 128ms [ 46.934788] balance_dirty_pages sleeped 124ms [ 47.066778] balance_dirty_pages sleeped 124ms [ 47.198774] balance_dirty_pages sleeped 128ms [ 47.330763] balance_dirty_pages sleeped 124ms [ 47.462768] balance_dirty_pages sleeped 128ms [ 47.594768] balance_dirty_pages sleeped 124ms [ 47.662763] balance_dirty_pages sleeped 60ms [ 47.798781] balance_dirty_pages sleeped 132ms [ 47.871435] balance_dirty_pages sleeped 64ms [ 48.002749] balance_dirty_pages sleeped 124ms [ 48.138787] balance_dirty_pages sleeped 132ms [ 48.270824] balance_dirty_pages sleeped 124ms [ 48.410762] balance_dirty_pages sleeped 128ms [ 48.542758] balance_dirty_pages sleeped 128ms [ 48.678786] balance_dirty_pages sleeped 132ms [ 48.810781] balance_dirty_pages sleeped 124ms [ 48.946755] balance_dirty_pages sleeped 124ms [ 49.182753] balance_dirty_pages sleeped 228ms [ 49.318773] balance_dirty_pages sleeped 128ms [ 49.666784] balance_dirty_pages sleeped 324ms [ 49.914774] balance_dirty_pages sleeped 228ms [ 79.998354] balance_dirty_pages sleeped 30068ms [ 80.062346] balance_dirty_pages sleeped 60ms [ 80.290414] balance_dirty_pages sleeped 224ms [ 80.542413] balance_dirty_pages sleeped 228ms [ 80.782384] balance_dirty_pages sleeped 228ms [ 81.142379] balance_dirty_pages sleeped 336ms [ 116.005926] balance_dirty_pages sleeped 34852ms [ 141.049584] balance_dirty_pages sleeped 25040ms
--- mm/page-writeback.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/page-writeback.c +6 -1
@@ -490,6 +490,7 @@ static void balance_dirty_pages(struct aunsigned long bdi_thresh;unsigned long pages_written =0 ;unsigned long pause =1 ; +unsigned long start = jiffies;struct backing_dev_info *bdi = mapping->backing_dev_info;
@@ -566,7 +567,8 @@ static void balance_dirty_pages(struct aif (pages_written >= write_chunk)break ;/* We've done our duty */ -schedule_timeout_interruptible (pause); +__set_current_state (TASK_INTERRUPTIBLE ); +io_schedule_timeout (pause);/* * Increase the delay for each loop, up to our previous
@@ -577,6 +579,9 @@ static void balance_dirty_pages(struct a pause =HZ /10 ; } +if (pause >1 ) +printk ("balance_dirty_pages sleeped %lums\n" , (jiffies - start) *1000 /HZ ); +if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && bdi->dirty_exceeded) bdi->dirty_exceeded =0 ;
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 6 lines · Wu Fengguang
Quoted 4 lines
Quoted 4 lines
Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writingwrite_chunk... So the time we spend inbalance_dirty_pages()is no longer limited, right?Right, this is a legitimate concern.
Quite.
Quoted 21 lines
Quoted 3 lines
Wu was saying that without the loop
nr_writebackwasn't limited, but sincebdi_writeback_wakeup()is driven from writeout completion, I'm not sure how again that was so.Let me summarize the ideas :)
There are two cases:
- there are no bdi or block io queue to limit
nr_writebackThis must be fixed. It either letnr_writebackgrow todirty_thresh(with loop) and thus squeezenr_dirty, or grow out of control totally (without loop). Current state is, thenr_writebackwait queue for NFS is there; the one for btrfs is still missing.- there is a
nr_writebacklimit, but is larger thandirty_threshIn this casenr_dirtywill be close to 0 regardless of the loop. The loop will help to keep
nr_dirty+nr_writeback+nr_unstable<dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by thenr_writebacklimit).
Quoted 8 lines
Quoted 3 lines
We can move all of
bdi_dirtytobdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.Yes, this explains the second case. It's some trade-off like: the
nr_writebacklimit can not be trusted in small memory systems, so do the loop to impose thedirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..
Ok, so I'm still puzzled.
set_page_dirty()
balance_dirty_pages_ratelimited()
balance_dirty_pages_ratelimited_nr(1)
balance_dirty_pages(nr);So we call balance_dirty_pages() with an appropriate count for each set_page_dirty() successful invocation, right?
balance_dirty_pages() guarantees that:
nr_dirty + nr_writeback + nr_unstable < dirty_thresh && (nr_dirty + nr_writeback + nr_unstable <
(dirty_thresh + background_thresh)/2 ||
bdi_dirty + bdi_writeback + bdi_unstable < bdi_thresh)
Now without loop, without writeback limit, I still see no way to actually generate more 'dirty' pages than dirty_thresh.
As soon as we hit dirty_thresh a process will wait for exactly the same amount of pages to get cleaned (writeback completed) as were dirtied (+/- the ratelimit fuzz which should even out over processes).
That should bound things to dirty_thresh -- the wait is on writeback complete, so nr_writeback is bounded too.
[ I forgot the exact semantics of unstable, if we clear writeback before unstable, we need to fix something ]
Now, a nr_writeback queue that limits writeback will still be useful, esp for high speed devices. Once they ramp up and bdi_thresh exceeds the queue size, it'll take effect. So you reap the benefits when needed.
Re: [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages()
Quoted 35 lines · Peter Zijlstra
Quoted 6 lines · Wu Fengguang
Quoted 4 lines
Quoted 4 lines
Hmm, probably you've discussed this in some other email but why do we
cycle in this loop until we get below dirty limit? We used to leave the loop after writingwrite_chunk... So the time we spend inbalance_dirty_pages()is no longer limited, right?Right, this is a legitimate concern.
Quite.
Quoted 25 lines
Quoted 3 lines
Wu was saying that without the loop
nr_writebackwasn't limited, but sincebdi_writeback_wakeup()is driven from writeout completion, I'm not sure how again that was so.Let me summarize the ideas :)
There are two cases:
- there are no bdi or block io queue to limit
nr_writebackThis must be fixed. It either letnr_writebackgrow todirty_thresh(with loop) and thus squeezenr_dirty, or grow out of control totally (without loop). Current state is, thenr_writebackwait queue for NFS is there; the one for btrfs is still missing.- there is a
nr_writebacklimit, but is larger thandirty_threshIn this casenr_dirtywill be close to 0 regardless of the loop. The loop will help to keep
nr_dirty+nr_writeback+nr_unstable<dirty_thresh
Without the loop, the "real" dirty threshold would be larger (determined by thenr_writebacklimit).Quoted 3 lines
We can move all of
bdi_dirtytobdi_writeout, if the bdi writeout queue permits, but it cannot grow beyond the total limit, since we're actually waiting for writeout completion.
Quoted 6 lines
Quoted 4 lines
Yes, this explains the second case. It's some trade-off like: the
nr_writebacklimit can not be trusted in small memory systems, so do the loop to impose thedirty_thresh, which unfortunately can hurt responsiveness on all systems with prolonged wait time..Ok, so I'm still puzzled.
Big sorry - it's me that was confused (by some buggy tests).
Quoted 7 lines
set_page_dirty() balance_dirty_pages_ratelimited() balance_dirty_pages_ratelimited_nr(1) balance_dirty_pages(nr);So we call
balance_dirty_pages()with an appropriate count for eachset_page_dirty()successful invocation, right?
Right.
Quoted 5 lines
balance_dirty_pages()guarantees that:
nr_dirty+nr_writeback+nr_unstable<dirty_thresh&& (nr_dirty+nr_writeback+nr_unstable<(dirty_thresh + background_thresh)/2 ||
Quoted 8 lines
bdi_dirty+bdi_writeback+bdi_unstable<bdi_thresh)Now without loop, without writeback limit, I still see no way to actually generate more 'dirty' pages than
dirty_thresh.As soon as we hit
dirty_thresha process will wait for exactly the same amount of pages to get cleaned (writeback completed) as were dirtied (+/- the ratelimit fuzz which should even out over processes).
Ah yes, we now wait for writeback _completed_ in bdi_writeback_wait(), instead of _start_ writeback in the old fashioned writeback_inodes().
Quoted 2 lines
That should bound things to
dirty_thresh-- the wait is on writeback complete, sonr_writebackis bounded too.
Right. It was not bounded in the tests because bdi_writeback_wait() quits _prematurely_, because the background writeback finds it was already under background threshold, and so wakeup the throttled tasks and then quit. Fixed by simply removing the wakeup-all in background writeback and this change:
if (args->for_background && !over_bground_thresh() &&
+ list_empty(&wb->bdi->throttle_list))
break;So now
- the throttled tasks are guaranteed to be wakeup
- it will only be wakeup in __bdi_writeout_inc()
- once wakeup, at least write_chunk pages have been written on behalf of it
Quoted 2 lines
[ I forgot the exact semantics of unstable, if we clear writeback before unstable, we need to fix something ]
New tests show that NFS is working fine without loop and without NFS nr_writeback limit:
$ dd if=/dev/zero of=/mnt/test/zero3 bs=1M count=200 &
$ vmmon -d 1 nr_writeback nr_dirty nr_unstable nr_writeback nr_dirty nr_unstable
0 2 0
0 2 0
0 22477 65
2 20849 1697
2 19153 3393
2 17420 5126
27825 7 5979
27816 0 41
26925 0 907
31064 286 159
32531 0 213
32548 0 89
32405 0 155
32464 0 98
32517 0 45
32560 0 194
32534 0 220
32601 0 222
32490 0 72
32447 0 115
32511 0 48
32535 0 216
32535 0 216
32535 0 216
32535 0 216
31555 0 1180
29732 0 3003
29277 0 3458
27721 0 5014
25955 0 6780
24356 0 8379
22763 0 9972
21083 0 11652
19371 0 13364
17564 0 15171
15781 0 16954
14005 0 18730
12230 0 20505
12177 0 20558
11383 0 21352
9489 0 23246
7621 0 25115
5866 0 26870
4790 0 27947
2962 0 29773
1089 0 31646
0 0 32735
0 0 32735
0 0 0
0 0 0Quoted 3 lines
Now, a
nr_writebackqueue that limits writeback will still be useful, esp for high speed devices. Once they ramp up andbdi_threshexceeds the queue size, it'll take effect. So you reap the benefits when needed.
Right, the nr_writeback limit avoids
nr_writeback => dirty_thresh
+
nr_dirty + nr_writeback < dirty_thresh
=>
nr_dirty => 0
Thanks for the clarification, it looks less obscure now :)
Thanks,
Fengguang
Re: [PATCH 31/45] writeback: sync old inodes first in background writeback
In reply to Wu FengguangQuoted 2 lines · Wu Fengguang
A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.
I've looked at this a bit again after you pointed to this thread in the direct reclaim thread, and I think we should be even more aggressive in pushing out old inodes.
We basically have two types of I/O done from wb_do_writeback:
- either we want to write all inodes for a given bdi/superblock. That includes all WB_SYNC_ALL callers, but also things like writeback_inodes_sb and the wakeup_flusher_threads call from sys_sync.
- or we have a specific goal, like for the background writeback or the wakeup_flusher_threads from free_more_memory.
For the first case there's obviously no point in doing any older_than_this processing as we write out all inodes anyway.
For the second case we should always do a older_than_this pass _first_.
Rationale: we really should get the old inodes out ASAP, so that we keep the amount of changes lost on a crash in the boundaries. Furthermore the callers only need N pages cleaned, and they don't care where from. So if we can reach our goal with the older_than_this writeback we're fine. If the writeback loop is long enough we can keep doing more of these later on as well.
Doing this should also help cleaning the code up a bit by moving the wb_check_old_data_flush logic into wb_writeback and getting rid of the for_kupdate paramter in struct wb_writeback_work. I'm not even sure it's worth keeping it in struct writeback_control.
Re: [PATCH 31/45] writeback: sync old inodes first in background writeback
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Wu Fengguang
A background flush work may run for ever. So it's reasonable for it to mimic the kupdate behavior of syncing old/expired inodes first.
I've looked at this a bit again after you pointed to this thread in the direct reclaim thread, and I think we should be even more aggressive in pushing out old inodes.
Agreed.
Quoted 11 lines
We basically have two types of I/O done from
wb_do_writeback:- either we want to write all inodes for a given bdi/superblock. That includes all
WB_SYNC_ALLcallers, but also things likewriteback_inodes_sband thewakeup_flusher_threadscall fromsys_sync.
- or we have a specific goal, like for the background writeback or thewakeup_flusher_threadsfromfree_more_memory.For the first case there's obviously no point in doing any
older_than_thisprocessing as we write out all inodes anyway.
We may also do older_than_this even for the sync-the-whole-world case, as long as this simplifies wb_writeback() and/or other code. This may make a difference for slow devices.
Quoted 1 line
For the second case we should always do a
older_than_thispass _first_.
Agree in general.
Quoted 6 lines
Rationale: we really should get the old inodes out ASAP, so that we keep the amount of changes lost on a crash in the boundaries. Furthermore the callers only need N pages cleaned, and they don't care where from. So if we can reach our goal with the
older_than_thiswriteback we're fine. If the writeback loop is long enough we can keep doing more of these later on as well.
Right.
Quoted 4 lines
Doing this should also help cleaning the code up a bit by moving the
wb_check_old_data_flushlogic intowb_writebackand getting rid of thefor_kupdateparamter in structwb_writeback_work. I'm not even sure it's worth keeping it in structwriteback_control.
I'd also like to see less for_kupdate tests. Whether or not we can totally get rid of the explicit for_kupdate case, there are always four main writeback goals/semantics:
- periodic stop when all 30s-old inodes are written - background stop when background threshold is reached - nr_pages stop when nr_pages written (or when all clean) - sync stop when all older-than-sync-time inodes are written
Note that
- the "sync" goal is obviously a superset of the "periodic" goal
- the "background" goal may be expanded to include the "periodic" goal
- the latter three goals may all do some "periodic" goal loops, with a moving "old" criterion.
Thanks,
Fengguang