writeback: Tracing support V3
Subsystems (5)
cover writeback: Tracing support V3
Hi Jens,
This series is an update of the writeback tracing that I'd previously posted against mainline. This version is based on the block/for-2.6.36 branch.
The tracing provides visibility into the behaviour of the writeback path and is sufficient to understand questions such as what process is writing data, what process is blocking on congestion, etc. It also allows view the differences in behaviour for different sync modes and calling contexts.
Output (slighty trimmed) for a background flush of a clean bdi is as follows:
flush-8:0-[...]: wbc_writeback_start: bdi 8:0: towrt=1024 skip=0 mode=0 kupd=1 bgrd=0 reclm=0 cyclic=1 more=0 older=0xffff5d8e start=0x0 end=0x0 flush-8:0-[...]: wbc_writeback_written: bdi 8:0: towrt=1024 skip=0 mode=0 kupd=1 bgrd=0 reclm=0 cyclic=1 more=0 older=0xffff5d8e start=0x0 end=0x0 flush-8:0-[...]: writeback_pages_written: 0
Here we see that we are doing writeback for kupdate in cyclic mode with inodes that are older than 0xffff5d8e. The filesystem is clean, so there is nothing to write...
Output (slighty trimmed) for a sync(1) based flush of a clean bdi is as follows:
sync-[...]: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=0 sync_mode=0 kupdate=0 range_cyclic=0 background=0
flush-8:0-[...]: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=0 sync_mode=0 kupdate=0 range_cyclic=0 background=0
flush-8:0-[...]: writeback_pages_written: 0
sync-[...]: writeback_queue: bdi 8:0: sb_dev 8:1 nr_pages=7712 sync_mode=0 kupdate=0 range_cyclic=0 background=0flush-8:0-[...]: writeback_exec: bdi 8:0: sb_dev 8:1 nr_pages=7712 sync_mode=0 kupdate=0 range_cyclic=0 background=0 flush-8:0-[...]: wbc_writeback_start: bdi 8:0: towrt=1024 skip=0 mode=0 kupd=0 bgrd=0 reclm=0 cyclic=0 more=0 older=0x0 start=0x0 end=0x7fffffffffffffff flush-8:0-[...]: wbc_writeback_written: bdi 8:0: towrt=1024 skip=0 mode=0 kupd=0 bgrd=0 reclm=0 cyclic=0 more=0 older=0x0 start=0x0 end=0x7fffffffffffffff flush-8:0-[...]: writeback_pages_written: 0
sync-[...]: writeback_queue: bdi 8:0: sb_dev 8:1 nr_pages=9223372036854775807 sync_mode=1 kupdate=0 range_cyclic=0 background=0
flush-8:0-[...]: writeback_exec: bdi 8:0: sb_dev 8:1 nr_pages=9223372036854775807 sync_mode=1 kupdate=0 range_cyclic=0 background=0 flush-8:0-[...]: wbc_writeback_start: bdi 8:0: wbc=ffff88011cb4bd20 towrt=1024 skip=0 mode=1 kupd=0 bgrd=0 reclm=0 cyclic=0 more=0 older=0x0 start=0x0 end=0x7fffffffffffffff flush-8:0-[...]: wbc_writeback_written: bdi 8:0: wbc=ffff88011cb4bd20 towrt=1024 skip=0 mode=1 kupd=0 bgrd=0 reclm=0 cyclic=0 more=0 older=0x0 start=0x0 end=0x7fffffffffffffff flush-8:0-[...]: writeback_pages_written: 0
What we can see here is that:
- sync queued an async flush to the entire bdi
- the bdi flusher thread wrote no pages (i.e. clean)
- sync has issued an async flush of superblock 8:1 for 7712 pages
- the bdi flush ran writeback on the superblock, wrote no pages
- sync queued a synchronous flush of the superblock
- the bdi flush ran writeback on the superblock, wrote no pages
---
Version 3:
- renamed "wb_sched" tracepoints to "writeback"
- renamed trace class definitions to be more use specific
- made work structure mandatory for writeback work events
- removed unnecessary task/no_task wb_sched trace points.
- converted no_work wb_sched tracepoint to use the basic writeback tracepoint class
- removed check for null bdi in writeback control trace events
Version 2:
- rewrite of initial patch from previous 2.6.34 based code to match new bdi scheduling code as part of rebasing onto the block/for-2.6.36 tree.
- removed work structure address from output as it was mostly redundant and took up lots of space in the output.
- removed writeback control structure address from output as it was mostly redundant and took up lots of space in the output.
PATCH 1/3 writeback: Initial tracing support
Trace queue/sched/exec parts of the writeback loop. This provides insight into when and why flusher threads are scheduled to run. e.g a sync invocation leaves traces like:
sync-[...]: writeback_queue: bdi 8:0: sb_dev 8:1 nr_pages=7712 sync_mode=0 kupdate=0 range_cyclic=0 background=0
flush-8:0-[...]: writeback_exec: bdi 8:0: sb_dev 8:1 nr_pages=7712 sync_mode=0 kupdate=0 range_cyclic=0 background=0
This also lays the foundation for adding more writeback tracing to provide deeper insight into the whole writeback path.
The original tracing code is from Jens Axboe, though this version is a rewrite as a result of the code being traced changing significantly.
--- fs/fs-writeback.c | 38 ++++++++++++--- include/trace/events/writeback.h | 91 ++++++++++++++++++++++++++++++++++++++ mm/backing-dev.c | 3 + 3 files changed, 124 insertions(+), 8 deletions(-) create mode 100644 include/trace/events/writeback.h
fs/fs-writeback.c +30 -8
@@ -26,15 +26,9 @@#include <linux/blkdev.h> #include <linux/backing-dev.h> #include <linux/buffer_head.h> +#include <linux/tracepoint.h> #include "internal.h" -#define inode_to_bdi (inode) ((inode)->i_mapping->backing_dev_info) - -/* -* We don't actually have pdflush, but this one is exported though /proc... -*/ -int nr_pdflush_threads; -/* * Passed into wb_writeback(), essentially a subset of writeback_control */
@@ -50,6 +44,21 @@ struct wb_writeback_work {struct completion *done;/* set if the caller waits */ }; +/* +* Include the creation of the trace points after defining the +* wb_writeback_work structure so that the definition remains local to this +* file. +*/ +#define CREATE_TRACE_POINTS +#include <trace/events/writeback.h> + +#define inode_to_bdi (inode) ((inode)->i_mapping->backing_dev_info) + +/* +* We don't actually have pdflush, but this one is exported though /proc... +*/ +int nr_pdflush_threads; +/** * writeback_in_progress - determine whether there is writeback in progress * @bdi: the device's backing_dev_info structure.
@@ -65,6 +74,8 @@ int writeback_in_progress(struct backing_dev_info *bdi)static void bdi_queue_work (struct backing_dev_info *bdi,struct wb_writeback_work *work) { +trace_writeback_queue (bdi, work); +spin_lock (&bdi->wb_lock);list_add_tail (&work->list, &bdi->work_list);spin_unlock (&bdi->wb_lock);
@@ -74,6 +85,7 @@ static void bdi_queue_work(struct backing_dev_info *bdi,* it gets created and wakes up, we'll run this work. */ if (unlikely (!bdi->wb.task)) { +trace_writeback_nothread (bdi, work);wake_up_process (default_backing_dev_info.wb.task); }else {struct bdi_writeback *wb = &bdi->wb;
@@ -95,8 +107,10 @@ __bdi_start_writeback(struct backing_dev_info *bdi, long nr_pages,*/ work =kzalloc (sizeof (*work),GFP_ATOMIC );if (!work) { -if (bdi->wb.task) +if (bdi->wb.task) { +trace_writeback_nowork (bdi);wake_up_process (bdi->wb.task); + }return ; }
@@ -751,6 +765,8 @@ long wb_do_writeback(struct bdi_writeback *wb, int force_wait)if (force_wait) work->sync_mode =WB_SYNC_ALL ; +trace_writeback_exec (bdi, work); + wrote +=wb_writeback (wb, work);/*
@@ -805,9 +821,13 @@ int bdi_writeback_thread(void *data)smp_mb__after_clear_bit ();wake_up_bit (&bdi->state, BDI_pending); +trace_writeback_thread_start (bdi); +while (!kthread_should_stop ()) { pages_written =wb_do_writeback (wb,0 ); +trace_writeback_pages_written (pages_written); +if (pages_written) last_active = jiffies;else if (wait_jiffies !=-1UL ) {
@@ -845,6 +865,8 @@ int bdi_writeback_thread(void *data)*/ if (!list_empty (&bdi->work_list))wb_do_writeback (wb,1 ); + +trace_writeback_thread_stop (bdi);return 0 ; }
include/trace/events/writeback.h new file +91 -0
91 lines not on this page · raw message
mm/backing-dev.c +3 -0
@@ -10,6 +10,7 @@#include <linux/module.h> #include <linux/writeback.h> #include <linux/device.h> +#include <trace/events/writeback.h> static atomic_long_t bdi_seq =ATOMIC_LONG_INIT (0 );
@@ -518,6 +519,7 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,bdi_debug_register (bdi,dev_name (dev));set_bit (BDI_registered, &bdi->state); +trace_writeback_bdi_register (bdi); exit:return ret; }
@@ -578,6 +580,7 @@ static void bdi_prune_sb(struct backing_dev_info *bdi)void bdi_unregister (struct backing_dev_info *bdi) {if (bdi->dev) { +trace_writeback_bdi_unregister (bdi);bdi_prune_sb (bdi);if (!bdi_cap_flush_forker (bdi))
Signature · 1 line
1.7.1
PATCH 2/3 writeback: Add tracing to balance_dirty_pages
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Tracing high level background writeback events is good, but it doesn't give the entire picture. Add visibility into write throttling to catch IO dispatched by foreground throttling of processing dirtying lots of pages.
--- fs/fs-writeback.c | 5 +++ include/trace/events/writeback.h | 64 ++++++++++++++++++++++++++++++++++++++ mm/page-writeback.c | 4 ++ 3 files changed, 73 insertions(+), 0 deletions(-)
fs/fs-writeback.c +5 -0
@@ -656,10 +656,14 @@ static long wb_writeback(struct bdi_writeback *wb, wbc.more_io =0 ; wbc.nr_to_write =MAX_WRITEBACK_PAGES ; wbc.pages_skipped =0 ; + +trace_wbc_writeback_start (&wbc, wb->bdi);if (work->sb)__writeback_inodes_sb (work->sb, wb, &wbc);else writeback_inodes_wb (wb, &wbc); +trace_wbc_writeback_written (&wbc, wb->bdi); + work->nr_pages -=MAX_WRITEBACK_PAGES - wbc.nr_to_write; wrote +=MAX_WRITEBACK_PAGES - wbc.nr_to_write;
@@ -687,6 +691,7 @@ static long wb_writeback(struct bdi_writeback *wb,if (!list_empty (&wb->b_more_io)) { inode =list_entry (wb->b_more_io.prev, struct inode, i_list); +trace_wbc_writeback_wait (&wbc, wb->bdi);inode_wait_for_writeback (inode); }spin_unlock (&inode_lock);
include/trace/events/writeback.h +64 -0
70 lines not on this page · raw message
mm/page-writeback.c +4 -0
@@ -34,6 +34,7 @@#include <linux/syscalls.h> #include <linux/buffer_head.h> #include <linux/pagevec.h> +#include <trace/events/writeback.h> /* * After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
@@ -535,11 +536,13 @@ static void balance_dirty_pages(struct address_space *mapping,* threshold otherwise wait until the disk writes catch * up. */ +trace_wbc_balance_dirty_start (&wbc, bdi);if (bdi_nr_reclaimable > bdi_thresh) {writeback_inodes_wb (&bdi->wb, &wbc); pages_written += write_chunk - wbc.nr_to_write;get_dirty_limits (&background_thresh, &dirty_thresh, &bdi_thresh, bdi); +trace_wbc_balance_dirty_written (&wbc, bdi); }/*
@@ -565,6 +568,7 @@ static void balance_dirty_pages(struct address_space *mapping,if (pages_written >= write_chunk)break ;/* We've done our duty */ +trace_wbc_balance_dirty_wait (&wbc, bdi);__set_current_state (TASK_INTERRUPTIBLE );io_schedule_timeout (pause);
Signature · 1 line
1.7.1
PATCH 3/3 writeback: Add tracing to write_cache_pages
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Add a trace event to the ->writepage loop in write_cache_pages to give visibility into how the ->writepage call is changing variables within the writeback control structure. Of most interest is how wbc->nr_to_write changes from call to call, especially with filesystems that write multiple pages in ->writepage.
--- include/trace/events/writeback.h | 1 + mm/page-writeback.c | 1 + 2 files changed, 2 insertions(+), 0 deletions(-)
include/trace/events/writeback.h +1 -0
@@ -148,6 +148,7 @@ DEFINE_WBC_EVENT(wbc_writeback_wait);DEFINE_WBC_EVENT (wbc_balance_dirty_start);DEFINE_WBC_EVENT (wbc_balance_dirty_written);DEFINE_WBC_EVENT (wbc_balance_dirty_wait); +DEFINE_WBC_EVENT (wbc_writepage);#endif /* _TRACE_WRITEBACK_H */
mm/page-writeback.c +1 -0
@@ -933,6 +933,7 @@ continue_unlock:if (!clear_page_dirty_for_io (page)) goto continue_unlock; +trace_wbc_writepage (wbc, mapping->backing_dev_info); ret = (*writepage)(page, wbc, data);if (unlikely (ret)) {if (ret ==AOP_WRITEPAGE_ACTIVATE ) {
Signature · 1 line
1.7.1
Quoted 4 lines · Dave Chinner
Hi Jens,
This series is an update of the writeback tracing that I'd previously posted against mainline. This version is based on the
block/for-2.6.36branch.
Thanks, I have applied this version.
Signature · 1 line
Jens Axboe