Page Cache writeback too slow, SSD/noop scheduler/ext2
Subsystems (14)
A message that is not in the archive (sent off-list or not yet received): C5EB16F4.C318%[email protected]
Quoted 46 lines · Andrew Morton
Quoted 35 lines · On Fri, 20 Mar 2009 19:26:06 +0100 Jos Houtman <[email protected]> wrote:
Hi,
We have hit a problem where the page-cache writeback algorithm is not keeping up.
When memory gets low this will result in very irregular performance drops.Our setup is as follows:
30 x Quad core machine with 64GB ram.
These are single purpose machines running MySQL. Kernel version: 2.6.28.7
A dedicated SSD drive for the ext2 database partition Noop scheduler for the ssd drive.The current hypothesis is as follows:
Thewk_updatefunction does not write enough dirty pages, which allows the number of dirty pages to grow to thedirty_backgroundlimit. When memory is low,__background_writeout()comes around and__forcefully__writes dirty pages to disk.
This forced write fills the disk queue and starves read calls that MySQL is trying to do: basically killing performance for a few seconds. This pattern repeats as soon as the cleared memory is filled again.Decreasing the
dirty_writeback_centisecsto 100doesn__thelpI
don__tknow why this is, but I did some preliminary tracing using systemtap and it seems that the majority of timeswk_updatecalls decides to do nothing.Doubling
/sys/block/sdb/queue/nr_requeststo 256, seems to help abit: thenr_dirtypages is increasing more slowly.
But I am unsure of side-effects and am afraid of increasing the starvation problem for mysql.Quoted 8 lines
I__amvery much willing to work on this issue and see it fixed, but would like to tap into the knowledge of people here.
So:
* Have more people seen this or simular issues?
* Is the hypothesis above a viable one?
* Suggestions/pointers for further research and statistics I should measure to improve the understanding of this problem.
Quoted 4 lines
I don't think that noop-iosched tries to do anything to prevent writes-starve-reads. Do you get better behaviour from any of the other IO schedulers?
I did a quick stress test and cfq does not immediately seem to hurt performance, although some of my colleague's have tested this in the past with the opposite results (which is why we use noop).
But despite the scheduler, the real problem is in the writeback algorithm not keeping up.
We can grow 600K dirty pages during the day, and only ~300k is flushed to disk during the night hours.
While a quick look at the writeback algorithm let me to expect __wk_update()__ to flush ~1024 pages every 5 seconds, which is almost 3GB per hour. It obviously does not manage to do this in our setup.
I don¹t believe the speed of the ssd to be the problem, running sync manually only takes a few minutes to flush 800K dirty pages to disk.
With regards,
Jos
Quoted 59 lines · On Monday 23 March 2009 03:53:29 Jos Houtman wrote:
Quoted 45 lines · Andrew Morton
Quoted 36 lines · On Fri, 20 Mar 2009 19:26:06 +0100 Jos Houtman <[email protected]> wrote:
Hi,
We have hit a problem where the page-cache writeback algorithm is not keeping up.
When memory gets low this will result in very irregular performance drops.Our setup is as follows:
30 x Quad core machine with 64GB ram.
These are single purpose machines running MySQL. Kernel version: 2.6.28.7
A dedicated SSD drive for the ext2 database partition Noop scheduler for the ssd drive.The current hypothesis is as follows:
Thewk_updatefunction does not write enough dirty pages, which allows the number of dirty pages to grow to thedirty_backgroundlimit. When memory is low,__background_writeout()comes around and__forcefully__writes dirty pages to disk.
This forced write fills the disk queue and starves read calls that MySQL is trying to do: basically killing performance for a few seconds. This pattern repeats as soon as the cleared memory is filled again.Decreasing the
dirty_writeback_centisecsto 100doesn__thelpI
don__tknow why this is, but I did some preliminary tracing using systemtap and it seems that the majority of timeswk_updatecalls decides to do nothing.Doubling
/sys/block/sdb/queue/nr_requeststo 256, seems to help abit:
thenr_dirtypages is increasing more slowly.
But I am unsure of side-effects and am afraid of increasing the starvation problem for mysql.Quoted 7 lines
I__amvery much willing to work on this issue and see it fixed, but would like to tap into the knowledge of people here.
So:
* Have more people seen this or simular issues?
* Is the hypothesis above a viable one?
* Suggestions/pointers for further research and statistics I should measure to improve the understanding of this problem.Quoted 3 lines
I don't think that noop-iosched tries to do anything to prevent writes-starve-reads. Do you get better behaviour from any of the other IO schedulers?
I did a quick stress test and cfq does not immediately seem to hurt performance, although some of my colleague's have tested this in the past with the opposite results (which is why we use noop).
But despite the scheduler, the real problem is in the writeback algorithm not keeping up.
We can grow 600K dirty pages during the day, and only ~300k is flushed to disk during the night hours.
Quoted 6 lines
While a quick look at the writeback algorithm let me to expect __wk_update()__ to flush ~1024 pages every 5 seconds, which is almost 3GB per hour. It obviously does not manage to do this in our setup.
I don¹t believe the speed of the ssd to be the problem, running sync manually only takes a few minutes to flush 800K dirty pages to disk.
kupdate surely should just continue to keep trying to write back pages so long as there are more old pages to clean, and the queue isn't congested. That seems to be the intention anyway: MAX_WRITEBACK_PAGES is just the number to write back in a single call, but you see nr_to_write is set to the number of dirty pages in the system.
On your system, what must be happening is more_io is not being set. The logic in fs/fs-writeback.c might be busted.
Quoted 67 lines · Nick Piggin
Quoted 59 lines · On Monday 23 March 2009 03:53:29 Jos Houtman wrote:
Quoted 45 lines · Andrew Morton
Quoted 36 lines · On Fri, 20 Mar 2009 19:26:06 +0100 Jos Houtman…
Hi,
We have hit a problem where the page-cache writeback algorithm is not keeping up.
When memory gets low this will result in very irregular performance drops.Our setup is as follows:
30 x Quad core machine with 64GB ram.
These are single purpose machines running MySQL. Kernel version: 2.6.28.7
A dedicated SSD drive for the ext2 database partition Noop scheduler for the ssd drive.The current hypothesis is as follows:
Thewk_updatefunction does not write enough dirty pages, which allows the number of dirty pages to grow to thedirty_backgroundlimit. When memory is low,__background_writeout()comes around and__forcefully__writes dirty pages to disk.
This forced write fills the disk queue and starves read calls that MySQL is trying to do: basically killing performance for a few seconds. This pattern repeats as soon as the cleared memory is filled again.Decreasing the
dirty_writeback_centisecsto 100doesn__thelpI
don__tknow why this is, but I did some preliminary tracing using systemtap and it seems that the majority of timeswk_updatecalls decides to do nothing.Doubling
/sys/block/sdb/queue/nr_requeststo 256, seems to help abit:
thenr_dirtypages is increasing more slowly.
But I am unsure of side-effects and am afraid of increasing the starvation problem for mysql.Quoted 7 lines
I__amvery much willing to work on this issue and see it fixed, but would like to tap into the knowledge of people here.
So:
* Have more people seen this or simular issues?
* Is the hypothesis above a viable one?
* Suggestions/pointers for further research and statistics I should measure to improve the understanding of this problem.Quoted 3 lines
I don't think that noop-iosched tries to do anything to prevent writes-starve-reads. Do you get better behaviour from any of the other IO schedulers?
I did a quick stress test and cfq does not immediately seem to hurt performance, although some of my colleague's have tested this in the past with the opposite results (which is why we use noop).
But despite the scheduler, the real problem is in the writeback algorithm not keeping up.
We can grow 600K dirty pages during the day, and only ~300k is flushed to disk during the night hours.Quoted 6 lines
While a quick look at the writeback algorithm let me to expect __wk_update()__ to flush ~1024 pages every 5 seconds, which is almost 3GB per hour. It obviously does not manage to do this in our setup.
I don¹t believe the speed of the ssd to be the problem, running sync manually only takes a few minutes to flush 800K dirty pages to disk.
Quoted 8 lines
kupdate surely should just continue to keep trying to write back pages so long as there are more old pages to clean, and the queue isn't congested. That seems to be the intention anyway:
MAX_WRITEBACK_PAGESis just the number to write back in a single call, but you seenr_to_writeis set to the number of dirty pages in the system.On your system, what must be happening is
more_iois not being set. The logic infs/fs-writeback.cmight be busted.
Hi Jos,
I prepared a debugging patch for 2.6.28. (I cannot observe writeback problems on my local ext2 mount.)
You can view the states of all dirty inodes by doing
modprobe filecache
echo ls dirty > /proc/filecache
cat /proc/filecache
The 'age' field shows (jiffies - inode->dirtied_when), which may also be useful for debugging Jeff and Ian's case(if it keeps growing, then dirtied_when is stuck).
The detailed dirty writeback traces can be retrieved by doing
echo 1 > /proc/sys/fs/dirty_debug
sleep 6s
echo 0 > /proc/sys/fs/dirty_debug
dmesg
The dmesg trace should help identify the bug in periodic writeback.
Thanks,
Fengguang
include/linux/mm.h +1 -0
@@ -27,6 +27,7 @@ extern unsigned long max_mapnr;extern unsigned long num_physpages;extern void * high_memory;extern int page_cluster; +extern char *const zone_names[];#ifdef CONFIG_SYSCTL extern int sysctl_legacy_va_layout;
mm/page_alloc.c +1 -1
@@ -104,7 +104,7 @@ int sysctl_lowmem_reserve_ratio[MAX_NR_ZEXPORT_SYMBOL (totalram_pages); -static char *const zone_names[MAX_NR_ZONES ] = { +char *const zone_names[MAX_NR_ZONES ] = { #ifdef CONFIG_ZONE_DMA"DMA" , #endif
fs/dcache.c +4 -1
@@ -1943,7 +1943,10 @@ char *__d_path(const struct path *path,if (dentry == root->dentry && vfsmnt == root->mnt)break ; -if (dentry == vfsmnt->mnt_root ||IS_ROOT (dentry)) { +if (unlikely (!vfsmnt)) { +if (IS_ROOT (dentry)) +break ; + }else if (dentry == vfsmnt->mnt_root ||IS_ROOT (dentry)) {/* Global root? */ if (vfsmnt->mnt_parent == vfsmnt) { goto global_root;
lib/radix-tree.c +0 -2
@@ -564,7 +564,6 @@ out: }EXPORT_SYMBOL (radix_tree_tag_clear); -#ifndef __KERNEL__/* Only the test harness uses this at present */ /** * radix_tree_tag_get - get a tag on a radix tree node * @root: radix tree root
@@ -627,7 +626,6 @@ int radix_tree_tag_get(struct radix_tree } }EXPORT_SYMBOL (radix_tree_tag_get); -#endif /** * radix_tree_next_hole - find the next hole (not-present entry)
fs/inode.c +26 -1
@@ -82,6 +82,10 @@ static struct hlist_head *inode_hashtabl*/ DEFINE_SPINLOCK (inode_lock); +EXPORT_SYMBOL (inode_in_use); +EXPORT_SYMBOL (inode_unused); +EXPORT_SYMBOL (inode_lock); +/* * iprune_mutex provides exclusion between the kswapd or try_to_free_pages * icache shrinking path, and the umount path. Without this exclusion,
@@ -108,6 +112,14 @@ static void wake_up_inode(struct inode *wake_up_bit (&inode->i_state, __I_LOCK); } +static inline void inode_created_by (struct inode *inode,struct task_struct *task) +{ +#ifdef CONFIG_PROC_FILECACHE_EXTRAS + inode->i_cuid = task->uid; +memcpy (inode->i_comm, task->comm,sizeof (task->comm)); +#endif +} +static struct inode *alloc_inode (struct super_block *sb) {static const struct address_space_operations empty_aops;
@@ -142,7 +154,7 @@ static struct inode *alloc_inode(struct inode->i_bdev =NULL ; inode->i_cdev =NULL ; inode->i_rdev =0 ; - inode->dirtied_when =0 ; + inode->dirtied_when = jiffies;if (security_inode_alloc (inode)) {if (inode->i_sb->s_op->destroy_inode) inode->i_sb->s_op->destroy_inode (inode);
@@ -183,6 +195,7 @@ static struct inode *alloc_inode(struct } inode->i_private =NULL ; inode->i_mapping = mapping; +inode_created_by (inode, current); }return inode; }
@@ -247,6 +260,8 @@ void __iget(struct inode * inode) inodes_stat.nr_unused--; } +EXPORT_SYMBOL (__iget); +/** * clear_inode - clear an inode * @inode: inode to clear
@@ -1353,6 +1368,16 @@ void inode_double_unlock(struct inode *i }EXPORT_SYMBOL (inode_double_unlock); + +struct hlist_head *get_inode_hash_budget (unsigned long index) +{ +if (index >= (1 << i_hash_shift)) +return NULL ; + +return inode_hashtable + index; +} +EXPORT_SYMBOL_GPL (get_inode_hash_budget); +static __initdata unsigned long ihash_entries;static int __init set_ihash_entries (char *str) {
fs/super.c +3 -0
@@ -45,6 +45,9 @@LIST_HEAD (super_blocks);DEFINE_SPINLOCK (sb_lock); +EXPORT_SYMBOL (super_blocks); +EXPORT_SYMBOL (sb_lock); +/** * alloc_super - create new superblock * @type: filesystem type superblock should belong to
mm/vmscan.c +1 -0
@@ -230,6 +230,7 @@ unsigned long shrink_slab(unsigned longup_read (&shrinker_rwsem);return ret; } +EXPORT_SYMBOL (shrink_slab);/* Called without lock on whether page is mapped, so answer is unstable */ static inline int page_mapping_inuse (struct page *page)
mm/swap_state.c +1 -0
@@ -44,6 +44,7 @@ struct address_space swapper_space = { .i_mmap_nonlinear = LIST_HEAD_INIT(swapper_space.i_mmap_nonlinear), .backing_dev_info = &swap_backing_dev_info, }; +EXPORT_SYMBOL_GPL (swapper_space);#define INC_CACHE_INFO (x) do { swap_cache_info.x++; } while (0)
Documentation/filesystems/proc.txt +83 -0
95 lines not on this page · raw message
fs/proc/filecache.c new file +1046 -0
1,046 lines not on this page · raw message
include/linux/fs.h +14 -0
@@ -685,6 +685,12 @@ struct inode {void *i_security;#endif void *i_private;/* fs or device private pointer */ + +#ifdef CONFIG_PROC_FILECACHE_EXTRAS +unsigned int i_access_count;/* opened how many times? */ +uid_t i_cuid;/* opened first by which user? */ +char i_comm[16 ];/* opened first by which app? */ +#endif };/*
@@ -773,6 +779,13 @@ static inline unsigned imajor(const strureturn MAJOR (inode->i_rdev); } +static inline void inode_accessed (struct inode *inode) +{ +#ifdef CONFIG_PROC_FILECACHE_EXTRAS + inode->i_access_count++; +#endif +} +extern struct block_device *I_BDEV (struct inode *inode);struct fown_struct {
@@ -1907,6 +1920,7 @@ extern void remove_inode_hash(struct inostatic inline void insert_inode_hash (struct inode *inode) {__insert_inode_hash (inode, inode->i_ino); } +struct hlist_head *get_inode_hash_budget (unsigned long index);extern struct file *get_empty_filp (void );extern void file_move (struct file *f,struct list_head *list);
fs/open.c +1 -0
@@ -828,6 +828,7 @@ static struct file *__dentry_open(struct goto cleanup_all; } +inode_accessed (inode); f->f_flags &= ~(O_CREAT |O_EXCL |O_NOCTTY |O_TRUNC );file_ra_state_init (&f->f_ra, f->f_mapping->host->i_mapping);
fs/Kconfig +30 -0
@@ -750,6 +750,36 @@ config CONFIGFS_FS Both sysfs and configfs can and should exist together on the same system. One is not a replacement for the other. +config PROC_FILECACHE + tristate "/proc/filecache support" + default m + depends on PROC_FS + help + This option creates a file /proc/filecache which enables one to + query/drop the cached files in memory. + + A quick start guide: + + # echo 'ls' > /proc/filecache + # head /proc/filecache + + # echo 'cat /bin/bash' > /proc/filecache + # head /proc/filecache + + # echo 'drop pagecache' > /proc/filecache + # echo 'drop slabcache' > /proc/filecache + + For more details, please check Documentation/filesystems/proc.txt . + + It can be a handy tool for sysadms and desktop users. + +config PROC_FILECACHE_EXTRAS + bool "track extra states" + default y + depends on PROC_FILECACHE + help + Track extra states that costs a little more time/space. + endmenu menu "Miscellaneous filesystems"
fs/proc/Makefile +2 -1
@@ -2,7 +2,8 @@ # Makefile for the Linux proc filesystem routines. # -obj-$(CONFIG_PROC_FS) += proc.o +obj-$(CONFIG_PROC_FS) += proc.o +obj-$(CONFIG_PROC_FILECACHE) += filecache.o proc-y := nommu.o task_nommu.o proc-$(CONFIG_MMU) := mmu.o task_mmu.o
fs/fs-writeback.c +81 -2
118 lines not on this page · raw message
include/linux/writeback.h +1 -0
@@ -159,5 +159,6 @@ void writeback_set_ratelimit(void);extern int nr_pdflush_threads;/* Global so it can be exported to sysctl read-only. */ +extern int sysctl_dirty_debug;#endif /* WRITEBACK_H */
kernel/sysctl.c +8 -0
@@ -1344,6 +1344,14 @@ static struct ctl_table fs_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "dirty_debug", + .data = &sysctl_dirty_debug, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, #if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE) { .ctl_name = CTL_UNNUMBERED,
mm/page-writeback.c +40 -0
@@ -104,6 +104,35 @@ EXPORT_SYMBOL(laptop_mode);/* End of sysctl-exported parameters */ +#define writeback_debug_report (n, wbc) do { \ + if(sysctl_dirty_debug) \ + __writeback_debug_report(n, wbc, \ + __FILE__, __LINE__, __FUNCTION__); \ +} while (0) + +void print_writeback_control (struct writeback_control *wbc) +{ +printk (KERN_DEBUG +"global dirty %lu writeback %lu nfs %lu " +"flags %c%c towrite %ld skipped %ld\n" , +global_page_state (NR_FILE_DIRTY ), +global_page_state (NR_WRITEBACK ), +global_page_state (NR_UNSTABLE_NFS ), + wbc->encountered_congestion ?'C' :'_' , + wbc->more_io ?'M' :'_' , + wbc->nr_to_write, + wbc->pages_skipped); +} + +void __writeback_debug_report (long n,struct writeback_control *wbc, +const char *file,int line,const char *func) +{ +printk (KERN_DEBUG "%s %d %s: %s(%d) %ld\n" , + file, line, func, + current->comm, current->pid, + n); +print_writeback_control (wbc); +}static void background_writeout (unsigned long _min_pages);
@@ -476,6 +505,7 @@ static void balance_dirty_pages(struct a pages_written += write_chunk - wbc.nr_to_write;get_dirty_limits (&background_thresh, &dirty_thresh, &bdi_thresh, bdi); +writeback_debug_report (pages_written, &wbc); }/*
@@ -502,6 +532,7 @@ static void balance_dirty_pages(struct abreak ;/* We've done our duty */ congestion_wait (WRITE ,HZ /10 ); +writeback_debug_report (-pages_written, &wbc); } if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh &&
@@ -596,6 +627,11 @@ void throttle_vm_writeout(gfp_t gfp_mask global_page_state(NR_WRITEBACK) <= dirty_thresh)break ;congestion_wait (WRITE ,HZ /10 ); +printk (KERN_DEBUG "throttle_vm_writeout: " +"congestion_wait on %lu+%lu > %lu\n" , +global_page_state (NR_UNSTABLE_NFS ), +global_page_state (NR_WRITEBACK ), + dirty_thresh);/* * The caller might hold locks which can prevent IO completion
@@ -645,7 +681,9 @@ static void background_writeout(unsignedelse break ; } +writeback_debug_report (min_pages, &wbc); } +writeback_debug_report (min_pages, &wbc); }/*
@@ -718,7 +756,9 @@ static void wb_kupdate(unsigned long argbreak ;/* All the old data is written */ } nr_to_write -=MAX_WRITEBACK_PAGES - wbc.nr_to_write; +writeback_debug_report (nr_to_write, &wbc); } +writeback_debug_report (nr_to_write, &wbc);if (time_before (next_jif, jiffies +HZ )) next_jif = jiffies +HZ ;if (dirty_writeback_interval)
Hi,
Quoted 6 lines
Quoted 6 lines
kupdate surely should just continue to keep trying to write back pages so long as there are more old pages to clean, and the queue isn't congested. That seems to be the intention anyway:
MAX_WRITEBACK_PAGESis just the number to write back in a single call, but you seenr_to_writeis set to the number of dirty pages in the system.
And when it's congested it should just wait a little bit before continuing.
Quoted 2 lines
Quoted 2 lines
On your system, what must be happening is
more_iois not being set. The logic infs/fs-writeback.cmight be busted.
I don't know about more_io, but I agree that the logic seems busted.
Quoted 5 lines
Hi Jos,
I prepared a debugging patch for 2.6.28. (I cannot observe writeback problems on my local ext2 mount.)
Thanx for the patch, but for the next time: How should I apply it? it seems to be context aware (@@) and broke on all kernel versions I tried 2.6.28/2.6.28.7/2.6.29
Because I saw the patch only a few hour ago and didn't want to block on your reply I decided to patch it manually and in the process ported it to 2.6.29.
As for the information the patch provided: It is most helpful.
Attached you will find a list of files containing dirty pages and the count of there dirty pages, there is also a dmesg output where I trace the writeback for 40 seconds.
I did some testing on my own using printk's and what I saw is that for the inodes located on sdb1 (the database) a lot of times they would pass
http://lxr.linux.no/linux+v2.6.29/fs/fs-writeback.c#L335
And then redirty_tail would be called, I haven't had the time to dig deeper, but that is my primary suspect for the moment.
Thanx again,
Jos
grep dirty /proc/vmstat; for i in $( echo ls > /proc/filecache; cat /proc/filecache | awk '{ if ($6 ~ /[D]/) print $0 }' | awk '{if ($12 ~ /database/) print $12}' ); do echo -n "$i dirty: "; echo /var/lib/mysql/$i > /proc/filecache; cat /proc/filecache | grep D | wc -l; done; echo ls > /proc/filecache; cat /proc/filecache | awk '{ if ($6 ~ /[D]/) print $0 }' nr_dirty 494902
/database/tbltable3.MYD dirty: 58101
/database/tbltable3.MYI dirty: 146806
/database/tbltable4.MYD dirty: 85737
/database/tbltable4.MYI dirty: 101299
/database/tbltable1.MYI dirty: 1727
/database/tbltable5.MYD dirty: 27189
/database/tbltable5.MYI dirty: 16847
/database/tbltable2.MYI dirty: 3
/database/tbltable2.MYD dirty: 3
/database/tbltable1.MYD dirty: 4
0 5242880 1072 0 0 D- 6 0 0 mount 00:02(bdev) (fd:02)
0 2097152 884 0 0 D- 6 0 0 mount 00:02(bdev) (fd:03)
0 5242880 5532 0 0 D- 23 0 0 mount 00:02(bdev) (fd:01)
196632 1 4 100 1 D- 5 1 60 mysqld fd:01(dm-1) /run/mysqld/relay-log.info
196634 306501 306504 100 1 D- 15 1 60 mysqld fd:01(dm-1) /run/mysqld/mysqld-relay-bin.000002
49163 466 48 10 1 D- 12 1 0 syslog-ng fd:02(dm-2) /messages
23 1 4 100 0 D- 11 104 0 bash fd:03(dm-3) /differenceWithLastTime_dm-3
22 1 4 100 0 D- 11 104 0 bash fd:03(dm-3) /differenceWithLastTime_dm-2
21 1 4 100 0 D- 11 104 0 bash fd:03(dm-3) /differenceWithLastTime_dm-1
20 1 4 100 0 D- 11 104 0 bash fd:03(dm-3) /differenceWithLastTime_dm-0
19 1 4 100 0 D- 11 104 0 bash fd:03(dm-3) /differenceWithLastTime_sdb1
18 1 4 100 0 D- 12 104 0 bash fd:03(dm-3) /differenceWithLastTime_sda3
17 1 4 100 0 D- 12 104 0 bash fd:03(dm-3) /differenceWithLastTime_sda2
16 1 4 100 0 D- 12 104 0 bash fd:03(dm-3) /differenceWithLastTime_sda1
14 1 4 100 0 D- 12 104 0 sh fd:03(dm-3) /netdev_old
15 1 4 100 0 D- 12 104 0 perl fd:03(dm-3) /.oldnetstat
13 1 4 100 0 D- 12 520 0 sh fd:03(dm-3) /netdev_new
5865478 3778125 251852 6 2 D- 10 12 0 du 08:11(sdb1) /database/tbltable3.MYD
5865487 12871243 1132552 8 1 D- 10 10 0 du 08:11(sdb1) /database/tbltable3.MYI
5865475 21707331 503688 2 2 D- 10 11 0 du 08:11(sdb1) /database/tbltable4.MYD
5865495 16414241 909488 5 1 D- 10 10 0 du 08:11(sdb1) /database/tbltable4.MYI
11 1 4 100 1 D- 15 1 60 mysqld 08:11(sdb1) /master.info
5865477 150818 51952 34 1 D- 15 10 0 du 08:11(sdb1) /database/tbltable1.MYI
5865479 495759 131276 26 1 D- 15 10 0 du 08:11(sdb1) /database/tbltable5.MYD
5865497 903791 276828 30 1 D- 15 10 0 du 08:11(sdb1) /database/tbltable5.MYI
5865499 118388 88 0 1 D- 17 10 0 du 08:11(sdb1) /database/tbltable2.MYI
5865480 325977 120 0 1 D- 17 10 0 du 08:11(sdb1) /database/tbltable2.MYD
5865489 78503 3300 4 2 D- 18 10 0 du 08:11(sdb1) /database/tbltable1.MYD
175 1 4 100 1 D- 3100 1161 0 udevd 00:0e(tmpfs) /.udev/uevent_seqnumredirty_tail line 539: inode 10144
inode 10144(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 447069 global dirty 438821 writeback 0 nfs 0 flags __ towrite 981 skipped 0 redirty_tail line 417: inode 5865495
inode 5865495(sdb1/tbltable4.MYI) count 2,1 size 16807325696 pages 197538
0 2 0 UD_P_ 1340 2 0 UD_P_ 1367 2 0 UD_P_ 1417 2 0 UD_P_ 3578 2 0 UD_P_ 3745 2 0 UD_P_ 3928 2 0 UD_P_ 4116 2 0 UD_P_ 4154 2 0 UD_P_ 4207 2 0 UD_P_ 4838 2 0 UD_P_ 4839 2 0 UD_P_ 4840 2 0 UD_P_ 5600 2 0 UD_P_
redirty_tail line 539: inode 1522
inode 1522(sysfs/0000:01:00.0) count 1,5 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 448366 global dirty 440133 writeback 91 nfs 0 flags C_ towrite 904 skipped 0 redirty_tail line 417: inode 5865475
inode 5865475(sdb1/tbltable4.MYD) count 2,2 size 22227138512 pages 106269
373 2 0 UD_P_ 1041 2 0 UD_P_ 2134 2 0 UD_P_ 3075 2 0 UD_P_ 4563 2 0 UD_P_ 5306 2 0 UD_P_ 5426 2 0 UD_P_ 5473 2 0 UD_P_ 5657 2 0 UD_P_ 5770 2 0 UD_P_ 5983 2 0 UD_P_ 7883 2 0 UD_P_ 9061 2 0 UD_P_ 9241 2 0 UD_P_
redirty_tail line 539: inode 10297
inode 10297(sysfs/subsystem) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 448354 global dirty 440133 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 417: inode 5865487
inode 5865487(sdb1/tbltable3.MYI) count 2,1 size 13179606016 pages 250251
0 2 0 UD_P_ 65 2 0 UD_P_ 109 2 0 UD_P_ 195 2 0 UD_P_ 200 2 0 UD_P_ 368 2 0 UD_P_ 473 2 0 UD_P_ 481 2 0 UD_P_ 530 2 0 UD_P_ 547 2 0 UD_P_ 695 2 0 UD_P_ 735 2 0 UD_P_ 751 2 0 UD_P_ 970 2 0 UD_P_
redirty_tail line 539: inode 10302
inode 10302(sysfs/driver) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 448342 global dirty 440133 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 417: inode 5865478
inode 5865478(sdb1/tbltable3.MYD) count 2,2 size 3868679145 pages 55009
159 2 0 UD_P_ 195 2 0 UD_P_ 207 2 0 UD_P_ 211 2 0 UD_P_ 220 2 0 UD_P_ 282 2 0 UD_P_ 283 2 0 UD_P_ 324 2 0 UD_P_ 436 2 0 UD_P_ 439 2 0 UD_P_ 548 2 0 UD_P_ 553 2 0 UD_P_ 682 2 0 UD_P_ 934 2 0 UD_P_
redirty_tail line 539: inode 10066
inode 10066(sysfs/timeout) count 1,0 size 4096 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 448329 global dirty 440042 writeback 91 nfs 0 flags C_ towrite 1011 skipped 0 redirty_tail line 539: inode 10289
inode 10289(sysfs/timeout) count 1,0 size 4096 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 448329 global dirty 440042 writeback 91 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 539: inode 8766
inode 8766(sysfs/ram0) count 1,14 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 448543 global dirty 440295 writeback 0 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 539: inode 8778
inode 8778(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 448873 global dirty 440625 writeback 0 nfs 0 flags __ towrite 1016 skipped 0 redirty_tail line 539: inode 8780
inode 8780(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 449115 global dirty 440867 writeback 0 nfs 0 flags __ towrite 1021 skipped 0 redirty_tail line 539: inode 8781
inode 8781(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 449441 global dirty 441193 writeback 0 nfs 0 flags __ towrite 984 skipped 0 redirty_tail line 417: inode 5865497
inode 5865497(sdb1/tbltable5.MYI) count 2,1 size 925447168 pages 63265
0 2 0 UD_P_ 1172 2 0 UD_P_ 1187 2 0 UD_P_ 1522 2 0 UD_P_ 1567 2 0 UD_P_ 1720 2 0 UD_P_ 1760 2 0 UD_P_ 1807 2 0 UD_P_ 1811 2 0 UD_P_ 2272 2 0 UD_P_ 2377 2 0 UD_P_ 2534 2 0 UD_P_ 2647 2 0 UD_P_ 2777 2 0 UD_P_
redirty_tail line 539: inode 8791
inode 8791(sysfs/ram1) count 1,14 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449463 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 564 skipped 0 redirty_tail line 417: inode 5865479
inode 5865479(sdb1/tbltable5.MYD) count 2,1 size 507638336 pages 29268
0 2 0 UD_P_ 1 2 0 UD_P_ 2 2 0 UD_P_ 3 2 0 UD_P_ 4 2 0 UD_P_ 8 2 0 UD_P_ 12 2 0 UD_P_ 16 2 0 UD_P_ 22 2 0 UD_P_ 26 2 0 UD_P_ 32 2 0 UD_P_ 33 2 0 UD_P_ 40 2 0 UD_P_ 41 2 0 UD_P_
redirty_tail line 539: inode 8803
inode 8803(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449448 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1009 skipped 0 redirty_tail line 539: inode 8805
inode 8805(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449448 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1024 skipped 0 redirty_tail line 539: inode 8806
inode 8806(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449448 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1024 skipped 0 redirty_tail line 539: inode 8816
inode 8816(sysfs/ram2) count 1,14 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449448 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1024 skipped 0 redirty_tail line 539: inode 8828
inode 8828(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449448 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1024 skipped 0 redirty_tail line 417: inode 5865477
inode 5865477(sdb1/tbltable1.MYI) count 2,1 size 154434560 pages 11790
0 2 0 UD_P_ 68 2 0 UD_P_ 106 2 0 UD_P_ 131 2 0 UD_P_ 185 2 0 UD_P_ 244 2 0 UD_P_ 252 2 0 UD_P_ 331 2 0 UD_P_ 393 2 0 UD_P_ 394 2 0 UD_P_ 395 2 0 UD_P_ 400 2 0 UD_P_ 482 2 0 UD_P_ 781 2 0 UD_P_
redirty_tail line 539: inode 8830
inode 8830(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449436 global dirty 441220 writeback 182 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 539: inode 8831
inode 8831(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 449436 global dirty 441220 writeback 182 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 417: inode 5865495
inode 5865495(sdb1/tbltable4.MYI) count 2,1 size 16807348224 pages 197993
0 2 0 UD_P_ 1340 2 0 UD_P_ 1367 2 0 UD_P_ 1417 2 0 UD_P_ 3578 2 0 UD_P_ 3745 2 0 UD_P_ 3928 2 0 UD_P_ 4116 2 0 UD_P_ 4154 2 0 UD_P_ 4207 2 0 UD_P_ 4838 2 0 UD_P_ 4839 2 0 UD_P_ 4840 2 0 UD_P_ 5600 2 0 UD_P_
redirty_tail line 539: inode 8841
inode 8841(sysfs/ram3) count 1,14 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449551 global dirty 441343 writeback 91 nfs 0 flags C_ towrite 893 skipped 0 redirty_tail line 417: inode 5865475
inode 5865475(sdb1/tbltable4.MYD) count 2,2 size 22227169324 pages 106507
373 2 0 UD_P_ 1041 2 0 UD_P_ 2134 2 0 UD_P_ 3075 2 0 UD_P_ 4563 2 0 UD_P_ 5306 2 0 UD_P_ 5426 2 0 UD_P_ 5473 2 0 UD_P_ 5657 2 0 UD_P_ 5770 2 0 UD_P_ 5983 2 0 UD_P_ 7883 2 0 UD_P_ 9061 2 0 UD_P_ 9241 2 0 UD_P_
redirty_tail line 539: inode 8853
inode 8853(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449538 global dirty 441343 writeback 91 nfs 0 flags C_ towrite 1011 skipped 0 redirty_tail line 417: inode 5865487
inode 5865487(sdb1/tbltable3.MYI) count 2,1 size 13179631616 pages 251244
0 2 0 UD_P_ 65 2 0 UD_P_ 109 2 0 UD_P_ 195 2 0 UD_P_ 200 2 0 UD_P_ 368 2 0 UD_P_ 473 2 0 UD_P_ 481 2 0 UD_P_ 530 2 0 UD_P_ 547 2 0 UD_P_ 695 2 0 UD_P_ 735 2 0 UD_P_ 751 2 0 UD_P_ 970 2 0 UD_P_
redirty_tail line 539: inode 8855
inode 8855(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449526 global dirty 441343 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 417: inode 5865478
inode 5865478(sdb1/tbltable3.MYD) count 2,2 size 3868689540 pages 55174
159 2 0 UD_P_ 195 2 0 UD_P_ 207 2 0 UD_P_ 211 2 0 UD_P_ 220 2 0 UD_P_ 282 2 0 UD_P_ 283 2 0 UD_P_ 324 2 0 UD_P_ 436 2 0 UD_P_ 439 2 0 UD_P_ 548 2 0 UD_P_ 553 2 0 UD_P_ 682 2 0 UD_P_ 934 2 0 UD_P_
redirty_tail line 539: inode 8856
inode 8856(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 449514 global dirty 441343 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 539: inode 8866
inode 8866(sysfs/ram4) count 1,14 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 449514 global dirty 441343 writeback 91 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 539: inode 8878
inode 8878(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 449732 global dirty 441484 writeback 0 nfs 0 flags __ towrite 1000 skipped 0 redirty_tail line 539: inode 8880
inode 8880(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450062 global dirty 441814 writeback 0 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 539: inode 8881
inode 8881(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450496 global dirty 442248 writeback 0 nfs 0 flags __ towrite 1014 skipped 0 redirty_tail line 539: inode 8891
inode 8891(sysfs/ram5) count 1,14 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450655 global dirty 442407 writeback 0 nfs 0 flags __ towrite 1018 skipped 0 redirty_tail line 417: inode 5865497
inode 5865497(sdb1/tbltable5.MYI) count 2,1 size 925447168 pages 63374
0 2 0 UD_P_ 1172 2 0 UD_P_ 1187 2 0 UD_P_ 1522 2 0 UD_P_ 1567 2 0 UD_P_ 1720 2 0 UD_P_ 1760 2 0 UD_P_ 1807 2 0 UD_P_ 1811 2 0 UD_P_ 2272 2 0 UD_P_ 2377 2 0 UD_P_ 2534 2 0 UD_P_ 2647 2 0 UD_P_ 2777 2 0 UD_P_
redirty_tail line 539: inode 8903
inode 8903(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450809 global dirty 442608 writeback 91 nfs 0 flags C_ towrite 886 skipped 0 redirty_tail line 417: inode 5865479
inode 5865479(sdb1/tbltable5.MYD) count 2,1 size 507638658 pages 29338
0 2 0 UD_P_ 1 2 0 UD_P_ 2 2 0 UD_P_ 3 2 0 UD_P_ 4 2 0 UD_P_ 8 2 0 UD_P_ 12 2 0 UD_P_ 16 2 0 UD_P_ 22 2 0 UD_P_ 26 2 0 UD_P_ 32 2 0 UD_P_ 33 2 0 UD_P_ 40 2 0 UD_P_ 41 2 0 UD_P_
redirty_tail line 539: inode 8905
inode 8905(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450793 global dirty 442517 writeback 91 nfs 0 flags C_ towrite 1008 skipped 0 redirty_tail line 417: inode 5865477
inode 5865477(sdb1/tbltable1.MYI) count 2,1 size 154434560 pages 11815
0 2 0 UD_P_ 68 2 0 UD_P_ 106 2 0 UD_P_ 131 2 0 UD_P_ 185 2 0 UD_P_ 244 2 0 UD_P_ 252 2 0 UD_P_ 331 2 0 UD_P_ 393 2 0 UD_P_ 394 2 0 UD_P_ 395 2 0 UD_P_ 400 2 0 UD_P_ 482 2 0 UD_P_ 781 2 0 UD_P_
redirty_tail line 539: inode 8906
inode 8906(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450781 global dirty 442517 writeback 182 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 539: inode 8916
inode 8916(sysfs/ram6) count 1,14 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450781 global dirty 442517 writeback 182 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 417: inode 5865495
inode 5865495(sdb1/tbltable4.MYI) count 2,1 size 16807370752 pages 198676
0 2 0 UD_P_ 1340 2 0 UD_P_ 1367 2 0 UD_P_ 1417 2 0 UD_P_ 3578 2 0 UD_P_ 3745 2 0 UD_P_ 3928 2 0 UD_P_ 4116 2 0 UD_P_ 4154 2 0 UD_P_ 4207 2 0 UD_P_ 4838 2 0 UD_P_ 4839 2 0 UD_P_ 4840 2 0 UD_P_ 5600 2 0 UD_P_
redirty_tail line 539: inode 8928
inode 8928(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450654 global dirty 442401 writeback 91 nfs 0 flags C_ towrite 644 skipped 0 redirty_tail line 417: inode 5865475
inode 5865475(sdb1/tbltable4.MYD) count 2,2 size 22227197236 pages 106846
373 2 0 UD_P_ 1041 2 0 UD_P_ 2134 2 0 UD_P_ 3075 2 0 UD_P_ 4563 2 0 UD_P_ 5306 2 0 UD_P_ 5426 2 0 UD_P_ 5473 2 0 UD_P_ 5657 2 0 UD_P_ 5770 2 0 UD_P_ 5983 2 0 UD_P_ 7883 2 0 UD_P_ 9061 2 0 UD_P_ 9241 2 0 UD_P_
redirty_tail line 539: inode 8930
inode 8930(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450641 global dirty 442401 writeback 91 nfs 0 flags C_ towrite 1011 skipped 0 redirty_tail line 417: inode 5865487
inode 5865487(sdb1/tbltable3.MYI) count 2,1 size 13179643904 pages 251935
0 2 0 UD_P_ 65 2 0 UD_P_ 109 2 0 UD_P_ 195 2 0 UD_P_ 200 2 0 UD_P_ 368 2 0 UD_P_ 473 2 0 UD_P_ 481 2 0 UD_P_ 530 2 0 UD_P_ 547 2 0 UD_P_ 695 2 0 UD_P_ 735 2 0 UD_P_ 751 2 0 UD_P_ 970 2 0 UD_P_
redirty_tail line 539: inode 8931
inode 8931(sysfs/slaves) count 1,0 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450629 global dirty 442401 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 417: inode 5865478
inode 5865478(sdb1/tbltable3.MYD) count 2,2 size 3868692045 pages 55323
159 2 0 UD_P_ 195 2 0 UD_P_ 207 2 0 UD_P_ 211 2 0 UD_P_ 220 2 0 UD_P_ 282 2 0 UD_P_ 283 2 0 UD_P_ 324 2 0 UD_P_ 436 2 0 UD_P_ 439 2 0 UD_P_ 548 2 0 UD_P_ 553 2 0 UD_P_ 682 2 0 UD_P_ 934 2 0 UD_P_
redirty_tail line 539: inode 8941
inode 8941(sysfs/ram7) count 1,14 size 0 pages 0 mm/page-writeback.c 829 wb_kupdate: pdflush(361) 450617 global dirty 442401 writeback 91 nfs 0 flags C_ towrite 1012 skipped 0 redirty_tail line 539: inode 8953
inode 8953(sysfs/power) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450617 global dirty 442401 writeback 91 nfs 0 flags __ towrite 1024 skipped 0 redirty_tail line 539: inode 8955
inode 8955(sysfs/holders) count 1,0 size 0 pages 0 mm/page-writeback.c 831 wb_kupdate: pdflush(361) 450881 global dirty 442633 writeback 0 nfs 0 flags __ towrite 977 skipped 0
Quoted 17 lines · Jos Houtman
Hi,
Quoted 6 lines
Quoted 6 lines
kupdate surely should just continue to keep trying to write back pages so long as there are more old pages to clean, and the queue isn't congested. That seems to be the intention anyway:
MAX_WRITEBACK_PAGESis just the number to write back in a single call, but you seenr_to_writeis set to the number of dirty pages in the system.And when it's congested it should just wait a little bit before continuing.
Quoted 2 lines
Quoted 2 lines
On your system, what must be happening is
more_iois not being set. The logic infs/fs-writeback.cmight be busted.I don't know about
more_io, but I agree that the logic seems busted.Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Hi Jos,
I prepared a debugging patch for 2.6.28. (I cannot observe writeback problems on my local ext2 mount.)
Thanx for the patch, but for the next time: How should I apply it? it seems to be context aware (@@) and broke on all kernel versions I tried 2.6.28/2.6.28.7/2.6.29
Do you mean that the patch applies after removing " @@.*$"?
To be safe, I created the patch with quilt as well as git, for 2.6.29.
Quoted 8 lines
Because I saw the patch only a few hour ago and didn't want to block on your reply I decided to patch it manually and in the process ported it to 2.6.29.
As for the information the patch provided: It is most helpful.
Attached you will find a list of files containing dirty pages and the count of there dirty pages, there is also a dmesg output where I trace the writeback for 40 seconds.
They helped, thank you!
Quoted 5 lines
I did some testing on my own using printk's and what I saw is that for the inodes located on sdb1 (the database) a lot of times they would pass
http://lxr.linux.no/linux+v2.6.29/fs/fs-writeback.c#L335
And thenredirty_tailwould be called, I haven't had the time to dig deeper, but that is my primary suspect for the moment.
You are right. In your case, there are several big dirty files in sdb1, and the sdb write queue is constantly (almost-)congested. The SSD write speed is so slow, that in each round of sdb1 writeback, it begins with an uncongested queue, but quickly fills up after writing some pages. Hence all the inodes will get redirtied because of (nr_to_write > 0).
The following quick fix should solve the slow-writeback-on-congested-SSD problem. However the writeback sequence is suboptimal: it sync-and-requeue each file until congested (in your case about 3~600 pages) instead of until MAX_WRITEBACK_PAGES=1024 pages.
A more complete fix would be turning MAX_WRITEBACK_PAGES into an exact per-file limit. It has been sitting in my todo list for quite a while...
Thanks,
Fengguang
--- fs/fs-writeback.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/fs-writeback.c +2 -1
@@ -325,7 +325,8 @@ __sync_single_inode(struct inode *inode,* soon as the queue becomes uncongested. */ inode->i_state |=I_DIRTY_PAGES ; -if (wbc->nr_to_write <=0 ) { +if (wbc->nr_to_write <=0 || + wbc->encountered_congestion) {/* * slice used up: queue for next turn */
fs/fs-writeback.c +2 -1
@@ -325,7 +325,8 @@ __sync_single_inode(struct inode *inode, struct writeback_control *wbc)* soon as the queue becomes uncongested. */ inode->i_state |=I_DIRTY_PAGES ; -if (wbc->nr_to_write <=0 ) { +if (wbc->nr_to_write <=0 || + wbc->encountered_congestion) {/* * slice used up: queue for next turn */
Quoted 6 lines
Quoted 4 lines
Thanx for the patch, but for the next time: How should I apply it? it seems to be context aware (@@) and broke on all kernel versions I tried 2.6.28/2.6.28.7/2.6.29
Do you mean that the patch applies after removing " @@.*$"?
I didn't try that, but this time it worked. So it was probably my error.
Quoted 11 lines
You are right. In your case, there are several big dirty files in sdb1, and the sdb write queue is constantly (almost-)congested. The SSD write speed is so slow, that in each round of sdb1 writeback, it begins with an uncongested queue, but quickly fills up after writing some pages. Hence all the inodes will get redirtied because of (
nr_to_write> 0).The following quick fix should solve the slow-writeback-on-congested-SSD problem. However the writeback sequence is suboptimal: it sync-and-requeue each file until congested (in your case about 3~600 pages) instead of until MAX_WRITEBACK_PAGES=1024 pages.
Yeah that fixed it, but performance dropped due to the more constant congestion. So I will need to try some different io schedulers.
Next to that I was wondering if there are any plans to make sure that not all dirty-files are written back in the same interval.
In my case all database files are written back each 30 seconds, while I would prefer them to be more divided over the interval.
Thanks,
Jos
Quoted 16 lines · Jos Houtman
Quoted 6 lines
Quoted 4 lines
Thanx for the patch, but for the next time: How should I apply it? it seems to be context aware (@@) and broke on all kernel versions I tried 2.6.28/2.6.28.7/2.6.29
Do you mean that the patch applies after removing " @@.*$"?
I didn't try that, but this time it worked. So it was probably my error.
Quoted 7 lines
You are right. In your case, there are several big dirty files in sdb1, and the sdb write queue is constantly (almost-)congested. The SSD write speed is so slow, that in each round of sdb1 writeback, it begins with an uncongested queue, but quickly fills up after writing some pages. Hence all the inodes will get redirtied because of (
nr_to_write> 0).
Quoted 7 lines
Quoted 4 lines
The following quick fix should solve the slow-writeback-on-congested-SSD problem. However the writeback sequence is suboptimal: it sync-and-requeue each file until congested (in your case about 3~600 pages) instead of until MAX_WRITEBACK_PAGES=1024 pages.
Yeah that fixed it, but performance dropped due to the more constant congestion. So I will need to try some different io schedulers.
Read performance or write performance?
Quoted 5 lines
Next to that I was wondering if there are any plans to make sure that not all dirty-files are written back in the same interval.
In my case all database files are written back each 30 seconds, while I would prefer them to be more divided over the interval.
pdflush will wake up every 5s to sync files dirtied more than 30s. So the writeback of inodes should be distributed(somehow randomly) into these 5s-interval-wakeups due to varied dirty times.
However the distribution may well be uneven in may cases. It seems to be conflicting goals for HDD and SSD: one favors somehow small busty writeback, another favors smooth writeback streams. I guess the better scheme would be bursty pdflush writebacks plus IO scheduler level QoS.
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
Next to that I was wondering if there are any plans to make sure that not all dirty-files are written back in the same interval.
In my case all database files are written back each 30 seconds, while I would prefer them to be more divided over the interval.
There another question I have: does the writeback go through the io scheduler? Because no matter the io scheduler or the tuning done, the writeback algorithm totally starves the reads.
See the url below for an example with CFQ, but deadline or noop all show this behaviour:
http://94.100.113.33/535450001-535500000/535451701-535451800/535451800_6_L7g
t.jpeg
Is there anything I can do about this behaviour by creating a better interleaving of the reads and writes?
Jos
Quoted 10 lines · Jos Houtman
Quoted 6 lines
Next to that I was wondering if there are any plans to make sure that not all dirty-files are written back in the same interval.
In my case all database files are written back each 30 seconds, while I would prefer them to be more divided over the interval.
There another question I have: does the writeback go through the io scheduler? Because no matter the io scheduler or the tuning done, the writeback algorithm totally starves the reads.
I noticed this annoying writes-starve-reads problem too. I'll look into it.
Quoted 7 lines
See the url below for an example with CFQ, but deadline or noop all show this behaviour:
http://94.100.113.33/535450001-535500000/535451701-535451800/535451800_6_L7g
t.jpegIs there anything I can do about this behaviour by creating a better interleaving of the reads and writes?
I guess it should be handled in the generic block io layer. Once we solved the writes-starve-reads problem, the bursty-writeback behavior becomes a no-problem for SSD.
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
Quoted 4 lines
There another question I have: does the writeback go through the io scheduler? Because no matter the io scheduler or the tuning done, the writeback algorithm totally starves the reads.
I noticed this annoying writes-starve-reads problem too. I'll look into it.
Thanks
Quoted 7 lines
Quoted 3 lines
Is there anything I can do about this behaviour by creating a better interleaving of the reads and writes?
I guess it should be handled in the generic block io layer. Once we solved the writes-starve-reads problem, the bursty-writeback behavior becomes a no-problem for SSD.
Yeah this was the part where I figured the io-schedulers kicked in, but obviously I was wrong :P.
If I can do anything more to help this along, let me know.
Thanks
Jos