writeback: add dirty_background_time per bdi variable
Subsystems (4)
PATCH 1/2 writeback: add dirty_background_time per bdi variable
From: Namjae Jeon <[email protected]>
This patch is based on suggestion by Wu Fengguang:
https://lkml.org/lkml/2011/8/19/19
kernel has mechanism to do writeback as per dirty_ratio and dirty_background ratio. It also maintains per task dirty rate limit to keep balance of dirty pages at any given instance by doing bdi bandwidth estimation.
Kernel also has max_ratio/min_ratio tunables to specify percentage of writecache to control per bdi dirty limits and task throttling.
However, there might be a usecase where user wants a writeback tuning parameter to flush dirty data at desired/tuned time interval especially at NFS server.
dirty_background_time provides an interface where user can tune background writeback start time using /sys/block/sda/bdi/dirty_background_time
dirty_background_time is used alongwith average bdi write bandwidth estimation to start background writeback.
One of the use case to demonstrate the patch functionality can be on NFS setup:-
We have a NFS setup with ethernet line of 100Mbps, while the USB disk is attached to server, which has a local speed of 25MBps. Server and client both are arm target boards.
Now if we perform a write operation over NFS (client to server), as per the network speed, data can travel at max speed of 100Mbps. But if we check the default write speed of USB hdd over NFS it comes around to 8MB/sec, far below the speed of network.
Reason being is as per the NFS logic, during write operation, initially pages are dirtied on NFS client side, then after reaching the dirty threshold/writeback limit (or in case of sync) data is actually sent to NFS server (so now again pages are dirtied on server side). This will be done in COMMIT call from client to server i.e if 100MB of data is dirtied and sent then it will take minimum 100MB/10Mbps ~ 8-9 seconds.
After the data is received, now it will take approx 100/25 ~4 Seconds to write the data to USB Hdd on server side. Hence making the overall time to write this much of data ~12 seconds, which in practically comes out to be near 7 to 8MB/second. After this a COMMIT response will be sent to NFS client.
However we may improve this write performace by making the use of NFS server idle time i.e while data is being received from the client, simultaneously initiate the writeback thread on server side. So instead of waiting for the complete data to come and then start the writeback, we can work in parallel while the network is still busy in receiving the data. Hence in this way overall performace will be improved.
If we tune dirty_background_time, we can see there
is increase in the performace and it comes out to be ~ 11MB/seconds. Results are:-
==========================================================
Case:1 - Normal setup without any changes ./performancetest_arm ./100MB write
RecSize WriteSpeed RanWriteSpeed
10485760 7.93MB/sec 8.11MB/sec
1048576 8.21MB/sec 7.80MB/sec
524288 8.71MB/sec 8.39MB/sec
262144 8.91MB/sec 7.83MB/sec
131072 8.91MB/sec 8.95MB/sec
65536 8.95MB/sec 8.90MB/sec
32768 8.76MB/sec 8.93MB/sec
16384 8.78MB/sec 8.67MB/sec
8192 8.90MB/sec 8.52MB/sec
4096 8.89MB/sec 8.28MB/secAverage speed is near 8MB/seconds.
Case:2 - Modified /sys/block/sda/bdi/dirty_background_time
on *** NFS Server only - not on NFS Client****
./performancetest_arm ./100MB write
RecSize WriteSpeed RanWriteSpeed
10485760 10.56MB/sec 10.37MB/sec
1048576 10.43MB/sec 10.33MB/sec
524288 10.32MB/sec 10.02MB/sec
262144 10.52MB/sec 10.19MB/sec
131072 10.34MB/sec 10.07MB/sec
65536 10.31MB/sec 10.06MB/sec
32768 10.27MB/sec 10.24MB/sec
16384 10.54MB/sec 10.03MB/sec
8192 10.41MB/sec 10.38MB/sec
4096 10.34MB/sec 10.12MB/secwe can see, average write speed is increased to ~10-11MB/sec.
============================================================
This patch provides the changes per block devices. So that we may modify the dirty_background_time as per the device and overall system is not impacted by the changes and we get improved perforamace in certain use cases.
NOTE: dirty_background_time is used alongwith average bdi write bandwidth estimation to start background writeback. But, bdi write bandwidth estimation is an _estimation_ and may become wildly wrong. So the dirty_background_time tuning may not always work to the user expectations. dirty_background_time will require careful tuning by users on NFS Server.
Changes since v1:
* make default value of 'dirty_background_time = 0' so that there is no change in default writeback behaviour.
* Add description of dirty_background_time in documentation
--- fs/fs-writeback.c | 21 +++++++++++++++++++-- include/linux/backing-dev.h | 1 + include/linux/writeback.h | 1 + mm/backing-dev.c | 21 +++++++++++++++++++++ mm/page-writeback.c | 3 ++- 5 files changed, 44 insertions(+), 3 deletions(-)
fs/fs-writeback.c +19 -2
@@ -769,6 +769,22 @@ static bool over_bground_thresh(struct backing_dev_info *bdi)return false; } +bool over_dirty_bground_time (struct backing_dev_info *bdi) +{ +unsigned long background_thresh; + +if (!bdi->dirty_background_time) +return false; + + background_thresh = bdi->avg_write_bandwidth * + bdi->dirty_background_time /1000 ; + +if (bdi_stat (bdi,BDI_RECLAIMABLE ) > background_thresh) +return true; + +return false; +} +/* * Called under wb->list_lock. If there are multiple wb per bdi, * only the flusher working on the first wb should do it.
@@ -828,7 +844,8 @@ static long wb_writeback(struct bdi_writeback *wb,* For background writeout, stop when we are below the * background dirty threshold */ -if (work->for_background && !over_bground_thresh (wb->bdi)) +if (work->for_background && !over_bground_thresh (wb->bdi) && + !over_dirty_bground_time (wb->bdi))break ;/*
@@ -920,7 +937,7 @@ static unsigned long get_nr_dirty_pages(void)static long wb_check_background_flush (struct bdi_writeback *wb) { -if (over_bground_thresh (wb->bdi)) { +if (over_bground_thresh (wb->bdi) ||over_dirty_bground_time (wb->bdi)) {struct wb_writeback_work work = { .nr_pages = LONG_MAX,
include/linux/backing-dev.h +1 -0
@@ -95,6 +95,7 @@ struct backing_dev_info {unsigned int min_ratio;unsigned int max_ratio, max_prop_frac; +unsigned int dirty_background_time;struct bdi_writeback wb;/* default writeback info for this bdi */ spinlock_t wb_lock;/* protects work_list */
include/linux/writeback.h +1 -0
@@ -96,6 +96,7 @@ long writeback_inodes_wb(struct bdi_writeback *wb, long nr_pages,long wb_do_writeback (struct bdi_writeback *wb,int force_wait);void wakeup_flusher_threads (long nr_pages,enum wb_reason reason);void inode_wait_for_writeback (struct inode *inode); +bool over_dirty_bground_time (struct backing_dev_info *bdi);/* writeback.h requires fs.h; it, too, is not included from here. */ static inline void wait_on_inode (struct inode *inode)
mm/backing-dev.c +21 -0
@@ -221,12 +221,32 @@ static ssize_t max_ratio_store(struct device *dev, } BDI_SHOW(max_ratio, bdi->max_ratio) +static ssize_t dirty_background_time_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct backing_dev_info *bdi =dev_get_drvdata (dev); +unsigned int msec; +ssize_t ret; + + ret =kstrtouint (buf,10 , &msec); +if (ret <0 ) +return ret; + + bdi->dirty_background_time = msec; +if (over_dirty_bground_time (bdi)) +bdi_start_background_writeback (bdi); + +return count; +} +BDI_SHOW(dirty_background_time, bdi->dirty_background_time) +#define __ATTR_RW (attr) __ATTR(attr, 0644, attr##_show, attr##_store)static struct device_attribute bdi_dev_attrs[] = {__ATTR_RW (read_ahead_kb),__ATTR_RW (min_ratio),__ATTR_RW (max_ratio), +__ATTR_RW (dirty_background_time), __ATTR_NULL, };
@@ -628,6 +648,7 @@ int bdi_init(struct backing_dev_info *bdi) bdi->min_ratio =0 ; bdi->max_ratio =100 ; bdi->max_prop_frac =FPROP_FRAC_BASE ; + bdi->dirty_background_time =0 ;spin_lock_init (&bdi->wb_lock);INIT_LIST_HEAD (&bdi->bdi_list);INIT_LIST_HEAD (&bdi->work_list);
mm/page-writeback.c +2 -1
@@ -1403,7 +1403,8 @@ pause:if (laptop_mode)return ; -if (nr_reclaimable > background_thresh) +if ((nr_reclaimable > background_thresh) || +over_dirty_bground_time (bdi))bdi_start_background_writeback (bdi); }
Signature · 1 line
1.7.9.5
Hi Namjae,
Quoted 9 lines · Namjae Jeon
From: Namjae Jeon <[email protected]>
This patch is based on suggestion by Wu Fengguang:
https://lkml.org/lkml/2011/8/19/19kernel has mechanism to do writeback as per
dirty_ratioanddirty_backgroundratio. It also maintains per task dirty rate limit to keep balance of dirty pages at any given instance by doing bdi bandwidth estimation.
Quoted 6 lines
Kernel also has max_ratio/min_ratio tunables to specify percentage of writecache to control per bdi dirty limits and task throttling.
However, there might be a usecase where user wants a writeback tuning parameter to flush dirty data at desired/tuned time interval especially at NFS server.
The dirty_background_time introduced by this patch is not an 'interval' thing, but a threshold to start background writeback.
Quoted 2 lines
dirty_background_timeprovides an interface where user can tune background writeback start time using/sys/block/sda/bdi/dirty_background_time
It's me that come up with the name "dirty_background_time". However it may confuse users what's the exact unit for "time". In this regard, it looks better to follow the established examples of "dirty_expire_centisecs"/"dirty_writeback_centisecs"?
Quoted 14 lines
dirty_background_timeis used alongwith average bdi write bandwidth estimation to start background writeback.One of the use case to demonstrate the patch functionality can be on NFS setup:-
We have a NFS setup with ethernet line of 100Mbps, while the USB disk is attached to server, which has a local speed of 25MBps. Server and client both are arm target boards.Now if we perform a write operation over NFS (client to server), as per the network speed, data can travel at max speed of 100Mbps. But if we check the default write speed of USB hdd over NFS it comes around to 8MB/sec, far below the speed of network.
Quoted 6 lines
Reason being is as per the NFS logic, during write operation, initially pages are dirtied on NFS client side, then after reaching the dirty threshold/writeback limit (or in case of sync) data is actually sent to NFS server (so now again pages are dirtied on server side). This will be done in COMMIT call from client to server i.e if 100MB of data is dirtied and sent then it will take minimum 100MB/10Mbps ~ 8-9 seconds.
To be exact, there are two steps:
- WRITE to send data from client to server
- COMMIT to sync data from server memory to disk
The nfs clients have logics to optimize away too frequent COMMITs. However in the case of continuous writing, the data are effectively COMMITed pretty soon after being WRITE to server memory. So "threshold" can keep the flusher more reliably busy than "interval".
It's better for the flusher to do the background writes because they are ASYNC IO, which generally performs better than SYNC COMMIT IO.
As for latency, there will be IO competitions between the flusher and nfsd which may impact some COMMITs' latency and improve some other COMMITs (because they'll do less SYNC IO).
Quoted 6 lines
After the data is received, now it will take approx 100/25 ~4 Seconds to write the data to USB Hdd on server side. Hence making the overall time to write this much of data ~12 seconds, which in practically comes out to be near 7 to 8MB/second. After this a COMMIT response will be sent to NFS client.
Quoted 6 lines
However we may improve this write performace by making the use of NFS server idle time i.e while data is being received from the client, simultaneously initiate the writeback thread on server side. So instead of waiting for the complete data to come and then start the writeback, we can work in parallel while the network is still busy in receiving the data. Hence in this way overall performace will be improved.
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Quoted 20 lines
If we tune
dirty_background_time, we can see there
is increase in the performace and it comes out to be ~ 11MB/seconds. Results are:-
==========================================================
Case:1 - Normal setup without any changes ./performancetest_arm ./100MB writeRecSize WriteSpeed RanWriteSpeed
10485760 7.93MB/sec 8.11MB/sec 1048576 8.21MB/sec 7.80MB/sec 524288 8.71MB/sec 8.39MB/sec 262144 8.91MB/sec 7.83MB/sec 131072 8.91MB/sec 8.95MB/sec 65536 8.95MB/sec 8.90MB/sec 32768 8.76MB/sec 8.93MB/sec 16384 8.78MB/sec 8.67MB/sec 8192 8.90MB/sec 8.52MB/sec 4096 8.89MB/sec 8.28MB/sec
Quoted 3 lines
Average speed is near 8MB/seconds.
Case:2 - Modified
/sys/block/sda/bdi/dirty_background_time
What's the exact threshold used? 1s?
Quoted 125 lines
on *** NFS Server only - not on NFS Client****
./performancetest_arm ./100MB writeRecSize WriteSpeed RanWriteSpeed
10485760 10.56MB/sec 10.37MB/sec 1048576 10.43MB/sec 10.33MB/sec 524288 10.32MB/sec 10.02MB/sec 262144 10.52MB/sec 10.19MB/sec 131072 10.34MB/sec 10.07MB/sec 65536 10.31MB/sec 10.06MB/sec 32768 10.27MB/sec 10.24MB/sec 16384 10.54MB/sec 10.03MB/sec 8192 10.41MB/sec 10.38MB/sec 4096 10.34MB/sec 10.12MB/secwe can see, average write speed is increased to ~10-11MB/sec.
============================================================This patch provides the changes per block devices. So that we may modify the
dirty_background_timeas per the device and overall system is not impacted by the changes and we get improved perforamace in certain use cases.NOTE:
dirty_background_timeis used alongwith average bdi write bandwidth estimation to start background writeback. But, bdi write bandwidth estimation is an _estimation_ and may become wildly wrong. So thedirty_background_timetuning may not always work to the user expectations.dirty_background_timewill require careful tuning by users on NFS Server.Changes since v1:
* make default value of 'dirty_background_time= 0' so that there is no change in default writeback behaviour.
* Add description ofdirty_background_timein documentationOriginal-patch-by Wu Fengguang <[email protected]>Signed-off-by Namjae Jeon <[email protected]>Signed-off-by Vivek Trivedi <[email protected]>--- fs/fs-writeback.c | 21 +++++++++++++++++++-- include/linux/backing-dev.h | 1 + include/linux/writeback.h | 1 + mm/backing-dev.c | 21 +++++++++++++++++++++ mm/page-writeback.c | 3 ++- 5 files changed, 44 insertions(+), 3 deletions(-)fs/fs-writeback.c +19
-2@@ -769,6 +769,22 @@ static bool over_bground_thresh(struct backing_dev_info *bdi)return false; } +bool over_dirty_bground_time (struct backing_dev_info *bdi) +{ +unsigned long background_thresh; + +if (!bdi->dirty_background_time) +return false; + + background_thresh = bdi->avg_write_bandwidth * + bdi->dirty_background_time /1000 ; + +if (bdi_stat (bdi,BDI_RECLAIMABLE ) > background_thresh) +return true; + +return false; +} +/* * Called under wb->list_lock. If there are multiple wb per bdi, * only the flusher working on the first wb should do it. @@ -828,7 +844,8 @@ static long wb_writeback(struct bdi_writeback *wb,* For background writeout, stop when we are below the * background dirty threshold */ -if (work->for_background && !over_bground_thresh (wb->bdi)) +if (work->for_background && !over_bground_thresh (wb->bdi) && + !over_dirty_bground_time (wb->bdi))break ;/* @@ -920,7 +937,7 @@ static unsigned long get_nr_dirty_pages(void)static long wb_check_background_flush (struct bdi_writeback *wb) { -if (over_bground_thresh (wb->bdi)) { +if (over_bground_thresh (wb->bdi) ||over_dirty_bground_time (wb->bdi)) {struct wb_writeback_work work = { .nr_pages = LONG_MAX,include/linux/backing-dev.h +1
-0@@ -95,6 +95,7 @@ struct backing_dev_info {unsigned int min_ratio;unsigned int max_ratio, max_prop_frac; +unsigned int dirty_background_time;struct bdi_writeback wb;/* default writeback info for this bdi */ spinlock_t wb_lock;/* protects work_list */ include/linux/writeback.h +1
-0@@ -96,6 +96,7 @@ long writeback_inodes_wb(struct bdi_writeback *wb, long nr_pages,long wb_do_writeback (struct bdi_writeback *wb,int force_wait);void wakeup_flusher_threads (long nr_pages,enum wb_reason reason);void inode_wait_for_writeback (struct inode *inode); +bool over_dirty_bground_time (struct backing_dev_info *bdi);/* writeback.h requires fs.h; it, too, is not included from here. */ static inline void wait_on_inode (struct inode *inode)mm/backing-dev.c +2
-0@@ -221,12 +221,32 @@ static ssize_t max_ratio_store(struct device *dev, } BDI_SHOW(max_ratio, bdi->max_ratio) +static ssize_t dirty_background_time_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count)
mm/backing-dev.c static ssize_t max_ratio_store(struct device *dev, · line 235
Quoted 10 lines
mm/backing-dev.c +10
-0+{ +struct backing_dev_info *bdi =dev_get_drvdata (dev); +unsigned int msec; +ssize_t ret; + + ret =kstrtouint (buf,10 , &msec); +if (ret <0 ) +return ret; + + bdi->dirty_background_time = msec;
As commented above, please change to 1/100 seconds unit and change variables/logics accordingly.
Quoted 30 lines
mm/backing-dev.c +9
-0+if (over_dirty_bground_time (bdi)) +bdi_start_background_writeback (bdi); + +return count; +} +BDI_SHOW(dirty_background_time, bdi->dirty_background_time) +#define __ATTR_RW (attr) __ATTR(attr, 0644, attr##_show, attr##_store)static struct device_attribute bdi_dev_attrs[] = {__ATTR_RW (read_ahead_kb),__ATTR_RW (min_ratio),__ATTR_RW (max_ratio), +__ATTR_RW (dirty_background_time), __ATTR_NULL, };@@ -628,6 +648,7 @@ int bdi_init(struct backing_dev_info *bdi) bdi->min_ratio =0 ; bdi->max_ratio =100 ; bdi->max_prop_frac =FPROP_FRAC_BASE ; + bdi->dirty_background_time =0 ;spin_lock_init (&bdi->wb_lock);INIT_LIST_HEAD (&bdi->bdi_list);INIT_LIST_HEAD (&bdi->work_list);mm/page-writeback.c +0
-0
mm/page-writeback.c pause: · line 1406
Quoted 5 lines
mm/page-writeback.c +1
-1@@ -1403,7 +1403,8 @@ pause:if (laptop_mode)return ; -if (nr_reclaimable > background_thresh) + if ((nr_reclaimable > background_thresh) ||
No need to add () here.
Quoted 6 lines
mm/page-writeback.c +1
-0+ over_dirty_bground_time(bdi))bdi_start_background_writeback (bdi); }Signature · 1 line
1.7.9.5
2012/9/11, Fengguang Wu <[email protected]>:
Quoted 1 line
Hi Namjae,
Hi. Wu.
Quoted 16 lines · Namjae Jeon
Quoted 14 lines
From: Namjae Jeon <[email protected]>
This patch is based on suggestion by Wu Fengguang:
https://lkml.org/lkml/2011/8/19/19kernel has mechanism to do writeback as per
dirty_ratioanddirty_background
ratio. It also maintains per task dirty rate limit to keep balance of dirty pages at any given instance by doing bdi bandwidth estimation.Kernel also has max_ratio/min_ratio tunables to specify percentage of writecache
to control per bdi dirty limits and task throttling.
Quoted 6 lines
Quoted 3 lines
However, there might be a usecase where user wants a writeback tuning parameter to flush dirty data at desired/tuned time interval especially at NFS server.
The
dirty_background_timeintroduced by this patch is not an 'interval' thing, but a threshold to start background writeback.
Agreed. I will update the description.
Quoted 9 lines
Quoted 4 lines
dirty_background_timeprovides an interface where user can tune background
writeback start time using/sys/block/sda/bdi/dirty_background_timeIt's me that come up with the name "
dirty_background_time". However it may confuse users what's the exact unit for "time". In this regard, it looks better to follow the established examples of "dirty_expire_centisecs"/"dirty_writeback_centisecs"?
Ok. I will change dirty_background_time -> dirty_background_centisecs and change variable logic to work in centisecs.
Quoted 33 lines
Quoted 16 lines
dirty_background_timeis used alongwith average bdi write bandwidth estimation
to start background writeback.One of the use case to demonstrate the patch functionality can be on NFS setup:-
We have a NFS setup with ethernet line of 100Mbps, while the USB disk is attached to server, which has a local speed of 25MBps. Server and client both are arm target boards.Now if we perform a write operation over NFS (client to server), as per the network speed, data can travel at max speed of 100Mbps. But if we check the default write speed of USB hdd over NFS it comes around to 8MB/sec, far below the speed of network.
Quoted 6 lines
Reason being is as per the NFS logic, during write operation, initially pages are dirtied on NFS client side, then after reaching the dirty threshold/writeback limit (or in case of sync) data is actually sent to NFS server (so now again pages are dirtied on server side). This will be done in COMMIT call from client to server i.e if 100MB of data is dirtied and sent then it will take minimum 100MB/10Mbps ~ 8-9 seconds.
To be exact, there are two steps:
- WRITE to send data from client to server
- COMMIT to sync data from server memory to diskThe nfs clients have logics to optimize away too frequent COMMITs. However in the case of continuous writing, the data are effectively COMMITed pretty soon after being WRITE to server memory. So "threshold" can keep the flusher more reliably busy than "interval".
Quoted 6 lines
It's better for the flusher to do the background writes because they are ASYNC IO, which generally performs better than SYNC COMMIT IO.
As for latency, there will be IO competitions between the flusher and nfsd which may impact some COMMITs' latency and improve some other COMMITs (because they'll do less SYNC IO).
Right! I will update this description also.
Quoted 14 lines
Quoted 7 lines
After the data is received, now it will take approx 100/25 ~4 Seconds to write the data to USB Hdd on server side. Hence making the overall time to write this much of data ~12 seconds, which in practically comes out to be near 7 to 8MB/second. After this a COMMIT response will be sent to NFS client.
Quoted 6 lines
However we may improve this write performace by making the use of NFS server idle time i.e while data is being received from the client, simultaneously initiate the writeback thread on server side. So instead of waiting for the complete data to come and then start the writeback, we can work in parallel while the network is still busy in receiving the data. Hence in this way overall performace will be improved.
Quoted 4 lines
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Actually, we use USB HDD and USB Flash devices at NFS server. There can be other similar users as well. So it might be useful to provide this tuning feature other.
As default value is zero, it is disabled by default and it should not impact normal writeback.
I will share large file writes test result on NFS client on USB HDD with/without tuning with patch.
Quoted 21 lines
Quoted 21 lines
If we tune
dirty_background_time, we can see there
is increase in the performace and it comes out to be ~ 11MB/seconds. Results are:-
==========================================================
Case:1 - Normal setup without any changes ./performancetest_arm ./100MB writeRecSize WriteSpeed RanWriteSpeed
10485760 7.93MB/sec 8.11MB/sec 1048576 8.21MB/sec 7.80MB/sec 524288 8.71MB/sec 8.39MB/sec 262144 8.91MB/sec 7.83MB/sec 131072 8.91MB/sec 8.95MB/sec 65536 8.95MB/sec 8.90MB/sec 32768 8.76MB/sec 8.93MB/sec 16384 8.78MB/sec 8.67MB/sec 8192 8.90MB/sec 8.52MB/sec 4096 8.89MB/sec 8.28MB/sec
Quoted 5 lines
Quoted 3 lines
Average speed is near 8MB/seconds.
Case:2 - Modified
/sys/block/sda/bdi/dirty_background_timeWhat's the exact threshold used? 1s?
Yes. We recommend to use it as 1 sec i.e. start writeback once dirty threashold reach bdi bandwidth at NFS Server.
As you know initial bandwidth is incorrect(default value 100 MB/sec), we have taken this result after some I/O work load on NFS Server.
Quoted 132 lines
Quoted 132 lines
on *** NFS Server only - not on NFS Client****
./performancetest_arm ./100MB writeRecSize WriteSpeed RanWriteSpeed
10485760 10.56MB/sec 10.37MB/sec 1048576 10.43MB/sec 10.33MB/sec 524288 10.32MB/sec 10.02MB/sec 262144 10.52MB/sec 10.19MB/sec 131072 10.34MB/sec 10.07MB/sec 65536 10.31MB/sec 10.06MB/sec 32768 10.27MB/sec 10.24MB/sec 16384 10.54MB/sec 10.03MB/sec 8192 10.41MB/sec 10.38MB/sec 4096 10.34MB/sec 10.12MB/secwe can see, average write speed is increased to ~10-11MB/sec.
============================================================This patch provides the changes per block devices. So that we may modify the
dirty_background_timeas per the device and overall system is not impacted
by the changes and we get improved perforamace in certain use cases.NOTE:
dirty_background_timeis used alongwith average bdi write bandwidth estimation to start background writeback. But, bdi write bandwidth estimation
is an _estimation_ and may become wildly wrong. So thedirty_background_time
tuning may not always work to the user expectations.dirty_background_time
will require careful tuning by users on NFS Server.Changes since v1:
* make default value of 'dirty_background_time= 0' so that there is no change
in default writeback behaviour.
* Add description ofdirty_background_timein documentationOriginal-patch-by Wu Fengguang <[email protected]>Signed-off-by Namjae Jeon <[email protected]>Signed-off-by Vivek Trivedi <[email protected]>--- fs/fs-writeback.c | 21 +++++++++++++++++++-- include/linux/backing-dev.h | 1 + include/linux/writeback.h | 1 + mm/backing-dev.c | 21 +++++++++++++++++++++ mm/page-writeback.c | 3 ++- 5 files changed, 44 insertions(+), 3 deletions(-)fs/fs-writeback.c +16
-0@@ -769,6 +769,22 @@ static bool over_bground_thresh(struct backing_dev_info *bdi)return false; } +bool over_dirty_bground_time (struct backing_dev_info *bdi) +{ +unsigned long background_thresh; + +if (!bdi->dirty_background_time) +return false; + + background_thresh = bdi->avg_write_bandwidth * + bdi->dirty_background_time /1000 ; + +if (bdi_stat (bdi,BDI_RECLAIMABLE ) > background_thresh) +return true; + +return false; +} +/* * Called under wb->list_lock. If there are multiple wb per bdi, * only the flusher working on the first wb should do it.
fs/fs-writeback.c +3
-2@@ -828,7 +844,8 @@ static long wb_writeback(struct bdi_writeback *wb,* For background writeout, stop when we are below the * background dirty threshold */ -if (work->for_background && !over_bground_thresh (wb->bdi)) +if (work->for_background && !over_bground_thresh (wb->bdi) && + !over_dirty_bground_time (wb->bdi))break ;/* @@ -920,7 +937,7 @@ static unsigned long get_nr_dirty_pages(void)static long wb_check_background_flush (struct bdi_writeback *wb) { -if (over_bground_thresh (wb->bdi)) { +if (over_bground_thresh (wb->bdi) ||over_dirty_bground_time (wb->bdi)) {struct wb_writeback_work work = { .nr_pages = LONG_MAX,include/linux/backing-dev.h +1
-0@@ -95,6 +95,7 @@ struct backing_dev_info {unsigned int min_ratio;unsigned int max_ratio, max_prop_frac; +unsigned int dirty_background_time;struct bdi_writeback wb;/* default writeback info for this bdi */ spinlock_t wb_lock;/* protects work_list */ include/linux/writeback.h +1
-0@@ -96,6 +96,7 @@ long writeback_inodes_wb(struct bdi_writeback *wb, long nr_pages, long wb_do_writeback(struct bdi_writeback *wb, int force_wait);void wakeup_flusher_threads (long nr_pages,enum wb_reason reason);void inode_wait_for_writeback (struct inode *inode); +bool over_dirty_bground_time (struct backing_dev_info *bdi);/* writeback.h requires fs.h; it, too, is not included from here. */ static inline void
wait_on_inode(struct inode *inode)mm/backing-dev.c +0
-0@@ -221,12 +221,32 @@ static ssize_t max_ratio_store(struct device *dev, }BDI_SHOW (max_ratio, bdi->max_ratio)
Quoted 15 lines
mm/backing-dev.c static ssize_t max_ratio_store(struct device *dev, · line 235
Quoted 12 lines
mm/backing-dev.c +12
-0+static ssize_t dirty_background_time_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct backing_dev_info *bdi =dev_get_drvdata (dev); +unsigned int msec; +ssize_t ret; + + ret =kstrtouint (buf,10 , &msec); +if (ret <0 ) +return ret; + + bdi->dirty_background_time = msec;As commented above, please change to 1/100 seconds unit and change variables/logics accordingly.
Ok. We will make changes as per suggestion.
Quoted 34 lines
Quoted 34 lines
mm/backing-dev.c +9
-0+if (over_dirty_bground_time (bdi)) +bdi_start_background_writeback (bdi); + +return count; +} +BDI_SHOW(dirty_background_time, bdi->dirty_background_time) +#define __ATTR_RW (attr) __ATTR(attr, 0644, attr##_show, attr##_store)static struct device_attribute bdi_dev_attrs[] = {__ATTR_RW (read_ahead_kb),__ATTR_RW (min_ratio),__ATTR_RW (max_ratio), +__ATTR_RW (dirty_background_time), __ATTR_NULL, };@@ -628,6 +648,7 @@ int bdi_init(struct backing_dev_info *bdi) bdi->min_ratio =0 ; bdi->max_ratio =100 ; bdi->max_prop_frac =FPROP_FRAC_BASE ; + bdi->dirty_background_time =0 ;spin_lock_init (&bdi->wb_lock);INIT_LIST_HEAD (&bdi->bdi_list);INIT_LIST_HEAD (&bdi->work_list);mm/page-writeback.c +0
-0@@ -1403,7 +1403,8 @@ pause:if (laptop_mode)return ;
Quoted 4 lines
mm/page-writeback.c pause: · line 1406
Quoted 2 lines
mm/page-writeback.c +1
-1-if (nr_reclaimable > background_thresh) + if ((nr_reclaimable > background_thresh) ||No need to add () here.
I will remove it.
Thanks.
Quoted 8 lines
Quoted 7 lines
mm/page-writeback.c +1
-1+ over_dirty_bground_time(bdi))bdi_start_background_writeback (bdi); } --1.7.9.5
Quoted 6 lines
Quoted 5 lines
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Quoted 8 lines
Actually, we use USB HDD and USB Flash devices at NFS server. There can be other similar users as well. So it might be useful to provide this tuning feature other.
As default value is zero, it is disabled by default and it should not impact normal writeback.I will share large file writes test result on NFS client on USB HDD with/without tuning with patch.
Hi. Wu.
I share 1GB continous write test result.
-> create a 1000 MB file
For continuous write - create 1 GB file
RecSize WriteSpeed
10485760 10.47MB/sec
1048576 10.35MB/sec
524288 10.48MB/sec
262144 10.48MB/sec
131072 10.52MB/sec
65536 10.56MB/sec
32768 10.64MB/sec
16384 10.31MB/sec
8192 10.52MB/sec
4096 10.45MB/secI will update changelog in patch.
Thanks Wu!.
Quoted 32 lines · Namjae Jeon
Quoted 6 lines
Quoted 5 lines
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Quoted 8 lines
Actually, we use USB HDD and USB Flash devices at NFS server. There can be other similar users as well. So it might be useful to provide this tuning feature other.
As default value is zero, it is disabled by default and it should not impact normal writeback.I will share large file writes test result on NFS client on USB HDD with/without tuning with patch.
Hi. Wu.
I share 1GB continous write test result.-> create a 1000 MB file
For continuous write - create 1 GB fileRecSize WriteSpeed 10485760 10.47MB/sec 1048576 10.35MB/sec 524288 10.48MB/sec 262144 10.48MB/sec 131072 10.52MB/sec 65536 10.56MB/sec 32768 10.64MB/sec 16384 10.31MB/sec 8192 10.52MB/sec 4096 10.45MB/sec
Quoted 1 line
I will update changelog in patch.
Thanks! What's the server side setting and can you give a comparison of different background writeback thresholds? This is this patch's target use cases, after all.
Thanks,
Fengguang
2012/9/12, Fengguang Wu <[email protected]>:
Quoted 33 lines · Namjae Jeon
Quoted 32 lines
Quoted 6 lines
Quoted 5 lines
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Quoted 8 lines
Actually, we use USB HDD and USB Flash devices at NFS server. There can be other similar users as well. So it might be useful to provide this tuning feature other.
As default value is zero, it is disabled by default and it should not impact normal writeback.I will share large file writes test result on NFS client on USB HDD with/without tuning with patch.
Hi. Wu.
I share 1GB continous write test result.-> create a 1000 MB file
For continuous write - create 1 GB fileRecSize WriteSpeed 10485760 10.47MB/sec 1048576 10.35MB/sec 524288 10.48MB/sec 262144 10.48MB/sec 131072 10.52MB/sec 65536 10.56MB/sec 32768 10.64MB/sec 16384 10.31MB/sec 8192 10.52MB/sec 4096 10.45MB/sec
Quoted 3 lines
Quoted 1 line
I will update changelog in patch.
Thanks! What's the server side setting
I missed server side setting.. I set dirty_background_centisecs to 1 sec.
Quoted 1 line
and can you give a comparison of different background writeback thresholds?
Okay, I will measure peformance per setting value of dirty_background_centisecs in nfs server.
This is this patch's
Quoted 1 line
target use cases, after all.
Sure.
Thanks.
Quoted 4 lines
Thanks,
Fengguang
2012/9/12, Namjae Jeon <[email protected]>:
Quoted 40 lines
2012/9/12, Fengguang Wu <[email protected]>:
Quoted 33 lines · Namjae Jeon
Quoted 32 lines
Quoted 6 lines
Quoted 5 lines
To be frank, no realistic NFS servers will use USB disk as backing storage. So that rational for reducing "initial" delays is weak. Continuous write performance to HDD is much more important. Do you have numbers for that?
Quoted 8 lines
Actually, we use USB HDD and USB Flash devices at NFS server. There can be other similar users as well. So it might be useful to provide this tuning feature other.
As default value is zero, it is disabled by default and it should not impact normal writeback.I will share large file writes test result on NFS client on USB HDD with/without tuning with patch.
Hi. Wu.
I share 1GB continous write test result.-> create a 1000 MB file
For continuous write - create 1 GB fileRecSize WriteSpeed 10485760 10.47MB/sec 1048576 10.35MB/sec 524288 10.48MB/sec 262144 10.48MB/sec 131072 10.52MB/sec 65536 10.56MB/sec 32768 10.64MB/sec 16384 10.31MB/sec 8192 10.52MB/sec 4096 10.45MB/secQuoted 3 lines
Quoted 1 line
I will update changelog in patch.
Thanks! What's the server side setting
I missed server side setting.. I set
dirty_background_centisecsto 1 sec.Quoted 2 lines
and can you give a comparison of different background writeback thresholds?
Quoted 7 lines
Okay, I will measure peformance per setting value of
dirty_background_centisecsin nfs server.This is this patch's
Quoted 1 line
target use cases, after all.
Sure.
Thanks.
Hi. Wu.
I shared write performance result as your suggestion.
-----------------------------------------------------------dirty_background_centisecs = 0, default value
-----------------------------------------------------------
[NFS Server]
$ cat /sys/block/sda/bdi/dirty_background_centisecs
0
[NFS Client]
$ ./performancetest_arm 1GB_file
WRT_SZ = 1000 MB
RecSize WriteSpeed
10485760 8.44MB/sec
1048576 8.48MB/sec
524288 8.37MB/sec
262144 8.16MB/sec
131072 8.48MB/sec
65536 8.38MB/sec
32768 8.65MB/sec
16384 8.27MB/sec
8192 8.52MB/sec
4096 8.20MB/sec-----------------------------------------------------------dirty_background_centisecs = 300, i.e. 3 seconds
-----------------------------------------------------------
[NFS Server]
$ echo 300 > /sys/block/sda/bdi/dirty_background_centisecs
$ cat /sys/block/sda/bdi/dirty_background_centisecs
300
[NFS Client]
$ ./performancetest_arm 1GB_file
RecSize WriteSpeed
10485760 8.60MB/sec
1048576 8.87MB/sec
524288 8.42MB/sec
262144 8.51MB/sec
131072 8.81MB/sec
65536 9.09MB/sec
32768 9.00MB/sec
16384 8.80MB/sec
8192 8.70MB/sec
4096 8.63MB/sec-----------------------------------------------------------dirty_background_centisecs = 200, i.e. 2 seconds
-----------------------------------------------------------
[NFS Server]
$ echo 200 > /sys/block/sda/bdi/dirty_background_centisecs
$ cat /sys/block/sda/bdi/dirty_background_centisecs
200
[NFS Client]
$ ./performancetest_arm 1GB_file
RecSize WriteSpeed
10485760 9.30MB/sec
1048576 9.31MB/sec
524288 9.84MB/sec
262144 9.52MB/sec
131072 9.42MB/sec
65536 9.76MB/sec
32768 9.57MB/sec
16384 9.39MB/sec
8192 9.40MB/sec
4096 9.80MB/sec-----------------------------------------------------------dirty_background_centisecs = 100, i.e. 1 second
-----------------------------------------------------------
[NFS Server]
$ echo 100 > /sys/block/sda/bdi/dirty_background_centisecs
$ cat /sys/block/sda/bdi/dirty_background_centisecs
100
[NFS Client]
$ ./performancetest_arm 1GB_file
RecSize WriteSpeed
10485760 10.27MB/sec
1048576 10.34MB/sec
524288 10.47MB/sec
262144 10.62MB/sec
131072 10.55MB/sec
65536 10.53MB/sec
32768 10.54MB/sec
16384 10.43MB/sec
8192 10.50MB/sec
4096 10.35MB/secI will update simple table for this result in changelog in patch.
Thanks.
Quoted 5 lines
Quoted 4 lines
Thanks,
Fengguang
Quoted 1 line
I will update simple table for this result in changelog in patch.
That would be good, thanks a lot for the testing!
Thanks,
Fengguang
Quoted 33 lines · Namjae Jeon
From: Namjae Jeon <[email protected]>
This patch is based on suggestion by Wu Fengguang:
https://lkml.org/lkml/2011/8/19/19kernel has mechanism to do writeback as per
dirty_ratioanddirty_backgroundratio. It also maintains per task dirty rate limit to keep balance of dirty pages at any given instance by doing bdi bandwidth estimation.Kernel also has max_ratio/min_ratio tunables to specify percentage of writecache to control per bdi dirty limits and task throttling.
However, there might be a usecase where user wants a writeback tuning parameter to flush dirty data at desired/tuned time interval especially at NFS server.
dirty_background_timeprovides an interface where user can tune background writeback start time using/sys/block/sda/bdi/dirty_background_time
dirty_background_timeis used alongwith average bdi write bandwidth estimation to start background writeback.One of the use case to demonstrate the patch functionality can be on NFS setup:-
We have a NFS setup with ethernet line of 100Mbps, while the USB disk is attached to server, which has a local speed of 25MBps. Server and client both are arm target boards.Now if we perform a write operation over NFS (client to server), as per the network speed, data can travel at max speed of 100Mbps. But if we check the default write speed of USB hdd over NFS it comes around to 8MB/sec, far below the speed of network.
Quoted 7 lines
Reason being is as per the NFS logic, during write operation, initially pages are dirtied on NFS client side, then after reaching the dirty threshold/writeback limit (or in case of sync) data is actually sent to NFS server (so now again pages are dirtied on server side). This will be done in COMMIT call from client to server i.e if 100MB of data is dirtied and sent then it will take minimum 100MB/10Mbps ~ 8-9 seconds.
can't the bdi on nfs client side indicate that the commit need to be send in smaller chunks ?. ie, we need the client to figure out that the NFS srever is backed by a slow usb disk and accordingly send the commit request ?
-aneesh