mm: let the bdi_writeout fraction respond more quickly
Subsystems (2)
RFC PATCH 1/1 mm: let the bdi_writeout fraction respond more quickly
Hi all,
The fraction of vm cache allowed to each BDI as calculated by get_dirty_limits (mm/page-writeback.c) respond very slowly to changes in workload.
Running a simple test that alternately writes 1Gb to sda then sdb, twice, shows the bdi_threshold taking approximately 15 seconds to reach a steady state value. This prevents a application from using all of the available cache and forces it to write to the physical disk earlier than strictly necessary.
As you can see from the attached graph, bdi_thresh_before.png, our current control system responds to this kind of workload very slowly.
The below patch speeds up the recalculation and lets it reach a steady state value in a couple of seconds. see bdi_thresh_after.png.
I get better throughput with this patch applied and have been running some variation of this on and off for some months without any obvious problems.
(These tests were all run on 2.6.35-rc3,
where dm-2 is a sata drive lvm/ext4 and sdb is ide ext4. I've got lots more results and graphs but won't bore you all with them ;) )
I see this as a considerable improvement but I have found the magic number of -4 empirically so it may just be tuned to my system. I'm not sure how to decide on a value that is suitable for everyone.
Does anyone have any suggestions or thoughts?
Unfortunately I don't have any other hardware to try this on, so I would be very interest to hear if anyone tries this on their favourite workload.
regards
Richard
patch against 2.6.35-rc3
mm/page-writeback.c +1 -1
@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }/*
Quoted 46 lines · Richard Kennedy
Hi all,
The fraction of vm cache allowed to each BDI as calculated byget_dirty_limits(mm/page-writeback.c) respond very slowly to changes in workload.Running a simple test that alternately writes 1Gb to sda then sdb, twice, shows the
bdi_thresholdtaking approximately 15 seconds to reach a steady state value. This prevents a application from using all of the available cache and forces it to write to the physical disk earlier than strictly necessary.
As you can see from the attached graph, bdi_thresh_before.png, our current control system responds to this kind of workload very slowly.The below patch speeds up the recalculation and lets it reach a steady state value in a couple of seconds. see bdi_thresh_after.png.
I get better throughput with this patch applied and have been running some variation of this on and off for some months without any obvious problems.
(These tests were all run on 2.6.35-rc3,
where dm-2 is a sata drive lvm/ext4 and sdb is ide ext4. I've got lots more results and graphs but won't bore you all with them ;) )I see this as a considerable improvement but I have found the magic number of -4 empirically so it may just be tuned to my system. I'm not sure how to decide on a value that is suitable for everyone.
Does anyone have any suggestions or thoughts?
Unfortunately I don't have any other hardware to try this on, so I would be very interest to hear if anyone tries this on their favourite workload.
regards
Richardpatch against 2.6.35-rc3
mm/page-writeback.c +0
-0
mm/page-writeback.c static int calc_period_shift(void) · line 150
Quoted 8 lines
mm/page-writeback.c +1
-1@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }/*
Fixed Jens email address. I can send you the graphs privately if you haven't already got them.
regards
Richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Richard Kennedy
mm/page-writeback.c static int calc_period_shift(void) · line 148
Quoted 11 lines
mm/page-writeback.c +1
-1@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }
IIRC I suggested similar things in the past and all we needed to do was find people doing the measurements on different bits of hardware or so..
I don't have any problems with the approach, all we need to make sure is that we never return 0 or a negative number (possibly ensure a minimum positive shift value).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Peter Zijlstra
Quoted 11 lines · Richard Kennedy
mm/page-writeback.c static int calc_period_shift(void) · line 148
Quoted 11 lines
mm/page-writeback.c +1
-1@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }IIRC I suggested similar things in the past and all we needed to do was find people doing the measurements on different bits of hardware or so..
Quoted 3 lines
I don't have any problems with the approach, all we need to make sure is that we never return 0 or a negative number (possibly ensure a minimum positive shift value).
Yep that sounds reasonable. would minimum shift of 4 be ok ?
something like
max ( (ilog2(dirty_total - 1)- 4) , 4);
Unfortunately volunteers don't seem to be leaping out of the woodwork, maybe Andrew could be persuaded to try this in his tree for a while and see if any one squeaks ?
regards
Richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · Richard Kennedy
Quoted 16 lines · Peter Zijlstra
Quoted 11 lines · Richard Kennedy
mm/page-writeback.c static int calc_period_shift(void) · line 148
Quoted 11 lines
mm/page-writeback.c +1
-1@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }IIRC I suggested similar things in the past and all we needed to do was find people doing the measurements on different bits of hardware or so..
Quoted 3 lines
I don't have any problems with the approach, all we need to make sure is that we never return 0 or a negative number (possibly ensure a minimum positive shift value).
Yep that sounds reasonable. would minimum shift of 4 be ok ?
Quoted 7 lines
something like
max ( (ilog2(dirty_total - 1)- 4) , 4);Unfortunately volunteers don't seem to be leaping out of the woodwork, maybe Andrew could be persuaded to try this in his tree for a while and see if any one squeaks ?
I'm pretty sure that most volunteers are curious what to actually test, so they shy away from it. If you added a good explanation of an easy way to test the before and after, then it would be more approachable.
I'll give it a spin here.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 32 lines · Jens Axboe
Quoted 23 lines · Richard Kennedy
Quoted 16 lines · Peter Zijlstra
Quoted 11 lines · Richard Kennedy
mm/page-writeback.c static int calc_period_shift(void) · line 148
Quoted 11 lines
mm/page-writeback.c +1
-1@@ -144,7 +144,7 @@ static int calc_period_shift(void)else dirty_total = (vm_dirty_ratio *determine_dirtyable_memory ()) /100 ; -return 2 +ilog2 (dirty_total -1 ); +return ilog2 (dirty_total -1 ) -4 ; }IIRC I suggested similar things in the past and all we needed to do was find people doing the measurements on different bits of hardware or so..
Quoted 3 lines
I don't have any problems with the approach, all we need to make sure is that we never return 0 or a negative number (possibly ensure a minimum positive shift value).
Yep that sounds reasonable. would minimum shift of 4 be ok ?
Quoted 7 lines
something like
max ( (ilog2(dirty_total - 1)- 4) , 4);Unfortunately volunteers don't seem to be leaping out of the woodwork, maybe Andrew could be persuaded to try this in his tree for a while and see if any one squeaks ?
Quoted 6 lines
I'm pretty sure that most volunteers are curious what to actually test, so they shy away from it. If you added a good explanation of an easy way to test the before and after, then it would be more approachable.
I'll give it a spin here.
Ah - sorry. but I thought what it did was obvious ;)
Finding a test that's going to show a difference isn't going to be that easy, It isn't going to have any effect on writing to a single bdi, but only workloads writing to 2 (or more) disks.
Calc_period_shift controls the speed that the bdi dirty threshold gets updated, which in turn controls how much of the vm_dirty cache a bdi can use.
The first graph shows that currently it is rather slow in reacting to
change so that when you switch the writes from sda to sdb, the threshold doesn't react quickly enough and sdb isn't allowed to use it's fair share of the cache and is forced to write to the spinning disk sooner. Therefore it's slower overall. But the speed difference is highly dependent on the size of the write v. the size of the cache and the speed of the disk v. speed of writing to memory.
The tests I run here are writing a large file to one disk then after a small delay start a small write to the second disk, but it's not easy to get repeatable results from them.
I don't have a simple test, but the patch will improve the fairness of the vm_dirty cache sharing. I had in mind the sort of server workloads where some disks are dedicated to particular applications and others to general use. There may also be some desktop improvements but they are difficult to pin down.
I'm sorry I wasn't clearer before and hope this has explained what I've been trying to do.
regards
Richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>