Flexible proportions for BDIs
Subsystems (4)
Jan Kara
maintainerv2 cover Flexible proportions for BDIs
Hello,
this is the second iteration of my patches for flexible proportions. Since
previous submission, I've converted BDI proportion calculations to use flexible proportions so now we can test proportions in kernel. Fengguang, can you give them a run in your JBOD setup? You might try to tweak VM_COMPLETIONS_PERIOD_LEN if things are fluctuating too much... I'm not yet completely decided how to set that constant. Thanks!
Honza
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jan Kara
maintainerPATCH v2 1/2 lib: Proportions with flexible period
Implement code computing proportions of events of different type (like code in lib/proportions.c) but allowing periods to have different lengths. This allows us to have aging periods of fixed wallclock time which gives better proportion estimates given the hugely varying throughput of different devices - previous measuring of aging period by number of events has the problem that a reasonable period length for a system with low-end USB stick is not a reasonable period length for a system with high-end storage array resulting either in too slow proportion updates or too fluctuating proportion updates.
--- include/linux/flex_proportions.h | 91 +++++++++++++++ lib/Makefile | 2 +- lib/flex_proportions.c | 238 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 330 insertions(+), 1 deletions(-) create mode 100644 include/linux/flex_proportions.h create mode 100644 lib/flex_proportions.c
include/linux/flex_proportions.h new file +91 -0
@@ -0,0 +1,91 @@ +/* +* Floating proportions with flexible aging period +* +* Copyright (C) 2011, SUSE, Jan Kara <[email protected]> +*/ + +#ifndef _LINUX_FLEX_PROPORTIONS_H +#define _LINUX_FLEX_PROPORTIONS_H + +#include <linux/percpu_counter.h> +#include <linux/spinlock.h> +#include <linux/seqlock.h> + +/* +* ---- Global proportion definitions ---- +*/ +struct fprop_global { +/* Number of events in the current period */ +struct percpu_counter events; +/* Current period */ +unsigned int period; +/* Synchronization with period transitions */ +seqcount_t sequence; +}; + +int fprop_global_init (struct fprop_global *p); +void fprop_global_destroy (struct fprop_global *p); +void fprop_new_period (struct fprop_global *p); + +/* +* ---- SINGLE ---- +*/ +struct fprop_local_single { +/* the local events counter */ +unsigned long events; +/* Period in which we last updated events */ +unsigned int period; +raw_spinlock_t lock;/* Protect period and numerator */ +}; + +#define INIT_FPROP_LOCAL_SINGLE (name) \ +{ .lock = __RAW_SPIN_LOCK_UNLOCKED(name.lock), \ +} + +int fprop_local_init_single (struct fprop_local_single *pl); +void __fprop_inc_single (struct fprop_global *p,struct fprop_local_single *pl); +void fprop_fraction_single (struct fprop_global *p, +struct fprop_local_single *pl,unsigned long *numerator, +unsigned long *denominator); + +static inline +void fprop_inc_single (struct fprop_global *p,struct fprop_local_single *pl) +{ +unsigned long flags; + +local_irq_save (flags); +__fprop_inc_single (p, pl); +local_irq_restore (flags); +} + +/* +* ---- PERCPU ---- +*/ +struct fprop_local_percpu { +/* the local events counter */ +struct percpu_counter events; +/* Period in which we last updated events */ +unsigned int period; +raw_spinlock_t lock;/* Protect period and numerator */ +}; + +int fprop_local_init_percpu (struct fprop_local_percpu *pl); +void fprop_local_destroy_percpu (struct fprop_local_percpu *pl); +void __fprop_inc_percpu (struct fprop_global *p,struct fprop_local_percpu *pl); +void __fprop_inc_percpu_max (struct fprop_global *p,struct fprop_local_percpu *pl, +int max_frac); +void fprop_fraction_percpu (struct fprop_global *p, +struct fprop_local_percpu *pl,unsigned long *numerator, +unsigned long *denominator); + +static inline +void fprop_inc_percpu (struct fprop_global *p,struct fprop_local_percpu *pl) +{ +unsigned long flags; + +local_irq_save (flags); +__fprop_inc_percpu (p, pl); +local_irq_restore (flags); +} + +#endif
lib/Makefile +1 -1
@@ -11,7 +11,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \ rbtree.o radix-tree.o dump_stack.o timerqueue.o\ idr.o int_sqrt.o extable.o prio_tree.o \ sha1.o md5.o irq_regs.o reciprocal_div.o argv_split.o \ - proportions.o prio_heap.o ratelimit.o show_mem.o \ + proportions.o flex_proportions.o prio_heap.o ratelimit.o show_mem.o \ is_single_threaded.o plist.o decompress.o lib-$(CONFIG_MMU) += ioremap.o
lib/flex_proportions.c new file +238 -0
238 lines not on this page · raw message
Signature · 8 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jan Kara
maintainerPATCH v2 2/2 block: Convert BDI proportion calculations to flexible proportions
In reply to Jan KaraConvert calculations of proportion of writeback each bdi does to new flexible proportion code. That allows us to use aging period of fixed wallclock time which gives better proportion estimates given the hugely varying throughput of different devices.
--- include/linux/backing-dev.h | 6 ++-- mm/backing-dev.c | 5 +-- mm/page-writeback.c | 80 ++++++++++++++++++++---------------------- 3 files changed, 43 insertions(+), 48 deletions(-)
include/linux/backing-dev.h +3 -3
@@ -10,7 +10,7 @@#include <linux/percpu_counter.h> #include <linux/log2.h> -#include <linux/proportions.h> +#include <linux/flex_proportions.h> #include <linux/kernel.h> #include <linux/fs.h> #include <linux/sched.h>
@@ -89,11 +89,11 @@ struct backing_dev_info {unsigned long dirty_ratelimit;unsigned long balanced_dirty_ratelimit; -struct prop_local_percpu completions; +struct fprop_local_percpu completions;int dirty_exceeded;unsigned int min_ratio; -unsigned int max_ratio, max_prop_frac; +unsigned int max_ratio;struct bdi_writeback wb;/* default writeback info for this bdi */ spinlock_t wb_lock;/* protects work_list */
mm/backing-dev.c +2 -3
@@ -677,7 +677,6 @@ int bdi_init(struct backing_dev_info *bdi) bdi->min_ratio =0 ; bdi->max_ratio =100 ; - bdi->max_prop_frac =PROP_FRAC_BASE ;spin_lock_init (&bdi->wb_lock);INIT_LIST_HEAD (&bdi->bdi_list);INIT_LIST_HEAD (&bdi->work_list);
@@ -700,7 +699,7 @@ int bdi_init(struct backing_dev_info *bdi) bdi->write_bandwidth =INIT_BW ; bdi->avg_write_bandwidth =INIT_BW ; - err =prop_local_init_percpu (&bdi->completions); + err =fprop_local_init_percpu (&bdi->completions);if (err) { err:
@@ -744,7 +743,7 @@ void bdi_destroy(struct backing_dev_info *bdi)for (i =0 ; i <NR_BDI_STAT_ITEMS ; i++)percpu_counter_destroy (&bdi->bdi_stat[i]); -prop_local_destroy_percpu (&bdi->completions); +fprop_local_destroy_percpu (&bdi->completions); }EXPORT_SYMBOL (bdi_destroy);
mm/page-writeback.c +38 -42
143 lines not on this page · raw message
Signature · 8 lines
1.7.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/2] block: Convert BDI proportion calculations to flexible proportions
Quoted 14 lines · Jan Kara
Convert calculations of proportion of writeback each bdi does to new flexible proportion code. That allows us to use aging period of fixed wallclock time which gives better proportion estimates given the hugely varying throughput of different devices.
Signed-off-by Jan Kara <[email protected]>--- include/linux/backing-dev.h | 6 ++-- mm/backing-dev.c | 5 +-- mm/page-writeback.c | 80 ++++++++++++++++++++---------------------- 3 files changed, 43 insertions(+), 48 deletions(-)mm/page-writeback.c +2
-0+static void vm_completions_period (struct work_struct *work); +/* Work for aging of vm_completions */
mm/page-writeback.c EXPORT_SYMBOL_GPL(bdi_writeout_inc); · line 429
Quoted 10 lines
mm/page-writeback.c +9
-0+static DECLARE_DEFERRED_WORK(vm_completions_period_work, vm_completions_period); + +static void vm_completions_period (struct work_struct *work) +{ +fprop_new_period (&vm_completions); +schedule_delayed_work (&vm_completions_period_work, +VM_COMPLETIONS_PERIOD_LEN ); +} +
Is it possible to optimize away the periodic work when there are no disk writes?
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/2] block: Convert BDI proportion calculations to flexible proportions
Quoted 17 lines · Fengguang Wu
Quoted 15 lines · Jan Kara
Convert calculations of proportion of writeback each bdi does to new flexible proportion code. That allows us to use aging period of fixed wallclock time which gives better proportion estimates given the hugely varying throughput of different devices.
Signed-off-by Jan Kara <[email protected]>--- include/linux/backing-dev.h | 6 ++-- mm/backing-dev.c | 5 +-- mm/page-writeback.c | 80 ++++++++++++++++++++---------------------- 3 files changed, 43 insertions(+), 48 deletions(-)mm/page-writeback.c +3
-0+static void vm_completions_period (struct work_struct *work); +/* Work for aging of vm_completions */ +static DECLARE_DEFERRED_WORK(vm_completions_period_work, vm_completions_period);
Quoted 11 lines
mm/page-writeback.c EXPORT_SYMBOL_GPL(bdi_writeout_inc); · line 429
Quoted 8 lines
mm/page-writeback.c +8
-0+ +static void vm_completions_period (struct work_struct *work) +{ +fprop_new_period (&vm_completions); +schedule_delayed_work (&vm_completions_period_work, +VM_COMPLETIONS_PERIOD_LEN ); +} +Is it possible to optimize away the periodic work when there are no disk writes?
That should really be a timer, nothing in there requires scheduling so the entire addition of the workqueue muck is pure overhead.
You could keep a second period counter that tracks the last observed period and whenever the period and last_observed_period are further apart than BITS_PER_LONG you can stop the timer.
You'll have to restart it when updating last_observed_period.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jan Kara
maintainerRe: [PATCH 2/2] block: Convert BDI proportion calculations to flexible proportions
Quoted 30 lines · Peter Zijlstra
Quoted 17 lines · Fengguang Wu
Quoted 15 lines · Jan Kara
Convert calculations of proportion of writeback each bdi does to new flexible proportion code. That allows us to use aging period of fixed wallclock time which gives better proportion estimates given the hugely varying throughput of different devices.
Signed-off-by Jan Kara <[email protected]>--- include/linux/backing-dev.h | 6 ++-- mm/backing-dev.c | 5 +-- mm/page-writeback.c | 80 ++++++++++++++++++++---------------------- 3 files changed, 43 insertions(+), 48 deletions(-)mm/page-writeback.c +3
-0+static void vm_completions_period (struct work_struct *work); +/* Work for aging of vm_completions */ +static DECLARE_DEFERRED_WORK(vm_completions_period_work, vm_completions_period);Quoted 11 lines
mm/page-writeback.c EXPORT_SYMBOL_GPL(bdi_writeout_inc); · line 429
Quoted 8 lines
mm/page-writeback.c +8
-0+ +static void vm_completions_period (struct work_struct *work) +{ +fprop_new_period (&vm_completions); +schedule_delayed_work (&vm_completions_period_work, +VM_COMPLETIONS_PERIOD_LEN ); +} +Is it possible to optimize away the periodic work when there are no disk writes?
Quoted 8 lines
That should really be a timer, nothing in there requires scheduling so the entire addition of the workqueue muck is pure overhead.
You could keep a second period counter that tracks the last observed period and whenever the period and
last_observed_periodare further apart thanBITS_PER_LONGyou can stop the timer.You'll have to restart it when updating
last_observed_period.
Good points. I'll improve this in the next version.
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Jan Kara
Hello,
this is the second iteration of my patches for flexible proportions. Since
previous submission, I've converted BDI proportion calculations to use flexible proportions so now we can test proportions in kernel. Fengguang, can you give them a run in your JBOD setup? You might try to tweakVM_COMPLETIONS_PERIOD_LENif things are fluctuating too much... I'm not yet completely decided how to set that constant. Thanks!
Kara, I've got some results and it's working great. Overall performance remains good. The default VM_COMPLETIONS_PERIOD_LEN = 0.5s is obviously too small, so I tried increasing it to 3s and then 8s. Results for xfs (which has most fluctuating IO completions and ditto for bdi_setpoint) are attached. The XFS result of vanilla 3.3 is also attached. The graphs are all for case bay/JBOD-2HDD-thresh=1000M/xfs-10dd.
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable bdi_setpoint as the vanilla kernel, while being able to adapt to the balanced bdi_setpoint much more fast (actually now the bdi_setpoint is immediately close to the balanced value when balance_dirty_pages() starts throttling, while the vanilla kernel takes about 20 seconds for bdi_setpoint to grow up).
3.4.0-rc2 3.4.0-rc2-prop8+
------------------------ ------------------------
195.86 +0.0% 195.87 bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.4.0-rc2
196.68 +0.1% 196.81 bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
187.39 -1.1% 185.28 bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.4.0-rc2
191.94 +0.7% 193.27 bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.4.0-rc2
193.01 -0.1% 192.76 bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.4.0-rc2
197.28 -0.1% 197.07 bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2
197.09 -0.2% 196.74 bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
160.16 -1.2% 158.21 bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.4.0-rc2
192.10 -0.1% 191.97 bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.4.0-rc2
163.35 +0.3% 163.79 bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.4.0-rc2
194.90 +0.0% 194.99 bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.4.0-rc2
191.10 +0.2% 191.53 bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
183.38 +2.6% 188.12 bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.4.0-rc2
196.02 -0.0% 195.99 bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2
170.18 +0.3% 170.70 bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
180.79 +1.6% 183.72 bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.4.0-rc2
189.00 -0.2% 188.68 bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.4.0-rc2
97.57 +0.0% 97.61 bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
96.69 +0.8% 97.50 bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.4.0-rc2
96.99 +0.9% 97.86 bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2
97.53 +0.2% 97.71 bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
95.85 -0.0% 95.80 bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.4.0-rc2
97.11 +0.1% 97.23 bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.4.0-rc2
99.38 -0.0% 99.37 bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
98.50 +0.2% 98.65 bay/thresh=1000M/ext4-1dd-1-3.4.0-rc2
97.54 -0.0% 97.50 bay/thresh=1000M/xfs-10dd-1-3.4.0-rc2
99.67 -0.1% 99.56 bay/thresh=1000M/xfs-1dd-1-3.4.0-rc2
99.34 -0.0% 99.33 bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
97.56 -0.1% 97.44 bay/thresh=100M/ext4-1dd-1-3.4.0-rc2
87.62 +0.2% 87.83 bay/thresh=100M/xfs-10dd-1-3.4.0-rc2
99.18 +0.0% 99.21 bay/thresh=100M/xfs-1dd-1-3.4.0-rc2
86.99 -0.8% 86.33 bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
91.65 -0.2% 91.42 bay/thresh=10M/ext4-1dd-1-3.4.0-rc2
69.54 +0.2% 69.72 bay/thresh=10M/xfs-10dd-1-3.4.0-rc2
91.73 +0.4% 92.10 bay/thresh=10M/xfs-1dd-1-3.4.0-rc2
3.31 -2.7% 3.22 bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
84.17 +0.7% 84.76 bay/thresh=1M/ext4-1dd-1-3.4.0-rc2
65.87 -2.2% 64.44 bay/thresh=1M/xfs-10dd-1-3.4.0-rc2
75.02 -0.6% 74.59 bay/thresh=1M/xfs-1dd-1-3.4.0-rc2
5109.05 +0.1% 5114.70 TOTAL write_bw 3723146.77 +0.0% 3723782.19 TOTAL io_wkB_s
12136.02 -0.3% 12096.63 TOTAL io_w_s
6246.22 +0.5% 6280.40 TOTAL io_wrqm_s
7.86 -5.6% 7.42 TOTAL io_rkB_s
2.00 -5.0% 1.90 TOTAL io_r_s
0.02 -38.5% 0.01 TOTAL io_rrqm_s
35432.39 -0.0% 35417.66 TOTAL io_avgrq_sz
2804.24 +0.9% 2830.82 TOTAL io_avgqu_sz
14045.59 +1.2% 14220.95 TOTAL io_await
150.05 +28.4% 192.65 TOTAL io_svctm
14048.25 +1.3% 14223.99 TOTAL io_util
97.79 +0.4% 98.20 TOTAL cpu_user
0.00 0.00 TOTAL cpu_nice
1150.46 -0.8% 1141.17 TOTAL cpu_system
1311.93 +0.5% 1319.01 TOTAL cpu_iowait
0.00 0.00 TOTAL cpu_steal
1439.81 +0.1% 1441.63 TOTAL cpu_idleThanks,
Fengguang
Jan Kara
maintainerHello,
Quoted 8 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
this is the second iteration of my patches for flexible proportions. Since
previous submission, I've converted BDI proportion calculations to use flexible proportions so now we can test proportions in kernel. Fengguang, can you give them a run in your JBOD setup? You might try to tweakVM_COMPLETIONS_PERIOD_LENif things are fluctuating too much... I'm not yet completely decided how to set that constant. Thanks!
Quoted 6 lines
Kara, I've got some results and it's working great. Overall performance remains good. The default
VM_COMPLETIONS_PERIOD_LEN= 0.5s is obviously too small, so I tried increasing it to 3s and then 8s. Results for xfs (which has most fluctuating IO completions and ditto forbdi_setpoint) are attached. The XFS result of vanilla 3.3 is also attached. The graphs are all for case bay/JBOD-2HDD-thresh=1000M/xfs-10dd.
Thanks for testing! I agree that 0.5s period is probably on the low end.
OTOH 8s seems a bit too much. Consider two bdi's with vastly different speeds - say their throughput ratio is 1:32 (e.g. an USB stick and a raid backed storage). When you write to the fast storage, then stop and start writing to the USB stick, then it will take 5 periods for bdi writeout ratio to become 1:1 and another 4-5 periods to be close to real current situation which is no IO to storage 100% io to USB stick. So with 8s period this will give you total transition time ~80s with seems like too much to me.
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).
Which graph is from which kernel? All four graphs have the same name so
I'm not sure...
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Jan Kara
Hello,
Quoted 8 lines · Wu Fengguang
Quoted 6 lines · Jan Kara
this is the second iteration of my patches for flexible proportions. Since
previous submission, I've converted BDI proportion calculations to use flexible proportions so now we can test proportions in kernel. Fengguang, can you give them a run in your JBOD setup? You might try to tweakVM_COMPLETIONS_PERIOD_LENif things are fluctuating too much... I'm not yet completely decided how to set that constant. Thanks!Quoted 6 lines
Kara, I've got some results and it's working great. Overall performance remains good. The default
VM_COMPLETIONS_PERIOD_LEN= 0.5s is obviously too small, so I tried increasing it to 3s and then 8s. Results for xfs (which has most fluctuating IO completions and ditto forbdi_setpoint) are attached. The XFS result of vanilla 3.3 is also attached. The graphs are all for case bay/JBOD-2HDD-thresh=1000M/xfs-10dd.
Quoted 9 lines
Thanks for testing! I agree that 0.5s period is probably on the low end.
OTOH 8s seems a bit too much. Consider two bdi's with vastly different speeds - say their throughput ratio is 1:32 (e.g. an USB stick and a raid backed storage). When you write to the fast storage, then stop and start writing to the USB stick, then it will take 5 periods for bdi writeout ratio to become 1:1 and another 4-5 periods to be close to real current situation which is no IO to storage 100% io to USB stick. So with 8s period this will give you total transition time ~80s with seems like too much to me.
OK, got it.
Quoted 9 lines
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).Which graph is from which kernel? All four graphs have the same name so
I'm not sure...
They are for test cases:
0.5s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop+/balance_dirty_pages-pages+.png
3s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop3+/balance_dirty_pages-pages+.png
8s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop8+/balance_dirty_pages-pages+.png
vanilla
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0/balance_dirty_pages-pages+.pngQuoted 7 lines
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).
OK. I'll test this scheme and report back.
loop {
dd to disk 1 for 30s
dd to disk 2 for 30s
}
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 29 lines
Quoted 9 lines
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).Which graph is from which kernel? All four graphs have the same name so
I'm not sure...They are for test cases:
0.5s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop+/balance_dirty_pages-pages+.png 3s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop3+/balance_dirty_pages-pages+.png 8s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop8+/balance_dirty_pages-pages+.png vanilla bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0/balance_dirty_pages-pages+.pngQuoted 7 lines
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).
Quoted 6 lines
OK. I'll test this scheme and report back.
loop {
dd to disk 1 for 30s
dd to disk 2 for 30s
}
Here are the new results. For simplicity I run the dd dirtiers continuously, and start another dd reader to knock down the write bandwidth from time to time:
loop {
dd from disk 1 for 30s
dd from disk 2 for 30s
}
The first attached iostat graph shows the resulting read/write bandwidth for one of the two disks.
The followed graphs are for
- 3s period
- 8s period
- vanilla
in order. The test case is (xfs-1dd, mem=2GB, 2 disks JBOD).
Thanks,
Fengguang
Quoted 45 lines · Fengguang Wu
Quoted 29 lines
Quoted 9 lines
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).Which graph is from which kernel? All four graphs have the same name so
I'm not sure...They are for test cases:
0.5s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop+/balance_dirty_pages-pages+.png 3s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop3+/balance_dirty_pages-pages+.png 8s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop8+/balance_dirty_pages-pages+.png vanilla bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0/balance_dirty_pages-pages+.pngQuoted 7 lines
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).Quoted 6 lines
OK. I'll test this scheme and report back.
loop {
dd to disk 1 for 30s
dd to disk 2 for 30s
}Here are the new results. For simplicity I run the dd dirtiers continuously, and start another dd reader to knock down the write bandwidth from time to time:
loop {
dd from disk 1 for 30s
dd from disk 2 for 30s
}
Quoted 8 lines
The first attached iostat graph shows the resulting read/write bandwidth for one of the two disks.
The followed graphs are for
- 3s period
- 8s period
- vanilla
in order. The test case is (xfs-1dd, mem=2GB, 2 disks JBOD).
Here are more results for another test box with mem=256G running 4 SSDs. This time I run 8 dd readers to better disturb the writes.
The first 3 graphs are for cases:
lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop3+ lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop8+ lkp-nex04/alternant_read_8/xfs-10dd-2-3.3.0
The last graph shows how the write bandwidth is squeezed by reads over time:
lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop8+/iostat-bw.png
The observations for this box are
- the 3s and 8s periods result in roughly the same adaption speed
- the patch makes a really *big* difference in systems with big memory:bandwidth ratio. It's sweet! In comparison, the vanilla kernel adapts to new write bandwidth so much slower.
Thanks,
Fengguang
Jan Kara
maintainerQuoted 68 lines · Wu Fengguang
Quoted 45 lines · Fengguang Wu
Quoted 29 lines
Quoted 9 lines
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).Which graph is from which kernel? All four graphs have the same name so
I'm not sure...They are for test cases:
0.5s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop+/balance_dirty_pages-pages+.png 3s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop3+/balance_dirty_pages-pages+.png 8s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop8+/balance_dirty_pages-pages+.png vanilla bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0/balance_dirty_pages-pages+.pngQuoted 7 lines
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).Quoted 6 lines
OK. I'll test this scheme and report back.
loop {
dd to disk 1 for 30s
dd to disk 2 for 30s
}Here are the new results. For simplicity I run the dd dirtiers continuously, and start another dd reader to knock down the write bandwidth from time to time:
loop {
dd from disk 1 for 30s
dd from disk 2 for 30s
}Quoted 8 lines
The first attached iostat graph shows the resulting read/write bandwidth for one of the two disks.
The followed graphs are for
- 3s period
- 8s period
- vanilla
in order. The test case is (xfs-1dd, mem=2GB, 2 disks JBOD).Here are more results for another test box with mem=256G running 4 SSDs. This time I run 8 dd readers to better disturb the writes.
The first 3 graphs are for cases:
lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop3+ lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop8+ lkp-nex04/alternant_read_8/xfs-10dd-2-3.3.0
The last graph shows how the write bandwidth is squeezed by reads over time:
lkp-nex04/alternant_read_8/xfs-10dd-2-3.4.0-rc5-prop8+/iostat-bw.png
Quoted 7 lines
The observations for this box are
- the 3s and 8s periods result in roughly the same adaption speed
- the patch makes a really *big* difference in systems with big memory:bandwidth ratio. It's sweet! In comparison, the vanilla kernel adapts to new write bandwidth so much slower.
Yes, in this configuration the benefit of the new algorithm can be clearly
seen. Together with the results of previous test I'd say 3s period is the best candidate.
Just I was thinking whether the period shouldn't be somehow set
automatically because I'm not convinced 3s will be right for everybody... Maybe something based on how big fluctuations in completion rate we observe. But it would be tricky given the load itself changes as well. So for now we'll have to live with a hardwired period I guess.
Thanks for the tests Fengguang! So is anybody against merging this?
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Jan Kara
So is anybody against merging this?
I'd like to see the timer disable itself stuff first.. other than that, no.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jan Kara
maintainerQuoted 5 lines · Peter Zijlstra
Quoted 1 line · Jan Kara
So is anybody against merging this?
I'd like to see the timer disable itself stuff first.. other than that, no.
Ah, my fault. I have the code written already but forgot to send it...
I'll post v3 in a minute.
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Jan,
Quoted 10 lines
Quoted 7 lines
The observations for this box are
- the 3s and 8s periods result in roughly the same adaption speed
- the patch makes a really *big* difference in systems with big memory:bandwidth ratio. It's sweet! In comparison, the vanilla kernel adapts to new write bandwidth so much slower.
Yes, in this configuration the benefit of the new algorithm can be clearly
seen. Together with the results of previous test I'd say 3s period is the best candidate.
Agreed. I'm fine with the fixed 3s period.
Quoted 5 lines
Just I was thinking whether the period shouldn't be somehow set
automatically because I'm not convinced 3s will be right for everybody... Maybe something based on how big fluctuations in completion rate we observe. But it would be tricky given the load itself changes as well. So for now we'll have to live with a hardwired period I guess.
Yeah, simple fixed periods should be good enough.
Quoted 1 line
Thanks for the tests Fengguang! So is anybody against merging this?
No problem for me, when Peter's concern is addressed.
Thanks!
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Jan Kara
maintainerQuoted 45 lines · Wu Fengguang
Quoted 29 lines
Quoted 9 lines
Quoted 7 lines
Look at the gray "bdi setpoint" lines. The VM_COMPLETIONS_PERIOD_LEN=8s kernel is able to achieve roughly the same stable
bdi_setpointas the vanilla kernel, while being able to adapt to the balancedbdi_setpointmuch more fast (actually now thebdi_setpointis immediately close to the balanced value whenbalance_dirty_pages()starts throttling, while the vanilla kernel takes about 20 seconds forbdi_setpointto grow up).Which graph is from which kernel? All four graphs have the same name so
I'm not sure...They are for test cases:
0.5s period
bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop+/balance_dirty_pages-pages+.png 3s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop3+/balance_dirty_pages-pages+.png 8s period bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.4.0-rc2-prop8+/balance_dirty_pages-pages+.png vanilla bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0/balance_dirty_pages-pages+.pngQuoted 7 lines
The faster (almost immediate) initial adaptation to bdi's writeout fraction
is mostly an effect of better normalization with my patches. Although it is pleasant, it happens just at the moment when there is a small number of periods with non-zero number of events. So more important for practice is in my opininion to compare transition of computed fractions when workload changes (i.e. we start writing to one bdi while writing to another bdi or so).Quoted 6 lines
OK. I'll test this scheme and report back.
loop {
dd to disk 1 for 30s
dd to disk 2 for 30s
}Here are the new results. For simplicity I run the dd dirtiers continuously, and start another dd reader to knock down the write bandwidth from time to time:
loop {
dd from disk 1 for 30s
dd from disk 2 for 30s
}
Quoted 8 lines
The first attached iostat graph shows the resulting read/write bandwidth for one of the two disks.
The followed graphs are for
- 3s period
- 8s period
- vanilla
in order. The test case is (xfs-1dd, mem=2GB, 2 disks JBOD).
Thanks for the test! So here 3s period adapts to changed throughput
fairly quickly, similarly as vanilla kernel, 8s period takes a bit more time. Random variations in computed proportions for 3s period are about the same as for vanilla kernel and in a reasonable range I'd say. For 8s period variations are even smaller as expected.
So all in all I'd say 3s period did fine here, although it did not offer much benefit over the previous algorithm. 8s period was a bit too slow to adapt.
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>