throttle: Allow I/O bursts for a user-defined period of time
cover throttle: Allow I/O bursts for a user-defined period of time
Hello everyone,
the current throttling code in QEMU allows limiting the I/O rate on block devices. Limits can be set in operations per second (IOPS) or bytes per second, allowing separate limits for read and write operations on both cases.
In its basic usage the user can set a limit of, say, 100 IOPS on a block device passing this option to -drive:
throttling.iops-total=100
In addition to that, QEMU can also allow the user to do I/O bursts exceeding that limit up to a configurable rate:
throttling.iops-total=100,throttling.iops-total-max=2000
With this, the user can do a burst of 2000 IOPS before they're throttled down to 100 IOPS. Then, after a sufficiently long period of unused I/O they will be able to do a burst again.
This patch series introduces the possibility to do bursts for longer period of times. A new setting called throttling.iops-total-max-length is used to define for how long bursts can be sustained.
So adding throttling.iops-total-max-length=60 to the previous configuration allows the user to do I/O at a rate of 2000 IOPS for 1 minute before going down to the base rate of 100 IOPS.
This is essentially the same as described in this AWS blog post:
https://aws.amazon.com/blogs/aws/new-ssd-backed-elastic-block-storage/
As described in the article, a use case for this feature is to allow better performance when booting the OS or restarting a service while keeping the average I/O limits lower the rest of the time.
Comments:
- There are 6 different settings for setting I/O limits: iops-total, iops-read, iops-write, bps-total, bps-read, bps-write. This series adds one new setting to set the length for each one of those.
I don't know if there's a good use case that requires such fine-grained control. It's of course also possible to make it simpler and add just one 'burst-length' setting that would apply for all cases, but the current solution is IMHO simple enough and consistent with the current API, and if we need to extend it later the result is probably going to be ugly.
- With this series we set "a maximum of X operations/second for a period of T seconds". If would also be possible to make it "a maximum of X operations/second up to a total of Y operations". It would be equivalent (Y = X * T) but I thought the current proposal makes a more clear API.
And I think that's all.
Thanks!
Berto
Alberto Garcia (13):
throttle: Make throttle_compute_timer() static
throttle: Make throttle_conflicting() set errp
throttle: Make throttle_max_is_missing_limit() set errp
throttle: Make throttle_is_valid() set errp
throttle: Set always an average value when setting a maximum value
throttle: Merge all functions that check the configuration into one
throttle: Use throttle_config_init() to initialize ThrottleConfig
throttle: Add support for burst periods
throttle: Add command-line settings to define the burst periods
qapi: Add burst length parameters to block_set_io_throttle
qapi: Add burst length fields to BlockDeviceInfo
throttle: Check that burst_level leaks correctly
throttle: Test throttle_compute_wait() during bursts
block/qapi.c | 20 ++++++++ blockdev.c | 99 ++++++++++++++++++++++++++---------- hmp.c | 12 +++++ include/qemu/throttle.h | 55 ++++++++++++++------ qapi/block-core.json | 90 ++++++++++++++++++++++++++++----- tests/test-throttle.c | 88 ++++++++++++++++++++++++-------- util/throttle.c | 132 ++++++++++++++++++++++++++++++++---------------- 7 files changed, 377 insertions(+), 119 deletions(-)
Signature · 1 line
2.7.0
PATCH 1/13 throttle: Make throttle_compute_timer() static
This function is only used internally in throttle.c
--- include/qemu/throttle.h | 6 ------ util/throttle.c | 8 ++++---- 2 files changed, 4 insertions(+), 10 deletions(-)
include/qemu/throttle.h +0 -6
@@ -84,12 +84,6 @@ void throttle_leak_bucket(LeakyBucket *bkt, int64_t delta);int64_t throttle_compute_wait (LeakyBucket *bkt); -/* expose timer computation function for unit tests */ -bool throttle_compute_timer (ThrottleState *ts, -bool is_write, -int64_t now, -int64_t *next_timestamp); -/* init/destroy cycle */ void throttle_init (ThrottleState *ts);
util/throttle.c +4 -4
@@ -137,10 +137,10 @@ static int64_t throttle_compute_wait_for(ThrottleState *ts,* @next_timestamp: the resulting timer * @ret: true if a timer must be set */ -bool throttle_compute_timer (ThrottleState *ts, -bool is_write, -int64_t now, -int64_t *next_timestamp) +static bool throttle_compute_timer (ThrottleState *ts, +bool is_write, +int64_t now, +int64_t *next_timestamp) {int64_t wait;
Signature · 1 line
2.7.0
PATCH 2/13 throttle: Make throttle_conflicting() set errp
In reply to Alberto GarciaThe caller does not need to set it, and this will allow us to refactor this function later.
--- blockdev.c | 4 +--- include/qemu/throttle.h | 2 +- tests/test-throttle.c | 12 ++++++------ util/throttle.c | 11 +++++++++-- 4 files changed, 17 insertions(+), 12 deletions(-)
blockdev.c +1 -3
@@ -345,9 +345,7 @@ static bool parse_stats_intervals(BlockAcctStats *stats, QList *intervals,static bool check_throttle_config (ThrottleConfig *cfg,Error **errp) { -if (throttle_conflicting (cfg)) { -error_setg (errp,"bps/iops/max total values and read/write values" -" cannot be used at the same time" ); +if (throttle_conflicting (cfg, errp)) {return false; }
include/qemu/throttle.h +1 -1
@@ -106,7 +106,7 @@ bool throttle_timers_are_initialized(ThrottleTimers *tt);/* configuration */ bool throttle_enabled (ThrottleConfig *cfg); -bool throttle_conflicting (ThrottleConfig *cfg); +bool throttle_conflicting (ThrottleConfig *cfg,Error **errp);bool throttle_is_valid (ThrottleConfig *cfg);
tests/test-throttle.c +6 -6
@@ -254,31 +254,31 @@ static void test_conflicts_for_one_set(bool is_max, int write) {memset (&cfg,0 ,sizeof (cfg)); -g_assert (!throttle_conflicting (&cfg)); +g_assert (!throttle_conflicting (&cfg,NULL ));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 ); -g_assert (throttle_conflicting (&cfg)); +g_assert (throttle_conflicting (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, write,1 ); -g_assert (throttle_conflicting (&cfg)); +g_assert (throttle_conflicting (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 ); -g_assert (throttle_conflicting (&cfg)); +g_assert (throttle_conflicting (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 ); -g_assert (!throttle_conflicting (&cfg)); +g_assert (!throttle_conflicting (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 ); -g_assert (!throttle_conflicting (&cfg)); +g_assert (!throttle_conflicting (&cfg,NULL )); }static void test_conflicting_config (void )
util/throttle.c +9 -2
@@ -252,8 +252,9 @@ bool throttle_enabled(ThrottleConfig *cfg)* * @cfg: the throttling configuration to inspect * @ret: true if any conflict detected else false +* @errp: error object */ -bool throttle_conflicting (ThrottleConfig *cfg) +bool throttle_conflicting (ThrottleConfig *cfg,Error **errp) {bool bps_flag, ops_flag;bool bps_max_flag, ops_max_flag;
@@ -274,7 +275,13 @@ bool throttle_conflicting(ThrottleConfig *cfg) (cfg->buckets[THROTTLE_OPS_READ ].max || cfg->buckets[THROTTLE_OPS_WRITE ].max); -return bps_flag || ops_flag || bps_max_flag || ops_max_flag; +if (bps_flag || ops_flag || bps_max_flag || ops_max_flag) { +error_setg (errp,"bps/iops/max total values and read/write values" +" cannot be used at the same time" ); +return true; + } + +return false; }/* check if a throttling configuration is valid
Signature · 1 line
2.7.0
PATCH 3/13 throttle: Make throttle_max_is_missing_limit() set errp
In reply to Alberto GarciaThe caller does not need to set it, and this will allow us to refactor this function later.
--- blockdev.c | 4 +--- include/qemu/throttle.h | 2 +- tests/test-throttle.c | 6 +++--- util/throttle.c | 5 ++++- 4 files changed, 9 insertions(+), 8 deletions(-)
blockdev.c +1 -3
@@ -355,9 +355,7 @@ static bool check_throttle_config(ThrottleConfig *cfg, Error **errp)return false; } -if (throttle_max_is_missing_limit (cfg)) { -error_setg (errp,"bps_max/iops_max require corresponding" -" bps/iops values" ); +if (throttle_max_is_missing_limit (cfg, errp)) {return false; }
include/qemu/throttle.h +1 -1
@@ -110,7 +110,7 @@ bool throttle_conflicting(ThrottleConfig *cfg, Error **errp);bool throttle_is_valid (ThrottleConfig *cfg); -bool throttle_max_is_missing_limit (ThrottleConfig *cfg); +bool throttle_max_is_missing_limit (ThrottleConfig *cfg,Error **errp); void throttle_config(ThrottleState *ts, ThrottleTimers *tt,
tests/test-throttle.c +3 -3
@@ -337,15 +337,15 @@ static void test_max_is_missing_limit(void)memset (&cfg,0 ,sizeof (cfg)); cfg.buckets[i].max =100 ; cfg.buckets[i].avg =0 ; -g_assert (throttle_max_is_missing_limit (&cfg)); +g_assert (throttle_max_is_missing_limit (&cfg,NULL )); cfg.buckets[i].max =0 ; cfg.buckets[i].avg =0 ; -g_assert (!throttle_max_is_missing_limit (&cfg)); +g_assert (!throttle_max_is_missing_limit (&cfg,NULL )); cfg.buckets[i].max =0 ; cfg.buckets[i].avg =100 ; -g_assert (!throttle_max_is_missing_limit (&cfg)); +g_assert (!throttle_max_is_missing_limit (&cfg,NULL )); } }
util/throttle.c +4 -1
@@ -306,13 +306,16 @@ bool throttle_is_valid(ThrottleConfig *cfg)/* check if bps_max/iops_max is used without bps/iops * @cfg: the throttling configuration to inspect +* @errp: error object */ -bool throttle_max_is_missing_limit (ThrottleConfig *cfg) +bool throttle_max_is_missing_limit (ThrottleConfig *cfg,Error **errp) {int i;for (i =0 ; i <BUCKETS_COUNT ; i++) {if (cfg->buckets[i].max && !cfg->buckets[i].avg) { +error_setg (errp,"bps_max/iops_max require corresponding" +" bps/iops values" );return true; } }
Signature · 1 line
2.7.0
PATCH 4/13 throttle: Make throttle_is_valid() set errp
In reply to Alberto GarciaThe caller does not need to set it, and this will allow us to refactor this function later.
--- blockdev.c | 4 +--- include/qemu/throttle.h | 2 +- tests/test-throttle.c | 2 +- util/throttle.c | 5 ++++- 4 files changed, 7 insertions(+), 6 deletions(-)
blockdev.c +1 -3
@@ -349,9 +349,7 @@ static bool check_throttle_config(ThrottleConfig *cfg, Error **errp)return false; } -if (!throttle_is_valid (cfg)) { -error_setg (errp,"bps/iops/max values must be within [0, %lld]" , -THROTTLE_VALUE_MAX ); +if (!throttle_is_valid (cfg, errp)) {return false; }
include/qemu/throttle.h +1 -1
@@ -108,7 +108,7 @@ bool throttle_enabled(ThrottleConfig *cfg);bool throttle_conflicting (ThrottleConfig *cfg,Error **errp); -bool throttle_is_valid (ThrottleConfig *cfg); +bool throttle_is_valid (ThrottleConfig *cfg,Error **errp);bool throttle_max_is_missing_limit (ThrottleConfig *cfg,Error **errp);
tests/test-throttle.c +1 -1
@@ -314,7 +314,7 @@ static void test_is_valid_for_value(int value, bool should_be_valid)for (index =0 ; index <BUCKETS_COUNT ; index++) {memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, index, value); -g_assert (throttle_is_valid (&cfg) == should_be_valid); +g_assert (throttle_is_valid (&cfg,NULL ) == should_be_valid); } } }
util/throttle.c +4 -1
@@ -287,8 +287,9 @@ bool throttle_conflicting(ThrottleConfig *cfg, Error **errp)/* check if a throttling configuration is valid * @cfg: the throttling configuration to inspect * @ret: true if valid else false +* @errp: error object */ -bool throttle_is_valid (ThrottleConfig *cfg) +bool throttle_is_valid (ThrottleConfig *cfg,Error **errp) {int i;
@@ -297,6 +298,8 @@ bool throttle_is_valid(ThrottleConfig *cfg) cfg->buckets[i].max < 0 || cfg->buckets[i].avg > THROTTLE_VALUE_MAX || cfg->buckets[i].max > THROTTLE_VALUE_MAX) { +error_setg (errp,"bps/iops/max values must be within [0, %lld]" , +THROTTLE_VALUE_MAX );return false; } }
Signature · 1 line
2.7.0
PATCH 5/13 throttle: Set always an average value when setting a maximum value
In reply to Alberto GarciaWhen testing the ranges of valid values, set_cfg_value() creates sometimes invalid throttling configurations by setting bucket.max while leaving bucket.avg uninitialized.
While this doesn't break the current tests, it will as soon as we unify all functions that check the validity of the throttling configuration.
This patch ensures that the value of bucket.avg is valid when setting bucket.max.
--- tests/test-throttle.c | 2 ++ 1 file changed, 2 insertions(+)
tests/test-throttle.c +2 -0
@@ -221,6 +221,8 @@ static void set_cfg_value(bool is_max, int index, int value) {if (is_max) { cfg.buckets[index].max = value; +/* If max is set, avg should never be 0 */ + cfg.buckets[index].avg =MAX (cfg.buckets[index].avg,1 ); }else { cfg.buckets[index].avg = value; }
Signature · 1 line
2.7.0
PATCH 6/13 throttle: Merge all functions that check the configuration into one
In reply to Alberto GarciaThere's no need to keep throttle_conflicting(), throttle_is_valid() and throttle_max_is_missing_limit() as separate functions, so this patch merges all three into one.
As a consequence, check_throttle_config() becomes redundant and can be replaced with throttle_is_valid().
--- blockdev.c | 21 ++------------------- include/qemu/throttle.h | 4 ---- tests/test-throttle.c | 18 +++++++++--------- util/throttle.c | 40 ++++++++-------------------------------- 4 files changed, 19 insertions(+), 64 deletions(-)
blockdev.c +2 -19
@@ -343,23 +343,6 @@ static bool parse_stats_intervals(BlockAcctStats *stats, QList *intervals,return true; } -static bool check_throttle_config (ThrottleConfig *cfg,Error **errp) -{ -if (throttle_conflicting (cfg, errp)) { -return false; - } - -if (!throttle_is_valid (cfg, errp)) { -return false; - } - -if (throttle_max_is_missing_limit (cfg, errp)) { -return false; - } - -return true; -} -typedef enum {MEDIA_DISK ,MEDIA_CDROM }DriveMediaType ;/* All parameters but @opts are optional and may be set to NULL. */
@@ -434,7 +417,7 @@ static void extract_common_blockdev_options(QemuOpts *opts, int *bdrv_flags, throttle_cfg->op_size =qemu_opt_get_number (opts,"throttling.iops-size" ,0 ); -if (!check_throttle_config (throttle_cfg, errp)) { +if (!throttle_is_valid (throttle_cfg, errp)) {return ; } }
@@ -2652,7 +2635,7 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd, cfg.op_size = iops_size; } -if (!check_throttle_config (&cfg, errp)) { +if (!throttle_is_valid (&cfg, errp)) { goto out; }
include/qemu/throttle.h +0 -4
@@ -106,12 +106,8 @@ bool throttle_timers_are_initialized(ThrottleTimers *tt);/* configuration */ bool throttle_enabled (ThrottleConfig *cfg); -bool throttle_conflicting (ThrottleConfig *cfg,Error **errp); -bool throttle_is_valid (ThrottleConfig *cfg,Error **errp); -bool throttle_max_is_missing_limit (ThrottleConfig *cfg,Error **errp); -void throttle_config (ThrottleState *ts,ThrottleTimers *tt,ThrottleConfig *cfg);
tests/test-throttle.c +9 -9
@@ -256,31 +256,31 @@ static void test_conflicts_for_one_set(bool is_max, int write) {memset (&cfg,0 ,sizeof (cfg)); -g_assert (!throttle_conflicting (&cfg,NULL )); +g_assert (throttle_is_valid (&cfg,NULL ));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 ); -g_assert (throttle_conflicting (&cfg,NULL )); +g_assert (!throttle_is_valid (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, write,1 ); -g_assert (throttle_conflicting (&cfg,NULL )); +g_assert (!throttle_is_valid (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 ); -g_assert (throttle_conflicting (&cfg,NULL )); +g_assert (!throttle_is_valid (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, total,1 ); -g_assert (!throttle_conflicting (&cfg,NULL )); +g_assert (throttle_is_valid (&cfg,NULL ));memset (&cfg,0 ,sizeof (cfg));set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 ); -g_assert (!throttle_conflicting (&cfg,NULL )); +g_assert (throttle_is_valid (&cfg,NULL )); }static void test_conflicting_config (void )
@@ -339,15 +339,15 @@ static void test_max_is_missing_limit(void)memset (&cfg,0 ,sizeof (cfg)); cfg.buckets[i].max =100 ; cfg.buckets[i].avg =0 ; -g_assert (throttle_max_is_missing_limit (&cfg,NULL )); +g_assert (!throttle_is_valid (&cfg,NULL )); cfg.buckets[i].max =0 ; cfg.buckets[i].avg =0 ; -g_assert (!throttle_max_is_missing_limit (&cfg,NULL )); +g_assert (throttle_is_valid (&cfg,NULL )); cfg.buckets[i].max =0 ; cfg.buckets[i].avg =100 ; -g_assert (!throttle_max_is_missing_limit (&cfg,NULL )); +g_assert (throttle_is_valid (&cfg,NULL )); } }
util/throttle.c +8 -32
@@ -248,14 +248,14 @@ bool throttle_enabled(ThrottleConfig *cfg)return false; } -/* return true if any two throttling parameters conflicts -* +/* check if a throttling configuration is valid * @cfg: the throttling configuration to inspect -* @ret: true if any conflict detected else false +* @ret: true if valid else false * @errp: error object */ -bool throttle_conflicting (ThrottleConfig *cfg,Error **errp) +bool throttle_is_valid (ThrottleConfig *cfg,Error **errp) { +int i;bool bps_flag, ops_flag;bool bps_max_flag, ops_max_flag;
@@ -278,21 +278,9 @@ bool throttle_conflicting(ThrottleConfig *cfg, Error **errp)if (bps_flag || ops_flag || bps_max_flag || ops_max_flag) {error_setg (errp,"bps/iops/max total values and read/write values" " cannot be used at the same time" ); -return true; +return false; } -return false; -} - -/* check if a throttling configuration is valid -* @cfg: the throttling configuration to inspect -* @ret: true if valid else false -* @errp: error object -*/ -bool throttle_is_valid (ThrottleConfig *cfg,Error **errp) -{ -int i; -for (i =0 ; i <BUCKETS_COUNT ; i++) { if (cfg->buckets[i].avg < 0 || cfg->buckets[i].max < 0 ||
@@ -302,27 +290,15 @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) THROTTLE_VALUE_MAX);return false; } - } -return true; -} - -/* check if bps_max/iops_max is used without bps/iops -* @cfg: the throttling configuration to inspect -* @errp: error object -*/ -bool throttle_max_is_missing_limit (ThrottleConfig *cfg,Error **errp) -{ -int i; - -for (i =0 ; i <BUCKETS_COUNT ; i++) {if (cfg->buckets[i].max && !cfg->buckets[i].avg) {error_setg (errp,"bps_max/iops_max require corresponding" " bps/iops values" ); -return true; +return false; } } -return false; + +return true; }/* fix bucket parameters */
Signature · 1 line
2.7.0
PATCH 7/13 throttle: Use throttle_config_init() to initialize ThrottleConfig
In reply to Alberto GarciaWe can currently initialize ThrottleConfig by zeroing all its fields, but this will change with the new fields to define the length of the burst periods.
This patch introduces a new throttle_config_init() function and uses it to replace all memset() calls that initialize ThrottleConfig directly.
--- blockdev.c | 4 ++-- include/qemu/throttle.h | 2 ++ tests/test-throttle.c | 28 +++++++++++++++++----------- util/throttle.c | 10 ++++++++++ 4 files changed, 31 insertions(+), 13 deletions(-)
blockdev.c +2 -2
@@ -387,7 +387,7 @@ static void extract_common_blockdev_options(QemuOpts *opts, int *bdrv_flags, }if (throttle_cfg) { -memset (throttle_cfg,0 ,sizeof (*throttle_cfg)); +throttle_config_init (throttle_cfg); throttle_cfg->buckets[THROTTLE_BPS_TOTAL ].avg =qemu_opt_get_number (opts,"throttling.bps-total" ,0 ); throttle_cfg->buckets[THROTTLE_BPS_READ].avg =
@@ -2603,7 +2603,7 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd, goto out; } -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg); cfg.buckets[THROTTLE_BPS_TOTAL ].avg = bps; cfg.buckets[THROTTLE_BPS_READ ].avg = bps_rd; cfg.buckets[THROTTLE_BPS_WRITE ].avg = bps_wr;
include/qemu/throttle.h +2 -0
@@ -114,6 +114,8 @@ void throttle_config(ThrottleState *ts,void throttle_get_config (ThrottleState *ts,ThrottleConfig *cfg); +void throttle_config_init (ThrottleConfig *cfg); +/* usage */ bool throttle_schedule_timer(ThrottleState *ts, ThrottleTimers *tt,
tests/test-throttle.c +17 -11
@@ -34,6 +34,9 @@ static bool double_cmp(double x, double y)/* tests for single bucket operations */ static void test_leak_bucket (void ) { +throttle_config_init (&cfg); + bkt = cfg.buckets[THROTTLE_BPS_TOTAL ]; +/* set initial value */ bkt.avg =150 ; bkt.max =15 ;
@@ -63,6 +66,9 @@ static void test_compute_wait(void)int64_t wait;int64_t result; +throttle_config_init (&cfg); + bkt = cfg.buckets[THROTTLE_BPS_TOTAL ]; +/* no operation limit set */ bkt.avg =0 ; bkt.max =15 ;
@@ -232,17 +238,17 @@ static void test_enabled(void) {int i; -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);g_assert (!throttle_enabled (&cfg));for (i =0 ; i <BUCKETS_COUNT ; i++) { -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (false, i,150 );g_assert (throttle_enabled (&cfg)); }for (i =0 ; i <BUCKETS_COUNT ; i++) { -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (false, i,-150 );g_assert (!throttle_enabled (&cfg)); }
@@ -255,29 +261,29 @@ static void test_conflicts_for_one_set(bool is_max, int read, int write) { -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);g_assert (throttle_is_valid (&cfg,NULL ));set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 );g_assert (!throttle_is_valid (&cfg,NULL )); -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (is_max, total,1 );set_cfg_value (is_max, write,1 );g_assert (!throttle_is_valid (&cfg,NULL )); -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (is_max, total,1 );set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 );g_assert (!throttle_is_valid (&cfg,NULL )); -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (is_max, total,1 );g_assert (throttle_is_valid (&cfg,NULL )); -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (is_max, read,1 );set_cfg_value (is_max, write,1 );g_assert (throttle_is_valid (&cfg,NULL ));
@@ -314,7 +320,7 @@ static void test_is_valid_for_value(int value, bool should_be_valid)int is_max, index;for (is_max =0 ; is_max <2 ; is_max++) {for (index =0 ; index <BUCKETS_COUNT ; index++) { -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg);set_cfg_value (is_max, index, value);g_assert (throttle_is_valid (&cfg,NULL ) == should_be_valid); }
@@ -336,7 +342,7 @@ static void test_max_is_missing_limit(void)int i;for (i =0 ; i <BUCKETS_COUNT ; i++) { -memset (&cfg,0 ,sizeof (cfg)); +throttle_config_init (&cfg); cfg.buckets[i].max =100 ; cfg.buckets[i].avg =0 ;g_assert (!throttle_is_valid (&cfg,NULL ));
@@ -551,7 +557,7 @@ static void test_groups(void)g_assert (bdrv1->throttle_state == bdrv3->throttle_state);/* Setting the config of a group member affects the whole group */ -memset (&cfg1,0 ,sizeof (cfg1)); +throttle_config_init (&cfg1); cfg1.buckets[THROTTLE_BPS_READ ].avg =500000 ; cfg1.buckets[THROTTLE_BPS_WRITE ].avg =285000 ; cfg1.buckets[THROTTLE_OPS_READ ].avg =20000 ;
util/throttle.c +10 -0
@@ -171,10 +171,20 @@ void throttle_timers_attach_aio_context(ThrottleTimers *tt, tt->write_timer_cb, tt->timer_opaque); } +/* +* Initialize the ThrottleConfig structure to a valid state +* @cfg: the config to initialize +*/ +void throttle_config_init (ThrottleConfig *cfg) +{ +memset (cfg,0 ,sizeof (*cfg)); +} +/* To be called first on the ThrottleState */ void throttle_init (ThrottleState *ts) {memset (ts,0 ,sizeof (ThrottleState)); +throttle_config_init (&ts->cfg); }/* To be called first on the ThrottleTimers */
Signature · 1 line
2.7.0
PATCH 8/13 throttle: Add support for burst periods
In reply to Alberto GarciaThis patch adds support for burst periods to the throttling code. With this feature the user can keep performing bursts as defined by the LeakyBucket.max rate for a configurable period of time.
--- include/qemu/throttle.h | 41 +++++++++++++++++++++++---- util/throttle.c | 73 ++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 96 insertions(+), 18 deletions(-)
include/qemu/throttle.h +36 -5
@@ -2,7 +2,7 @@* QEMU throttling infrastructure * * Copyright (C) Nodalink, EURL. 2013-2014 -* Copyright (C) Igalia, S.L. 2015 +* Copyright (C) Igalia, S.L. 2015-2016 * * Authors: * Benoît Canet <[email protected]>
@@ -42,16 +42,47 @@ typedef enum { } BucketType;/* -* The max parameter of the leaky bucket throttling algorithm can be used to -* allow the guest to do bursts. -* The max value is a pool of I/O that the guest can use without being throttled -* at all. Throttling is triggered once this pool is empty. +* This module implements I/O limits using the leaky bucket +* algorithm. The code is independent of the I/O units, but it is +* currently used for bytes per second and operations per second. +* +* Three parameters can be set by the user: +* +* - avg: the desired I/O limits in units per second. +* - max: the limit during bursts, also in units per second. +* - burst_length: the maximum length of the burst period, in seconds. +* +* Here's how it works: +* +* - The bucket level (number of performed I/O units) is kept in +* bkt.level and leaks at a rate of bkt.avg units per second. +* +* - The size of the bucket is bkt.max * bkt.burst_length. Once the +* bucket is full no more I/O is performed until the bucket leaks +* again. This is what makes the I/O rate bkt.avg. +* +* - The bkt.avg rate does not apply until the bucket is full, +* allowing the user to do bursts until then. The I/O limit during +* bursts is bkt.max. To enforce this limit we keep an additional +* bucket in bkt.burst_length that leaks at a rate of bkt.max units +* per second. +* +* - Because of all of the above, the user can perform I/O at a +* maximum of bkt.max units per second for at most bkt.burst_length +* seconds in a row. After that the bucket will be full and the I/O +* rate will go down to bkt.avg. +* +* - Since the bucket always leaks at a rate of bkt.avg, this also +* determines how much the user needs to wait before being able to +* do bursts again. */ typedef struct LeakyBucket {double avg;/* average goal in units per second */ double max;/* leaky bucket max burst in units */ double level;/* bucket level in units */ +double burst_level;/* bucket level in units (for computing bursts) */ +unsigned burst_length;/* max length of the burst period, in seconds */ }LeakyBucket ;/* The following structure is used to configure a ThrottleState
util/throttle.c +60 -13
121 lines not on this page · raw message
Signature · 1 line
2.7.0
Re: [Qemu-devel] [PATCH 08/13] throttle: Add support for burst periods
Am 05.02.2016 um 11:59 hat Alberto Garcia geschrieben:
Quoted 55 lines
This patch adds support for burst periods to the throttling code. With this feature the user can keep performing bursts as defined by the LeakyBucket.max rate for a configurable period of time.
Signed-off-by Alberto Garcia <[email protected]>--- include/qemu/throttle.h | 41 +++++++++++++++++++++++---- util/throttle.c | 73 ++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 96 insertions(+), 18 deletions(-)include/qemu/throttle.h +25
-5@@ -2,7 +2,7 @@* QEMU throttling infrastructure * * Copyright (C) Nodalink, EURL. 2013-2014 -* Copyright (C) Igalia, S.L. 2015 +* Copyright (C) Igalia, S.L. 2015-2016 * * Authors: * Benoît Canet <[email protected]> @@ -42,16 +42,47 @@ typedef enum { } BucketType;/* -* The max parameter of the leaky bucket throttling algorithm can be used to -* allow the guest to do bursts. -* The max value is a pool of I/O that the guest can use without being throttled -* at all. Throttling is triggered once this pool is empty. +* This module implements I/O limits using the leaky bucket +* algorithm. The code is independent of the I/O units, but it is +* currently used for bytes per second and operations per second. +* +* Three parameters can be set by the user: +* +* - avg: the desired I/O limits in units per second. +* - max: the limit during bursts, also in units per second. +* - burst_length: the maximum length of the burst period, in seconds. +* +* Here's how it works: +* +* - The bucket level (number of performed I/O units) is kept in +* bkt.level and leaks at a rate of bkt.avg units per second. +* +* - The size of the bucket is bkt.max * bkt.burst_length. Once the +* bucket is full no more I/O is performed until the bucket leaks +* again. This is what makes the I/O rate bkt.avg. +* +* - The bkt.avg rate does not apply until the bucket is full, +* allowing the user to do bursts until then. The I/O limit during +* bursts is bkt.max. To enforce this limit we keep an additional +* bucket in bkt.burst_length that leaks at a rate of bkt.max units +* per second.
include/qemu/throttle.h typedef enum { · line 78
Quoted 10 lines
include/qemu/throttle.h +9
-0+* +* - Because of all of the above, the user can perform I/O at a +* maximum of bkt.max units per second for at most bkt.burst_length +* seconds in a row. After that the bucket will be full and the I/O +* rate will go down to bkt.avg. +* +* - Since the bucket always leaks at a rate of bkt.avg, this also +* determines how much the user needs to wait before being able to +* do bursts again. */
Good summary, thanks!
Quoted 37 lines
include/qemu/throttle.h +2
-0typedef struct LeakyBucket {double avg;/* average goal in units per second */ double max;/* leaky bucket max burst in units */ double level;/* bucket level in units */ +double burst_level;/* bucket level in units (for computing bursts) */ +unsigned burst_length;/* max length of the burst period, in seconds */ }LeakyBucket ;/* The following structure is used to configure a ThrottleState util/throttle.c +12
-1@@ -41,6 +41,14 @@ void throttle_leak_bucket(LeakyBucket *bkt, int64_t delta_ns)/* make the bucket leak */ bkt->level =MAX (bkt->level - leak,0 ); + +/* if we allow bursts for more than one second we also need to +* keep track of bkt->burst_level so the bkt->max goal per second +* is attained */ +if (bkt->burst_length >1 ) { + leak = (bkt->max * (double ) delta_ns) /NANOSECONDS_PER_SECOND ; + bkt->burst_level =MAX (bkt->burst_level - leak,0 ); + } }/* Calculate the time delta since last leak and make proportionals leaks @@ -91,13 +99,24 @@ int64_t throttle_compute_wait(LeakyBucket *bkt)return 0 ; } - extra = bkt->level - bkt->max; +/* If the bucket is full then we have to wait */ + extra = bkt->level - bkt->max * bkt->burst_length; +if (extra >0 ) { +return throttle_do_compute_wait (bkt->avg, extra);
util/throttle.c int64_t throttle_compute_wait(LeakyBucket *bkt) · line 113
Quoted 10 lines
util/throttle.c +7
-2+ } -if (extra <=0 ) { -return 0 ; +/* If the bucket is not full yet we have to make sure that we +* fulfill the goal of bkt->max units per second. */ +if (bkt->burst_length >1 ) { +/* We use 1/10 of the max value to smooth the throttling. +* See throttle_fix_bucket() for more details. */ + extra = bkt->burst_level - bkt->max /10 ;
I don't understand the connection between throttle_fix_bucket() and this.
throttle_fix_bucket() seems to set a default rate for bursts, which kind of makes sense to me (but what's the point when this is lower than the average rate?)
Here we work on a bkt->max that is either supplied by the user and should therefore be respected, or the default in throttle_fix_bucket() has already been applied.
What this line does is letting the request wait for more than would be strictly necessary, or in other words, for the last second before the bucket runs full, you only allow a tenth of the actual maximum rate.
I understand that having any burst at all helps, so the default that throttle_fix_bucket() sets used to make sense. I'm not so sure that it still makes sense with its max < avg setting (max used to be additional units on top of avg, now it's measured on its own). For the divison by 10 here, however, I'm still puzzled.
What am I missing?
util/throttle.c int64_t throttle_compute_wait(LeakyBucket *bkt) · line 120
Quoted 8 lines
util/throttle.c +4
-1+if (extra >0 ) { +return throttle_do_compute_wait (bkt->max, extra); + } } -return throttle_do_compute_wait (bkt->avg, extra); +return 0 ; }
Kevin
Re: [Qemu-devel] [PATCH 08/13] throttle: Add support for burst periods
Quoted 7 lines · Kevin Wolf
util/throttle.c int64_t throttle_compute_wait(LeakyBucket *bkt) · line 113
Quoted 6 lines
util/throttle.c +6
-0+/* If the bucket is not full yet we have to make sure that we +* fulfill the goal of bkt->max units per second. */ +if (bkt->burst_length >1 ) { +/* We use 1/10 of the max value to smooth the throttling. +* See throttle_fix_bucket() for more details. */ + extra = bkt->burst_level - bkt->max /10 ;
Quoted 6 lines
I don't understand the connection between
throttle_fix_bucket()and this.
throttle_fix_bucket()seems to set a default rate for bursts, which kind of makes sense to me (but what's the point when this is lower than the average rate?)
The point is to smoothen the throttling: if you request a max rate of 2000 operations per second, what you actually get is a rate of 200 operations per tenth of a second.
With the current code, if you set bkg->avg to 1000 but not bkt->max you'll get exactly this. I'm just applying the same logic to bursts.
Berto
PATCH 9/13 throttle: Add command-line settings to define the burst periods
In reply to Alberto GarciaThis patch adds all the throttling.*-max-length command-line parameters to define the length of the burst periods.
--- blockdev.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
blockdev.c +37 -0
@@ -414,6 +414,19 @@ static void extract_common_blockdev_options(QemuOpts *opts, int *bdrv_flags, throttle_cfg->buckets[THROTTLE_OPS_WRITE ].max =qemu_opt_get_number (opts,"throttling.iops-write-max" ,0 ); + throttle_cfg->buckets[THROTTLE_BPS_TOTAL ].burst_length = +qemu_opt_get_number (opts,"throttling.bps-total-max-length" ,1 ); + throttle_cfg->buckets[THROTTLE_BPS_READ ].burst_length = +qemu_opt_get_number (opts,"throttling.bps-read-max-length" ,1 ); + throttle_cfg->buckets[THROTTLE_BPS_WRITE ].burst_length = +qemu_opt_get_number (opts,"throttling.bps-write-max-length" ,1 ); + throttle_cfg->buckets[THROTTLE_OPS_TOTAL ].burst_length = +qemu_opt_get_number (opts,"throttling.iops-total-max-length" ,1 ); + throttle_cfg->buckets[THROTTLE_OPS_READ ].burst_length = +qemu_opt_get_number (opts,"throttling.iops-read-max-length" ,1 ); + throttle_cfg->buckets[THROTTLE_OPS_WRITE ].burst_length = +qemu_opt_get_number (opts,"throttling.iops-write-max-length" ,1 ); + throttle_cfg->op_size =qemu_opt_get_number (opts,"throttling.iops-size" ,0 );
@@ -4064,6 +4077,30 @@ QemuOptsList qemu_common_drive_opts = { .type = QEMU_OPT_NUMBER, .help = "total bytes write burst", },{ + .name = "throttling.iops-total-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-total-max burst period, in seconds", + },{ + .name = "throttling.iops-read-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-read-max burst period, in seconds", + },{ + .name = "throttling.iops-write-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-write-max burst period, in seconds", + },{ + .name = "throttling.bps-total-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-total-max burst period, in seconds", + },{ + .name = "throttling.bps-read-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-read-max burst period, in seconds", + },{ + .name = "throttling.bps-write-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-write-max burst period, in seconds", + },{ .name = "throttling.iops-size", .type = QEMU_OPT_NUMBER, .help = "when limiting by iops max size of an I/O in bytes",
Signature · 1 line
2.7.0
PATCH 10/13 qapi: Add burst length parameters to block_set_io_throttle
In reply to Alberto GarciaThis patch adds the new bps_*_max_length and iops_*_max_length parameters to the block_set_io_throttle command.
--- blockdev.c | 31 +++++++++++++++++++++++++++++++ hmp.c | 12 ++++++++++++ qapi/block-core.json | 51 +++++++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 88 insertions(+), 6 deletions(-)
blockdev.c +31 -0
@@ -2590,6 +2590,18 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd,int64_t iops_rd_max, bool has_iops_wr_max, int64_t iops_wr_max, + bool has_bps_max_length, + int64_t bps_max_length, + bool has_bps_rd_max_length, + int64_t bps_rd_max_length, + bool has_bps_wr_max_length, + int64_t bps_wr_max_length, + bool has_iops_max_length, + int64_t iops_max_length, + bool has_iops_rd_max_length, + int64_t iops_rd_max_length, + bool has_iops_wr_max_length, + int64_t iops_wr_max_length, bool has_iops_size, int64_t iops_size, bool has_group,
@@ -2644,6 +2656,25 @@ void qmp_block_set_io_throttle(const char *device, int64_t bps, int64_t bps_rd, cfg.buckets[THROTTLE_OPS_WRITE ].max = iops_wr_max; } +if (has_bps_max_length) { + cfg.buckets[THROTTLE_BPS_TOTAL ].burst_length = bps_max_length; + } +if (has_bps_rd_max_length) { + cfg.buckets[THROTTLE_BPS_READ ].burst_length = bps_rd_max_length; + } +if (has_bps_wr_max_length) { + cfg.buckets[THROTTLE_BPS_WRITE ].burst_length = bps_wr_max_length; + } +if (has_iops_max_length) { + cfg.buckets[THROTTLE_OPS_TOTAL ].burst_length = iops_max_length; + } +if (has_iops_rd_max_length) { + cfg.buckets[THROTTLE_OPS_READ ].burst_length = iops_rd_max_length; + } +if (has_iops_wr_max_length) { + cfg.buckets[THROTTLE_OPS_WRITE ].burst_length = iops_wr_max_length; + } +if (has_iops_size) { cfg.op_size = iops_size; }
hmp.c +12 -0
@@ -1411,6 +1411,18 @@ void hmp_block_set_io_throttle(Monitor *mon, const QDict *qdict)0 , false,0 , + false,/* no burst length via HMP */ +0 , + false, +0 , + false, +0 , + false, +0 , + false, +0 , + false, +0 , false,/* No default I/O size */ 0 , false,
qapi/block-core.json +45 -6
68 lines not on this page · raw message
Signature · 1 line
2.7.0
PATCH 11/13 qapi: Add burst length fields to BlockDeviceInfo
In reply to Alberto GarciaThis patch adds the new bps_*_max_length and iops_*_max_length parameters to the BlockDeviceInfo struct.
--- block/qapi.c | 20 ++++++++++++++++++++ qapi/block-core.json | 39 +++++++++++++++++++++++++++++++++------ 2 files changed, 53 insertions(+), 6 deletions(-)
block/qapi.c +20 -0
@@ -92,6 +92,26 @@ BlockDeviceInfo *bdrv_block_device_info(BlockDriverState *bs, Error **errp) info->has_iops_wr_max = cfg.buckets[THROTTLE_OPS_WRITE ].max; info->iops_wr_max = cfg.buckets[THROTTLE_OPS_WRITE ].max; + info->has_bps_max_length = info->has_bps_max; + info->bps_max_length = + cfg.buckets[THROTTLE_BPS_TOTAL ].burst_length; + info->has_bps_rd_max_length = info->has_bps_rd_max; + info->bps_rd_max_length = + cfg.buckets[THROTTLE_BPS_READ ].burst_length; + info->has_bps_wr_max_length = info->has_bps_wr_max; + info->bps_wr_max_length = + cfg.buckets[THROTTLE_BPS_WRITE ].burst_length; + + info->has_iops_max_length = info->has_iops_max; + info->iops_max_length = + cfg.buckets[THROTTLE_OPS_TOTAL ].burst_length; + info->has_iops_rd_max_length = info->has_iops_rd_max; + info->iops_rd_max_length = + cfg.buckets[THROTTLE_OPS_READ ].burst_length; + info->has_iops_wr_max_length = info->has_iops_wr_max; + info->iops_wr_max_length = + cfg.buckets[THROTTLE_OPS_WRITE ].burst_length; + info->has_iops_size = cfg.op_size; info->iops_size = cfg.op_size;
qapi/block-core.json +33 -6
@@ -273,17 +273,41 @@ # # @image: the info of image used (since: 1.6) # -# @bps_max: #optional total max in bytes (Since 1.7) +# @bps_max: #optional total throughput limit during bursts, +# in bytes (Since 1.7) # -# @bps_rd_max: #optional read max in bytes (Since 1.7) +# @bps_rd_max: #optional read throughput limit during bursts, +# in bytes (Since 1.7) # -# @bps_wr_max: #optional write max in bytes (Since 1.7) +# @bps_wr_max: #optional write throughput limit during bursts, +# in bytes (Since 1.7) # -# @iops_max: #optional total I/O operations max (Since 1.7) +# @iops_max: #optional total I/O operations per second during bursts, +# in bytes (Since 1.7) # -# @iops_rd_max: #optional read I/O operations max (Since 1.7) +# @iops_rd_max: #optional read I/O operations per second during bursts, +# in bytes (Since 1.7) # -# @iops_wr_max: #optional write I/O operations max (Since 1.7) +# @iops_wr_max: #optional write I/O operations per second during bursts, +# in bytes (Since 1.7) +# +# @bps_max_length: #optional maximum length of the @bps_max burst +# period, in seconds. (Since 2.6) +# +# @bps_rd_max_length: #optional maximum length of the @bps_rd_max +# burst period, in seconds. (Since 2.6) +# +# @bps_wr_max_length: #optional maximum length of the @bps_wr_max +# burst period, in seconds. (Since 2.6) +# +# @iops_max_length: #optional maximum length of the @iops burst +# period, in seconds. (Since 2.6) +# +# @iops_rd_max_length: #optional maximum length of the @iops_rd_max +# burst period, in seconds. (Since 2.6) +# +# @iops_wr_max_length: #optional maximum length of the @iops_wr_max +# burst period, in seconds. (Since 2.6) # # @iops_size: #optional an I/O size in bytes (Since 1.7) #
@@ -308,6 +332,9 @@ '*bps_max': 'int', '*bps_rd_max': 'int', '*bps_wr_max': 'int', '*iops_max': 'int', '*iops_rd_max': 'int', '*iops_wr_max': 'int', + '*bps_max_length': 'int', '*bps_rd_max_length': 'int', + '*bps_wr_max_length': 'int', '*iops_max_length': 'int', + '*iops_rd_max_length': 'int', '*iops_wr_max_length': 'int', '*iops_size': 'int', '*group': 'str', 'cache': 'BlockdevCacheInfo', 'write_threshold': 'int' } }
Signature · 1 line
2.7.0
PATCH 12/13 throttle: Check that burst_level leaks correctly
In reply to Alberto GarciaThis patch expands test_leak_bucket() to check that burst_level leaks correctly.
--- tests/test-throttle.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
tests/test-throttle.c +16 -0
@@ -59,6 +59,22 @@ static void test_leak_bucket(void)g_assert (bkt.avg ==150 );g_assert (bkt.max ==15 );g_assert (double_cmp (bkt.level,0 )); + +/* check that burst_level leaks correctly */ + bkt.burst_level =6 ; + bkt.max =250 ; + bkt.burst_length =2 ;/* otherwise burst_level will not leak */ +throttle_leak_bucket (&bkt,NANOSECONDS_PER_SECOND /100 ); +g_assert (double_cmp (bkt.burst_level,3.5 )); + +throttle_leak_bucket (&bkt,NANOSECONDS_PER_SECOND /100 ); +g_assert (double_cmp (bkt.burst_level,1 )); + +throttle_leak_bucket (&bkt,NANOSECONDS_PER_SECOND /100 ); +g_assert (double_cmp (bkt.burst_level,0 )); + +throttle_leak_bucket (&bkt,NANOSECONDS_PER_SECOND /100 ); +g_assert (double_cmp (bkt.burst_level,0 )); }static void test_compute_wait (void )
Signature · 1 line
2.7.0
PATCH 13/13 throttle: Test throttle_compute_wait() during bursts
In reply to Alberto GarciaThis test simulates an I/O burst for more than two seconds and checks that it works as expected.
--- tests/test-throttle.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
tests/test-throttle.c +22 -0
@@ -79,6 +79,7 @@ static void test_leak_bucket(void)static void test_compute_wait (void ) { +unsigned i;int64_t wait;int64_t result;
@@ -114,6 +115,27 @@ static void test_compute_wait(void)/* time required to do half an operation */ result = (int64_t )NANOSECONDS_PER_SECOND /150 /2 ;g_assert (wait == result); + +/* Perform I/O for 2.2 seconds at a rate of bkt.max */ + bkt.burst_length =2 ; + bkt.level =0 ; + bkt.avg =10 ; + bkt.max =200 ; +for (i =0 ; i <22 ; i++) { +double units = bkt.max /10 ; + bkt.level += units; + bkt.burst_level += units; +throttle_leak_bucket (&bkt,NANOSECONDS_PER_SECOND /10 ); + wait =throttle_compute_wait (&bkt); +g_assert (double_cmp (bkt.burst_level,0 )); +g_assert (double_cmp (bkt.level, (i +1 ) * (bkt.max - bkt.avg) /10 )); +/* We can do bursts for the 2 seconds we have configured in +* burst_length. We have 100 extra miliseconds of burst +* because bkt.level has been leaking during this time. +* After that, we have to wait. */ + result = i <21 ?0 :1.8 *NANOSECONDS_PER_SECOND ; +g_assert (wait == result); + } }/* functions to test ThrottleState initialization/destroy methods */
Signature · 1 line
2.7.0
Am 05.02.2016 um 11:59 hat Alberto Garcia geschrieben:
Quoted 50 lines
Hello everyone,
the current throttling code in QEMU allows limiting the I/O rate on block devices. Limits can be set in operations per second (IOPS) or bytes per second, allowing separate limits for read and write operations on both cases.
In its basic usage the user can set a limit of, say, 100 IOPS on a block device passing this option to -drive:
throttling.iops-total=100
In addition to that, QEMU can also allow the user to do I/O bursts exceeding that limit up to a configurable rate:
throttling.iops-total=100,throttling.iops-total-max=2000
With this, the user can do a burst of 2000 IOPS before they're throttled down to 100 IOPS. Then, after a sufficiently long period of unused I/O they will be able to do a burst again.
This patch series introduces the possibility to do bursts for longer period of times. A new setting called throttling.iops-total-max-length is used to define for how long bursts can be sustained.
So adding throttling.iops-total-max-length=60 to the previous configuration allows the user to do I/O at a rate of 2000 IOPS for 1 minute before going down to the base rate of 100 IOPS.
This is essentially the same as described in this AWS blog post:
https://aws.amazon.com/blogs/aws/new-ssd-backed-elastic-block-storage/
As described in the article, a use case for this feature is to allow better performance when booting the OS or restarting a service while keeping the average I/O limits lower the rest of the time.
Comments:
- There are 6 different settings for setting I/O limits: iops-total, iops-read, iops-write, bps-total, bps-read, bps-write. This series adds one new setting to set the length for each one of those.
I don't know if there's a good use case that requires such fine-grained control. It's of course also possible to make it simpler and add just one 'burst-length' setting that would apply for all cases, but the current solution is IMHO simple enough and consistent with the current API, and if we need to extend it later the result is probably going to be ugly.
Quoted 7 lines
- With this series we set "a maximum of X operations/second for a period of T seconds". If would also be possible to make it "a maximum of X operations/second up to a total of Y operations". It would be equivalent (Y = X * T) but I thought the current proposal makes a more clear API.
And I think that's all.
Congratulations, you found the unmaintained spot in the block layer! :-)
Anyway, the cover letter makes sense to me. And thanks for this great writeup! Can we turn it into documentation? Throttling seems to be quite underdocumented, and your explanation above let me understand for the first time what these *-max options actually were.
I'll have to see whether I can review the meat of this series next week, but for now:
Patches 1-7: Reviewed-by: Kevin Wolf <[email protected]>
Quoted 4 lines · Kevin Wolf
Anyway, the cover letter makes sense to me. And thanks for this great writeup! Can we turn it into documentation? Throttling seems to be quite underdocumented, and your explanation above let me understand for the first time what these *-max options actually were.
Yeah, I have it half written, it will go in the next version of the series.
Quoted 1 line
Patches 1-7: Reviewed-by: Kevin Wolf <[email protected]>
Thanks!
Berto
Quoted 5 lines · Alberto Garcia
- With this series we set "a maximum of X operations/second for a period of T seconds". If would also be possible to make it "a maximum of X operations/second up to a total of Y operations". It would be equivalent (Y = X * T) but I thought the current proposal makes a more clear API.
I find the diagram in the blog post clear. The QEMU code is a little harder to understand, it seems like there are too many variables and special cases. There are 4 core variables:
1. Refill rate (aka avg), e.g. 30 IOPS
2. Max bucket level (aka max * burst_length), e.g. 5.4 million IOPS
3. Burst rate (aka max), e.g. 3000 IOPS
4. Current bucket level
However, your patch to the core algorithm is small and makes sense to me. Maybe I just don't appreciate some of the subtleties that require the existing code to use more variables.
Looks good to me.
Quoted 7 lines · Stefan Hajnoczi
Quoted 5 lines · Alberto Garcia
- With this series we set "a maximum of X operations/second for a period of T seconds". If would also be possible to make it "a maximum of X operations/second up to a total of Y operations". It would be equivalent (Y = X * T) but I thought the current proposal makes a more clear API.
Quoted 8 lines
I find the diagram in the blog post clear. The QEMU code is a little harder to understand, it seems like there are too many variables and special cases. There are 4 core variables:
1. Refill rate (aka avg), e.g. 30 IOPS
2. Max bucket level (aka max *burst_length), e.g. 5.4 million IOPS
3. Burst rate (aka max), e.g. 3000 IOPS
4. Current bucket level
The blog post uses the token bucket algorithm but QEMU uses the leaky bucket. They're equivalent, but one is the reverse of the other: in the former the bucket is full and the performed I/O is the water that leaks from the bucket; in the latter the bucket is empty and the performed I/O is the water that goes into the bucket.
We have these 3 variables:
- Leak rate (avg), 30 IOPS
- Fill-up rate (max). 3000 IOPS
- Fill-up time (burst_length), 30 minutes
We allow I/O until the bucket is full and we disallow it afterwards. The bucket leaks at 30 IOPS so that's the reason why we get that I/O rate. We need to keep track of the bucket level (bkt->level) and make it leak at 30 IOPS.
With this series we also want to limit the amount of water per second that goes into the bucket before it's full. So how do we make sure that we don't allow more than 3000 IOPS when adding water to the bucket? We actually have the same problem as in the previous paragraph, except that the I/O rate here is different. That's why we need bkt->burst_level that leaks at 3000 IOPS.
It's easier to see if you think about it as a second bucket on top of the main one that leaks at 3000 IOPS.
Berto
Quoted 12 lines · Alberto Garcia
Quoted 11 lines · Stefan Hajnoczi
Quoted 5 lines · Alberto Garcia
- With this series we set "a maximum of X operations/second for a period of T seconds". If would also be possible to make it "a maximum of X operations/second up to a total of Y operations". It would be equivalent (Y = X * T) but I thought the current proposal makes a more clear API.
I find the diagram in the blog post clear. The QEMU code is a little harder to understand, it seems like there are too many variables and special cases. There are 4 core variables:
Quoted 7 lines
Quoted 4 lines
1. Refill rate (aka avg), e.g. 30 IOPS
2. Max bucket level (aka max *burst_length), e.g. 5.4 million IOPS
3. Burst rate (aka max), e.g. 3000 IOPS
4. Current bucket levelThe blog post uses the token bucket algorithm but QEMU uses the leaky bucket. They're equivalent, but one is the reverse of the other:
Yes, I find the token bucket approach clearer :).
Stefan