migration: Rate limit inside host pages
PATCH 1/1 migration: Rate limit inside host pages
From: "Dr. David Alan Gilbert" <[email protected]>
When using hugepages, rate limiting is necessary within each huge page, since a 1G huge page can take a significant time to send, so you end up with bursty behaviour.
--- migration/migration.c | 57 ++++++++++++++++++++++++------------------ migration/migration.h | 1 + migration/ram.c | 2 ++ migration/trace-events | 4 +-- 4 files changed, 37 insertions(+), 27 deletions(-)
migration/migration.c +32 -25
@@ -3224,6 +3224,37 @@ void migration_consume_urgent_request(void)qemu_sem_wait (&migrate_get_current ()->rate_limit_sem); } +/* Returns true if the rate limiting was broken by an urgent request */ +bool migration_rate_limit (void ) +{ +int64_t now =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +MigrationState *s =migrate_get_current (); + +bool urgent = false; +migration_update_counters (s, now); +if (qemu_file_rate_limit (s->to_dst_file)) { +/* +* Wait for a delay to do rate limiting OR +* something urgent to post the semaphore. +*/ +int ms = s->iteration_start_time +BUFFER_DELAY - now; +trace_migration_rate_limit_pre (ms); +if (qemu_sem_timedwait (&s->rate_limit_sem, ms) ==0 ) { +/* +* We were woken by one or more urgent things but +* the timedwait will have consumed one of them. +* The service routine for the urgent wake will dec +* the semaphore itself for each item it consumes, +* so add this one we just eat back. +*/ +qemu_sem_post (&s->rate_limit_sem); + urgent = true; + } +trace_migration_rate_limit_post (urgent); + } +return urgent; +} +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel.
@@ -3290,8 +3321,6 @@ static void *migration_thread(void *opaque)trace_migration_thread_setup_complete ();while (migration_is_active (s)) { -int64_t current_time; -if (urgent || !qemu_file_rate_limit (s->to_dst_file)) {MigIterateState iter_state =migration_iteration_run (s);if (iter_state ==MIG_ITERATE_SKIP ) {
@@ -3318,29 +3347,7 @@ static void *migration_thread(void *opaque)update_iteration_initial_status (s); } - current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); - -migration_update_counters (s, current_time); - - urgent = false; -if (qemu_file_rate_limit (s->to_dst_file)) { -/* Wait for a delay to do rate limiting OR -* something urgent to post the semaphore. -*/ -int ms = s->iteration_start_time +BUFFER_DELAY - current_time; -trace_migration_thread_ratelimit_pre (ms); -if (qemu_sem_timedwait (&s->rate_limit_sem, ms) ==0 ) { -/* We were worken by one or more urgent things but -* the timedwait will have consumed one of them. -* The service routine for the urgent wake will dec -* the semaphore itself for each item it consumes, -* so add this one we just eat back. -*/ -qemu_sem_post (&s->rate_limit_sem); - urgent = true; - } -trace_migration_thread_ratelimit_post (urgent); - } + urgent =migration_rate_limit (); }trace_migration_thread_after_loop ();
migration/migration.h +1 -0
@@ -341,5 +341,6 @@ int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);void migration_make_urgent_request (void );void migration_consume_urgent_request (void ); +bool migration_rate_limit (void );#endif
migration/ram.c +2 -0
@@ -2616,6 +2616,8 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss, pages += tmppages; pss->page++; +/* Allow rate limiting to happen in the middle of huge pages */ +migration_rate_limit (); }while ((pss->page & (pagesize_bits -1 )) &&offset_in_ramblock (pss->block, pss->page <<TARGET_PAGE_BITS ));
migration/trace-events +2 -2
@@ -138,12 +138,12 @@ migrate_send_rp_recv_bitmap(char *name, int64_t size) "block '%s' size 0x%"PRIi6 migration_completion_file_err(void) "" migration_completion_postcopy_end(void) "" migration_completion_postcopy_end_after_complete(void) "" +migration_rate_limit_pre(int ms) "%d ms" +migration_rate_limit_post(int urgent) "urgent: %d" migration_return_path_end_before(void) "" migration_return_path_end_after(int rp_error) "%d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" -migration_thread_ratelimit_pre(int ms) "%d ms" -migration_thread_ratelimit_post(int urgent) "urgent: %d" migration_thread_setup_complete(void) "" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) ""
Signature · 1 line
2.23.0
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When using hugepages, rate limiting is necessary within each huge page, since a 1G huge page can take a significant time to send, so you end up with bursty behaviour.
Fixes 4c011c37ecb3 ("postcopy: Send whole huge pages")Reported-by Lin Ma <[email protected]>Signed-off-by Dr. David Alan Gilbert <[email protected]>---
I can agree that rate limit needs to be done for huge pages.
migration/ram.c static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss, · line 2623
Quoted 13 lines
migration/ram.c +2
-0@@ -2616,6 +2616,8 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss, pages += tmppages; pss->page++; +/* Allow rate limiting to happen in the middle of huge pages */ +migration_rate_limit (); }while ((pss->page & (pagesize_bits -1 )) &&offset_in_ramblock (pss->block, pss->page <<TARGET_PAGE_BITS ));
But is doing the rate limit for each page, no? Even when not using huge pages.
Not that it should be a big issue (performance wise). Have you done any meassuremnet?
Later, Juan.
Quoted 38 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When using hugepages, rate limiting is necessary within each huge page, since a 1G huge page can take a significant time to send, so you end up with bursty behaviour.
Fixes 4c011c37ecb3 ("postcopy: Send whole huge pages")Reported-by Lin Ma <[email protected]>Signed-off-by Dr. David Alan Gilbert <[email protected]>--- migration/migration.c | 57 ++++++++++++++++++++++++------------------ migration/migration.h | 1 + migration/ram.c | 2 ++ migration/trace-events | 4 +-- 4 files changed, 37 insertions(+), 27 deletions(-)migration/migration.c +14
-0@@ -3224,6 +3224,37 @@ void migration_consume_urgent_request(void)qemu_sem_wait (&migrate_get_current ()->rate_limit_sem); } +/* Returns true if the rate limiting was broken by an urgent request */ +bool migration_rate_limit (void ) +{ +int64_t now =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +MigrationState *s =migrate_get_current (); + +bool urgent = false; +migration_update_counters (s, now); +if (qemu_file_rate_limit (s->to_dst_file)) { +/* +* Wait for a delay to do rate limiting OR +* something urgent to post the semaphore. +*/ +int ms = s->iteration_start_time +BUFFER_DELAY - now;
migration/migration.c void migration_consume_urgent_request(void) · line 3250
Quoted 10 lines
migration/migration.c +10
-0+trace_migration_rate_limit_pre (ms); +if (qemu_sem_timedwait (&s->rate_limit_sem, ms) ==0 ) { +/* +* We were woken by one or more urgent things but +* the timedwait will have consumed one of them. +* The service routine for the urgent wake will dec +* the semaphore itself for each item it consumes, +* so add this one we just eat back. +*/ +qemu_sem_post (&s->rate_limit_sem);
I remembered I've commented around this when it was first introduced on whether we can avoid this post(). IMHO we can if with something like an eventfd, so when we queue the page we write the eventfd to 1, here we poll() on the eventfd with the same timeout, then clear it after the poll no matter what. When unqueue, we can probably simply do nothing. I'm not sure about Windows or other OS, though..
Anyway this patch is not changing that part but to fix huge page issue, so that's another story for sure.
Thanks,
Signature · 1 line
Peter Xu
Quoted 22 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When using hugepages, rate limiting is necessary within each huge page, since a 1G huge page can take a significant time to send, so you end up with bursty behaviour.
Fixes 4c011c37ecb3 ("postcopy: Send whole huge pages")Reported-by Lin Ma <[email protected]>Signed-off-by Dr. David Alan Gilbert <[email protected]>---
Reviewed-by Juan Quintela <[email protected]>I can agree that rate limit needs to be done for huge pages.
Quoted 6 lines
migration/ram.c +0
-0@@ -2616,6 +2616,8 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss,
Quoted 10 lines
migration/ram.c static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss, · line 2623
Quoted 7 lines
migration/ram.c +2
-0pages += tmppages; pss->page++; +/* Allow rate limiting to happen in the middle of huge pages */ +migration_rate_limit (); }while ((pss->page & (pagesize_bits -1 )) &&offset_in_ramblock (pss->block, pss->page <<TARGET_PAGE_BITS ));But is doing the rate limit for each page, no? Even when not using huge pages.
Right.
Quoted 2 lines
Not that it should be a big issue (performance wise). Have you done any meassuremnet?
I've just given it a quick run; it still seems to be hitting ~9.5Gbps on my 10Gbps interface; so it doesn't seem to be the limit on that.
Dave
Quoted 3 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK