migratioin/ram: leave RAMBlock->bmap blank on allocating
PATCH 1/1 migratioin/ram: leave RAMBlock->bmap blank on allocating
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().
Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().
--- migration/ram.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
migration/ram.c +1 -7
@@ -3151,12 +3151,7 @@ static int ram_state_init(RAMState **rsp)qemu_mutex_init (&(*rsp)->src_page_req_mutex);QSIMPLEQ_INIT (&(*rsp)->src_page_requests); -/* -* Count the total number of pages used by ram blocks not including any -* gaps due to alignment or unplugs. -*/ - (*rsp)->migration_dirty_pages =ram_bytes_total () >>TARGET_PAGE_BITS ; - + (*rsp)->migration_dirty_pages =0 ;ram_state_reset (*rsp);return 0 ;
@@ -3172,7 +3167,6 @@ static void ram_list_init_bitmaps(void)RAMBLOCK_FOREACH_NOT_IGNORED (block) { pages = block->max_length >>TARGET_PAGE_BITS ; block->bmap =bitmap_new (pages); -bitmap_set (block->bmap,0 , pages);if (migrate_postcopy_ram ()) { block->unsentmap =bitmap_new (pages);bitmap_set (block->unsentmap,0 , pages);
Signature · 1 line
2.19.1
Quoted 7 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Quoted 6 lines
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().Signed-off-by Wei Yang <[email protected]>
I've looked at this for a while, and I think it's OK, so
Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
Dave
Quoted 33 lines
--- migration/ram.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)migration/ram.c +1
-7@@ -3151,12 +3151,7 @@ static int ram_state_init(RAMState **rsp)qemu_mutex_init (&(*rsp)->src_page_req_mutex);QSIMPLEQ_INIT (&(*rsp)->src_page_requests); -/* -* Count the total number of pages used by ram blocks not including any -* gaps due to alignment or unplugs. -*/ - (*rsp)->migration_dirty_pages =ram_bytes_total () >>TARGET_PAGE_BITS ; - + (*rsp)->migration_dirty_pages =0 ;ram_state_reset (*rsp);return 0 ;@@ -3172,7 +3167,6 @@ static void ram_list_init_bitmaps(void)RAMBLOCK_FOREACH_NOT_IGNORED (block) { pages = block->max_length >>TARGET_PAGE_BITS ; block->bmap =bitmap_new (pages); -bitmap_set (block->bmap,0 , pages);if (migrate_postcopy_ram ()) { block->unsentmap =bitmap_new (pages);bitmap_set (block->unsentmap,0 , pages);Signature · 1 line
2.19.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().
Quoted 8 lines
Quoted 1 line
Signed-off-by Wei Yang <[email protected]>I've looked at this for a while, and I think it's OK, so
Reviewed-by Dr. David Alan Gilbert <[email protected]>Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:
https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html
Regards,
Signature · 1 line
Peter Xu
Quoted 18 lines · Peter Xu
Quoted 17 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Quoted 6 lines
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().Signed-off-by Wei Yang <[email protected]>I've looked at this for a while, and I think it's OK, so
Quoted 9 lines
Quoted 4 lines
Reviewed-by Dr. David Alan Gilbert <[email protected]>Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:
I have done some migration cases, like migrate a linux guest through tcp.
Other cases suggested to do?
Quoted 6 lines
https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html
Regards,
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 24 lines · Wei Yang
Quoted 23 lines · Peter Xu
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Quoted 6 lines
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().Signed-off-by Wei Yang <[email protected]>I've looked at this for a while, and I think it's OK, so
Reviewed-by Dr. David Alan Gilbert <[email protected]>Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
Quoted 6 lines
Quoted 4 lines
I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:
I have done some migration cases, like migrate a linux guest through tcp.
When did you start the migration? Have you tried to migrate during some workload?
Quoted 2 lines
Other cases suggested to do?
Could you also help answer the question I raised below in the link?
Thanks,
Quoted 1 line
Signature · 1 line
Peter Xu
Quoted 30 lines · Peter Xu
Quoted 29 lines · Wei Yang
Quoted 27 lines · Peter Xu
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().Quoted 8 lines
Quoted 1 line
Signed-off-by Wei Yang <[email protected]>I've looked at this for a while, and I think it's OK, so
Reviewed-by Dr. David Alan Gilbert <[email protected]>Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:
Quoted 5 lines
Quoted 1 line
I have done some migration cases, like migrate a linux guest through tcp.
When did you start the migration? Have you tried to migrate during some workload?
I tried kernel build in guest.
Quoted 9 lines
Quoted 2 lines
Other cases suggested to do?
Could you also help answer the question I raised below in the link?
Thanks,
Quoted 1 line
I took a look into this link, hope my understanding is correct.
You concern is this thread/patch is based on one prerequisite -- dirty all the bitmap at start.
My answer is we already did it in ram_block_add() for each RAMBlock. And then the bitmap is synced by migration_bitmap_sync_precopy() from ram_list.dirty_memory to RAMBlock.bmap.
Quoted 2 lines
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 49 lines · Wei Yang
Quoted 30 lines · Peter Xu
Quoted 29 lines · Wei Yang
Quoted 27 lines · Peter Xu
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:
During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in
cpu_physical_memory_sync_dirty_bitmap().Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.
Leaving
RAMBlock->bmapblank on allocating would have a side effect onmigration_dirty_pages, since it is calculated from the result ofcpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to setmigration_dirty_pagesto 0 inram_state_init().Quoted 8 lines
Quoted 1 line
Signed-off-by Wei Yang <[email protected]>I've looked at this for a while, and I think it's OK, so
Reviewed-by Dr. David Alan Gilbert <[email protected]>Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.
I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:
Quoted 5 lines
Quoted 1 line
I have done some migration cases, like migrate a linux guest through tcp.
When did you start the migration? Have you tried to migrate during some workload?
I tried kernel build in guest.
Quoted 9 lines
Quoted 2 lines
Other cases suggested to do?
Could you also help answer the question I raised below in the link?
Thanks,
Quoted 1 line
Quoted 8 lines
I took a look into this link, hope my understanding is correct.
You concern is this thread/patch is based on one prerequisite -- dirty all the bitmap at start.
My answer is we already did it in
ram_block_add()for each RAMBlock. And then the bitmap is synced bymigration_bitmap_sync_precopy()from ram_list.dirty_memory to RAMBlock.bmap.
Ah I see, thanks for the pointer. Then I would agree it's fine.
I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.
And even if you want to remove this, I still have two suggestions:
(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap
(2) imho you can move "migration_dirty_pages = 0" into ram_list_init_bitmaps() too to let them be together
Signature · 1 line
Peter Xu
Quoted 13 lines · Peter Xu
Ah I see, thanks for the pointer. Then I would agree it's fine.
I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.
I am not sure about the background here, so I didn't make a change at this place.
Quoted 10 lines
And even if you want to remove this, I still have two suggestions:
(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap
(2) imho you can move "
migration_dirty_pages= 0" intoram_list_init_bitmaps()too to let them be together
I will address these two comments and send v2.
Thanks.
Quoted 2 lines
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 15 lines · Wei Yang
Quoted 13 lines · Peter Xu
Ah I see, thanks for the pointer. Then I would agree it's fine.
I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.
Quoted 13 lines
I am not sure about the background here, so I didn't make a change at this place.
Quoted 10 lines
And even if you want to remove this, I still have two suggestions:
(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap
(2) imho you can move "
migration_dirty_pages= 0" intoram_list_init_bitmaps()too to let them be together
I took a look into this one.
ram_list_init_bitmaps() setup bitmap for each RAMBlock, while ram_state_init() setup RAMState. Since migration_dirty_pages belongs to RAMState, it maybe more proper to leave it at the original place.
Do you feel good about this?
Quoted 11 lines
I will address these two comments and send v2.
Thanks.
Quoted 2 lines
Signature · 1 line
Peter XuSignature · 2 lines
Wei Yang Help you, Help me
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 30 lines · Wei Yang
Quoted 15 lines · Wei Yang
Quoted 13 lines · Peter Xu
Ah I see, thanks for the pointer. Then I would agree it's fine.
I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.
Quoted 13 lines
I am not sure about the background here, so I didn't make a change at this place.
Quoted 10 lines
And even if you want to remove this, I still have two suggestions:
(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap
(2) imho you can move "
migration_dirty_pages= 0" intoram_list_init_bitmaps()too to let them be together
Quoted 7 lines
I took a look into this one.
ram_list_init_bitmaps()setup bitmap for each RAMBlock, whileram_state_init()setup RAMState. Sincemigration_dirty_pagesbelongs to RAMState, it maybe more proper to leave it at the original place.Do you feel good about this?
Yes it's ok to me. Thanks,
Signature · 1 line
Peter Xu