Skip to contentSkip to conversation

migratioin/ram: leave RAMBlock->bmap blank on allocating

PATCH Superseded Wei Yang 3 participants 10 messages May 7, 2019 – Jun 3, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH 1/1 migratioin/ram: leave RAMBlock->bmap blank on allocating

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

---
 migration/ram.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
migration/ram.c +1 -7
index 95c51109d2..417874707d 100644
@@ -3151,12 +3151,7 @@ static int ram_state_init(RAMState **rsp)
     qemu_mutex_init(&(*rsp)->src_page_req_mutex);
     QSIMPLEQ_INIT(&(*rsp)->src_page_requests);
 
-    /*
-     * Count the total number of pages used by ram blocks not including any
-     * gaps due to alignment or unplugs.
-     */
-    (*rsp)->migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
-
+    (*rsp)->migration_dirty_pages = 0;
     ram_state_reset(*rsp);
 
     return 0;
@@ -3172,7 +3167,6 @@ static void ram_list_init_bitmaps(void)
         RAMBLOCK_FOREACH_NOT_IGNORED(block) {
             pages = block->max_length >> TARGET_PAGE_BITS;
             block->bmap = bitmap_new(pages);
-            bitmap_set(block->bmap, 0, pages);
             if (migrate_postcopy_ram()) {
                 block->unsentmap = bitmap_new(pages);
                 bitmap_set(block->unsentmap, 0, pages);
Signature · 1 line
2.19.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 7 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Quoted 6 lines

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

I've looked at this for a while, and I think it's OK, so

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

Dave

Quoted 33 lines
---
 migration/ram.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
migration/ram.c +1 -7
index 95c51109d2..417874707d 100644
@@ -3151,12 +3151,7 @@ static int ram_state_init(RAMState **rsp)
     qemu_mutex_init(&(*rsp)->src_page_req_mutex);
     QSIMPLEQ_INIT(&(*rsp)->src_page_requests);
 
-    /*
-     * Count the total number of pages used by ram blocks not including any
-     * gaps due to alignment or unplugs.
-     */
-    (*rsp)->migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
-
+    (*rsp)->migration_dirty_pages = 0;
     ram_state_reset(*rsp);
 
     return 0;
@@ -3172,7 +3167,6 @@ static void ram_list_init_bitmaps(void)
         RAMBLOCK_FOREACH_NOT_IGNORED(block) {
             pages = block->max_length >> TARGET_PAGE_BITS;
             block->bmap = bitmap_new(pages);
-            bitmap_set(block->bmap, 0, pages);
             if (migrate_postcopy_ram()) {
                 block->unsentmap = bitmap_new(pages);
                 bitmap_set(block->unsentmap, 0, pages);
Signature · 1 line
2.19.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#2 lore raw
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

Quoted 8 lines
Quoted 1 line

I've looked at this for a while, and I think it's OK, so

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html

Regards,

Signature · 1 line
Peter Xu
author
#3 lore raw
Quoted 18 lines · Peter Xu
Quoted 17 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Quoted 6 lines

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

I've looked at this for a while, and I think it's OK, so

Quoted 9 lines
Quoted 4 lines

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:

I have done some migration cases, like migrate a linux guest through tcp.

Other cases suggested to do?

Quoted 6 lines

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html

Regards,

Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me
#4 lore raw
Quoted 24 lines · Wei Yang
Quoted 23 lines · Peter Xu
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Quoted 6 lines

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

I've looked at this for a while, and I think it's OK, so

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

Quoted 6 lines
Quoted 4 lines

I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:

I have done some migration cases, like migrate a linux guest through tcp.

When did you start the migration? Have you tried to migrate during some workload?

Quoted 2 lines

Other cases suggested to do?

Could you also help answer the question I raised below in the link?

Thanks,

Quoted 1 line
Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html

Signature · 1 line
Peter Xu
author
#5 lore raw
Quoted 30 lines · Peter Xu
Quoted 29 lines · Wei Yang
Quoted 27 lines · Peter Xu
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

Quoted 8 lines
Quoted 1 line

I've looked at this for a while, and I think it's OK, so

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:

Quoted 5 lines
Quoted 1 line

I have done some migration cases, like migrate a linux guest through tcp.

When did you start the migration? Have you tried to migrate during some workload?

I tried kernel build in guest.

Quoted 9 lines
Quoted 2 lines

Other cases suggested to do?

Could you also help answer the question I raised below in the link?

Thanks,

Quoted 1 line
Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html

I took a look into this link, hope my understanding is correct.

You concern is this thread/patch is based on one prerequisite -- dirty all the bitmap at start.

My answer is we already did it in ram_block_add() for each RAMBlock. And then the bitmap is synced by migration_bitmap_sync_precopy() from ram_list.dirty_memory to RAMBlock.bmap.

Quoted 2 lines
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me
#6 lore raw
Quoted 49 lines · Wei Yang
Quoted 30 lines · Peter Xu
Quoted 29 lines · Wei Yang
Quoted 27 lines · Peter Xu
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Wei Yang ([email protected]) wrote:

During migration, we would sync bitmap from ram_list.dirty_memory to RAMBlock.bmap in cpu_physical_memory_sync_dirty_bitmap().

Since we set RAMBlock.bmap and ram_list.dirty_memory both to all 1, this means at the first round this sync is meaningless and is a duplicated work.

Leaving RAMBlock->bmap blank on allocating would have a side effect on migration_dirty_pages, since it is calculated from the result of cpu_physical_memory_sync_dirty_bitmap(). To keep it right, we need to set migration_dirty_pages to 0 in ram_state_init().

Quoted 8 lines
Quoted 1 line

I've looked at this for a while, and I think it's OK, so

Peter, Juan: Can you just see if there's arny reason this would be bad, but I think it's actually more sensible than what we have.

I really suspect it will work in all cases... Wei, have you done any test (or better, thorough tests) with this change? My reasoning of why we should need the bitmap all set is here:

Quoted 5 lines
Quoted 1 line

I have done some migration cases, like migrate a linux guest through tcp.

When did you start the migration? Have you tried to migrate during some workload?

I tried kernel build in guest.

Quoted 9 lines
Quoted 2 lines

Other cases suggested to do?

Could you also help answer the question I raised below in the link?

Thanks,

Quoted 1 line
Quoted 1 line

https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg07361.html

Quoted 8 lines

I took a look into this link, hope my understanding is correct.

You concern is this thread/patch is based on one prerequisite -- dirty all the bitmap at start.

My answer is we already did it in ram_block_add() for each RAMBlock. And then the bitmap is synced by migration_bitmap_sync_precopy() from ram_list.dirty_memory to RAMBlock.bmap.

Ah I see, thanks for the pointer. Then I would agree it's fine.

I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.

And even if you want to remove this, I still have two suggestions:

(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap

(2) imho you can move "migration_dirty_pages = 0" into ram_list_init_bitmaps() too to let them be together

Signature · 1 line
Peter Xu
author
#7 lore raw
Quoted 13 lines · Peter Xu

Ah I see, thanks for the pointer. Then I would agree it's fine.

I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.

I am not sure about the background here, so I didn't make a change at this place.

Quoted 10 lines

And even if you want to remove this, I still have two suggestions:

(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap

(2) imho you can move "migration_dirty_pages = 0" into ram_list_init_bitmaps() too to let them be together

I will address these two comments and send v2.

Thanks.

Quoted 2 lines
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me
author
#8 lore raw
Quoted 15 lines · Wei Yang
Quoted 13 lines · Peter Xu

Ah I see, thanks for the pointer. Then I would agree it's fine.

I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.

Quoted 13 lines

I am not sure about the background here, so I didn't make a change at this place.

Quoted 10 lines

And even if you want to remove this, I still have two suggestions:

(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap

(2) imho you can move "migration_dirty_pages = 0" into ram_list_init_bitmaps() too to let them be together

I took a look into this one.

ram_list_init_bitmaps() setup bitmap for each RAMBlock, while ram_state_init() setup RAMState. Since migration_dirty_pages belongs to RAMState, it maybe more proper to leave it at the original place.

Do you feel good about this?

Quoted 11 lines

I will address these two comments and send v2.

Thanks.

Quoted 2 lines
Signature · 1 line
Peter Xu
Signature · 2 lines
Wei Yang
Help you, Help me
Signature · 2 lines
Wei Yang
Help you, Help me
#9 lore raw
Quoted 30 lines · Wei Yang
Quoted 15 lines · Wei Yang
Quoted 13 lines · Peter Xu

Ah I see, thanks for the pointer. Then I would agree it's fine.

I'm not an expert of TCG - I'm curious on why all those three dirty bitmaps need to be set at the very beginning. IIUC at least the VGA bitmap should not require that (so IMHO we should be fine to have all zeros with VGA bitmap for ramblocks, and we only set them when the guest touches them). Migration bitmap should be special somehow but I don't know much on TCG/TLB part I'd confess so I can't say. In other words, if migration is the only one that requires this "all-1" initialization then IMHO we may consider to remove the other part rather than here in migration because that's what we'd better to be sure with.

Quoted 13 lines

I am not sure about the background here, so I didn't make a change at this place.

Quoted 10 lines

And even if you want to remove this, I still have two suggestions:

(1) proper comment here above bmap on the above fact that although bmap is not set here but it's actually set somewhere else because we'll sooner or later copy all 1s from the ramblock bitmap

(2) imho you can move "migration_dirty_pages = 0" into ram_list_init_bitmaps() too to let them be together

Quoted 7 lines

I took a look into this one.

ram_list_init_bitmaps() setup bitmap for each RAMBlock, while ram_state_init() setup RAMState. Since migration_dirty_pages belongs to RAMState, it maybe more proper to leave it at the original place.

Do you feel good about this?

Yes it's ok to me. Thanks,

Signature · 1 line
Peter Xu