kvm: Take into account the unaligned section size when preparing bitmap
PATCH 1/1 kvm: Take into account the unaligned section size when preparing bitmap
The kernel KVM_CLEAR_DIRTY_LOG interface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).
If the size is unaligned, let's go through the slow path to manipulate a temp bitmap for the interface so that we won't bother with those unaligned bits at the end of bitmap.
I don't think this can happen in practice since the upper layer would provide us with the alignment guarantee. I'm not sure if kvm-all could rely on it. And this patch is mainly intended to address correctness of the specific algorithm used inside kvm_log_clear_one_slot().
--- accel/kvm/kvm-all.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
accel/kvm/kvm-all.c +5 -2
@@ -747,7 +747,7 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,assert (bmap_start %BITS_PER_LONG ==0 );/* We should never do log_clear before log_sync */ assert (mem->dirty_bmap); -if (start_delta) { +if (start_delta || bmap_npages - size / psize) {/* Slow path - we need to manipulate a temp bitmap */ bmap_clear =bitmap_new (bmap_npages); bitmap_copy_with_src_offset(bmap_clear, mem->dirty_bmap,
@@ -760,7 +760,10 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,bitmap_clear (bmap_clear,0 , start_delta); d.dirty_bitmap = bmap_clear; }else { -/* Fast path - start address aligns well with BITS_PER_LONG */ +/* +* Fast path - both start and size align well with BITS_PER_LONG +* (or the end of memory slot) +*/ d.dirty_bitmap = mem->dirty_bmap +BIT_WORD (bmap_start); }
Signature · 1 line
2.19.1
Hi, Zenghui,
Quoted 6 lines · Zenghui Yu
The kernel
KVM_CLEAR_DIRTY_LOGinterface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).
Thanks for the patch, though my understanding is that this is not a bug.
Please have a look at kvm_memslot_init_dirty_bitmap() where we'll allocate the dirty bitmap to be aligned to 8 bytes (assuming that's the possible max of the value sizeof(unsigned long)). That exactly covers 64 pages.
So here as long as start_delta==0 (so the value of "bmap_npages - size / psize" won't really matter a lot, imho), then we'll definitely have KVMSlot.dirty_bmap long enough to cover the range we'd like to clear.
Note that the size of KVMSlot.dirty_bmap can be bigger than the actually size of the kvm memslot, however since kvm_memslot_init_dirty_bitmap() initialized it to all zero so the extra bits will always be zero for the whole lifecycle of the vm/bitmap.
Thanks,
Quoted 35 lines
Signed-off-by Zenghui Yu <[email protected]>--- accel/kvm/kvm-all.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)accel/kvm/kvm-all.c +5
-2@@ -747,7 +747,7 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,assert (bmap_start %BITS_PER_LONG ==0 );/* We should never do log_clear before log_sync */ assert (mem->dirty_bmap); -if (start_delta) { +if (start_delta || bmap_npages - size / psize) {/* Slow path - we need to manipulate a temp bitmap */ bmap_clear =bitmap_new (bmap_npages); bitmap_copy_with_src_offset(bmap_clear, mem->dirty_bmap,@@ -760,7 +760,10 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,bitmap_clear (bmap_clear,0 , start_delta); d.dirty_bitmap = bmap_clear; }else { -/* Fast path - start address aligns well with BITS_PER_LONG */ +/* +* Fast path - both start and size align well with BITS_PER_LONG +* (or the end of memory slot) +*/ d.dirty_bitmap = mem->dirty_bmap +BIT_WORD (bmap_start); }Signature · 1 line
2.19.1
Signature · 1 line
Peter Xu
Hi Peter,
Thanks for having a look at it.
Quoted 12 lines · Peter Xu
Hi, Zenghui,
Quoted 6 lines · Zenghui Yu
The kernel
KVM_CLEAR_DIRTY_LOGinterface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).Thanks for the patch, though my understanding is that this is not a bug.
Quoted 7 lines
Please have a look at
kvm_memslot_init_dirty_bitmap()where we'll allocate the dirty bitmap to be aligned to 8 bytes (assuming that's the possible max of the value sizeof(unsigned long)). That exactly covers 64 pages.So here as long as start_delta==0 (so the value of "
bmap_npages- size / psize" won't really matter a lot, imho), then we'll definitely have KVMSlot.dirty_bmap long enough to cover the range we'd like to clear.
I agree. But actually I'm not saying that KVMSlot.dirty_bmap is not long enough. What I was having in mind is something like:
// psize = qemu_real_host_page_size;
// slot.start_addr = 0;
// slot.memory_size = 64 * psize;
kvm_log_clear_one_slot(slot, as, 0 * psize, 32 * psize); --> [1]
kvm_log_clear_one_slot(slot, as, 32 * psize, 32 * psize); --> [2]So the @size is not aligned with 64 pages. Before this patch, we'll clear dirty status for all pages(0-63) through [1]. It looks to me that this violates the caller's expectation since we only want to clear pages(0-31).
As I said, I don't think this will happen in practice -- the migration code should always provide us with a 64-page aligned section (right?). I'm just thinking about the correctness of the specific algorithm used by kvm_log_clear_one_slot().
Or maybe I had missed some other points obvious ;-) ?
Thanks,
Zenghui
Quoted 43 lines
Note that the size of KVMSlot.dirty_bmap can be bigger than the actually size of the kvm memslot, however since
kvm_memslot_init_dirty_bitmap()initialized it to all zero so the extra bits will always be zero for the whole lifecycle of the vm/bitmap.Thanks,
Quoted 35 lines
Signed-off-by Zenghui Yu <[email protected]>--- accel/kvm/kvm-all.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)accel/kvm/kvm-all.c +5
-2@@ -747,7 +747,7 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,assert (bmap_start %BITS_PER_LONG ==0 );/* We should never do log_clear before log_sync */ assert (mem->dirty_bmap); -if (start_delta) { +if (start_delta || bmap_npages - size / psize) {/* Slow path - we need to manipulate a temp bitmap */ bmap_clear =bitmap_new (bmap_npages); bitmap_copy_with_src_offset(bmap_clear, mem->dirty_bmap,@@ -760,7 +760,10 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,bitmap_clear (bmap_clear,0 , start_delta); d.dirty_bitmap = bmap_clear; }else { -/* Fast path - start address aligns well with BITS_PER_LONG */ +/* +* Fast path - both start and size align well with BITS_PER_LONG +* (or the end of memory slot) +*/ d.dirty_bitmap = mem->dirty_bmap +BIT_WORD (bmap_start); }Signature · 1 line
2.19.1
Peter Xu
Acked-byQuoted 35 lines · Zenghui Yu
Hi Peter,
Thanks for having a look at it.
Quoted 12 lines · Peter Xu
Hi, Zenghui,
Quoted 6 lines · Zenghui Yu
The kernel
KVM_CLEAR_DIRTY_LOGinterface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).Thanks for the patch, though my understanding is that this is not a bug.
Quoted 7 lines
Please have a look at
kvm_memslot_init_dirty_bitmap()where we'll allocate the dirty bitmap to be aligned to 8 bytes (assuming that's the possible max of the value sizeof(unsigned long)). That exactly covers 64 pages.So here as long as start_delta==0 (so the value of "
bmap_npages- size / psize" won't really matter a lot, imho), then we'll definitely have KVMSlot.dirty_bmap long enough to cover the range we'd like to clear.I agree. But actually I'm not saying that KVMSlot.dirty_bmap is not long enough. What I was having in mind is something like:
// psize = qemu_real_host_page_size; // slot.start_addr = 0; // slot.memory_size = 64 * psize; kvm_log_clear_one_slot(slot, as, 0 * psize, 32 * psize); --> [1] kvm_log_clear_one_slot(slot, as, 32 * psize, 32 * psize); --> [2]
Quoted 4 lines
So the @size is not aligned with 64 pages. Before this patch, we'll clear dirty status for all pages(0-63) through [1]. It looks to me that this violates the caller's expectation since we only want to clear pages(0-31).
Now I see; I think you're right. :)
Quoted 3 lines
As I said, I don't think this will happen in practice -- the migration code should always provide us with a 64-page aligned section (right?).
Yes, migration is the major consumer, and that should be guaranteed indeed, see CLEAR_BITMAP_SHIFT_MIN.
Not sure about VGA - that should try to do log clear even without migration, but I guess that satisfies the 64-page alignment too, since it's not a huge number (256KB). The VGA buffer size could be related to screen resolution, then N*1024*768 could still guarantee a safe use of the fast path.
Quoted 2 lines
I'm just thinking about the correctness of the specific algorithm used by
kvm_log_clear_one_slot().
Yeah, then I think it's okay to have this, just in case someday we'll hit it.
(It would be nicer if above example could be squashed into commit message)
Thanks,
Signature · 1 line
Peter Xu
Quoted 55 lines · Peter Xu
Quoted 35 lines · Zenghui Yu
Hi Peter,
Thanks for having a look at it.
Quoted 12 lines · Peter Xu
Hi, Zenghui,
Quoted 6 lines · Zenghui Yu
The kernel
KVM_CLEAR_DIRTY_LOGinterface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).Thanks for the patch, though my understanding is that this is not a bug.
Quoted 7 lines
Please have a look at
kvm_memslot_init_dirty_bitmap()where we'll allocate the dirty bitmap to be aligned to 8 bytes (assuming that's the possible max of the value sizeof(unsigned long)). That exactly covers 64 pages.So here as long as start_delta==0 (so the value of "
bmap_npages- size / psize" won't really matter a lot, imho), then we'll definitely have KVMSlot.dirty_bmap long enough to cover the range we'd like to clear.I agree. But actually I'm not saying that KVMSlot.dirty_bmap is not long enough. What I was having in mind is something like:
// psize = qemu_real_host_page_size; // slot.start_addr = 0; // slot.memory_size = 64 * psize; kvm_log_clear_one_slot(slot, as, 0 * psize, 32 * psize); --> [1] kvm_log_clear_one_slot(slot, as, 32 * psize, 32 * psize); --> [2]Quoted 4 lines
So the @size is not aligned with 64 pages. Before this patch, we'll clear dirty status for all pages(0-63) through [1]. It looks to me that this violates the caller's expectation since we only want to clear pages(0-31).
Now I see; I think you're right. :)
Quoted 3 lines
As I said, I don't think this will happen in practice -- the migration code should always provide us with a 64-page aligned section (right?).
Yes, migration is the major consumer, and that should be guaranteed indeed, see
CLEAR_BITMAP_SHIFT_MIN.Not sure about VGA - that should try to do log clear even without migration, but I guess that satisfies the 64-page alignment too, since it's not a huge number (256KB). The VGA buffer size could be related to screen resolution, then N*1024*768 could still guarantee a safe use of the fast path.
Quoted 6 lines
Quoted 2 lines
I'm just thinking about the correctness of the specific algorithm used by
kvm_log_clear_one_slot().Yeah, then I think it's okay to have this, just in case someday we'll hit it.
Acked-by Peter Xu <[email protected]>
Thanks for that.
Quoted 1 line
(It would be nicer if above example could be squashed into commit message)
I'll squash it if v2 is needed.
Thanks,
Zenghui
Hi,
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Thanks,
Keqian
Quoted 96 lines · Zenghui Yu
Hi Peter,
Thanks for having a look at it.
Quoted 12 lines · Peter Xu
Hi, Zenghui,
Quoted 6 lines · Zenghui Yu
The kernel
KVM_CLEAR_DIRTY_LOGinterface has align requirement on both the start and the size of the given range of pages. We have been careful to handle the unaligned cases when performing CLEAR on one slot. But it seems that we forget to take the unaligned *size* case into account when preparing bitmap for the interface, and we may end up clearing dirty status for pages outside of [start, start + size).Thanks for the patch, though my understanding is that this is not a bug.
Quoted 7 lines
Please have a look at
kvm_memslot_init_dirty_bitmap()where we'll allocate the dirty bitmap to be aligned to 8 bytes (assuming that's the possible max of the value sizeof(unsigned long)). That exactly covers 64 pages.So here as long as start_delta==0 (so the value of "
bmap_npages- size / psize" won't really matter a lot, imho), then we'll definitely have KVMSlot.dirty_bmap long enough to cover the range we'd like to clear.I agree. But actually I'm not saying that KVMSlot.dirty_bmap is not long enough. What I was having in mind is something like:
// psize = qemu_real_host_page_size; // slot.start_addr = 0; // slot.memory_size = 64 * psize; kvm_log_clear_one_slot(slot, as, 0 * psize, 32 * psize); --> [1] kvm_log_clear_one_slot(slot, as, 32 * psize, 32 * psize); --> [2]So the @size is not aligned with 64 pages. Before this patch, we'll clear dirty status for all pages(0-63) through [1]. It looks to me that this violates the caller's expectation since we only want to clear pages(0-31).
As I said, I don't think this will happen in practice -- the migration code should always provide us with a 64-page aligned section (right?). I'm just thinking about the correctness of the specific algorithm used by
kvm_log_clear_one_slot().Or maybe I had missed some other points obvious ;-) ?
Thanks,
ZenghuiQuoted 33 lines
Note that the size of KVMSlot.dirty_bmap can be bigger than the actually size of the kvm memslot, however since
kvm_memslot_init_dirty_bitmap()initialized it to all zero so the extra bits will always be zero for the whole lifecycle of the vm/bitmap.Thanks,
Quoted 26 lines
Signed-off-by Zenghui Yu <[email protected]>--- accel/kvm/kvm-all.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)accel/kvm/kvm-all.c +3
-2@@ -747,7 +747,7 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,assert (bmap_start %BITS_PER_LONG ==0 );/* We should never do log_clear before log_sync */ assert (mem->dirty_bmap); -if (start_delta) { +if (start_delta || bmap_npages - size / psize) {/* Slow path - we need to manipulate a temp bitmap */ bmap_clear =bitmap_new (bmap_npages); bitmap_copy_with_src_offset(bmap_clear, mem->dirty_bmap,@@ -760,7 +760,10 @@ static int kvm_log_clear_one_slot(KVMSlot *mem, int as_id, uint64_t start,bitmap_clear (bmap_clear,0 , start_delta); d.dirty_bitmap = bmap_clear; }else { -/* Fast path - start address aligns well with BITS_PER_LONG */ +/* +* Fast path - both start and size align well with BITS_PER_LONG Quoted 9 lines
Quoted 8 lines
accel/kvm/kvm-all.c +2
-0+* (or the end of memory slot) +*/ d.dirty_bitmap = mem->dirty_bmap +BIT_WORD (bmap_start); } --2.19.1
.
Keqian,
Quoted 4 lines · zhukeqian
Hi,
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If start_delta != 0, kvm_log_clear_one_slot() should already go the slow path.
Thanks,
Signature · 1 line
Peter Xu
Quoted 5 lines · Peter Xu
Keqian,
Quoted 2 lines · zhukeqian
Hi,
Quoted 9 lines
Quoted 2 lines
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
start_delta /= psize;
If start is not PAGE aligned, then start_delta is not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.
I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
Maybe I miss something?
Keqian.
Quoted 10 lines · zhukeqian
Quoted 7 lines · Peter Xu
Keqian,
Quoted 4 lines · zhukeqian
Hi,
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Ah, I really don't anticipate this to happen but ... The question is what's the expectation of the caller if a non-PAGE aligned @start is given. Should we clear dirty state for the page which covers the unaligned @start? Or just skip it?
Quoted 10 lines
Quoted 7 lines
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
Quoted 7 lines
start_delta/= psize;If start is not PAGE aligned, then
start_deltais not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
I don't object to do so (though haven't checked the code carefully). I also don't think there is much we can do given that the caller doesn't care about which *exact pages* to clear.
Thanks,
Zenghui
Quoted 20 lines · zhukeqian
Quoted 5 lines · Peter Xu
Keqian,
Quoted 2 lines · zhukeqian
Hi,
Quoted 9 lines
Quoted 2 lines
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
Quoted 7 lines
start_delta/= psize;If start is not PAGE aligned, then
start_deltais not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
Callers should be with TARGET_PAGE_SIZE aligned on the size, so at least x86_64 should be pretty safe since host/guest page sizes match.
Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
Signature · 1 line
Peter Xu
Quoted 29 lines · Peter Xu
Quoted 20 lines · zhukeqian
Quoted 5 lines · Peter Xu
Keqian,
Quoted 2 lines · zhukeqian
Hi,
Quoted 9 lines
Quoted 2 lines
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
Quoted 7 lines
start_delta/= psize;If start is not PAGE aligned, then
start_deltais not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
Quoted 8 lines
Callers should be with
TARGET_PAGE_SIZEaligned on the size, so at leastx86_64should be pretty safe since host/guest page sizes match.Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
Hi,
Indeed, we handle target_page_size aligned @start and @size in general. Maybe we'd better add explicit function comments about alignment requirement, and explicit alignment assert on @start and @size?
Keqian.
Quoted 39 lines · zhukeqian
Quoted 29 lines · Peter Xu
Quoted 20 lines · zhukeqian
Quoted 5 lines · Peter Xu
Keqian,
Quoted 2 lines · zhukeqian
Hi,
Quoted 9 lines
Quoted 2 lines
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
Quoted 7 lines
start_delta/= psize;If start is not PAGE aligned, then
start_deltais not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
Quoted 8 lines
Callers should be with
TARGET_PAGE_SIZEaligned on the size, so at leastx86_64should be pretty safe since host/guest page sizes match.Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
Quoted 5 lines
Hi,
Indeed, we handle
target_page_sizealigned @start and @size in general. Maybe we'd better add explicit function comments about alignment requirement, and explicit alignment assert on @start and @size?
Yes we can, but I think it's not strongly necessary. As Zenghui pointed out, the callers of memory_region_clear_dirty_bitmap() should always be aware of the fact that dirty bitmap is always page size based.
OTOH I'm more worried on the other question on how we handle guest psize != host psize case for migration now...
Thanks,
Signature · 1 line
Peter Xu
Quoted 40 lines · Peter Xu
Quoted 39 lines · zhukeqian
Quoted 37 lines · Peter Xu
Quoted 20 lines · zhukeqian
Quoted 5 lines · Peter Xu
Keqian,
Quoted 2 lines · zhukeqian
Hi,
Quoted 9 lines
Quoted 2 lines
I see that if start or size is not PAGE aligned, it also clears areas which beyond caller's expectation, so do we also need to consider this?
Could you elaborate?
If
start_delta!= 0,kvm_log_clear_one_slot()should already go the slow path.Thanks,
Hi Peter,
Quoted 7 lines
start_delta/= psize;If start is not PAGE aligned, then
start_deltais not PAGE aligned. so I think the above code will implicitly extend our start to be PAGE aligned.I suggest that we should shrink the start and (start + size) to be PAGE aligned at beginning of this function.
Callers should be with
TARGET_PAGE_SIZEaligned on the size, so at leastx86_64should be pretty safe since host/guest page sizes match.Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
Quoted 6 lines
Quoted 5 lines
Hi,
Indeed, we handle
target_page_sizealigned @start and @size in general. Maybe we'd better add explicit function comments about alignment requirement, and explicit alignment assert on @start and @size?
Hi Peter,
Quoted 3 lines
Yes we can, but I think it's not strongly necessary. As Zenghui pointed out, the callers of
memory_region_clear_dirty_bitmap()should always be aware of the fact that dirty bitmap is always page size based.
Agree.
Quoted 3 lines
OTOH I'm more worried on the other question on how we handle guest psize != host psize case for migration now...
I think it does not matter when guest_psize != host_psize, as we only need to interact with stage2 page tables during migration. Stage2 is enough to tracking guest dirty memory, and even if guest close stage1, we also can do a successful migration.
Please point out if I misunderstood what you meant.
Thanks,
Keqian
Quoted 3 lines
Thanks,
On Mon, Dec 14, 2020 at 10:14:11AM +0800, zhukeqian wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
[1]
Quoted 1 line
Hi Peter,
Keqian,
Quoted 5 lines
Quoted 2 lines
OTOH I'm more worried on the other question on how we handle guest psize != host psize case for migration now...
I think it does not matter when
guest_psize!=host_psize, as we only need to interact with stage2 page tables during migration. Stage2 is enough to tracking guest dirty memory, and even if guest close stage1, we also can do a successful migration.
I don't know why 2-stage matters here, since I believe KVM can track dirty pages either using two dimentional paging or shadowing, however it's always done in host small page size. The question I'm confused is there seems to have a size mismatch between qemu migration and what kvm does [1]. For example, how migration works on ARM64 where host has psize==4K while guest has psize=64K.
Thanks,
Signature · 1 line
Peter Xu
Quoted 21 lines · Peter Xu
On Mon, Dec 14, 2020 at 10:14:11AM +0800, zhukeqian wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
[1]
Quoted 1 line
Hi Peter,
Keqian,
Quoted 5 lines
Quoted 2 lines
OTOH I'm more worried on the other question on how we handle guest psize != host psize case for migration now...
I think it does not matter when
guest_psize!=host_psize, as we only need to interact with stage2 page tables during migration. Stage2 is enough to tracking guest dirty memory, and even if guest close stage1, we also can do a successful migration.
Quoted 6 lines
I don't know why 2-stage matters here, since I believe KVM can track dirty pages either using two dimentional paging or shadowing, however it's always done in host small page size. The question I'm confused is there seems to have a size mismatch between qemu migration and what kvm does [1]. For example, how migration works on ARM64 where host has psize==4K while guest has psize=64K.
Hi Peter,
OK, I got it ;-) Do you mean qemu_real_host_page_size != TARGET_PAGE_SIZE?
After my analysis, I see that when qemu_real_host_page_size != TARGET_PAGE_SIZE, there are some problems indeed. I have send out some patches, please check whether they solve this problem, thanks!
Keqian.
Quoted 2 lines
Thanks,
Hi Keqian, Peter,
Quoted 29 lines · zhukeqian
Quoted 21 lines · Peter Xu
On Mon, Dec 14, 2020 at 10:14:11AM +0800, zhukeqian wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Though indeed I must confess I don't know how it worked in general when host page size != target page size, at least for migration. For example, I believe kvm dirty logging is host page size based, though migration should be migrating pages in guest page size granule when it spots a dirty bit set.
[1]
Quoted 1 line
Hi Peter,
Keqian,
Quoted 5 lines
Quoted 2 lines
OTOH I'm more worried on the other question on how we handle guest psize != host psize case for migration now...
I think it does not matter when
guest_psize!=host_psize, as we only need to interact with stage2 page tables during migration. Stage2 is enough to tracking guest dirty memory, and even if guest close stage1, we also can do a successful migration.Quoted 6 lines
I don't know why 2-stage matters here, since I believe KVM can track dirty pages either using two dimentional paging or shadowing, however it's always done in host small page size. The question I'm confused is there seems to have a size mismatch between qemu migration and what kvm does [1]. For example, how migration works on ARM64 where host has psize==4K while guest has psize=64K.
Quoted 6 lines
Hi Peter,
OK, I got it ;-) Do you mean
qemu_real_host_page_size!=TARGET_PAGE_SIZE?
After my analysis, I see that whenqemu_real_host_page_size!=TARGET_PAGE_SIZE, there are some problems indeed. I have send out some patches, please check whether they solve this problem, thanks!
Now I see what your concern is :) Thanks both for the explanation and the further fix!
Zenghui