migration: Bug fixes (prepare for preempt-full)
cover migration: Bug fixes (prepare for preempt-full)
This patchset does bug fixes that I found when testing preempt-full.
Patch 1 should fix a possible deadloop when I hit when testing the preempt-full code. I didn't verify it because it's so hard to trigger, but the logic should be explained in the patch.
Patch 2 fixes a race condition I can easily trigger with the latest preempt-full code when running with recovery+tls test. The bug hides quite deep and took time to debug. Fundamentally it's about qemufile API, I hope someday we can have something better than that but still so far there's no strong enough reason to rework the whole thing.
Patch 3-4 are two patches to disable either postcopy or preempt mode only for xbzrle/compression.
Patch 5 is something nice to have to optimize the bitmap ops.
The last two patches are actually part of my preempt-full RFC series.
I picked these patches out explicitly from preempt-full series, because at least patches 1-4 fix real bugs in current code base, so they should get more focus.
Thanks,
Peter Xu (5):
migration: Fix possible deadloop of ram save process
migration: Fix race on qemu_file_shutdown()
migration: Disallow xbzrle with postcopy
migration: Disallow postcopy preempt to be used with compress
migration: Use non-atomic ops for clear log bitmap
include/exec/ram_addr.h | 11 +++++----- include/exec/ramblock.h | 3 +++ include/qemu/bitmap.h | 1 + migration/migration.c | 16 +++++++++++++++ migration/qemu-file.c | 27 ++++++++++++++++++++++--- migration/ram.c | 16 +++++++++++---- util/bitmap.c | 45 +++++++++++++++++++++++++++++++++++++++++ 7 files changed, 107 insertions(+), 12 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/5 migration: Fix possible deadloop of ram save process
When starting ram saving procedure (especially at the completion phase), always set last_seen_block to non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".
Then we'll guarantee both last_seen_block and pss.block will be valid always before the loop starts.
See the comment in the code for some details.
--- migration/ram.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
migration/ram.c +12 -4
@@ -2546,14 +2546,22 @@ static int ram_find_and_save_block(RAMState *rs)return pages; } +/* +* Always keep last_seen_block/last_page valid during this procedure, +* because find_dirty_block() relies on these values (e.g., we compare +* last_seen_block with pss.block to see whether we searched all the +* ramblocks) to detect the completion of migration. Having NULL value +* of last_seen_block can conditionally cause below loop to run forever. +*/ +if (!rs->last_seen_block) { + rs->last_seen_block =QLIST_FIRST_RCU (&ram_list.blocks); + rs->last_page =0 ; + } + pss.block = rs->last_seen_block; pss.page = rs->last_page; pss.complete_round = false; -if (!pss.block) { - pss.block =QLIST_FIRST_RCU (&ram_list.blocks); - } - do { again = true; found = get_queued_page(rs, &pss);
Signature · 1 line
2.32.0
Re: [PATCH 1/5] migration: Fix possible deadloop of ram save process
Quoted 10 lines · * Peter Xu ([email protected]) wrote:
When starting ram saving procedure (especially at the completion phase), always set
last_seen_blockto non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".Then we'll guarantee both
last_seen_blockand pss.block will be valid always before the loop starts.See the comment in the code for some details.
Signed-off-by Peter Xu <[email protected]>
Yeh I guess it can currently only happen during restart?
Quoted 38 lines
--- migration/ram.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)migration/ram.c +12
-4@@ -2546,14 +2546,22 @@ static int ram_find_and_save_block(RAMState *rs)return pages; } +/* +* Always keep last_seen_block/last_page valid during this procedure, +* because find_dirty_block() relies on these values (e.g., we compare +* last_seen_block with pss.block to see whether we searched all the +* ramblocks) to detect the completion of migration. Having NULL value +* of last_seen_block can conditionally cause below loop to run forever. +*/ +if (!rs->last_seen_block) { + rs->last_seen_block =QLIST_FIRST_RCU (&ram_list.blocks); + rs->last_page =0 ; + } + pss.block = rs->last_seen_block; pss.page = rs->last_page; pss.complete_round = false; -if (!pss.block) { - pss.block =QLIST_FIRST_RCU (&ram_list.blocks); - } - do { again = true; found = get_queued_page(rs, &pss);Signature · 1 line
2.32.0
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 1/5] migration: Fix possible deadloop of ram save process
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Peter Xu ([email protected]) wrote:
When starting ram saving procedure (especially at the completion phase), always set
last_seen_blockto non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".
Quoted 8 lines
Quoted 6 lines
Then we'll guarantee both
last_seen_blockand pss.block will be valid always before the loop starts.See the comment in the code for some details.
Signed-off-by Peter Xu <[email protected]>Yeh I guess it can currently only happen during restart?
There're only two places to clear last_seen_block:
ram_state_reset[2683] rs->last_seen_block = NULL;
ram_postcopy_send_discard_bitmap[2876] rs->last_seen_block = NULL;
Where for the reset case:
ram_state_init[2994] ram_state_reset(*rsp);
ram_state_resume_prepare[3110] ram_state_reset(rs);
ram_save_iterate[3271] ram_state_reset(rs);
So I think it can at least happen in two places, either (1) postcopy just started (assume when postcopy starts accidentally when all dirty pages were migrated?), or (2) postcopy recover from failure.
In my case I triggered this deadloop when I was debugging the other bug fixed by the next patch where it was postcopy recovery (on tls), but only once.. So currently I'm still not 100% sure whether this is the same problem, but logically it could trigger.
I also remember I used to hit very rare deadloops before too, maybe they're the same thing because I did test recovery a lot.
Quoted 2 lines
Reviewed-by Dr. David Alan Gilbert <[email protected]>
Thanks!
Signature · 1 line
Peter Xu
Re: [PATCH 1/5] migration: Fix possible deadloop of ram save process
Quoted 22 lines · * Peter Xu ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Peter Xu ([email protected]) wrote:
When starting ram saving procedure (especially at the completion phase), always set
last_seen_blockto non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".Quoted 8 lines
Quoted 6 lines
Then we'll guarantee both
last_seen_blockand pss.block will be valid always before the loop starts.See the comment in the code for some details.
Signed-off-by Peter Xu <[email protected]>Yeh I guess it can currently only happen during restart?
There're only two places to clear
last_seen_block:ram_state_reset[2683]
rs->last_seen_block= NULL;
ram_postcopy_send_discard_bitmap[2876]rs->last_seen_block= NULL;Where for the reset case:
Quoted 7 lines
ram_state_init[2994]
ram_state_reset(*rsp);
ram_state_resume_prepare[3110]ram_state_reset(rs);
ram_save_iterate[3271]ram_state_reset(rs);So I think it can at least happen in two places, either (1) postcopy just started (assume when postcopy starts accidentally when all dirty pages were migrated?), or (2) postcopy recover from failure.
Oh, (1) is a more general problem then; yeh.
Quoted 7 lines
In my case I triggered this deadloop when I was debugging the other bug fixed by the next patch where it was postcopy recovery (on tls), but only once.. So currently I'm still not 100% sure whether this is the same problem, but logically it could trigger.
I also remember I used to hit very rare deadloops before too, maybe they're the same thing because I did test recovery a lot.
Note; 'deadlock' not 'deadloop'.
Dave
Quoted 8 lines
Quoted 2 lines
Reviewed-by Dr. David Alan Gilbert <[email protected]>Thanks!
Signature · 1 line
Peter Xu
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 1/5] migration: Fix possible deadloop of ram save process
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 22 lines · * Peter Xu ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Peter Xu ([email protected]) wrote:
When starting ram saving procedure (especially at the completion phase), always set
last_seen_blockto non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".Quoted 8 lines
Quoted 6 lines
Then we'll guarantee both
last_seen_blockand pss.block will be valid always before the loop starts.See the comment in the code for some details.
Signed-off-by Peter Xu <[email protected]>Yeh I guess it can currently only happen during restart?
There're only two places to clear
last_seen_block:ram_state_reset[2683]
rs->last_seen_block= NULL;
ram_postcopy_send_discard_bitmap[2876]rs->last_seen_block= NULL;Where for the reset case:
Quoted 7 lines
ram_state_init[2994]
ram_state_reset(*rsp);
ram_state_resume_prepare[3110]ram_state_reset(rs);
ram_save_iterate[3271]ram_state_reset(rs);So I think it can at least happen in two places, either (1) postcopy just started (assume when postcopy starts accidentally when all dirty pages were migrated?), or (2) postcopy recover from failure.
Oh, (1) is a more general problem then; yeh.
Quoted 5 lines
In my case I triggered this deadloop when I was debugging the other bug fixed by the next patch where it was postcopy recovery (on tls), but only once.. So currently I'm still not 100% sure whether this is the same problem, but logically it could trigger.
Quoted 4 lines
Quoted 2 lines
I also remember I used to hit very rare deadloops before too, maybe they're the same thing because I did test recovery a lot.
Note; 'deadlock' not 'deadloop'.
(Oops I somehow forgot there's still this series pending..)
Here it's not about a lock, or maybe I should add a space ("dead loop")?
Signature · 1 line
Peter Xu
Re: [PATCH 1/5] migration: Fix possible deadloop of ram save process
Quoted 39 lines · * Peter Xu ([email protected]) wrote:
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 22 lines · * Peter Xu ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Peter Xu ([email protected]) wrote:
When starting ram saving procedure (especially at the completion phase), always set
last_seen_blockto non-NULL to make sure we can always correctly detect the case where "we've migrated all the dirty pages".Quoted 8 lines
Quoted 6 lines
Then we'll guarantee both
last_seen_blockand pss.block will be valid always before the loop starts.See the comment in the code for some details.
Signed-off-by Peter Xu <[email protected]>Yeh I guess it can currently only happen during restart?
There're only two places to clear
last_seen_block:ram_state_reset[2683]
rs->last_seen_block= NULL;
ram_postcopy_send_discard_bitmap[2876]rs->last_seen_block= NULL;Where for the reset case:
Quoted 7 lines
ram_state_init[2994]
ram_state_reset(*rsp);
ram_state_resume_prepare[3110]ram_state_reset(rs);
ram_save_iterate[3271]ram_state_reset(rs);So I think it can at least happen in two places, either (1) postcopy just started (assume when postcopy starts accidentally when all dirty pages were migrated?), or (2) postcopy recover from failure.
Oh, (1) is a more general problem then; yeh.
Quoted 5 lines
In my case I triggered this deadloop when I was debugging the other bug fixed by the next patch where it was postcopy recovery (on tls), but only once.. So currently I'm still not 100% sure whether this is the same problem, but logically it could trigger.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I also remember I used to hit very rare deadloops before too, maybe they're the same thing because I did test recovery a lot.
Note; 'deadlock' not 'deadloop'.
(Oops I somehow forgot there's still this series pending..)
Here it's not about a lock, or maybe I should add a space ("dead loop")?
So the normal phrases I'm used to are:
'deadlock' - two threads waiting for each other 'livelock' - two threads spinning for each other
Dave
Quoted 3 lines
Signature · 1 line
Peter Xu
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 2/5 migration: Fix race on qemu_file_shutdown()
In reply to Peter XuIn qemu_file_shutdown(), there's a possible race if with current order of operation. There're two major things to do:
(1) Do real shutdown() (e.g. shutdown() syscall on socket)
(2) Update qemufile's last_error
We must do (2) before (1) otherwise there can be a race condition like:
page receiver other thread
------------- ------------
qemu_get_buffer()
do shutdown()
returns 0 (buffer all zero) (meanwhile we didn't check this retcode)
try to detect IO error
last_error==NULL, IO okay
install ALL-ZERO page
set last_error
--> guest crash!
To fix this, we can also check retval of qemu_get_buffer(), but not all APIs can be properly checked and ultimately we still need to go back to qemu_file_get_error(). E.g. qemu_get_byte() doesn't return error.
Maybe some day a rework of qemufile API is really needed, but for now keep using qemu_file_get_error() and fix it by not allowing that race condition to happen. Here shutdown() is indeed special because the last_error was emulated. For real -EIO errors it'll always be set when e.g. sendmsg() error triggers so we won't miss those ones, only shutdown() is a bit tricky here.
--- migration/qemu-file.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)
migration/qemu-file.c +24 -3
@@ -79,6 +79,30 @@ int qemu_file_shutdown(QEMUFile *f)int ret =0 ; f->shutdown = true; + +/* +* We must set qemufile error before the real shutdown(), otherwise +* there can be a race window where we thought IO all went though +* (because last_error==NULL) but actually IO has already stopped. +* +* If without correct ordering, the race can happen like this: +* +* page receiver other thread +* ------------- ------------ +* qemu_get_buffer() +* do shutdown() +* returns 0 (buffer all zero) +* (we didn't check this retcode) +* try to detect IO error +* last_error==NULL, IO okay +* install ALL-ZERO page +* set last_error +* --> guest crash! +*/ +if (!f->last_error) { +qemu_file_set_error (f, -EIO ); + } +if (!qio_channel_has_feature (f->ioc,QIO_CHANNEL_FEATURE_SHUTDOWN )) {return -ENOSYS ;
@@ -88,9 +112,6 @@ int qemu_file_shutdown(QEMUFile *f) ret = -EIO ; } -if (!f->last_error) { -qemu_file_set_error (f, -EIO ); - }return ret; }
Signature · 1 line
2.32.0
Re: [PATCH 2/5] migration: Fix race on qemu_file_shutdown()
Quoted 31 lines · * Peter Xu ([email protected]) wrote:
In
qemu_file_shutdown(), there's a possible race if with current order of operation. There're two major things to do:(1) Do real
shutdown()(e.g.shutdown()syscall on socket)
(2) Update qemufile'slast_errorWe must do (2) before (1) otherwise there can be a race condition like:
page receiver other thread
------------- ------------
qemu_get_buffer()
doshutdown()
returns 0 (buffer all zero) (meanwhile we didn't check this retcode)
try to detect IO error
last_error==NULL, IO okay
install ALL-ZERO page
setlast_error
--> guest crash!To fix this, we can also check retval of
qemu_get_buffer(), but not all APIs can be properly checked and ultimately we still need to go back toqemu_file_get_error(). E.g.qemu_get_byte()doesn't return error.Maybe some day a rework of qemufile API is really needed, but for now keep using
qemu_file_get_error()and fix it by not allowing that race condition to happen. Hereshutdown()is indeed special because thelast_errorwas emulated. For real-EIOerrors it'll always be set when e.g.sendmsg()error triggers so we won't miss those ones, onlyshutdown()is a bit tricky here.
Quoted 2 lines
Cc Daniel P. Berrange <[email protected]>Signed-off-by Peter Xu <[email protected]>
Oh that's kind of fun,
Quoted 52 lines
--- migration/qemu-file.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)migration/qemu-file.c +24
-3@@ -79,6 +79,30 @@ int qemu_file_shutdown(QEMUFile *f)int ret =0 ; f->shutdown = true; + +/* +* We must set qemufile error before the real shutdown(), otherwise +* there can be a race window where we thought IO all went though +* (because last_error==NULL) but actually IO has already stopped. +* +* If without correct ordering, the race can happen like this: +* +* page receiver other thread +* ------------- ------------ +* qemu_get_buffer() +* do shutdown() +* returns 0 (buffer all zero) +* (we didn't check this retcode) +* try to detect IO error +* last_error==NULL, IO okay +* install ALL-ZERO page +* set last_error +* --> guest crash! +*/ +if (!f->last_error) { +qemu_file_set_error (f, -EIO ); + } +if (!qio_channel_has_feature (f->ioc,QIO_CHANNEL_FEATURE_SHUTDOWN )) {return -ENOSYS ;@@ -88,9 +112,6 @@ int qemu_file_shutdown(QEMUFile *f) ret = -EIO ; } -if (!f->last_error) { -qemu_file_set_error (f, -EIO ); - }return ret; }Signature · 1 line
2.32.0
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 2/5] migration: Fix race on qemu_file_shutdown()
In reply to Peter XuQuoted 24 lines · Peter Xu
In
qemu_file_shutdown(), there's a possible race if with current order of operation. There're two major things to do:(1) Do real
shutdown()(e.g.shutdown()syscall on socket)
(2) Update qemufile'slast_errorWe must do (2) before (1) otherwise there can be a race condition like:
page receiver other thread
------------- ------------
qemu_get_buffer()
doshutdown()
returns 0 (buffer all zero) (meanwhile we didn't check this retcode)
try to detect IO error
last_error==NULL, IO okay
install ALL-ZERO page
setlast_error
--> guest crash!To fix this, we can also check retval of
qemu_get_buffer(), but not all APIs can be properly checked and ultimately we still need to go back toqemu_file_get_error(). E.g.qemu_get_byte()doesn't return error.
Quoted 6 lines
Maybe some day a rework of qemufile API is really needed, but for now keep using
qemu_file_get_error()and fix it by not allowing that race condition to happen. Hereshutdown()is indeed special because thelast_errorwas emulated. For real-EIOerrors it'll always be set when e.g.sendmsg()error triggers so we won't miss those ones, onlyshutdown()is a bit tricky here.
The ultimate answer really is to stop using QEMUFile entirely and just do migration with the QIOChannel objects directly. The work I did in the last cycle to remove the QEMUFileOps callbacks was another piece of the puzzle in moving in that direction, by ensuring that every QEMUFile is now backed by a QIOChannel. All QEMUFile is doing now is adding the I/O caching layer and some convenience APIs for I/O operations.
So the next step would be to add a QIOChannelCached class that can wrap over another QIOChannel, to add the I/O buffering functionality that currently exists in QEMUFile. Once that's done, it'd be at the stage where we could look at how to use QIOChannel APIs for VMstate. It would likely involve wiring up an Error **errp parameter into a great many places so we get synchronous error propagation instead of out-of-band error checking, so a phased transition would need to be figured out.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 2/5] migration: Fix race on qemu_file_shutdown()
Quoted 39 lines · Daniel P. Berrangé
Quoted 24 lines · Peter Xu
In
qemu_file_shutdown(), there's a possible race if with current order of operation. There're two major things to do:(1) Do real
shutdown()(e.g.shutdown()syscall on socket)
(2) Update qemufile'slast_errorWe must do (2) before (1) otherwise there can be a race condition like:
page receiver other thread
------------- ------------
qemu_get_buffer()
doshutdown()
returns 0 (buffer all zero) (meanwhile we didn't check this retcode)
try to detect IO error
last_error==NULL, IO okay
install ALL-ZERO page
setlast_error
--> guest crash!To fix this, we can also check retval of
qemu_get_buffer(), but not all APIs can be properly checked and ultimately we still need to go back toqemu_file_get_error(). E.g.qemu_get_byte()doesn't return error.Quoted 6 lines
Maybe some day a rework of qemufile API is really needed, but for now keep using
qemu_file_get_error()and fix it by not allowing that race condition to happen. Hereshutdown()is indeed special because thelast_errorwas emulated. For real-EIOerrors it'll always be set when e.g.sendmsg()error triggers so we won't miss those ones, onlyshutdown()is a bit tricky here.The ultimate answer really is to stop using QEMUFile entirely and just do migration with the QIOChannel objects directly. The work I did in the last cycle to remove the QEMUFileOps callbacks was another piece of the puzzle in moving in that direction, by ensuring that every QEMUFile is now backed by a QIOChannel. All QEMUFile is doing now is adding the I/O caching layer and some convenience APIs for I/O operations.
Quoted 7 lines
So the next step would be to add a QIOChannelCached class that can wrap over another QIOChannel, to add the I/O buffering functionality that currently exists in QEMUFile. Once that's done, it'd be at the stage where we could look at how to use QIOChannel APIs for VMstate. It would likely involve wiring up an Error **errp parameter into a great many places so we get synchronous error propagation instead of out-of-band error checking, so a phased transition would need to be figured out.
Yes, Error** sounds very good to have.
So far I'm not satisfied with qemufile api majorly because of that error handling, especially on *get() interfaces.
Besides that, do you have anything else in mind that would make QIOChannelCached better than qemufile (e.g. on how we do caching)?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/5] migration: Fix race on qemu_file_shutdown()
Quoted 48 lines · Peter Xu
Quoted 39 lines · Daniel P. Berrangé
Quoted 24 lines · Peter Xu
In
qemu_file_shutdown(), there's a possible race if with current order of operation. There're two major things to do:(1) Do real
shutdown()(e.g.shutdown()syscall on socket)
(2) Update qemufile'slast_errorWe must do (2) before (1) otherwise there can be a race condition like:
page receiver other thread
------------- ------------
qemu_get_buffer()
doshutdown()
returns 0 (buffer all zero) (meanwhile we didn't check this retcode)
try to detect IO error
last_error==NULL, IO okay
install ALL-ZERO page
setlast_error
--> guest crash!To fix this, we can also check retval of
qemu_get_buffer(), but not all APIs can be properly checked and ultimately we still need to go back toqemu_file_get_error(). E.g.qemu_get_byte()doesn't return error.Quoted 6 lines
Maybe some day a rework of qemufile API is really needed, but for now keep using
qemu_file_get_error()and fix it by not allowing that race condition to happen. Hereshutdown()is indeed special because thelast_errorwas emulated. For real-EIOerrors it'll always be set when e.g.sendmsg()error triggers so we won't miss those ones, onlyshutdown()is a bit tricky here.The ultimate answer really is to stop using QEMUFile entirely and just do migration with the QIOChannel objects directly. The work I did in the last cycle to remove the QEMUFileOps callbacks was another piece of the puzzle in moving in that direction, by ensuring that every QEMUFile is now backed by a QIOChannel. All QEMUFile is doing now is adding the I/O caching layer and some convenience APIs for I/O operations.
Quoted 7 lines
So the next step would be to add a QIOChannelCached class that can wrap over another QIOChannel, to add the I/O buffering functionality that currently exists in QEMUFile. Once that's done, it'd be at the stage where we could look at how to use QIOChannel APIs for VMstate. It would likely involve wiring up an Error **errp parameter into a great many places so we get synchronous error propagation instead of out-of-band error checking, so a phased transition would need to be figured out.
Quoted 7 lines
Yes, Error** sounds very good to have.
So far I'm not satisfied with qemufile api majorly because of that error handling, especially on
*get()interfaces.Besides that, do you have anything else in mind that would make QIOChannelCached better than qemufile (e.g. on how we do caching)?
Depends what you mean by better ? I think the caching code would be a bit easier to understand, because QEMUFile gets a bit confusing about which logic is used for read side and which is used for the write side.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 2/5] migration: Fix race on qemu_file_shutdown()
Quoted 7 lines · Daniel P. Berrangé
Quoted 2 lines
Besides that, do you have anything else in mind that would make QIOChannelCached better than qemufile (e.g. on how we do caching)?
Depends what you mean by better ? I think the caching code would be a bit easier to understand, because QEMUFile gets a bit confusing about which logic is used for read side and which is used for the write side.
I see, looking forward to it. Thanks,
Signature · 1 line
Peter Xu
PATCH 3/5 migration: Disallow xbzrle with postcopy
In reply to Peter XuIt's not supported since the 1st day, as ram_load_postcopy does not handle RAM_SAVE_FLAG_XBZRLE. Mark it disabled explicitly.
--- migration/migration.c | 5 +++++ 1 file changed, 5 insertions(+)
migration/migration.c +5 -0
@@ -1268,6 +1268,11 @@ static bool migrate_caps_check(bool *cap_list,error_setg (errp,"Postcopy is not compatible with ignore-shared" );return false; } + +if (cap_list[MIGRATION_CAPABILITY_XBZRLE ]) { +error_setg (errp,"Postcopy is not compatible with xbzrle" ); +return false; + } }if (cap_list[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT ]) {
Signature · 1 line
2.32.0
Re: [PATCH 3/5] migration: Disallow xbzrle with postcopy
Quoted 2 lines · * Peter Xu ([email protected]) wrote:
It's not supported since the 1st day, as
ram_load_postcopydoes not handleRAM_SAVE_FLAG_XBZRLE. Mark it disabled explicitly.
We've already got a check in ram_save_page:
if (rs->xbzrle_enabled && !migration_in_postcopy()) {
pages = save_xbzrle_page(rs, &p, current_addr, block,
offset);so that's supposed to allow you to enable xbzrle with postcopy and take advantage of xbzrle during the precopy phase.
Dave
Quoted 24 lines
Signed-off-by Peter Xu <[email protected]>--- migration/migration.c | 5 +++++ 1 file changed, 5 insertions(+)migration/migration.c +5
-0@@ -1268,6 +1268,11 @@ static bool migrate_caps_check(bool *cap_list,error_setg (errp,"Postcopy is not compatible with ignore-shared" );return false; } + +if (cap_list[MIGRATION_CAPABILITY_XBZRLE ]) { +error_setg (errp,"Postcopy is not compatible with xbzrle" ); +return false; + } }if (cap_list[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT ]) {Signature · 1 line
2.32.0
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH 3/5] migration: Disallow xbzrle with postcopy
Quoted 4 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Peter Xu ([email protected]) wrote:
It's not supported since the 1st day, as
ram_load_postcopydoes not handleRAM_SAVE_FLAG_XBZRLE. Mark it disabled explicitly.
Quoted 8 lines
We've already got a check in
ram_save_page:if (rs->xbzrle_enabled && !migration_in_postcopy()) { pages = save_xbzrle_page(rs, &p, current_addr, block, offset);so that's supposed to allow you to enable xbzrle with postcopy and take advantage of xbzrle during the precopy phase.
Ah! Makes sense. I'll drop this one.
Signature · 1 line
Peter Xu
PATCH 4/5 migration: Disallow postcopy preempt to be used with compress
In reply to Peter XuThe preempt mode requires the capability to assign channel for each of the page, while the compression logic will currently assign pages to different compress thread/local-channel so potentially they're incompatible.
--- migration/migration.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
migration/migration.c +11 -0
@@ -1341,6 +1341,17 @@ static bool migrate_caps_check(bool *cap_list,error_setg (errp,"Postcopy preempt requires postcopy-ram" );return false; } + +/* +* Preempt mode requires urgent pages to be sent in separate +* channel, OTOH compression logic will disorder all pages into +* different compression channels, which is not compatible with the +* preempt assumptions on channel assignments. +*/ +if (cap_list[MIGRATION_CAPABILITY_COMPRESS ]) { +error_setg (errp,"Postcopy preempt not compatible with compress" ); +return false; + } }return true;
Signature · 1 line
2.32.0
Re: [PATCH 4/5] migration: Disallow postcopy preempt to be used with compress
Quoted 5 lines · * Peter Xu ([email protected]) wrote:
The preempt mode requires the capability to assign channel for each of the page, while the compression logic will currently assign pages to different compress thread/local-channel so potentially they're incompatible.
Signed-off-by Peter Xu <[email protected]>
Quoted 29 lines
--- migration/migration.c | 11 +++++++++++ 1 file changed, 11 insertions(+)migration/migration.c +11
-0@@ -1341,6 +1341,17 @@ static bool migrate_caps_check(bool *cap_list,error_setg (errp,"Postcopy preempt requires postcopy-ram" );return false; } + +/* +* Preempt mode requires urgent pages to be sent in separate +* channel, OTOH compression logic will disorder all pages into +* different compression channels, which is not compatible with the +* preempt assumptions on channel assignments. +*/ +if (cap_list[MIGRATION_CAPABILITY_COMPRESS ]) { +error_setg (errp,"Postcopy preempt not compatible with compress" ); +return false; + } }return true;Signature · 1 line
2.32.0
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 5/5 migration: Use non-atomic ops for clear log bitmap
In reply to Peter XuSince we already have bitmap_mutex to protect either the dirty bitmap or the clear log bitmap, we don't need atomic operations to set/clear/test on the clear log bitmap. Switching all ops from atomic to non-atomic versions, meanwhile touch up the comments to show which lock is in charge.
Introduced non-atomic version of bitmap_test_and_clear_atomic(), mostly the same as the atomic version but simplified a few places, e.g. dropped the "old_bits" variable, and also the explicit memory barriers.
--- include/exec/ram_addr.h | 11 +++++----- include/exec/ramblock.h | 3 +++ include/qemu/bitmap.h | 1 + util/bitmap.c | 45 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 5 deletions(-)
include/exec/ram_addr.h +6 -5
@@ -42,7 +42,8 @@ static inline long clear_bmap_size(uint64_t pages, uint8_t shift) }/** -* clear_bmap_set: set clear bitmap for the page range +* clear_bmap_set: set clear bitmap for the page range. Must be with +* bitmap_mutex held. * * @rb: the ramblock to operate on * @start: the start page number
@@ -55,12 +56,12 @@ static inline void clear_bmap_set(RAMBlock *rb, uint64_t start, {uint8_t shift = rb->clear_bmap_shift; -bitmap_set_atomic (rb->clear_bmap, start >> shift, -clear_bmap_size (npages, shift)); +bitmap_set (rb->clear_bmap, start >> shift,clear_bmap_size (npages, shift)); }/** -* clear_bmap_test_and_clear: test clear bitmap for the page, clear if set +* clear_bmap_test_and_clear: test clear bitmap for the page, clear if set. +* Must be with bitmap_mutex held. * * @rb: the ramblock to operate on * @page: the page number to check
@@ -71,7 +72,7 @@ static inline bool clear_bmap_test_and_clear(RAMBlock *rb, uint64_t page) {uint8_t shift = rb->clear_bmap_shift; -return bitmap_test_and_clear_atomic (rb->clear_bmap, page >> shift,1 ); +return bitmap_test_and_clear (rb->clear_bmap, page >> shift,1 ); }static inline bool offset_in_ramblock (RAMBlock *b,ram_addr_t offset)
include/exec/ramblock.h +3 -0
@@ -53,6 +53,9 @@ struct RAMBlock {* and split clearing of dirty bitmap on the remote node (e.g., * KVM). The bitmap will be set only when doing global sync. * +* It is only used during src side of ram migration, and it is +* protected by the global ram_state.bitmap_mutex. +* * NOTE: this bitmap is different comparing to the other bitmaps * in that one bit can represent multiple guest pages (which is * decided by the `clear_bmap_shift' variable below). On
include/qemu/bitmap.h +1 -0
@@ -253,6 +253,7 @@ void bitmap_set(unsigned long *map, long i, long len);void bitmap_set_atomic (unsigned long *map,long i,long len);void bitmap_clear (unsigned long *map,long start,long nr);bool bitmap_test_and_clear_atomic (unsigned long *map,long start,long nr); +bool bitmap_test_and_clear (unsigned long *map,long start,long nr);void bitmap_copy_and_clear_atomic (unsigned long *dst,unsigned long *src,long nr); unsigned long bitmap_find_next_zero_area(unsigned long *map,
util/bitmap.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.32.0