Skip to contentSkip to conversation

tools/libxenguest: Fix migration's debug option

PATCH In review Andrew Cooper 3 participants 11 messages Jul 2, 2021 – Jul 5, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tools/libxenguest: Fix migration's debug option

The code has gone through many refactors, but the first refactor was the one which broke it by inverting the check with respect to checkpointed streams.

Fixes 7449fb36c6c8 ("migration/save: pass checkpointed_stream from libxl to libxc")

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Olaf Hering <[email protected]>

xl migrate --debug might not be perfect, but this at least brings it back to mostly working.

I don't think dropping it is a sensible move. In particular, it is invaluable for testing the logdirty infrastructure when migrating a memtest VM.

If anyone has a clever idea to fix the grant problem, then we can. It is after all a debug option, without any specific prescribed behaviour.

---
 tools/libs/guest/xg_sr_save.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libs/guest/xg_sr_save.c +1 -1
index 2ba7c3200cd5..f0e2bd048d37 100644
@@ -752,7 +752,7 @@ static int send_domain_memory_live(struct xc_sr_context *ctx)
     if ( rc )
         goto out;
 
-    if ( ctx->save.debug && ctx->stream_type != XC_STREAM_PLAIN )
+    if ( ctx->save.debug && ctx->stream_type == XC_STREAM_PLAIN )
     {
         rc = verify_frames(ctx);
         if ( rc )
Signature · 1 line
2.11.0
Acked-by on v1 1/1
#1 lore raw

Am Fri, 2 Jul 2021 20:03:42 +0100
schrieb Andrew Cooper <[email protected]>:

Quoted 6 lines

The code has gone through many refactors, but the first refactor was the one which broke it by inverting the check with respect to checkpointed streams.

Fixes 7449fb36c6c8 ("migration/save: pass checkpointed_stream from libxl to libxc")

Olaf

Reviewed-by on v1 1/1
#2 lore raw
In reply to Andrew Cooper
Quoted 6 lines · Andrew Cooper

The code has gone through many refactors, but the first refactor was the one which broke it by inverting the check with respect to checkpointed streams.

Fixes 7449fb36c6c8 ("migration/save: pass checkpointed_stream from libxl to libxc")
Quoted 5 lines

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Olaf Hering <[email protected]>

Quoted 8 lines

xl migrate --debug might not be perfect, but this at least brings it back to mostly working.

I don't think dropping it is a sensible move. In particular, it is invaluable for testing the logdirty infrastructure when migrating a memtest VM.

If anyone has a clever idea to fix the grant problem, then we can. It is after all a debug option, without any specific prescribed behaviour.

What is "the grant problem" referring to here? Neither anything above nor the offending original commit has any reference to grants, or a problem with them.

Jan

#3 lore raw

Am Mon, 5 Jul 2021 09:57:21 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 3 lines

What is "the grant problem" referring to here? Neither anything above nor the offending original commit has any reference to grants, or a problem with them.

When the guest is paused during final transit, the backends will continue to write into domU memory. As a result the final additional iteration to verify memory on both sides will always see errors. The code has no way to know for which pfn such mismatches in page content can safely be ignored.

Olaf

#4 lore raw
Quoted 10 lines · Olaf Hering

Am Mon, 5 Jul 2021 09:57:21 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 3 lines

What is "the grant problem" referring to here? Neither anything above nor the offending original commit has any reference to grants, or a problem with them.

When the guest is paused during final transit, the backends will continue to write into domU memory. As a result the final additional iteration to verify memory on both sides will always see errors.

I see. A similar problem then exists with at least the FIFO event channel per-vCPU control blocks?

Quoted 2 lines

The code has no way to know for which pfn such mismatches in page content can safely be ignored.

Well, in principle this can be known, but it's expensive: For a paused domain the grant table can't change anymore. Any pages referenced by a valid non-r/o grant table entry could in principle change.

Jan

#5 lore raw

Am Mon, 5 Jul 2021 10:23:00 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 2 lines

I see. A similar problem then exists with at least the FIFO event channel per-vCPU control blocks?

I have not done any debugging how the pages differ and what they are actually used for.

My guess was that it might be activity from the backends, particularly netback. I found no API to query the usage of a page, so I declared the interface broken..

Olaf

#6 lore raw
Quoted 11 lines · Olaf Hering

Am Mon, 5 Jul 2021 10:23:00 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 2 lines

I see. A similar problem then exists with at least the FIFO event channel per-vCPU control blocks?

I have not done any debugging how the pages differ and what they are actually used for.

My guess was that it might be activity from the backends, particularly netback. I found no API to query the usage of a page, so I declared the interface broken..

"The interface" being which one? The tool stack can map the guest's grant table, so it is in the position to find out about all grants without further hypervisor help. Afaict the same isn't true for the FIFO event channel control blocks, though, and maybe there are further ways by which pages may get modified for a paused guest.

Jan

#7 lore raw

Am Mon, 5 Jul 2021 11:19:59 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 3 lines

"The interface" being which one? The tool stack can map the guest's grant table, so it is in the position to find out about all grants without further hypervisor help.

The interface means the code behind verify_frames.

If there are indeed ways to query which pages belong to grants, how would the toolstack need to do that?

Olaf

#8 lore raw
Quoted 10 lines · Olaf Hering

Am Mon, 5 Jul 2021 11:19:59 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 3 lines

"The interface" being which one? The tool stack can map the guest's grant table, so it is in the position to find out about all grants without further hypervisor help.

The interface means the code behind verify_frames.

If there are indeed ways to query which pages belong to grants, how would the toolstack need to do that?

Map the grant table of the guest and walk it, recording any MFN for which at least one valid r/w grant exists.

Jan

#9 lore raw
Quoted 4 lines · Jan Beulich
Quoted 3 lines · Olaf Hering

Am Mon, 5 Jul 2021 11:19:59 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

"The interface" being which one? The tool stack can map the guest's grant table, so it is in the position to find out about all grants without further hypervisor help.

The interface means the code behind verify_frames.

If there are indeed ways to query which pages belong to grants, how would the toolstack need to do that?

Map the grant table of the guest and walk it, recording any MFN for which at least one valid r/w grant exists.

That doesn't help - Its still racy with in-flight IO.  Also with updates from Xen such as the wallclocks.

The only way to fix the IO problem is to disconnect the blk/net rings before doing the final sweep for frames, but that clobbers any ability to restart the VM on the source side if things go wrong at the destination.

I don't have an answer at all for the vcpu info frames.

~Andrew

#10 lore raw
Quoted 8 lines · Andrew Cooper
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Olaf Hering

Am Mon, 5 Jul 2021 11:19:59 +0200
schrieb Jan Beulich <[email protected]>:

Quoted 3 lines

"The interface" being which one? The tool stack can map the guest's grant table, so it is in the position to find out about all grants without further hypervisor help.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The interface means the code behind verify_frames.

If there are indeed ways to query which pages belong to grants, how would the toolstack need to do that?

Map the grant table of the guest and walk it, recording any MFN for which at least one valid r/w grant exists.

That doesn't help - Its still racy with in-flight IO.

Well, I meant the recorded data to be used to simply not verify those frames.

Quoted 1 line

  Also with updates from Xen such as the wallclocks.

This doesn't occur for a paused domain, does it?

Quoted 5 lines

The only way to fix the IO problem is to disconnect the blk/net rings before doing the final sweep for frames, but that clobbers any ability to restart the VM on the source side if things go wrong at the destination.

I don't have an answer at all for the vcpu info frames.

Yeah, they fall in the same category as the FIFO control pages, as they contain evtchn_{pending_sel,upcall_pending}.

Jan