Skip to contentSkip to conversation

Postcopy implementation

PATCH Superseded Dr. David Alan Gilbert (git) 8 participants 209 messages Jun 16, 2015 – Oct 20, 2015 Page 1 of 2

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover Postcopy implementation

From: "Dr. David Alan Gilbert" <[email protected]>

This is the 7th cut of my version of postcopy; it is designed for use with
the Linux kernel additions posted by Andrea Arcangeli here:

git clone --reference linux -b userfault21
git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git

Note this API is slightly different from the last version; but the code is now in the linux-mm tree, the API is getting stable and the kernel code is just getting fixes now.

This qemu series can be found at:

https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v7 tag.

It addresses most of the previous review comments, but there are still one or two I'm working.

As with v6, the userfaultfd.h isn't included in the tree, so you'll need a userfaultfd.h header (and syscall define) from Andrea's kernel.

This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/

v7
updated to Andrea's userfault 21 interface
Don't restart the source after an error if we entered postcopy (thanks to Li Liang for reporting) Killed off dead local_tmp_page (thanks to Christoph Seifert for spotting that)
Made sure the incoming page mutex is only initialised once
Moved request alignment from destination to source; destination now just verifies the request is aligned Got rid of the 'dup' on the return-fd
Just use the same fd and make the forward path own the fd
Reworked the consumption of request pages off the queue to handle hosts with larger
pages more cleanly
Disallowed enabling postcopy+compression
This is probably fixable, but it needs reworking of the decompression threads
to place the pages atomically
Disallow postcopy+RDMA
This is probably trickier to fix; RDMA drops the received data straight into memory using DMA; we would need to find a way to stop that. I can see a way to make it so that the precopy phase still uses RDMA and then stops using it later, but that would
make it harder when a real fix came along.
Minor fixups from review
6 of the smaller patches from v6 are now already in head

TODO
Testing on machines with hps!=tps
Tidy up the hps!=tps code on the receive side

Dr. David Alan Gilbert (42):
Start documenting how postcopy works.
Provide runtime Target page information
Init page sizes in qtest
qemu_ram_block_from_host
Add qemu_get_buffer_less_copy to avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile
ram_debug_dump_bitmap: Dump a migration bitmap as text
migrate_init: Call from savevm
Rename save_live_complete to save_live_complete_precopy Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modify save_live_pending for postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Add qemu_savevm_state_complete_postcopy
Postcopy: Maintain sentmap and calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy end in migration_thread
Page request: Add MIG_RP_MSG_REQ_PAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration Don't sync dirty bitmaps in postcopy
Host page!=target page: Cleanup bitmaps
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_handle_ commands
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopy

 balloon.c                        |  11 +
 docs/migration.txt               | 167 +++++++
 exec.c                           |  64 ++-
 hmp-commands.hx                  |  15 +
 hmp.c                            |   7 +
 hmp.h                            |   1 +
 hw/ppc/spapr.c                   |   2 +-
 hw/virtio/virtio-balloon.c       |   4 +-
 include/exec/cpu-all.h           |   2 -
 include/exec/cpu-common.h        |   3 +
 include/migration/migration.h    | 112 ++++-
 include/migration/postcopy-ram.h |  88 ++++
 include/migration/qemu-file.h    |  10 +
 include/migration/vmstate.h      |   8 +-
 include/qemu/typedefs.h          |   3 +
 include/sysemu/balloon.h         |   2 +
 include/sysemu/sysemu.h          |  44 +-
 migration/Makefile.objs          |   2 +-
 migration/block.c                |   9 +-
 migration/migration.c            | 736 ++++++++++++++++++++++++++++--
 migration/postcopy-ram.c         | 706 +++++++++++++++++++++++++++++
 migration/qemu-file-unix.c       | 110 ++++-
 migration/qemu-file.c            |  74 ++++
 migration/ram.c                  | 933 ++++++++++++++++++++++++++++++++++++---
 migration/savevm.c               | 776 +++++++++++++++++++++++++++++---
 qapi-schema.json                 |  18 +-
 qmp-commands.hx                  |  19 +
 qtest.c                          |   1 +
 trace-events                     |  76 +++-
 29 files changed, 3788 insertions(+), 215 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
Signature · 1 line
2.4.3
#1 lore raw

PATCH v7 1/42 Start documenting how postcopy works.

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
docs/migration.txt +167 -0
index f6df4be..b4b93d1 100644
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.
Signature · 1 line
2.4.3
#2 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 24 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
docs/migration.txt +0 -0
index f6df4be..b4b93d1 100644
@@ -291,3 +291,170 @@ save/send this state when we are in the middle

of a pio operation
(that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent.

docs/migration.txt +8 -0
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the
Quoted 1 line

Postcopy destination

This line is a bit long O:-)

In general, we are too near to the 80 columns limit.

#3 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 29 lines · * Juan Quintela ([email protected]) wrote:
Quoted 24 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
docs/migration.txt +0 -0
index f6df4be..b4b93d1 100644
@@ -291,3 +291,170 @@ save/send this state when we are in the middle

of a pio operation
(that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent.

docs/migration.txt +8 -0
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the
Quoted 1 line

Postcopy destination

This line is a bit long O:-)

Quoted 1 line

In general, we are too near to the 80 columns limit.

Thanks, Fixed (interesting, the check-patch doesn't seem to moan about text files).

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

In reply to Dr. David Alan Gilbert (git)
Quoted 33 lines
docs/migration.txt index f6df4be..b4b93d1 renamed from docs/migration.txt +0 -0

100644

docs/migration.txt +0 -0
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a

pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent.

docs/migration.txt +6 -0
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd
+(although possibly with another fd or similar for some fast way of throwing

pages across).

docs/migration.txt +3 -0
+
+However, some uses need two way communication; in particular the
+Postcopy destination needs to be able to request pages on demand from

the source.

docs/migration.txt +13 -0
+
+For these scenarios there is a 'return path' from the destination to
+the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for
+the return path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND

postcopy

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 321

Quoted 8 lines
docs/migration.txt +8 -0
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to
+converge; its plus side is that there is an upper bound on the amount
+of migration traffic and time it takes, the down side is that during
+the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.

Hi David,

Do you have any idea or plan to deal with the failure happened during the postcopy phase?

Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.

Liang

#5 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 44 lines · * Li, Liang Z ([email protected]) wrote:
Quoted 33 lines
docs/migration.txt index f6df4be..b4b93d1 renamed from docs/migration.txt +0 -0

100644

docs/migration.txt +0 -0
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a

pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent.

docs/migration.txt +6 -0
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd
+(although possibly with another fd or similar for some fast way of throwing

pages across).

docs/migration.txt +3 -0
+
+However, some uses need two way communication; in particular the
+Postcopy destination needs to be able to request pages on demand from

the source.

docs/migration.txt +13 -0
+
+For these scenarios there is a 'return path' from the destination to
+the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for
+the return path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND

postcopy

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 321

Quoted 8 lines
docs/migration.txt +8 -0
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to
+converge; its plus side is that there is an upper bound on the amount
+of migration traffic and time it takes, the down side is that during
+the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.

Hi David,

Quoted 5 lines

Do you have any idea or plan to deal with the failure happened during the postcopy phase?

Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.

The VM memory image is still on the source VM, so you can restart the source, however that's not safe, because once the destination has started running it is sending out packets and also modifying the block storage. If you restarted the source at that point what block and net state can you accept being visible?

Dave

Quoted 2 lines

Liang

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#6 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

In reply to Li, Liang Z
Quoted 6 lines · Liang Z

Do you have any idea or plan to deal with the failure happened during the postcopy phase?

Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.

There's no solution to this problem, except for rollback to a previous snapshot.

To give an idea, an example of an intended usecase for postcopy is datacenter evacuation in 30 minutes after a tsunami alert. That's not a case where you care much about losing guests to network failures.

Why is there no solution? Let's look at one of the best surveys on migration,
http://courses.cs.vt.edu/~cs5204/fall05-kafura/Papers/Migration/ProcessMigration.pdf
(warning, 59 pages!):

[3.2] If only part of the task state is transferred to another node, the task can start executing sooner, and the initial migration costs are lower.

[3.4] Fault resilience can be improved in several ways. The impact of failures during migration can be reduced by maintaining process state on both the source and destination sites until the destination site instance is successfully promoted to a regular process and the source node is informed about this.

[3.5] Migration algorithms should avoid linear dependencies on the amount of state to be transferred. For example, the eager data transfer strategy has costs proportional to the address space size

"Pre"copy means "start copying *before* promoting the destination to be the primary host" and it has such a linear dependency on the amount of state to be transferred. "Post"copy means "delay some copying to *after* promoting the destination to be the primary host".

So we have:

Precopy Postcopy

   3.2 Performance            - (1)             - (2)
   3.4 Fault resilience       +                 -
   3.5 Scalability            -                 +

(1) smaller impact, longer freeze time
(2) larger impact, extremely short freeze time

Postcopy can also limit the length of the non-resilient phase, by starting with a precopy phase and only switching to postcopy after some time. Then you have:

Precopy Hybrid Postcopy

   3.2 Performance            - (1)          + (3)        - (2)
   3.4 Fault resilience       +              -            --
   3.5 Scalability            -              +            +

(3) intermediate impact, extremely short freeze time

but there is still going to be a phase where migration is not resilient to network faults.

Cloud operators can use a combination of precopy and postcopy. For example, I would not use postcopy for mass migration when doing host updates, but it can be used as a last resort before a scheduled downtime.

For example, say you're doing a rolling update and you want it complete by next Sunday. 90% of the guests are shut down by the customers or can be migrated successfully with precopy. The others do not converge and their SLA does not let you throttle them to complete precopy migration.

You then tell your customers that either they shutdown and restart their instances before Saturday 8:00 PM, or they might be shut down forcibly.
Then for customers who haven't rebooted you can do
postcopy---you have alerted them that something might go wrong. So even though postcopy would not be a first choice, it can still help cloud operators.

Paolo

#7 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 6 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines · Liang Z

Do you have any idea or plan to deal with the failure happened during the postcopy phase?

Quoted 6 lines
Quoted 3 lines

Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.

There's no solution to this problem, except for rollback to a previous snapshot.

Yes, and you might be able to avoid some of the pain if you COWd the disk data on the destination until the migration was finished; that would allow you to restart the source VM in the state prior to postcopy starting; although the network's view of it is going to be very messy.

Quoted 3 lines

To give an idea, an example of an intended usecase for postcopy is datacenter evacuation in 30 minutes after a tsunami alert. That's not a case where you care much about losing guests to network failures.

Well; you have to make a call as to what your best option is; you could always shut the VM down and boot it up fresh in your new safe data centre. Your preference is determined by your confidence that your VM would boot back up safely and how long it would take and the confidence in that network during the migration period and the pain of knowing what will happen if you explicitly shut the VM down.

Quoted 9 lines

Cloud operators can use a combination of precopy and postcopy. For example, I would not use postcopy for mass migration when doing host updates, but it can be used as a last resort before a scheduled downtime.

For example, say you're doing a rolling update and you want it complete by next Sunday. 90% of the guests are shut down by the customers or can be migrated successfully with precopy. The others do not converge and their SLA does not let you throttle them to complete precopy migration.

Indeed the interface lets you do that pretty easily; since as long as you have enabled postcopy, it starts in precopy mode and is fully recoverable until you issue the 'migrate_start_postcopy' which might be when it's tried 'n' times and you can see that the workload you have isn't going to converge.

Dave

Quoted 8 lines

You then tell your customers that either they shutdown and restart their instances before Saturday 8:00 PM, or they might be shut down forcibly.
Then for customers who haven't rebooted you can do
postcopy---you have alerted them that something might go wrong. So even though postcopy would not be a first choice, it can still help cloud operators.

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#8 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

In reply to Dr. David Alan Gilbert (git)

Hi Dave,

Quoted 2 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[...]

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325

Quoted 9 lines

+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;

docs/migration.txt +7 -0
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.

I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.

The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.

I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?

Quoted 136 lines
docs/migration.txt +135 -0
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.
Signature · 2 lines
Thanks,
Yang.
#9 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 16 lines · Yang Hongyang

Hi Dave,

Quoted 2 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[...]

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325

Quoted 9 lines

+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;

docs/migration.txt +7 -0
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
Quoted 7 lines

I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.

Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy. Here it does not need any cache at all, it just places the dirty pages where it will be accessed.

Quoted 147 lines

- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.

The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.

I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?

Quoted 136 lines
docs/migration.txt +135 -0
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.
#10 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 27 lines · zhanghailiang
Quoted 18 lines · Yang Hongyang

Hi Dave,

Quoted 2 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[...]

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325

Quoted 11 lines

+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of

docs/migration.txt +5 -0
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
Quoted 7 lines

I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.

Quoted 6 lines

Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.

I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]

Quoted 157 lines
Quoted 144 lines

- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.

The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.

I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?

Quoted 133 lines
(quoted diff) +2 -0
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if

precopy

docs/migration.txt +22 -0
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream

completely

docs/migration.txt +14 -0
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state

stream

docs/migration.txt +39 -0
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the

destination

(quoted diff) +2 -0
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running

behaviour

docs/migration.txt +49 -0
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
Quoted 9 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 460

Quoted 8 lines
docs/migration.txt +7 -0
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.

.

Signature · 2 lines
Thanks,
Yang.
#11 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 30 lines · * Yang Hongyang ([email protected]) wrote:
Quoted 27 lines · zhanghailiang
Quoted 25 lines · Yang Hongyang

Hi Dave,

Quoted 2 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[...]

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325

Quoted 11 lines

+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of

docs/migration.txt +5 -0
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.

I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.

Quoted 6 lines
Quoted 6 lines

Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.

Yes, zhanghailiang is correct; the source keeps sending other pages without being asked, however when asked it sends requested pages immediately. and the 'cache' is just the main memory from which the destination is working.

However, the idea of using a separate socket is one that we have been thinking about; one of the problems is that the urgent requested pages get delayed behind the background page transfer and that increases the latency; a separate socket should fix that.

Quoted 2 lines

I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]

Yes, I can add to the documentation; I've added the following text:

During postcopy the source scans the list of dirty pages and sends them to the destination without being requested (in much the same way as precopy), however when a page request is received from the destination the dirty page scanning restarts from the requested location. This causes requested pages to be sent quickly, and also causes pages directly after the requested page to be sent quickly in the hope that those pages are likely to be requested by the destination soon.

Dave

Quoted 155 lines
Quoted 155 lines
Quoted 153 lines

- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.

The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.

I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?

Quoted 141 lines
(quoted diff) +2 -0
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if

precopy

docs/migration.txt +22 -0
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream

completely

docs/migration.txt +14 -0
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state

stream

docs/migration.txt +39 -0
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the

destination

(quoted diff) +2 -0
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running

behaviour

docs/migration.txt +56 -0
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.
Quoted 6 lines
Quoted 2 lines

.

Signature · 2 lines
Thanks,
Yang.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#12 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 38 lines · Dr. David Alan Gilbert
Quoted 30 lines · * Yang Hongyang ([email protected]) wrote:
Quoted 27 lines · zhanghailiang
Quoted 25 lines · Yang Hongyang

Hi Dave,

Quoted 2 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[...]

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325

Quoted 11 lines

+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of

docs/migration.txt +5 -0
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.

I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.

Quoted 6 lines
Quoted 6 lines

Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.

Quoted 8 lines

Yes, zhanghailiang is correct; the source keeps sending other pages without being asked, however when asked it sends requested pages immediately. and the 'cache' is just the main memory from which the destination is working.

However, the idea of using a separate socket is one that we have been thinking about; one of the problems is that the urgent requested pages get delayed behind the background page transfer and that increases the latency; a separate socket should fix that.

That would be better.

Quoted 6 lines
Quoted 3 lines

I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]

Yes, I can add to the documentation; I've added the following text:

Quoted 7 lines

During postcopy the source scans the list of dirty pages and sends them to the destination without being requested (in much the same way as precopy), however when a page request is received from the destination the dirty page scanning restarts from the requested location. This causes requested pages to be sent quickly, and also causes pages directly after the requested page to be sent quickly in the hope that those pages are likely to be requested by the destination soon.

Looks clearer for me now :)

Quoted 168 lines

Dave

Quoted 155 lines
Quoted 155 lines
Quoted 153 lines

- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.

The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.

I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?

Quoted 141 lines
(quoted diff) +2 -0
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if

precopy

docs/migration.txt +22 -0
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream

completely

docs/migration.txt +14 -0
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state

stream

docs/migration.txt +39 -0
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the

destination

(quoted diff) +2 -0
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running

behaviour

docs/migration.txt +56 -0
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that the contents of the sentmap are sacrificed during the calculation
+of the discard set and thus aren't valid once in postcopy.  The dirtymap
+is still valid and is used to ensure that no page is sent more than once.  Any
+request for a page that has already been sent is ignored.  Duplicate requests
+such as this can happen as a page is sent at about the same time the
+destination accesses it.
Quoted 6 lines
Quoted 2 lines

.

--
Thanks,
Yang.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK .

Signature · 2 lines
Thanks,
Yang.
Reviewed-by on v7 1/42
#13 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

In reply to Dr. David Alan Gilbert (git)
Quoted 3 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

A few minor comments:

Quoted 27 lines
---
 docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
docs/migration.txt +15 -0
index f6df4be..b4b93d1 100644
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 318

Quoted 10 lines
docs/migration.txt +10 -0
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;

(or take too long to converge)

Quoted 4 lines
docs/migration.txt +4 -0
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 332

Quoted 10 lines
docs/migration.txt +10 -0
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):

How about this instead:

"To enable postcopy, issue this command ont he monitor prior to the start of migration:"

Otherwise, there's ambiguity that there is some way to enable this after a precopy migration has started.

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 336

Quoted 4 lines
docs/migration.txt +4 -0
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:

"A future migration will then start in precopy mode. However, issuing:"

?

Quoted 71 lines
docs/migration.txt +71 -0
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 417

Quoted 10 lines
docs/migration.txt +10 -0
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)

Adding:

"This gives the destination a chance to fail early if postcopy is not possible."

?

Quoted 6 lines
docs/migration.txt +6 -0
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 433

Quoted 10 lines
docs/migration.txt +10 -0
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).

indentation went off a bit

Amit

#14 lore raw

Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.

Quoted 40 lines · * Amit Shah ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

A few minor comments:

Quoted 31 lines
---
 docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
docs/migration.txt +19 -0
index f6df4be..b4b93d1 100644
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 318

Quoted 6 lines
docs/migration.txt +6 -0
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;

(or take too long to converge)

Added.

Quoted 18 lines
Quoted 5 lines
docs/migration.txt +4 -0
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 332

Quoted 10 lines
docs/migration.txt +10 -0
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):

How about this instead:

Quoted 5 lines

"To enable postcopy, issue this command ont he monitor prior to the start of migration:"

Otherwise, there's ambiguity that there is some way to enable this after a precopy migration has started.

Done.

Quoted 9 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 336

Quoted 4 lines
docs/migration.txt +4 -0
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:

"A future migration will then start in precopy mode. However, issuing:"

?

Ah yes, I see it's ambiguous because it doesn't say you still need to do the normal migration stuff to start migration;

I've changed it to:

The normal commands are then used to start a migration, which is still started in precopy mode. Issuing:

migrate_start_postcopy

will now cause the transition from precopy to postcopy.

Quoted 84 lines
Quoted 71 lines
docs/migration.txt +71 -0
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 417

Quoted 10 lines
docs/migration.txt +10 -0
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)

Adding:

Quoted 4 lines

"This gives the destination a chance to fail early if postcopy is not possible."

?

I added:
"The destination will fail early in migration at this point if the
required OS support is not present. "

Quoted 10 lines
Quoted 10 lines
docs/migration.txt +10 -0
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 433

Quoted 6 lines
docs/migration.txt +6 -0
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).

indentation went off a bit

Fixed.

Thanks,

Dave

Quoted 3 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#15 lore raw

PATCH v7 2/42 Provide runtime Target page information

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into migration/ram.c.

Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.

---
 exec.c                  | 10 ++++++++++
 include/sysemu/sysemu.h |  1 +
 2 files changed, 11 insertions(+)
exec.c +10 -0
index 76bfc4a..81a4481 100644
@@ -3313,6 +3313,16 @@ int cpu_memory_rw_debug(CPUState *cpu, target_ulong addr,
     }
     return 0;
 }
+
+/*
+ * Allows code that needs to deal with migration bitmaps etc to still be built
+ * target independent.
+ */
+size_t qemu_target_page_bits(void)
+{
+    return TARGET_PAGE_BITS;
+}
+
 #endif
 
 /*
include/sysemu/sysemu.h +1 -0
index ef793f7..7e42f76 100644
@@ -68,6 +68,7 @@ int qemu_reset_requested_get(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_devices_reset(void);
 void qemu_system_reset(bool report);
+size_t qemu_target_page_bits(void);
 
 void qemu_add_exit_notifier(Notifier *notify);
 void qemu_remove_exit_notifier(Notifier *notify);
Signature · 1 line
2.4.3
Reviewed-by on v7 2/42
#16 lore raw

Re: [Qemu-devel] [PATCH v7 02/42] Provide runtime Target page information

Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into migration/ram.c.

Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.

#17 lore raw

PATCH v7 3/42 Init page sizes in qtest

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

One of my patches used a loop that was based on host page size; it dies in qtest since qtest hadn't bothered init'ing it.

---
 qtest.c | 1 +
 1 file changed, 1 insertion(+)
qtest.c +1 -0
index 05cefd2..8e10340 100644
@@ -657,6 +657,7 @@ void qtest_init(const char *qtest_chrdev, const char *qtest_log, Error **errp)
 
     inbuf = g_string_new("");
     qtest_chr = chr;
+    page_size_init();
 }
 
 bool qtest_driver(void)
Signature · 1 line
2.4.3
#21 lore raw

PATCH v7 4/42 qemu_ram_block_from_host

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.

qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock and the global ram_addr_t value.

Rewrite qemu_ram_addr_from_host to use qemu_ram_block_from_host.

Provide qemu_ram_get_idstr since its the actual name text sent on the wire.

---
 exec.c                    | 54 +++++++++++++++++++++++++++++++++++++++--------
 include/exec/cpu-all.h    |  2 --
 include/exec/cpu-common.h |  3 +++
 include/qemu/typedefs.h   |  1 +
 4 files changed, 49 insertions(+), 11 deletions(-)
exec.c +45 -9
index 81a4481..d235001 100644
@@ -1265,6 +1265,11 @@ static RAMBlock *find_ram_block(ram_addr_t addr)
     return NULL;
 }
 
+const char *qemu_ram_get_idstr(RAMBlock *rb)
+{
+    return rb->idstr;
+}
+
 /* Called with iothread lock held.  */
 void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev)
 {
@@ -1755,8 +1760,16 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size)
     }
 }
 
-/* Some of the softmmu routines need to translate from a host pointer
- * (typically a TLB entry) back to a ram offset.
+/*
+ * Translates a host ptr back to a RAMBlock, a ram_addr and an offset
+ * in that RAMBlock.
+ *
+ * ptr: Host pointer to look up
+ * round_offset: If true round the result offset down to a page boundary
+ * *ram_addr: set to result ram_addr
+ * *offset: set to result offset within the RAMBlock
+ *
+ * Returns: RAMBlock (or NULL if not found)
  *
  * By the time this function returns, the returned pointer is not protected
  * by RCU anymore.  If the caller is not within an RCU critical section and
@@ -1764,18 +1777,22 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size)
  * pointer, such as a reference to the region that includes the incoming
  * ram_addr_t.
  */
-MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)
+RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,
+                                   ram_addr_t *ram_addr,
+                                   ram_addr_t *offset)
 {
     RAMBlock *block;
     uint8_t *host = ptr;
-    MemoryRegion *mr;
 
     if (xen_enabled()) {
         rcu_read_lock();
         *ram_addr = xen_ram_addr_from_mapcache(ptr);
-        mr = qemu_get_ram_block(*ram_addr)->mr;
+        block = qemu_get_ram_block(*ram_addr);
+        if (block) {
+            *offset = (host - block->host);
+        }
         rcu_read_unlock();
-        return mr;
+        return block;
     }
 
     rcu_read_lock();
@@ -1798,10 +1815,29 @@ MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)
     return NULL;
 
 found:
-    *ram_addr = block->offset + (host - block->host);
-    mr = block->mr;
+    *offset = (host - block->host);
+    if (round_offset) {
+        *offset &= TARGET_PAGE_MASK;
+    }
+    *ram_addr = block->offset + *offset;
     rcu_read_unlock();
-    return mr;
+    return block;
+}
+
+/* Some of the softmmu routines need to translate from a host pointer
+   (typically a TLB entry) back to a ram offset.  */
+MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)
+{
+    RAMBlock *block;
+    ram_addr_t offset; /* Not used */
+
+    block = qemu_ram_block_from_host(ptr, false, ram_addr, &offset);
+
+    if (!block) {
+        return NULL;
+    }
+
+    return block->mr;
 }
 
 static void notdirty_mem_write(void *opaque, hwaddr ram_addr,
include/exec/cpu-all.h +0 -2
index ac06c67..1f336e6 100644
@@ -266,8 +266,6 @@ CPUArchState *cpu_copy(CPUArchState *env);
 
 /* memory API */
 
-typedef struct RAMBlock RAMBlock;
-
 struct RAMBlock {
     struct rcu_head rcu;
     struct MemoryRegion *mr;
include/exec/cpu-common.h +3 -0
index de8a720..f1a045b 100644
@@ -62,8 +62,11 @@ typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);
 void qemu_ram_remap(ram_addr_t addr, ram_addr_t length);
 /* This should not be used by devices.  */
 MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr);
+RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,
+                                   ram_addr_t *ram_addr, ram_addr_t *offset);
 void qemu_ram_set_idstr(ram_addr_t addr, const char *name, DeviceState *dev);
 void qemu_ram_unset_idstr(ram_addr_t addr);
+const char *qemu_ram_get_idstr(RAMBlock *rb);
 
 void cpu_physical_memory_rw(hwaddr addr, uint8_t *buf,
                             int len, int is_write);
include/qemu/typedefs.h +1 -0
index 6fdcbcd..fc7c70e 100644
@@ -70,6 +70,7 @@ typedef struct QEMUSGList QEMUSGList;
 typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct QEMUTimerListGroup QEMUTimerListGroup;
 typedef struct QEMUTimer QEMUTimer;
+typedef struct RAMBlock RAMBlock;
 typedef struct Range Range;
 typedef struct SerialState SerialState;
 typedef struct SHPCDevice SHPCDevice;
Signature · 1 line
2.4.3
Reviewed-by on v7 4/42
#22 lore raw

Re: [Qemu-devel] [PATCH v7 04/42] qemu_ram_block_from_host

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.

qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock and the global ram_addr_t value.

Quoted 7 lines

Rewrite qemu_ram_addr_from_host to use qemu_ram_block_from_host.

Provide qemu_ram_get_idstr since its the actual name text sent on the wire.

Reviewed-by on v7 4/42
#23 lore raw

Re: [Qemu-devel] [PATCH v7 04/42] qemu_ram_block_from_host

In reply to Dr. David Alan Gilbert (git)
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.

qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock and the global ram_addr_t value.

Quoted 7 lines

Rewrite qemu_ram_addr_from_host to use qemu_ram_block_from_host.

Provide qemu_ram_get_idstr since its the actual name text sent on the wire.

Amit

#24 lore raw

PATCH v7 5/42 Add qemu_get_buffer_less_copy to avoid copies some of the time

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.

Thus 'qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.

---
 include/migration/qemu-file.h |  2 ++
 migration/qemu-file.c         | 47 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)
include/migration/qemu-file.h +2 -0
index 4f67d79..29a9d69 100644
@@ -162,6 +162,8 @@ int qemu_get_buffer(QEMUFile *f, uint8_t *buf, int size);
 ssize_t qemu_put_compression_data(QEMUFile *f, const uint8_t *p, size_t size,
                                   int level);
 int qemu_put_qemu_file(QEMUFile *f_des, QEMUFile *f_src);
+int qemu_get_buffer_less_copy(QEMUFile *f, uint8_t **buf, int size);
+
 /*
  * Note that you can only peek continuous bytes from where the current pointer
  * is; you aren't guaranteed to be able to peak to +n bytes unless you've
migration/qemu-file.c +47 -0

53 lines not on this page · raw message

Signature · 1 line
2.4.3
Reviewed-by on v7 5/42
#25 lore raw

Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time

Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.

Thus 'qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.

I don't know still where this function is used, but function is correct.

Can I suggest to change th ename to:

qemu_get_buffer_in_place()?

less_copy sounds ambigous to me :p

#26 lore raw

Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time

Quoted 15 lines · * Juan Quintela ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.

Thus 'qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.

Quoted 7 lines

I don't know still where this function is used, but function is correct.

Can I suggest to change th ename to:

qemu_get_buffer_in_place()?

less_copy sounds ambigous to me :p

Changed.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 5/42
#27 lore raw

Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time

In reply to Dr. David Alan Gilbert (git)
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.

Thus 'qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.

Amit

#28 lore raw

PATCH v7 6/42 Add wrapper for setting blocking status on a QEMUFile

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.

---
 include/migration/qemu-file.h |  1 +
 migration/qemu-file.c         | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
index 29a9d69..d43c835 100644
@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
+void qemu_file_change_blocking(QEMUFile *f, bool block);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
migration/qemu-file.c +15 -0
index c111a6b..c746129 100644
@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
 
     return res == len ? res : 0;
 }
+
+/*
+ * Change the blocking state of the QEMUFile.
+ * Note: On some transports the OS only keeps a single blocking state for
+ *       both directions, and thus changing the blocking on the main
+ *       QEMUFile can also affect the return path.
+ */
+void qemu_file_change_blocking(QEMUFile *f, bool block)
+{
+    if (block) {
+        qemu_set_block(qemu_get_fd(f));
+    } else {
+        qemu_set_nonblock(qemu_get_fd(f));
+    }
+}
Signature · 1 line
2.4.3
Reviewed-by on v7 6/42
#29 lore raw

Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.

Can we improve naming?

Quoted 23 lines
---
 include/migration/qemu-file.h |  1 +
 migration/qemu-file.c         | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
index 29a9d69..d43c835 100644
@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
+void qemu_file_change_blocking(QEMUFile *f, bool block);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
migration/qemu-file.c +0 -0
index c111a6b..c746129 100644
@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
 

migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661

Quoted 10 lines
migration/qemu-file.c +8 -0
     return res == len ? res : 0;
 }
+
+/*
+ * Change the blocking state of the QEMUFile.
+ * Note: On some transports the OS only keeps a single blocking state for
+ *       both directions, and thus changing the blocking on the main
+ *       QEMUFile can also affect the return path.
+ */
+void qemu_file_change_blocking(QEMUFile *f, bool block)

qemu_file_set_blocking?

It don't change the blocking, it just do whatever block says?

Quoted 7 lines
migration/qemu-file.c +7 -0
+{
+    if (block) {
+        qemu_set_block(qemu_get_fd(f));
+    } else {
+        qemu_set_nonblock(qemu_get_fd(f));
+    }
+}
#30 lore raw

Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile

Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.

Can we improve naming?

Quoted 23 lines
---
 include/migration/qemu-file.h |  1 +
 migration/qemu-file.c         | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
index 29a9d69..d43c835 100644
@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
+void qemu_file_change_blocking(QEMUFile *f, bool block);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
migration/qemu-file.c +0 -0
index c111a6b..c746129 100644
@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
 

migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661

Quoted 10 lines
migration/qemu-file.c +8 -0
     return res == len ? res : 0;
 }
+
+/*
+ * Change the blocking state of the QEMUFile.
+ * Note: On some transports the OS only keeps a single blocking state for
+ *       both directions, and thus changing the blocking on the main
+ *       QEMUFile can also affect the return path.
+ */
+void qemu_file_change_blocking(QEMUFile *f, bool block)

qemu_file_set_blocking?

It don't change the blocking, it just do whatever block says?

Quoted 7 lines

migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 668

Quoted 7 lines
migration/qemu-file.c +7 -0
+{
+    if (block) {
+        qemu_set_block(qemu_get_fd(f));
+    } else {
+        qemu_set_nonblock(qemu_get_fd(f));
+    }
+}

I worry about having a:
qemu_file_set_blocking
and a
qemu_set_block

it sounds a bit similar when one always 'sets' (i.e. turns on) and the other either turns on or off.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#31 lore raw

Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile

Quoted 68 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.

Can we improve naming?

Quoted 23 lines
---
 include/migration/qemu-file.h |  1 +
 migration/qemu-file.c         | 15 +++++++++++++++
 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
index 29a9d69..d43c835 100644
@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
+void qemu_file_change_blocking(QEMUFile *f, bool block);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
migration/qemu-file.c +0 -0
index c111a6b..c746129 100644
@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
 

migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661

Quoted 10 lines
migration/qemu-file.c +8 -0
     return res == len ? res : 0;
 }
+
+/*
+ * Change the blocking state of the QEMUFile.
+ * Note: On some transports the OS only keeps a single blocking state for
+ *       both directions, and thus changing the blocking on the main
+ *       QEMUFile can also affect the return path.
+ */
+void qemu_file_change_blocking(QEMUFile *f, bool block)

qemu_file_set_blocking?

It don't change the blocking, it just do whatever block says?

Quoted 7 lines

migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 668

Quoted 7 lines
migration/qemu-file.c +7 -0
+{
+    if (block) {
+        qemu_set_block(qemu_get_fd(f));
+    } else {
+        qemu_set_nonblock(qemu_get_fd(f));
+    }
+}

I worry about having a:
qemu_file_set_blocking
and a
qemu_set_block

Quoted 2 lines

it sounds a bit similar when one always 'sets' (i.e. turns on) and the other either turns on or off.

There is a parameter difference, but I am not writting the code, and don't care so much. I would expect a function with change in its name to change to the other plocking, whatever that is :P

Later, Juan.

#32 lore raw

PATCH v7 7/42 ram_debug_dump_bitmap: Dump a migration bitmap as text

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).

The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.

---
 include/migration/migration.h |  1 +
 migration/ram.c               | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/migration/migration.h +1 -0
index 9387c8c..b3a7f75 100644
@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +38 -0
index 57368e1..efc215a 100644
@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of; it won't bother printing lines that are all this value.
+ * If 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur + linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur + curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}
 
 /* Each of ram_save_setup, ram_save_iterate and ram_save_complete has
  * long-running RCU critical section.  When rcu-reclaims in the code
Signature · 1 line
2.4.3
#33 lore raw

Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 41 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).

The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.

---
 include/migration/migration.h |  1 +
 migration/ram.c               | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/migration/migration.h +1 -0
index 9387c8c..b3a7f75 100644
@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +6 -0
index 57368e1..efc215a 100644
@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of; it won't bother printing lines that are all this value.
+ * If 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)

migration/ram.c static void reset_ram_globals(void) · line 1069

Quoted 10 lines
migration/ram.c +10 -0
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }

Why? Just alssert that todump!= NULL?

Quoted 4 lines
migration/ram.c +4 -0
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;

migration/ram.c static void reset_ram_globals(void) · line 1083

Quoted 10 lines
migration/ram.c +10 -0
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur + linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur + curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';

Put 1 and 0? Why the dot?

migration/ram.c static void reset_ram_globals(void) · line 1091

Quoted 8 lines
migration/ram.c +8 -0
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}

And once here, why are we doing it this way? We have

find_first_bit(addr, nbits) and find_first_zero_bit(addr, nbits) and friends?

Doiwg the walk by hand looks weird, no?

Later, Juan.

#34 lore raw

Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 46 lines · * Juan Quintela ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).

The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.

---
 include/migration/migration.h |  1 +
 migration/ram.c               | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/migration/migration.h +1 -0
index 9387c8c..b3a7f75 100644
@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +10 -0
index 57368e1..efc215a 100644
@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of; it won't bother printing lines that are all this value.
+ * If 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
Quoted 8 lines

migration/ram.c static void reset_ram_globals(void) · line 1069

Quoted 6 lines
migration/ram.c +6 -0
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }

Why? Just alssert that todump!= NULL?

'migration_bitmap' is static to ram.c, so allowing NULL to get you a dump of the migration_bitmap means that if you call this dump routine from any error path you're debugging anywhere in qemu then you can dump the migration bitmap. e.g. I was adding calls to this in migration.c and migration/postcopy-ram.c in error paths I was trying to debug.

Quoted 14 lines

migration/ram.c static void reset_ram_globals(void) · line 1083

Quoted 13 lines
migration/ram.c +13 -0
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur + linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur + curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
Quoted 1 line

Put 1 and 0? Why the dot?

It's easier to see an occasional '1' in a big field of .'s.

Quoted 10 lines

migration/ram.c static void reset_ram_globals(void) · line 1091

Quoted 8 lines
migration/ram.c +8 -0
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}
Quoted 6 lines

And once here, why are we doing it this way? We have

find_first_bit(addr, nbits) and find_first_zero_bit(addr, nbits) and friends?

Doiwg the walk by hand looks weird, no?

Here's a compile-tested-only version using find_ - it's bigger, if you think it's better I can use this instead:

void ram_debug_dump_bitmap(unsigned long *todump, bool expected)

{
    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;

    int64_t cur;
    int64_t linelen = 128;

    if (!todump) {
        todump = migration_bitmap;
    }

    for (cur = 0; cur < ram_pages; cur += linelen) {
        int64_t curb;
        unsigned long next_bit;

        /*
         * Last line; catch the case where the line length
         * is longer than remaining ram
         */
        if (cur + linelen > ram_pages) {
            linelen = ram_pages - cur;
        }
        if (expected) {
            next_bit = find_next_bit(todump, cur + linelen, cur);
        } else {
            next_bit = find_next_zero_bit(todump, cur + linelen, cur);
        }
        if (next_bit >= (cur + linelen)) {
            continue;
        }

        for (curb = 0; curb < linelen; curb++) {
            bool thisbit = test_bit(cur + curb, todump);
            fputc(thisbit ? '1' : '.', stderr);
        }
        fputc('\n', stderr);
    }
}

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by
#35 lore raw

Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 56 lines · Dr. David Alan Gilbert
Quoted 46 lines · * Juan Quintela ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).

The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.

---
 include/migration/migration.h |  1 +
 migration/ram.c               | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/migration/migration.h +1 -0
index 9387c8c..b3a7f75 100644
@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +10 -0
index 57368e1..efc215a 100644
@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of; it won't bother printing lines that are all this value.
+ * If 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
Quoted 8 lines

migration/ram.c static void reset_ram_globals(void) · line 1069

Quoted 6 lines
migration/ram.c +6 -0
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }

Why? Just alssert that todump!= NULL?

Quoted 6 lines

'migration_bitmap' is static to ram.c, so allowing NULL to get you a dump of the migration_bitmap means that if you call this dump routine from any error path you're debugging anywhere in qemu then you can dump the migration bitmap. e.g. I was adding calls to this in migration.c and migration/postcopy-ram.c in error paths I was trying to debug.

ok.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
migration/ram.c +13 -0
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur + linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur + curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
Quoted 3 lines
Quoted 1 line

Put 1 and 0? Why the dot?

It's easier to see an occasional '1' in a big field of .'s.

ok.

Quoted 60 lines
Quoted 11 lines

migration/ram.c static void reset_ram_globals(void) · line 1091

Quoted 9 lines
migration/ram.c +8 -0
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}
Quoted 6 lines

And once here, why are we doing it this way? We have

find_first_bit(addr, nbits) and find_first_zero_bit(addr, nbits) and friends?

Doiwg the walk by hand looks weird, no?

Here's a compile-tested-only version using find_ - it's bigger, if you think it's better I can use this instead:

void ram_debug_dump_bitmap(unsigned long *todump, bool expected)

{
    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;

    int64_t cur;
    int64_t linelen = 128;

    if (!todump) {
        todump = migration_bitmap;
    }

    for (cur = 0; cur < ram_pages; cur += linelen) {
        int64_t curb;
        unsigned long next_bit;

        /*
         * Last line; catch the case where the line length
         * is longer than remaining ram
         */
        if (cur + linelen > ram_pages) {
            linelen = ram_pages - cur;
        }
        if (expected) {
            next_bit = find_next_bit(todump, cur + linelen, cur);
        } else {
            next_bit = find_next_zero_bit(todump, cur + linelen, cur);
        }
        if (next_bit >= (cur + linelen)) {
            continue;
        }

        for (curb = 0; curb < linelen; curb++) {
            bool thisbit = test_bit(cur + curb, todump);
            fputc(thisbit ? '1' : '.', stderr);
        }
        fputc('\n', stderr);
    }
}
Quoted 1 line

Dave

Quoted 6 lines
Quoted 4 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 7/42
#36 lore raw

Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text

In reply to Dr. David Alan Gilbert (git)
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).

The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.

Amit

#37 lore raw

PATCH v7 8/42 migrate_init: Call from savevm

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

---
 include/migration/migration.h | 4 +---
 include/qemu/typedefs.h       | 1 +
 migration/migration.c         | 2 +-
 migration/savevm.c            | 2 ++
 4 files changed, 5 insertions(+), 4 deletions(-)
include/migration/migration.h +1 -3
index b3a7f75..414c5cf 100644
@@ -41,10 +41,7 @@ struct MigrationParams {
     bool shared;
 };
 
-typedef struct MigrationState MigrationState;
-
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
-
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -115,6 +112,7 @@ int migrate_fd_close(MigrationState *s);
 
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
+MigrationState *migrate_init(const MigrationParams *params);
 bool migration_in_setup(MigrationState *);
 bool migration_has_finished(MigrationState *);
 bool migration_has_failed(MigrationState *);
include/qemu/typedefs.h +1 -0
index fc7c70e..8403856 100644
@@ -41,6 +41,7 @@ typedef struct MemoryRegion MemoryRegion;
 typedef struct MemoryRegionSection MemoryRegionSection;
 typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct MigrationParams MigrationParams;
+typedef struct MigrationState MigrationState;
 typedef struct Monitor Monitor;
 typedef struct MouseTransformInfo MouseTransformInfo;
 typedef struct MSIMessage MSIMessage;
migration/migration.c +1 -1
index b04b457..3cd7f4b 100644
@@ -489,7 +489,7 @@ bool migration_has_failed(MigrationState *s)
             s->state == MIGRATION_STATUS_FAILED);
 }
 
-static MigrationState *migrate_init(const MigrationParams *params)
+MigrationState *migrate_init(const MigrationParams *params)
 {
     MigrationState *s = migrate_get_current();
     int64_t bandwidth_limit = s->bandwidth_limit;
migration/savevm.c +2 -0
index 2091882..aaf8c5c 100644
@@ -903,6 +903,8 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
         .blk = 0,
         .shared = 0
     };
+    MigrationState *ms = migrate_init(&params);
+    ms->file = f;
 
     if (qemu_savevm_state_blocked(errp)) {
         return -EINVAL;
Signature · 1 line
2.4.3
Reviewed-by on v7 8/42
#38 lore raw

Re: [Qemu-devel] [PATCH v7 08/42] migrate_init: Call from savevm

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

#39 lore raw

Re: [Qemu-devel] [PATCH v7 08/42] migrate_init: Call from savevm

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

Amit

#40 lore raw

PATCH v7 9/42 Rename save_live_complete to save_live_complete_precopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious.

---
 hw/ppc/spapr.c              |  2 +-
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  2 +-
 migration/block.c           |  2 +-
 migration/migration.c       |  2 +-
 migration/ram.c             |  2 +-
 migration/savevm.c          | 10 +++++-----
 trace-events                |  2 +-
 8 files changed, 12 insertions(+), 12 deletions(-)
hw/ppc/spapr.c +1 -1
index f174e5a..2f8155d 100644
@@ -1375,7 +1375,7 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id)
 static SaveVMHandlers savevm_htab_handlers = {
     .save_live_setup = htab_save_setup,
     .save_live_iterate = htab_save_iterate,
-    .save_live_complete = htab_save_complete,
+    .save_live_complete_precopy = htab_save_complete,
     .load_state = htab_load,
 };
 
include/migration/vmstate.h +1 -1
index 7153b1e..074747c 100644
@@ -40,7 +40,7 @@ typedef struct SaveVMHandlers {
     SaveStateHandler *save_state;
 
     void (*cancel)(void *opaque);
-    int (*save_live_complete)(QEMUFile *f, void *opaque);
+    int (*save_live_complete_precopy)(QEMUFile *f, void *opaque);
 
     /* This runs both outside and inside the iothread lock.  */
     bool (*is_active)(void *opaque);
include/sysemu/sysemu.h +1 -1
index 7e42f76..6dae2db 100644
@@ -87,7 +87,7 @@ void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
 void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
-void qemu_savevm_state_complete(QEMUFile *f);
+void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
 int qemu_loadvm_state(QEMUFile *f);
migration/block.c +1 -1
index ddb59cc..3005668 100644
@@ -877,7 +877,7 @@ static SaveVMHandlers savevm_block_handlers = {
     .set_params = block_set_params,
     .save_live_setup = block_save_setup,
     .save_live_iterate = block_save_iterate,
-    .save_live_complete = block_save_complete,
+    .save_live_complete_precopy = block_save_complete,
     .save_live_pending = block_save_pending,
     .load_state = block_load,
     .cancel = block_migration_cancel,
migration/migration.c +1 -1
index 3cd7f4b..295f15a 100644
@@ -794,7 +794,7 @@ static void *migration_thread(void *opaque)
                 ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete(s->file);
+                    qemu_savevm_state_complete_precopy(s->file);
                 }
                 qemu_mutex_unlock_iothread();
 
migration/ram.c +1 -1
index efc215a..492ed8a 100644
@@ -1604,7 +1604,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
-    .save_live_complete = ram_save_complete,
+    .save_live_complete_precopy = ram_save_complete,
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
migration/savevm.c +5 -5
index aaf8c5c..f9168ac 100644
@@ -793,19 +793,19 @@ static bool should_send_vmdesc(void)
     return !machine->suppress_vmdesc;
 }
 
-void qemu_savevm_state_complete(QEMUFile *f)
+void qemu_savevm_state_complete_precopy(QEMUFile *f)
 {
     QJSON *vmdesc;
     int vmdesc_len;
     SaveStateEntry *se;
     int ret;
 
-    trace_savevm_state_complete();
+    trace_savevm_state_complete_precopy();
 
     cpu_synchronize_all_states();
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
-        if (!se->ops || !se->ops->save_live_complete) {
+        if (!se->ops || !se->ops->save_live_complete_precopy) {
             continue;
         }
         if (se->ops && se->ops->is_active) {
@@ -817,7 +817,7 @@ void qemu_savevm_state_complete(QEMUFile *f)
 
         save_section_header(f, se, QEMU_VM_SECTION_END);
 
-        ret = se->ops->save_live_complete(f, se->opaque);
+        ret = se->ops->save_live_complete_precopy(f, se->opaque);
         trace_savevm_section_end(se->idstr, se->section_id, ret);
         save_section_footer(f, se);
         if (ret < 0) {
@@ -923,7 +923,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
 
     ret = qemu_file_get_error(f);
     if (ret == 0) {
-        qemu_savevm_state_complete(f);
+        qemu_savevm_state_complete_precopy(f);
         ret = qemu_file_get_error(f);
     }
     if (ret != 0) {
trace-events +1 -1
index 1abca7a..d539528 100644
@@ -1188,7 +1188,7 @@ savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, sectio
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
-savevm_state_complete(void) ""
+savevm_state_complete_precopy(void) ""
 savevm_state_cancel(void) ""
 vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s"
 vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s"
Signature · 1 line
2.4.3
Reviewed-by on v7 9/42
#41 lore raw

Re: [Qemu-devel] [PATCH v7 09/42] Rename save_live_complete to save_live_complete_precopy

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious.

I will have called it save_live_precopy() and just drop the complete to make names shorter, but as I am not doing the coding ....

#42 lore raw

PATCH v7 10/42 Return path: Open a return path on QEMUFile for sockets

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's.

---
 include/migration/qemu-file.h |  7 +++++
 migration/qemu-file-unix.c    | 69 +++++++++++++++++++++++++++++++++++++------
 migration/qemu-file.c         | 12 ++++++++
 3 files changed, 79 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
index d43c835..7721c42 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                uint64_t *bytes_sent);
 
 /*
+ * Return a QEMUFile for comms in the opposite direction
+ */
+typedef QEMUFile *(QEMURetPathFunc)(void *opaque);
+
+/*
  * Stop any read or write (depending on flags) on the underlying
  * transport on the QEMUFile.
  * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
@@ -192,6 +198,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
+QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 void qemu_file_change_blocking(QEMUFile *f, bool block);
 
migration/qemu-file-unix.c +60 -9

84 lines not on this page · raw message

migration/qemu-file.c +12 -0
index c746129..7d9d983 100644
@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)
     return f->ops->shut_down(f->opaque, true, true);
 }
 
+/*
+ * Result: QEMUFile* for a 'return path' for comms in the opposite direction
+ *         NULL if not available
+ */
+QEMUFile *qemu_file_get_return_path(QEMUFile *f)
+{
+    if (!f->ops->get_return_path) {
+        return NULL;
+    }
+    return f->ops->get_return_path(f->opaque);
+}
+
 bool qemu_file_mode_is_not_valid(const char *mode)
 {
     if (mode == NULL ||
Signature · 1 line
2.4.3
Reviewed-by on v7 10/42
#43 lore raw

Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Quoted 19 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's.

---
 include/migration/qemu-file.h |  7 +++++
 migration/qemu-file-unix.c    | 69 +++++++++++++++++++++++++++++++++++++------
 migration/qemu-file.c         | 12 ++++++++
 3 files changed, 79 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +0 -0
index d43c835..7721c42 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Hi

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, · line 89

Quoted 5 lines
include/migration/qemu-file.h +5 -0
+/*
+ * Give a QEMUFile* off the same socket but data in the opposite
+ * direction.
+ */
+static QEMUFile *socket_dup_return_path(void *opaque)

We call it dup

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Quoted 11 lines
include/migration/qemu-file.h +11 -0
+{
+    QEMUFileSocket *forward = opaque;
+    QEMUFileSocket *reverse;
+
+    if (qemu_file_get_error(forward->file)) {
+        /* If the forward file is in error, don't try and open a return */
+        return NULL;
+    }
+
+    reverse = g_malloc0(sizeof(QEMUFileSocket));
+    reverse->fd = forward->fd;

But we don't dup it :p

For the cest, I am ok with the patch.

Quoted 64 lines
include/migration/qemu-file.h +19 -9
+    /* I don't think there's a better way to tell which direction 'this' is */
+    if (forward->file->ops->get_buffer != NULL) {
+        /* being called from the read side, so we need to be able to write */
+        return qemu_fopen_ops(reverse, &socket_return_write_ops);
+    } else {
+        return qemu_fopen_ops(reverse, &socket_return_read_ops);
+    }
+}
+
 static ssize_t unix_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
                                   int64_t pos)
 {
@@ -204,18 +254,19 @@ QEMUFile *qemu_fdopen(int fd, const char *mode)
 }
 
 static const QEMUFileOps socket_read_ops = {
-    .get_fd     = socket_get_fd,
-    .get_buffer = socket_get_buffer,
-    .close      = socket_close,
-    .shut_down  = socket_shutdown
-
+    .get_fd          = socket_get_fd,
+    .get_buffer      = socket_get_buffer,
+    .close           = socket_close,
+    .shut_down       = socket_shutdown,
+    .get_return_path = socket_dup_return_path
 };
 
 static const QEMUFileOps socket_write_ops = {
-    .get_fd        = socket_get_fd,
-    .writev_buffer = socket_writev_buffer,
-    .close         = socket_close,
-    .shut_down     = socket_shutdown
+    .get_fd          = socket_get_fd,
+    .writev_buffer   = socket_writev_buffer,
+    .close           = socket_close,
+    .shut_down       = socket_shutdown,
+    .get_return_path = socket_dup_return_path
 };
 
 QEMUFile *qemu_fopen_socket(int fd, const char *mode)
migration/qemu-file.c +12 -0
index c746129..7d9d983 100644
@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)
     return f->ops->shut_down(f->opaque, true, true);
 }
 
+/*
+ * Result: QEMUFile* for a 'return path' for comms in the opposite direction
+ *         NULL if not available
+ */
+QEMUFile *qemu_file_get_return_path(QEMUFile *f)
+{
+    if (!f->ops->get_return_path) {
+        return NULL;
+    }
+    return f->ops->get_return_path(f->opaque);
+}
+
 bool qemu_file_mode_is_not_valid(const char *mode)
 {
     if (mode == NULL ||
#44 lore raw

Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets

Quoted 31 lines · * Juan Quintela ([email protected]) wrote:

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Quoted 19 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's.

---
 include/migration/qemu-file.h |  7 +++++
 migration/qemu-file-unix.c    | 69 +++++++++++++++++++++++++++++++++++++------
 migration/qemu-file.c         | 12 ++++++++
 3 files changed, 79 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +0 -0
index d43c835..7721c42 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Hi

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, · line 89

Quoted 5 lines
include/migration/qemu-file.h +5 -0
+/*
+ * Give a QEMUFile* off the same socket but data in the opposite
+ * direction.
+ */
+static QEMUFile *socket_dup_return_path(void *opaque)

We call it dup

Quoted 13 lines

include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,

Quoted 11 lines
include/migration/qemu-file.h +11 -0
+{
+    QEMUFileSocket *forward = opaque;
+    QEMUFileSocket *reverse;
+
+    if (qemu_file_get_error(forward->file)) {
+        /* If the forward file is in error, don't try and open a return */
+        return NULL;
+    }
+
+    reverse = g_malloc0(sizeof(QEMUFileSocket));
+    reverse->fd = forward->fd;

But we don't dup it :p

Oh yeh, we used to :-) I've replaced _dup with _get

Quoted 3 lines

For the cest, I am ok with the patch.

Thanks.

Dave

Quoted 51 lines
Quoted 51 lines
include/migration/qemu-file.h +19 -9
+    /* I don't think there's a better way to tell which direction 'this' is */
+    if (forward->file->ops->get_buffer != NULL) {
+        /* being called from the read side, so we need to be able to write */
+        return qemu_fopen_ops(reverse, &socket_return_write_ops);
+    } else {
+        return qemu_fopen_ops(reverse, &socket_return_read_ops);
+    }
+}
+
 static ssize_t unix_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
                                   int64_t pos)
 {
@@ -204,18 +254,19 @@ QEMUFile *qemu_fdopen(int fd, const char *mode)
 }
 
 static const QEMUFileOps socket_read_ops = {
-    .get_fd     = socket_get_fd,
-    .get_buffer = socket_get_buffer,
-    .close      = socket_close,
-    .shut_down  = socket_shutdown
-
+    .get_fd          = socket_get_fd,
+    .get_buffer      = socket_get_buffer,
+    .close           = socket_close,
+    .shut_down       = socket_shutdown,
+    .get_return_path = socket_dup_return_path
 };
 
 static const QEMUFileOps socket_write_ops = {
-    .get_fd        = socket_get_fd,
-    .writev_buffer = socket_writev_buffer,
-    .close         = socket_close,
-    .shut_down     = socket_shutdown
+    .get_fd          = socket_get_fd,
+    .writev_buffer   = socket_writev_buffer,
+    .close           = socket_close,
+    .shut_down       = socket_shutdown,
+    .get_return_path = socket_dup_return_path
 };
 
 QEMUFile *qemu_fopen_socket(int fd, const char *mode)
migration/qemu-file.c +0 -0
index c746129..7d9d983 100644
@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)
     return f->ops->shut_down(f->opaque, true, true);
 }
 
Quoted 15 lines
Quoted 5 lines
migration/qemu-file.c +5 -0
+/*
+ * Result: QEMUFile* for a 'return path' for comms in the opposite direction
+ *         NULL if not available
+ */
+QEMUFile *qemu_file_get_return_path(QEMUFile *f)

migration/qemu-file.c int qemu_file_shutdown(QEMUFile *f) · line 60

Quoted 10 lines
migration/qemu-file.c +7 -0
+{
+    if (!f->ops->get_return_path) {
+        return NULL;
+    }
+    return f->ops->get_return_path(f->opaque);
+}
+
 bool qemu_file_mode_is_not_valid(const char *mode)
 {
     if (mode == NULL ||

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 10/42
#45 lore raw

Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's.

Thanks, this looks better than the dup way of doing it.

Amit

#46 lore raw

PATCH v7 11/42 Return path: socket_writev_buffer: Block even on non-blocking fd's

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

---
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +36 -5
index 561621e..b6c55ab 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN && err != EWOULDBLOCK) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
Signature · 1 line
2.4.3
Reviewed-by on v7 11/42
#47 lore raw

Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

Quoted 16 lines
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +2 -2
index 561621e..b6c55ab 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {

migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 51

Quoted 10 lines
migration/qemu-file-unix.c +7 -3
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }

ion_send() can return -1 on error

This looks a "hacky way" to look for it, althoght my understanding is that it is correct, so the review-by

Quoted 30 lines
migration/qemu-file-unix.c +27 -0
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN && err != EWOULDBLOCK) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
#48 lore raw

Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 10 lines · * Juan Quintela ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

Thanks,

Quoted 29 lines
Quoted 16 lines
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +2 -2
index 561621e..b6c55ab 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {

migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 51

Quoted 10 lines
migration/qemu-file-unix.c +7 -3
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }

ion_send() can return -1 on error

Quoted 2 lines

This looks a "hacky way" to look for it, althoght my understanding is that it is correct, so the review-by

I have to check the socket_error in two cases, one where len == -1 , and the other where it's only sent some of the data. Given I have to do it in both cases it doesn't seme worth explicitly checking for the -1.

Dave

Quoted 29 lines
Quoted 29 lines
migration/qemu-file-unix.c +27 -0
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN && err != EWOULDBLOCK) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
Quoted 1 line

migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 81

Quoted 1 line
migration/qemu-file-unix.c +0 -0
 static int socket_get_fd(void *opaque)

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 11/42
#49 lore raw

Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

Amit

#50 lore raw

PATCH v7 12/42 Migration commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

---
 include/migration/migration.h |  1 +
 include/sysemu/sysemu.h       |  7 +++++++
 migration/savevm.c            | 46 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 4 files changed, 55 insertions(+)
include/migration/migration.h +1 -0
index 414c5cf..8adaa45 100644
@@ -34,6 +34,7 @@
 #define QEMU_VM_SECTION_FULL         0x04
 #define QEMU_VM_SUBSECTION           0x05
 #define QEMU_VM_VMDESCRIPTION        0x06
+#define QEMU_VM_COMMAND              0x07
 #define QEMU_VM_SECTION_FOOTER       0x7e
 
 struct MigrationParams {
include/sysemu/sysemu.h +7 -0
index 6dae2db..5869607 100644
@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+/* Subcommands for QEMU_VM_COMMAND */
+enum qemu_vm_cmd {
+    MIG_CMD_INVALID = 0,   /* Must be 0 */
+};
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
+                              uint16_t len, uint8_t *data);
 int qemu_loadvm_state(QEMUFile *f);
 
 typedef enum DisplayType
migration/savevm.c +46 -0

64 lines not on this page · raw message

trace-events +1 -0
index d539528..73a65c3 100644
@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64
 qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_state_begin(void) ""
Signature · 1 line
2.4.3
#51 lore raw

Re: [Qemu-devel] [PATCH v7 12/42] Migration commands

Quoted 21 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

---
 include/migration/migration.h |  1 +
 include/sysemu/sysemu.h       |  7 +++++++
 migration/savevm.c            | 46 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 4 files changed, 55 insertions(+)
include/migration/migration.h +0 -0
index 414c5cf..8adaa45 100644

include/migration/migration.h line 37

Quoted 4 lines
include/migration/migration.h +1 -0
@@ -34,6 +34,7 @@
 #define QEMU_VM_SECTION_FULL         0x04
 #define QEMU_VM_SUBSECTION           0x05
 #define QEMU_VM_VMDESCRIPTION        0x06
+#define QEMU_VM_COMMAND              0x07

conflicts with configuration section just sent

Quoted 33 lines
include/migration/migration.h +0 -0
 #define QEMU_VM_SECTION_FOOTER       0x7e
 
 struct MigrationParams {
include/sysemu/sysemu.h +7 -0
index 6dae2db..5869607 100644
@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+/* Subcommands for QEMU_VM_COMMAND */
+enum qemu_vm_cmd {
+    MIG_CMD_INVALID = 0,   /* Must be 0 */
+};
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
+                              uint16_t len, uint8_t *data);
 int qemu_loadvm_state(QEMUFile *f);
 
 typedef enum DisplayType
migration/savevm.c +0 -0
index f9168ac..7ce9d21 100644

migration/savevm.c static bool check_section_footer(QEMUFile *f, SaveStateEntry *se) · line 690

Quoted 5 lines
migration/savevm.c +2 -0
@@ -686,6 +686,23 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)
     return true;
 }
 
+/* Send a 'QEMU_VM_COMMAND' type element with the command
+ * and associated data.

Please, use doxygen comments for new functions :p

Quoted 25 lines
migration/savevm.c +18 -0
+ */
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, (uint16_t)command);
+    qemu_put_be16(f, len);
+    if (len) {
+        qemu_put_buffer(f, data, len);
+    }
+    qemu_fflush(f);
+}
+
 bool qemu_savevm_state_blocked(Error **errp)
 {
     SaveStateEntry *se;
@@ -982,6 +999,29 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
+/*
+ * Process an incoming 'QEMU_VM_COMMAND'
+ * negative return on error (will issue error message)

migration/savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 1014

Quoted 10 lines
migration/savevm.c +10 -0
+ */
+static int loadvm_process_command(QEMUFile *f)
+{
+    uint16_t cmd;
+    uint16_t len;
+
+    cmd = qemu_get_be16(f);
+    len = qemu_get_be16(f);
+
+    trace_loadvm_process_command(cmd, len);

trace for load but not for sending?

Quoted 38 lines
migration/savevm.c +16 -0
+    switch (cmd) {
+
+    default:
+        error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
+        return -1;
+    }
+
+    return 0;
+}
+
 struct LoadStateEntry {
     QLIST_ENTRY(LoadStateEntry) entry;
     SaveStateEntry *se;
@@ -1114,6 +1154,12 @@ int qemu_loadvm_state(QEMUFile *f)
                 goto out;
             }
             break;
+        case QEMU_VM_COMMAND:
+            ret = loadvm_process_command(f);
+            if (ret < 0) {
+                goto out;
+            }
+            break;
         default:
             error_report("Unknown savevm section type %d", section_type);
             ret = -EINVAL;
trace-events +1 -0
index d539528..73a65c3 100644
@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64
 qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_state_begin(void) ""
#52 lore raw

Re: [Qemu-devel] [PATCH v7 12/42] Migration commands

Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

---
 include/migration/migration.h |  1 +
 include/sysemu/sysemu.h       |  7 +++++++
 migration/savevm.c            | 46 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 4 files changed, 55 insertions(+)
Quoted 11 lines

include/migration/migration.h line 37

Quoted 9 lines
include/migration/migration.h +1 -0
index 414c5cf..8adaa45 100644
@@ -34,6 +34,7 @@
 #define QEMU_VM_SECTION_FULL         0x04
 #define QEMU_VM_SUBSECTION           0x05
 #define QEMU_VM_VMDESCRIPTION        0x06
+#define QEMU_VM_COMMAND              0x07

conflicts with configuration section just sent

Inc'd to 8.

Quoted 36 lines
Quoted 36 lines
include/migration/migration.h +0 -0
 #define QEMU_VM_SECTION_FOOTER       0x7e
 
 struct MigrationParams {
include/sysemu/sysemu.h +7 -0
index 6dae2db..5869607 100644
@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+/* Subcommands for QEMU_VM_COMMAND */
+enum qemu_vm_cmd {
+    MIG_CMD_INVALID = 0,   /* Must be 0 */
+};
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
+                              uint16_t len, uint8_t *data);
 int qemu_loadvm_state(QEMUFile *f);
 
 typedef enum DisplayType
migration/savevm.c +0 -0
index f9168ac..7ce9d21 100644
@@ -686,6 +686,23 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)
     return true;
 }
 
Quoted 4 lines

migration/savevm.c static bool check_section_footer(QEMUFile *f, SaveStateEntry *se) · line 690

Quoted 2 lines
migration/savevm.c +2 -0
+/* Send a 'QEMU_VM_COMMAND' type element with the command
+ * and associated data.

Please, use doxygen comments for new functions :p

Done.

Quoted 23 lines
Quoted 23 lines
migration/savevm.c +15 -0
+ */
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, (uint16_t)command);
+    qemu_put_be16(f, len);
+    if (len) {
+        qemu_put_buffer(f, data, len);
+    }
+    qemu_fflush(f);
+}
+
 bool qemu_savevm_state_blocked(Error **errp)
 {
     SaveStateEntry *se;
@@ -982,6 +999,29 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
Quoted 15 lines

migration/savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 1014

Quoted 13 lines
migration/savevm.c +13 -0
+/*
+ * Process an incoming 'QEMU_VM_COMMAND'
+ * negative return on error (will issue error message)
+ */
+static int loadvm_process_command(QEMUFile *f)
+{
+    uint16_t cmd;
+    uint16_t len;
+
+    cmd = qemu_get_be16(f);
+    len = qemu_get_be16(f);
+
+    trace_loadvm_process_command(cmd, len);

trace for load but not for sending?

Added.

Dave

Quoted 30 lines
Quoted 30 lines
migration/savevm.c +16 -0
+    switch (cmd) {
+
+    default:
+        error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
+        return -1;
+    }
+
+    return 0;
+}
+
 struct LoadStateEntry {
     QLIST_ENTRY(LoadStateEntry) entry;
     SaveStateEntry *se;
@@ -1114,6 +1154,12 @@ int qemu_loadvm_state(QEMUFile *f)
                 goto out;
             }
             break;
+        case QEMU_VM_COMMAND:
+            ret = loadvm_process_command(f);
+            if (ret < 0) {
+                goto out;
+            }
+            break;
         default:
             error_report("Unknown savevm section type %d", section_type);
             ret = -EINVAL;
trace-events +0 -0
index d539528..73a65c3 100644
Quoted 8 lines

trace-events virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 · line 1189

Quoted 8 lines
trace-events +1 -0
@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64
 qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_state_begin(void) ""

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 12/42
#53 lore raw

Re: [Qemu-devel] [PATCH v7 12/42] Migration commands

In reply to Dr. David Alan Gilbert (git)
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

Amit

#54 lore raw

PATCH v7 13/42 Return path: Control commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination

---
 include/migration/migration.h |  2 ++
 include/sysemu/sysemu.h       |  6 ++++-
 migration/savevm.c            | 60 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  2 ++
 4 files changed, 69 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index 8adaa45..65fe5db 100644
@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    QEMUFile *return_path;
+
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 };
include/sysemu/sysemu.h +5 -1
index 5869607..d8875ca 100644
@@ -84,7 +84,9 @@ void qemu_announce_self(void);
 
 /* Subcommands for QEMU_VM_COMMAND */
 enum qemu_vm_cmd {
-    MIG_CMD_INVALID = 0,   /* Must be 0 */
+    MIG_CMD_INVALID = 0,       /* Must be 0 */
+    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
+    MIG_CMD_PING,              /* Request a PONG on the RP */
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -97,6 +99,8 @@ void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
+void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
+void qemu_savevm_send_open_return_path(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
 typedef enum DisplayType
migration/savevm.c +60 -0

86 lines not on this page · raw message

trace-events +2 -0
index 73a65c3..5967fdf 100644
@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
+loadvm_process_command_ping(uint32_t val) "%x"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
+savevm_send_ping(uint32_t val) "%x"
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
Signature · 1 line
2.4.3
#55 lore raw

Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands

Quoted 33 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination

---
 include/migration/migration.h |  2 ++
 include/sysemu/sysemu.h       |  6 ++++-
 migration/savevm.c            | 60 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  2 ++
 4 files changed, 69 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index 8adaa45..65fe5db 100644
@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    QEMUFile *return_path;
+
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 };
include/sysemu/sysemu.h +0 -0
index 5869607..d8875ca 100644

include/sysemu/sysemu.h void qemu_announce_self(void); · line 89

Quoted 7 lines
include/sysemu/sysemu.h +3 -1
@@ -84,7 +84,9 @@ void qemu_announce_self(void);
 
 /* Subcommands for QEMU_VM_COMMAND */
 enum qemu_vm_cmd {
-    MIG_CMD_INVALID = 0,   /* Must be 0 */
+    MIG_CMD_INVALID = 0,       /* Must be 0 */
+    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
+    MIG_CMD_PING,              /* Request a PONG on the RP */

Add MIG_CMD_MAX

struct cmd_args {
       int len;
       char *name;
} cmd_args[] ={

.[MIG_CMD_INVALID] = {
.len = 0,
.name = "CMD_INVALID"},
.[MIG_CMD_OPEN_RETURN_PATH] = {
.len = 0,
.name = "CMD_OPEN_RETURN_PATH"},
.....
}
}

I have done the initialization by hand, not sure if syntax is ok.

include/sysemu/sysemu.h void qemu_announce_self(void);

Quoted 11 lines
include/sysemu/sysemu.h +2 -0
 static int loadvm_process_command(QEMUFile *f)
 {
+    MigrationIncomingState *mis = migration_incoming_get_current();
     uint16_t cmd;
     uint16_t len;
+    uint32_t tmp32;
 
     cmd = qemu_get_be16(f);
     len = qemu_get_be16(f);
 
     trace_loadvm_process_command(cmd, len);

/* yes, this should go in previous patch */

if (cmd > MIG_CMD_MAX ) {
            error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
     .....
}

if (loadvm_process_command_simple_lencheck(cmd_args[cmd].name, len,

cmd_args[cmd].len) {
                   return -1;
}

switch (cmd) {
case MIG_CMD_OPEN_RETURN_PATH:

        if (mis->return_path) {
            error_report("CMD_OPEN_RETURN_PATH called when RP already open");
            /* Not really a problem, so don't give up */
            return 0;
        }
        mis->return_path = qemu_file_get_return_path(f);
        if (!mis->return_path) {
            error_report("CMD_OPEN_RETURN_PATH failed");
            return -1;
        }
        break;

.....
}

You get the idea. I normally even put the code from the command in a function pointer, but I know that other people don't like it. This way you factor the common code at the beginning of the function.

What do you think?

Quoted 50 lines
     switch (cmd) {
include/sysemu/sysemu.h +31 -0
+    case MIG_CMD_OPEN_RETURN_PATH:
+        if (loadvm_process_command_simple_lencheck("CMD_OPEN_RETURN_PATH",
+                                                   len, 0)) {
+            return -1;
+        }
+        if (mis->return_path) {
+            error_report("CMD_OPEN_RETURN_PATH called when RP already open");
+            /* Not really a problem, so don't give up */
+            return 0;
+        }
+        mis->return_path = qemu_file_get_return_path(f);
+        if (!mis->return_path) {
+            error_report("CMD_OPEN_RETURN_PATH failed");
+            return -1;
+        }
+        break;
+
+    case MIG_CMD_PING:
+        if (loadvm_process_command_simple_lencheck("CMD_PING", len,
+                                                   sizeof(tmp32))) {
+            return -1;
+        }
+        tmp32 = qemu_get_be32(f);
+        trace_loadvm_process_command_ping(tmp32);
+        if (!mis->return_path) {
+            error_report("CMD_PING (0x%x) received with no return path",
+                         tmp32);
+            return -1;
+        }
+        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        break;
 
     default:
         error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
trace-events +2 -0
index 73a65c3..5967fdf 100644
@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
+loadvm_process_command_ping(uint32_t val) "%x"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
+savevm_send_ping(uint32_t val) "%x"
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
#56 lore raw

Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands

Quoted 55 lines · * Juan Quintela ([email protected]) wrote:
Quoted 33 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination

---
 include/migration/migration.h |  2 ++
 include/sysemu/sysemu.h       |  6 ++++-
 migration/savevm.c            | 60 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  2 ++
 4 files changed, 69 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index 8adaa45..65fe5db 100644
@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    QEMUFile *return_path;
+
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 };
include/sysemu/sysemu.h +0 -0
index 5869607..d8875ca 100644

include/sysemu/sysemu.h void qemu_announce_self(void); · line 89

Quoted 7 lines
include/sysemu/sysemu.h +3 -1
@@ -84,7 +84,9 @@ void qemu_announce_self(void);
 
 /* Subcommands for QEMU_VM_COMMAND */
 enum qemu_vm_cmd {
-    MIG_CMD_INVALID = 0,   /* Must be 0 */
+    MIG_CMD_INVALID = 0,       /* Must be 0 */
+    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
+    MIG_CMD_PING,              /* Request a PONG on the RP */

Add MIG_CMD_MAX

struct cmd_args {
       int len;
       char *name;
} cmd_args[] ={
.[MIG_CMD_INVALID] = {
   .len = 0,
   .name = "CMD_INVALID"},
.[MIG_CMD_OPEN_RETURN_PATH] = {
   .len = 0,
   .name = "CMD_OPEN_RETURN_PATH"},
Quoted 5 lines

.....
}
}

I have done the initialization by hand, not sure if syntax is ok.

Pretty close, no '.] before the [] - we end up with (at the end of the series):

static struct mig_cmd_args {
ssize_t len; /* -1 = variable */
const char *name;

} mig_cmd_args[] = {
    [MIG_CMD_INVALID]          = { .len = -1, .name = "INVALID" },
    [MIG_CMD_OPEN_RETURN_PATH] = { .len =  0, .name = "OPEN_RETURN_PATH" },
    [MIG_CMD_PING]             = { .len = sizeof(uint32_t), .name = "PING" },
    [MIG_CMD_POSTCOPY_ADVISE]  = { .len = 16, .name = "POSTCOPY_ADVISE" },
    [MIG_CMD_POSTCOPY_LISTEN]  = { .len =  0, .name = "POSTCOPY_LISTEN" },
    [MIG_CMD_POSTCOPY_RUN]     = { .len =  0, .name = "POSTCOPY_RUN" },
    [MIG_CMD_POSTCOPY_RAM_DISCARD] =
                                 { .len = -1, .name = "POSTCOPY_RAM_DISCARD" },
    [MIG_CMD_MAX]              = { .len = -1, .name = "MAX" },
};
Quoted 7 lines
Quoted 7 lines
include/sysemu/sysemu.h +2 -0
 static int loadvm_process_command(QEMUFile *f)
 {
+    MigrationIncomingState *mis = migration_incoming_get_current();
     uint16_t cmd;
     uint16_t len;
+    uint32_t tmp32;
Quoted 6 lines
Quoted 4 lines
     cmd = qemu_get_be16(f);
     len = qemu_get_be16(f);
 
     trace_loadvm_process_command(cmd, len);

/* yes, this should go in previous patch */

Yes, done.

Quoted 24 lines
if (cmd > MIG_CMD_MAX ) {
            error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
     .....
}

if (loadvm_process_command_simple_lencheck(cmd_args[cmd].name, len,

cmd_args[cmd].len) {
                   return -1;
}

switch (cmd) {
case MIG_CMD_OPEN_RETURN_PATH:

        if (mis->return_path) {
            error_report("CMD_OPEN_RETURN_PATH called when RP already open");
            /* Not really a problem, so don't give up */
            return 0;
        }
        mis->return_path = qemu_file_get_return_path(f);
        if (!mis->return_path) {
            error_report("CMD_OPEN_RETURN_PATH failed");
            return -1;
        }
        break;
Quoted 8 lines

.....
}

You get the idea. I normally even put the code from the command in a function pointer, but I know that other people don't like it. This way you factor the common code at the beginning of the function.

What do you think?

Yep, done.

Dave

Quoted 44 lines
Quoted 44 lines

     switch (cmd) {
include/sysemu/sysemu.h +31 -0
+    case MIG_CMD_OPEN_RETURN_PATH:
+        if (loadvm_process_command_simple_lencheck("CMD_OPEN_RETURN_PATH",
+                                                   len, 0)) {
+            return -1;
+        }
+        if (mis->return_path) {
+            error_report("CMD_OPEN_RETURN_PATH called when RP already open");
+            /* Not really a problem, so don't give up */
+            return 0;
+        }
+        mis->return_path = qemu_file_get_return_path(f);
+        if (!mis->return_path) {
+            error_report("CMD_OPEN_RETURN_PATH failed");
+            return -1;
+        }
+        break;
+
+    case MIG_CMD_PING:
+        if (loadvm_process_command_simple_lencheck("CMD_PING", len,
+                                                   sizeof(tmp32))) {
+            return -1;
+        }
+        tmp32 = qemu_get_be32(f);
+        trace_loadvm_process_command_ping(tmp32);
+        if (!mis->return_path) {
+            error_report("CMD_PING (0x%x) received with no return path",
+                         tmp32);
+            return -1;
+        }
+        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        break;
 
     default:
         error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
trace-events +0 -0
index 73a65c3..5967fdf 100644
@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
Quoted 8 lines

trace-events qemu_loadvm_state_section(unsigned int section_type) "%d" · line 1193

Quoted 8 lines
trace-events +2 -0
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
+loadvm_process_command_ping(uint32_t val) "%x"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
+savevm_send_ping(uint32_t val) "%x"
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 13/42
#57 lore raw

Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination

Amit

#58 lore raw

PATCH v7 14/42 Return path: Send responses from destination to source

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index 65fe5db..36caab9 100644
@@ -42,12 +42,20 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+    MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +45 -0

57 lines not on this page · raw message

migration/savevm.c +1 -1
index a995014..d424c2a 100644
@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 5967fdf..5738e3f 100644
@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 1 line
2.4.3
#59 lore raw

Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source

Quoted 76 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index 65fe5db..36caab9 100644
@@ -42,12 +42,20 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+    MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +3 -0
index 295f15a..afb19a1 100644
@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
     QLIST_INIT(&mis_current->loadvm_handlers);
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)
     qemu_coroutine_enter(co, f);
 }
 
+/*
+ * Send a message on the return channel back to the source

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197

Quoted 10 lines
migration/migration.c +10 -0
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data)
+{
+    trace_migrate_send_rp_message((int)message_type, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)message_type);
+    qemu_put_be16(mis->return_path, len);

if (len) {

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198

Quoted 1 line
migration/migration.c +1 -0
+    qemu_put_buffer(mis->return_path, data, len);

}

?

We check for zero sized command on control commands but not on responses?

Quoted 59 lines
migration/migration.c +31 -0
+    qemu_fflush(mis->return_path);
+    qemu_mutex_unlock(&mis->rp_mutex);
+}
+
+/*
+ * Send a 'SHUT' message on the return channel with the given value
+ * to indicate that we've finished with the RP.  Non-0 value indicates
+ * error.
+ */
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_MSG_SHUT, sizeof(buf), &buf);
+}
+
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_MSG_PONG, sizeof(buf), &buf);
+}
+
 /* amount of nanoseconds we are willing to wait for migration to be down.
  * the choice of nanoseconds is because it is the maximum resolution that
  * get_clock() can achieve. It is an internal measure. All user-visible
migration/savevm.c +1 -1
index a995014..d424c2a 100644
@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 5967fdf..5738e3f 100644
@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
#60 lore raw

Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source

Quoted 89 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index 65fe5db..36caab9 100644
@@ -42,12 +42,20 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+    MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +3 -0
index 295f15a..afb19a1 100644
@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
     QLIST_INIT(&mis_current->loadvm_handlers);
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)
     qemu_coroutine_enter(co, f);
 }
 
+/*
+ * Send a message on the return channel back to the source

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197

Quoted 10 lines
migration/migration.c +10 -0
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data)
+{
+    trace_migrate_send_rp_message((int)message_type, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)message_type);
+    qemu_put_be16(mis->return_path, len);

if (len) {

Quoted 8 lines

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198

Quoted 1 line
migration/migration.c +1 -0
+    qemu_put_buffer(mis->return_path, data, len);

}

?

We check for zero sized command on control commands but not on responses?

Or should I remove the check in the control commands case? qemu_put_buffer looks like it's safe for size == 0

Dave

Quoted 59 lines
Quoted 59 lines
migration/migration.c +31 -0
+    qemu_fflush(mis->return_path);
+    qemu_mutex_unlock(&mis->rp_mutex);
+}
+
+/*
+ * Send a 'SHUT' message on the return channel with the given value
+ * to indicate that we've finished with the RP.  Non-0 value indicates
+ * error.
+ */
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_MSG_SHUT, sizeof(buf), &buf);
+}
+
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_MSG_PONG, sizeof(buf), &buf);
+}
+
 /* amount of nanoseconds we are willing to wait for migration to be down.
  * the choice of nanoseconds is because it is the maximum resolution that
  * get_clock() can achieve. It is an internal measure. All user-visible
migration/savevm.c +1 -1
index a995014..d424c2a 100644
@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 5967fdf..5738e3f 100644
@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
Quoted 1 line

trace-events migrate_fd_cleanup(void) "" · line 1405

Quoted 1 line
trace-events +0 -0
 # migration/rdma.c

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#61 lore raw

Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source

Quoted 94 lines · Dr. David Alan Gilbert
Quoted 93 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index 65fe5db..36caab9 100644
@@ -42,12 +42,20 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+    MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +3 -0
index 295f15a..afb19a1 100644
@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
     QLIST_INIT(&mis_current->loadvm_handlers);
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)
     qemu_coroutine_enter(co, f);
 }
 
+/*
+ * Send a message on the return channel back to the source

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197

Quoted 10 lines
migration/migration.c +10 -0
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data)
+{
+    trace_migrate_send_rp_message((int)message_type, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)message_type);
+    qemu_put_be16(mis->return_path, len);

if (len) {

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198

Quoted 1 line
migration/migration.c +1 -0
+    qemu_put_buffer(mis->return_path, data, len);

}

Quoted 7 lines
Quoted 4 lines

?

We check for zero sized command on control commands but not on responses?

Or should I remove the check in the control commands case? qemu_put_buffer looks like it's safe for size == 0

I would go for this just for consistence?

#62 lore raw

Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source

Quoted 97 lines · * Juan Quintela ([email protected]) wrote:
Quoted 96 lines · Dr. David Alan Gilbert
Quoted 95 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 migration/savevm.c            |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index 65fe5db..36caab9 100644
@@ -42,12 +42,20 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Messages sent on the return path from destination to source */
+enum mig_rp_message_type {
+    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
+    MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_decompress_threads(void);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +3 -0
index 295f15a..afb19a1 100644
@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
     QLIST_INIT(&mis_current->loadvm_handlers);
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)
     qemu_coroutine_enter(co, f);
 }
 
+/*
+ * Send a message on the return channel back to the source

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197

Quoted 10 lines
migration/migration.c +10 -0
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rp_message_type message_type,
+                             uint16_t len, void *data)
+{
+    trace_migrate_send_rp_message((int)message_type, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)message_type);
+    qemu_put_be16(mis->return_path, len);

if (len) {

migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198

Quoted 1 line
migration/migration.c +1 -0
+    qemu_put_buffer(mis->return_path, data, len);

}

?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

We check for zero sized command on control commands but not on responses?

Or should I remove the check in the control commands case? qemu_put_buffer looks like it's safe for size == 0

I would go for this just for consistence?

OK, check removed in both cases.

Dave

(Resend: I dropped the list off the first message) --
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 14/42
#63 lore raw

Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source

In reply to Dr. David Alan Gilbert (git)
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler

Amit

#64 lore raw

PATCH v7 15/42 Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 177 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  12 +++
 3 files changed, 196 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 36caab9..868f59a 100644
@@ -77,6 +77,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +176 -1

208 lines not on this page · raw message

trace-events +12 -0
index 5738e3f..282cde1 100644
@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
 flic_reset_failed(int err) "flic: reset failed %d"
 
 # migration.c
+await_return_path_close_on_source_close(void) ""
+await_return_path_close_on_source_joining(void) ""
 migrate_set_state(int new_state) "new state %d"
 migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+open_return_path_on_source(void) ""
+open_return_path_on_source_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 1 line
2.4.3
#65 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

migration/migration.c +6 -0
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 868

Quoted 10 lines
migration/migration.c +10 -0
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&

What can make rp == NULL?
THinking about that, could you mean *rp here?

Quoted 13 lines
migration/migration.c +13 -0
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 891

Quoted 10 lines
migration/migration.c +9 -0
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);

I know this is a big request, but getting an array with messages length and message names to be able to print nice error messages looks ilke good?

Quoted 6 lines
migration/migration.c +6 -0
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 907

Quoted 10 lines
migration/migration.c +10 -0
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the message and the data */
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);

make local variable and call it sibling_error or whatever you like?

Quoted 4 lines
migration/migration.c +4 -0
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 921

Quoted 10 lines
migration/migration.c +10 -0
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_MSG_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);

unused?
Althought I guess it is used somewhere to make sure that the value is the same that whatever we did the ping. credentials?

I can't see with this and previous patch what value is sent here.

Quoted 8 lines
migration/migration.c +8 -0
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (rp && qemu_file_get_error(rp)) {

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 939

Quoted 10 lines
migration/migration.c +10 -0
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */

unused_by_know attribute required O:-)

#66 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

migration/migration.c +11 -0
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 868

Quoted 5 lines
migration/migration.c +5 -0
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&

What can make rp == NULL?
THinking about that, could you mean *rp here?

I've reworked this; it was meant to catch the case of the rp being closed early, but was racy.

I've now got:

    while (!ms->rp_state.error && !qemu_file_get_error(rp) &&
           migration_already_active(ms)) {

and now the rp qemu_file gets closed at the end of the thread, anyone else that wants the rp_thread to exit sets the error flag.

Quoted 18 lines
Quoted 18 lines
migration/migration.c +17 -0
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 891

Quoted 5 lines
migration/migration.c +5 -0
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);

I know this is a big request, but getting an array with messages length and message names to be able to print nice error messages looks ilke good?

Done; (same way as for the commands on the forward path).

Quoted 10 lines
Quoted 10 lines
migration/migration.c +10 -0
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 907

Quoted 6 lines
migration/migration.c +6 -0
+        }
+
+        /* OK, we have the message and the data */
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);

make local variable and call it sibling_error or whatever you like?

Done.

Quoted 15 lines
Quoted 4 lines
migration/migration.c +4 -0
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 921

Quoted 10 lines
migration/migration.c +10 -0
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_MSG_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
Quoted 5 lines

unused?
Althought I guess it is used somewhere to make sure that the value is the same that whatever we did the ping. credentials?

I can't see with this and previous patch what value is sent here.

I'm not using the ping/pong messages for anything active, they exist primarily as a debug and tracing aid.

Quoted 12 lines
Quoted 12 lines
migration/migration.c +12 -0
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 939

Quoted 6 lines
migration/migration.c +6 -0
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */

unused_by_know attribute required O:-)

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#67 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 847

Quoted 13 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

migration/migration.c +5 -1
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)

Can you rename this to something like

mark_source_rp_bad()

?

Intent is clearer that way.

Also, the comment says caller will print something, but the invocations below are a mix of printfs and traces. Not saying the caller has to print always, but maybe only comment needs update.

Quoted 45 lines
migration/migration.c +44 -0
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);
+            source_return_path_bad(ms);

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 902

Quoted 10 lines
migration/migration.c +10 -0
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }

Amit

#68 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

Quoted 17 lines · * Amit Shah ([email protected]) wrote:

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 847

Quoted 13 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

migration/migration.c +5 -1
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)

Can you rename this to something like

Quoted 5 lines

mark_source_rp_bad()

?

Intent is clearer that way.

Done.

Quoted 3 lines

Also, the comment says caller will print something, but the invocations below are a mix of printfs and traces. Not saying the caller has to print always, but maybe only comment needs update.

Yes, I've changed the comment, and changed one of the traces into an error_report.

Thanks,

Dave

Quoted 50 lines
Quoted 50 lines
migration/migration.c +48 -0
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 902

Quoted 6 lines
migration/migration.c +6 -0
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#69 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)

Hi Dave,

Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
  include/migration/migration.h |   8 ++
  migration/migration.c         | 177 +++++++++++++++++++++++++++++++++++++++++-
  trace-events                  |  12 +++
  3 files changed, 196 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index 36caab9..868f59a 100644

include/migration/migration.h struct MigrationState · line 83

Quoted 7 lines
include/migration/migration.h +4 -0
@@ -77,6 +77,14 @@ struct MigrationState
 
      int state;
      MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;

There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.

But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.

Thanks,
zhanghailiang

Quoted 254 lines
include/migration/migration.h +4 -0
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
      double mbps;
      int64_t total_time;
      int64_t downtime;
migration/migration.c +176 -1
index afb19a1..fb2f491 100644
@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
      return params;
  }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIGRATION_STATUS_ACTIVE:
+    case MIGRATION_STATUS_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
  static void get_xbzrle_cache_stats(MigrationInfo *info)
  {
      if (migrate_use_xbzrle()) {
@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
      }
  }
 
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
  static void migrate_fd_cleanup(void *opaque)
  {
      MigrationState *s = opaque;
@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)
      qemu_bh_delete(s->cleanup_bh);
      s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
      if (s->file) {
          trace_migrate_fd_cleanup();
          qemu_mutex_unlock_iothread();
@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)
      QEMUFile *f = migrate_get_current()->file;
      trace_migrate_fd_cancel();
 
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);
+    }
+
      do {
          old_state = s->state;
          if (old_state != MIGRATION_STATUS_SETUP &&
@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)
      return s->xbzrle_cache_size;
  }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the message and the data */
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_MSG_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_return_path_on_source(MigrationState *ms)
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_return_path_on_source();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_return_path_on_source_continue();
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+/* Returns 0 if the RP was ok, otherwise there was an error on the RP */
+static int await_return_path_close_on_source(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->rp_state.file) {
+        qemu_file_shutdown(ms->rp_state.file);
+    }
+    trace_await_return_path_close_on_source_joining();
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    trace_await_return_path_close_on_source_close();
+    return ms->rp_state.error;
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
  static void *migration_thread(void *opaque)
  {
      MigrationState *s = opaque;
trace-events +12 -0
index 5738e3f..282cde1 100644
@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
  flic_reset_failed(int err) "flic: reset failed %d"
 
  # migration.c
+await_return_path_close_on_source_close(void) ""
+await_return_path_close_on_source_joining(void) ""
  migrate_set_state(int new_state) "new state %d"
  migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
  migrate_fd_error(void) ""
  migrate_fd_cancel(void) ""
  migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
  migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+open_return_path_on_source(void) ""
+open_return_path_on_source_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
  migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
  # migration/rdma.c
#70 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

Quoted 31 lines · * zhanghailiang ([email protected]) wrote:

Hi Dave,

Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 177 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  12 +++
 3 files changed, 196 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index 36caab9..868f59a 100644

include/migration/migration.h struct MigrationState · line 83

Quoted 7 lines
include/migration/migration.h +4 -0
@@ -77,6 +77,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;

There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.

Quoted 5 lines

But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.

Would the following be clearer:

On the source make the existing migration file:

       QEMUFile  *to_dst_file;
  and for the return path
       QEMUFile  *from_dst_dile;

and then on the destination, the incoming migration stream:

       QEMUFile  *from_src_file;
  and then the return path on the destionation:
       QEMUFile  *to_src_file;

Dave

Quoted 256 lines

Thanks,
zhanghailiang

Quoted 252 lines
include/migration/migration.h +4 -0
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +176 -1
index afb19a1..fb2f491 100644
@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     return params;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIGRATION_STATUS_ACTIVE:
+    case MIGRATION_STATUS_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }
 
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);
+    }
+
     do {
         old_state = s->state;
         if (old_state != MIGRATION_STATUS_SETUP &&
@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the message and the data */
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_MSG_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_return_path_on_source(MigrationState *ms)
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_return_path_on_source();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_return_path_on_source_continue();
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+/* Returns 0 if the RP was ok, otherwise there was an error on the RP */
+static int await_return_path_close_on_source(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->rp_state.file) {
+        qemu_file_shutdown(ms->rp_state.file);
+    }
+    trace_await_return_path_close_on_source_joining();
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    trace_await_return_path_close_on_source_close();
+    return ms->rp_state.error;
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
trace-events +12 -0
index 5738e3f..282cde1 100644
@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
 flic_reset_failed(int err) "flic: reset failed %d"
 
 # migration.c
+await_return_path_close_on_source_close(void) ""
+await_return_path_close_on_source_joining(void) ""
 migrate_set_state(int new_state) "new state %d"
 migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+open_return_path_on_source(void) ""
+open_return_path_on_source_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
Quoted 4 lines

trace-events flic_no_device_api(int err) "flic: no Device Contral API support %d" · line 1417

Quoted 2 lines
trace-events +0 -0
 # migration/rdma.c

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#71 lore raw

Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path

Quoted 32 lines · Dr. David Alan Gilbert
Quoted 31 lines · * zhanghailiang ([email protected]) wrote:

Hi Dave,

Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
  include/migration/migration.h |   8 ++
  migration/migration.c         | 177 +++++++++++++++++++++++++++++++++++++++++-
  trace-events                  |  12 +++
  3 files changed, 196 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index 36caab9..868f59a 100644

include/migration/migration.h struct MigrationState · line 83

Quoted 7 lines
include/migration/migration.h +4 -0
@@ -77,6 +77,14 @@ struct MigrationState
 
      int state;
      MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;

There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.

Quoted 8 lines
Quoted 5 lines

But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.

Would the following be clearer:

Yes, it is clearer and more graceful :)

Quoted 5 lines
   On the source make the existing migration file:
        QEMUFile  *to_dst_file;
   and for the return path
        QEMUFile  *from_dst_dile;
                            ^

from_dst_file

Quoted 272 lines

and then on the destination, the incoming migration stream:

        QEMUFile  *from_src_file;
   and then the return path on the destionation:
        QEMUFile  *to_src_file;

Dave

Quoted 256 lines

Thanks,
zhanghailiang

Quoted 252 lines
include/migration/migration.h +4 -0
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
      double mbps;
      int64_t total_time;
      int64_t downtime;
migration/migration.c +176 -1
index afb19a1..fb2f491 100644
@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
      return params;
  }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIGRATION_STATUS_ACTIVE:
+    case MIGRATION_STATUS_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
  static void get_xbzrle_cache_stats(MigrationInfo *info)
  {
      if (migrate_use_xbzrle()) {
@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
      }
  }
 
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
  static void migrate_fd_cleanup(void *opaque)
  {
      MigrationState *s = opaque;
@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)
      qemu_bh_delete(s->cleanup_bh);
      s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
      if (s->file) {
          trace_migrate_fd_cleanup();
          qemu_mutex_unlock_iothread();
@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)
      QEMUFile *f = migrate_get_current()->file;
      trace_migrate_fd_cancel();
 
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);
+    }
+
      do {
          old_state = s->state;
          if (old_state != MIGRATION_STATUS_SETUP &&
@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)
      return s->xbzrle_cache_size;
  }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_type;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_type = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+        case MIG_RP_MSG_PONG:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid message 0x%04x length 0x%04x",
+                    header_type, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received message 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_type, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the message and the data */
+        switch (header_type) {
+        case MIG_RP_MSG_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_MSG_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_return_path_on_source(MigrationState *ms)
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_return_path_on_source();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_return_path_on_source_continue();
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+/* Returns 0 if the RP was ok, otherwise there was an error on the RP */
+static int await_return_path_close_on_source(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->rp_state.file) {
+        qemu_file_shutdown(ms->rp_state.file);
+    }
+    trace_await_return_path_close_on_source_joining();
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    trace_await_return_path_close_on_source_close();
+    return ms->rp_state.error;
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
  static void *migration_thread(void *opaque)
  {
      MigrationState *s = opaque;
trace-events +12 -0
index 5738e3f..282cde1 100644
@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
  flic_reset_failed(int err) "flic: reset failed %d"
 
  # migration.c
+await_return_path_close_on_source_close(void) ""
+await_return_path_close_on_source_joining(void) ""
  migrate_set_state(int new_state) "new state %d"
  migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
  migrate_fd_error(void) ""
  migrate_fd_cancel(void) ""
  migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
  migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+open_return_path_on_source(void) ""
+open_return_path_on_source_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
  migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
Quoted 4 lines

trace-events flic_no_device_api(int err) "flic: no Device Contral API support %d" · line 1417

Quoted 2 lines
trace-events +0 -0
  # migration/rdma.c

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

.

#72 lore raw

PATCH v7 16/42 Rework loadvm path for subloops

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and cause the parent loops to exit as well.

---
 include/migration/migration.h |   6 ++
 migration/migration.c         |   2 +
 migration/savevm.c            | 131 +++++++++++++++++++++++-------------------
 trace-events                  |   4 ++
 4 files changed, 83 insertions(+), 60 deletions(-)
include/migration/migration.h +6 -0
index 868f59a..1bf78f6 100644
@@ -54,6 +54,12 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    /*
+     * Free at the start of the main state load, set as the main thread finishes
+     * loading state.
+     */
+    QemuEvent      main_thread_load_event;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 
migration/migration.c +2 -0
index fb2f491..a743018 100644
@@ -86,12 +86,14 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
     mis_current->file = f;
     QLIST_INIT(&mis_current->loadvm_handlers);
     qemu_mutex_init(&mis_current->rp_mutex);
+    qemu_event_init(&mis_current->main_thread_load_event, false);
 
     return mis_current;
 }
 
 void migration_incoming_state_destroy(void)
 {
+    qemu_event_destroy(&mis_current->main_thread_load_event);
     loadvm_free_handlers(mis_current);
     g_free(mis_current);
     mis_current = NULL;
migration/savevm.c +71 -60

205 lines not on this page · raw message

trace-events +4 -0
index 282cde1..5644cc2 100644
@@ -1181,7 +1181,11 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64
 
 # migration/savevm.c
 qemu_loadvm_state_section(unsigned int section_type) "%d"
+qemu_loadvm_state_section_command(int ret) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
+qemu_loadvm_state_main(void) ""
+qemu_loadvm_state_main_quit_parent(void) ""
+qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 loadvm_process_command_ping(uint32_t val) "%x"
Signature · 1 line
2.4.3
Reviewed-by on v7 16/42
#73 lore raw

Re: [Qemu-devel] [PATCH v7 16/42] Rework loadvm path for subloops

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 6 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and cause the parent loops to exit as well.

Reviewed-by on v7 16/42
#74 lore raw

Re: [Qemu-devel] [PATCH v7 16/42] Rework loadvm path for subloops

In reply to Dr. David Alan Gilbert (git)
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 6 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and cause the parent loops to exit as well.

Amit

#75 lore raw

PATCH v7 17/42 Add migration-capability boolean for postcopy-ram.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

---
 include/migration/migration.h |  1 +
 migration/migration.c         | 23 +++++++++++++++++++++++
 qapi-schema.json              |  6 +++++-
 3 files changed, 29 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 1bf78f6..da4b72f 100644
@@ -183,6 +183,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy_ram(void);
 bool migrate_zero_blocks(void);
 
 bool migrate_auto_converge(void);
migration/migration.c +23 -0
index a743018..cd89a9b 100644
@@ -408,6 +408,20 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     for (cap = params; cap; cap = cap->next) {
         s->enabled_capabilities[cap->value->capability] = cap->value->state;
     }
+
+    if (migrate_postcopy_ram()) {
+        if (migrate_use_compression()) {
+            /* The decompression threads asynchronously write into RAM
+             * rather than use the atomic copies needed to avoid
+             * userfaulting.  It should be possible to fix the decompression
+             * threads for compatibility in future.
+             */
+            error_report("Postcopy is not currently compatible with "
+                         "compression");
+            s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM] =
+                false;
+        }
+    }
 }
 
 void qmp_migrate_set_parameters(bool has_compress_level,
@@ -770,6 +784,15 @@ void qmp_migrate_set_downtime(double value, Error **errp)
     max_downtime = (uint64_t)value;
 }
 
+bool migrate_postcopy_ram(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM];
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
qapi-schema.json +5 -1
index 6e17a5c..0b6fe54 100644
@@ -526,11 +526,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.4)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
   'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
-           'compress'] }
+           'compress', 'x-postcopy-ram'] }
 
 ##
 # @MigrationCapabilityStatus
Signature · 1 line
2.4.3
Reviewed-by on v7 17/42
#76 lore raw

Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.

Quoted 17 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

---
 include/migration/migration.h |  1 +
 migration/migration.c         | 23 +++++++++++++++++++++++
 qapi-schema.json              |  6 +++++-
 3 files changed, 29 insertions(+), 1 deletion(-)

+++ b/qapi-schema.json

qapi-schema.json line 531

Quoted 6 lines
qapi-schema.json +3 -0
@@ -526,11 +526,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.4)

Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#77 lore raw

Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.

Quoted 16 lines · * Eric Blake ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

---
 include/migration/migration.h |  1 +
 migration/migration.c         | 23 +++++++++++++++++++++++
 qapi-schema.json              |  6 +++++-
 3 files changed, 29 insertions(+), 1 deletion(-)
Quoted 11 lines

qapi-schema.json line 531

Quoted 8 lines

+++ b/qapi-schema.json

qapi-schema.json +3 -0
@@ -526,11 +526,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.4)

Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?

It's easy enough to remove the x- once we're all happy; it seems pretty stable at the moment but when we're done I'll just submit a one liner to take the x- off.

Dave

Quoted 6 lines
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#78 lore raw

Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.

Quoted 29 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Eric Blake ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

---
 include/migration/migration.h |  1 +
 migration/migration.c         | 23 +++++++++++++++++++++++
 qapi-schema.json              |  6 +++++-
 3 files changed, 29 insertions(+), 1 deletion(-)
Quoted 11 lines

qapi-schema.json line 531

Quoted 8 lines

+++ b/qapi-schema.json

qapi-schema.json +3 -0
@@ -526,11 +526,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.4)

Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?

Quoted 3 lines

It's easy enough to remove the x- once we're all happy; it seems pretty stable at the moment but when we're done I'll just submit a one liner to take the x- off.

We shouldn't mark it stable till we have a released kernel (which freezes the kernel API for us).

We could pick this patchset without a released kernel, but this will have to remain x- if that happens.

Amit

Reviewed-by on v7 17/42
#79 lore raw

Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

Reviewed-by on v7 17/42
#80 lore raw

Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

Amit

#81 lore raw

PATCH v7 18/42 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  16 +++
 include/sysemu/sysemu.h       |  20 ++++
 migration/migration.c         |  13 +++
 migration/savevm.c            | 247 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  10 ++
 5 files changed, 306 insertions(+)
include/migration/migration.h +16 -0
index da4b72f..a5951ac 100644
@@ -50,6 +50,15 @@ enum mig_rp_message_type {
 };
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -62,6 +71,7 @@ struct MigrationIncomingState {
 
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
+    PostcopyState postcopy_state;
 
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
@@ -231,4 +241,10 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
 
 void ram_mig_init(void);
 void savevm_skip_section_footers(void);
+
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +20 -0
index d8875ca..c5738f5 100644
@@ -87,6 +87,17 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +112,15 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len,
+                                           uint64_t *start_list,
+                                           uint64_t *end_list);
+
 int qemu_loadvm_state(QEMUFile *f);
 
 typedef enum DisplayType
migration/migration.c +13 -0
index cd89a9b..34cd9a6 100644
@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}
+
migration/savevm.c +247 -0

277 lines not on this page · raw message

trace-events +10 -0
index 5644cc2..44ac831 100644
@@ -1187,11 +1187,21 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_postcopy_handle_advise(void) ""
+loadvm_postcopy_handle_listen(void) ""
+loadvm_postcopy_handle_run(void) ""
+loadvm_postcopy_ram_handle_discard(void) ""
+loadvm_postcopy_ram_handle_discard_end(void) ""
+loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 loadvm_process_command_ping(uint32_t val) "%x"
+qemu_savevm_send_postcopy_advise(void) ""
+qemu_savevm_send_postcopy_ram_discard(const char *id, uint16_t len) "%s: %ud"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_send_ping(uint32_t val) "%x"
+savevm_send_postcopy_listen(void) ""
+savevm_send_postcopy_run(void) ""
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
Signature · 1 line
2.4.3
#82 lore raw

Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 20 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  16 +++
 include/sysemu/sysemu.h       |  20 ++++
 migration/migration.c         |  13 +++
 migration/savevm.c            | 247 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  10 ++
 5 files changed, 306 insertions(+)
migration/migration.c +0 -0
index cd89a9b..34cd9a6 100644

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1134

Quoted 7 lines
migration/migration.c +4 -0
@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);

What is wrong with atomic_read() here?
As the set of the state is atomic, even a normal read would do (I think)

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1138

Quoted 4 lines
migration/migration.c +4 -0
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len,
+                                           uint64_t *start_list,
+                                           uint64_t *end_list)

I haven't looked at the following patches where this function is used, but it appears that getting an iovec could be a good idea?

migration/migration.c void migrate_fd_connect(MigrationState *s)

Quoted 8 lines
migration/migration.c +8 -0
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;
+    size_t name_len = strlen(name);
+
+    trace_qemu_savevm_send_postcopy_ram_discard(name, len);
+    buf = g_malloc0(len*16 + name_len + 3);

I would suggest
gmalloc0(1 + 1 + name_len + 1 + (8 + 8) * len)

just to be clear where things came from.

I think that we don't need the \0 at all. If \0 is not there, strlen() return is going to be "funny". So, we can just change the assert to name_len < 255?

migration/migration.c

Quoted 2 lines
migration/migration.c +2 -0
+    buf[0] = 0; /* Version */
+    assert(name_len < 256);

Can we move the assert before the malloc()?

My guess is that in a perfect world the assert would be a return -EINVAL, but I know that it is complicated.

migration/migration.c

Quoted 2 lines
migration/migration.c +2 -0
+    buf[1] = name_len;
+    memcpy(buf+2, name, name_len);

spaces around '+' (same around)

migration/migration.c

Quoted 8 lines
migration/migration.c +8 -0
+    tmplen = 2+name_len;
+    buf[tmplen++] = '\0';
+
+    for (t = 0; t < len; t++) {
+        cpu_to_be64w((uint64_t *)(buf + tmplen), start_list[t]);
+        tmplen += 8;
+        cpu_to_be64w((uint64_t *)(buf + tmplen), end_list[t]);
+        tmplen += 8;
           trace_qemu_savevm_send_postcopy_range(name, start_list[t], end_list[t]);

??

migration/migration.c

Quoted 6 lines
migration/migration.c +6 -0
+    /* We're expecting a
+     *    Version (0)
+     *    a RAM ID string (length byte, name, 0 term)
+     *    then at least 1 16 byte chunk
+    */
+    if (len < 20) { 1 +
       1+1+1+1+2*8

Humm, thinking about it, .... why are we not needing a length field of number of entries?

migration/migration.c

Quoted 6 lines
migration/migration.c +6 -0
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+
+    tmp = qemu_get_byte(mis->file);
+    if (tmp != 0) {

I think that a constant telling POSTCOPY_VERSION0 or whatever?

#83 lore raw

Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 11 lines · Juan Quintela

migration/savevm.c enum LoadVMExitCodes {

Quoted 6 lines
migration/savevm.c +6 -0
+    /* We're expecting a
+     *    Version (0)
+     *    a RAM ID string (length byte, name, 0 term)
+     *    then at least 1 16 byte chunk
+    */
+    if (len < 20) { 1 +
       1+1+1+1+2*8

Humm, thinking about it, .... why are we not needing a length field of number of entries?

hm, yea.

Quoted 8 lines

migration/savevm.c enum LoadVMExitCodes { · line 1169

Quoted 6 lines
migration/savevm.c +6 -0
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+
+    tmp = qemu_get_byte(mis->file);
+    if (tmp != 0) {

I think that a constant telling POSTCOPY_VERSION0 or whatever?

agreed.

Amit

#84 lore raw

Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Juan Quintela
Quoted 16 lines · * Juan Quintela ([email protected]) wrote:
Quoted 15 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  16 +++
 include/sysemu/sysemu.h       |  20 ++++
 migration/migration.c         |  13 +++
 migration/savevm.c            | 247 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  10 ++
 5 files changed, 306 insertions(+)
Quoted 15 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1134

Quoted 12 lines
migration/migration.c +4 -0
index cd89a9b..34cd9a6 100644
@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);

What is wrong with atomic_read() here?
As the set of the state is atomic, even a normal read would do (I think)

Actually, I made this an atomic_mb_read as per Paolo's comment on my v5 version (31st March).
I also added a comment documenting which threads read/write the state.

Quoted 7 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1138

Quoted 4 lines
migration/migration.c +4 -0
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len,
+                                           uint64_t *start_list,
+                                           uint64_t *end_list)

I haven't looked at the following patches where this function is used, but it appears that getting an iovec could be a good idea?

Yes, although I wouldn't want to make the wire format dependent on the host size_t or pointer size or anything.

Quoted 13 lines

migration/migration.c void migrate_fd_connect(MigrationState *s)

Quoted 9 lines
migration/migration.c +8 -0
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;
+    size_t name_len = strlen(name);
+
+    trace_qemu_savevm_send_postcopy_ram_discard(name, len);
+    buf = g_malloc0(len*16 + name_len + 3);

I would suggest
gmalloc0(1 + 1 + name_len + 1 + (8 + 8) * len)

Quoted 1 line

just to be clear where things came from.

Done.

Quoted 3 lines

I think that we don't need the \0 at all. If \0 is not there, strlen() return is going to be "funny". So, we can just change the assert to name_len < 255?

Dave Gibson asked for the \0 in a previous review.

Quoted 5 lines

migration/migration.c

Quoted 3 lines
migration/migration.c +2 -0
+    buf[0] = 0; /* Version */
+    assert(name_len < 256);

Can we move the assert before the malloc()?

Done.

Quoted 7 lines

My guess is that in a perfect world the assert would be a return -EINVAL, but I know that it is complicated.

migration/migration.c

Quoted 2 lines
migration/migration.c +2 -0
+    buf[1] = name_len;
+    memcpy(buf+2, name, name_len);

spaces around '+' (same around)

Done.

Quoted 9 lines

migration/migration.c

Quoted 9 lines
migration/migration.c +8 -0
+    tmplen = 2+name_len;
+    buf[tmplen++] = '\0';
+
+    for (t = 0; t < len; t++) {
+        cpu_to_be64w((uint64_t *)(buf + tmplen), start_list[t]);
+        tmplen += 8;
+        cpu_to_be64w((uint64_t *)(buf + tmplen), end_list[t]);
+        tmplen += 8;
Quoted 3 lines
           trace_qemu_savevm_send_postcopy_range(name, start_list[t], end_list[t]);

??

???

Quoted 8 lines

migration/migration.c

Quoted 6 lines
migration/migration.c +6 -0
+    /* We're expecting a
+     *    Version (0)
+     *    a RAM ID string (length byte, name, 0 term)
+     *    then at least 1 16 byte chunk
+    */
+    if (len < 20) { 1 +
       1+1+1+1+2*8

Done.

Quoted 2 lines

Humm, thinking about it, .... why are we not needing a length field of number of entries?

Because we've got the size of the whole message from the command header.

Quoted 8 lines

migration/migration.c

Quoted 6 lines
migration/migration.c +6 -0
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+
+    tmp = qemu_get_byte(mis->file);
+    if (tmp != 0) {

I think that a constant telling POSTCOPY_VERSION0 or whatever?

Done; (as a const postcopy_ram_discard_version)

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 18/42
#85 lore raw

Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

But:

Quoted 8 lines
migration/savevm.c +8 -0
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len,
+                                           uint64_t *start_list,
+                                           uint64_t *end_list)
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;

migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 766

Quoted 10 lines
migration/savevm.c +10 -0
+    size_t name_len = strlen(name);
+
+    trace_qemu_savevm_send_postcopy_ram_discard(name, len);
+    buf = g_malloc0(len*16 + name_len + 3);
+    buf[0] = 0; /* Version */
+    assert(name_len < 256);
+    buf[1] = name_len;
+    memcpy(buf+2, name, name_len);
+    tmplen = 2+name_len;
+    buf[tmplen++] = '\0';

whitespace around operators missing

migration/savevm.c enum LoadVMExitCodes { · line 1184

Quoted 4 lines
migration/savevm.c +3 -0
+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
+                                              uint16_t len)
 
+    len -= 3+strlen(ramid);

ditto

Amit

#86 lore raw

Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 10 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

Thanks,

Quoted 23 lines

But:

Quoted 8 lines
migration/savevm.c +8 -0
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len,
+                                           uint64_t *start_list,
+                                           uint64_t *end_list)
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;

migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 766

Quoted 10 lines
migration/savevm.c +10 -0
+    size_t name_len = strlen(name);
+
+    trace_qemu_savevm_send_postcopy_ram_discard(name, len);
+    buf = g_malloc0(len*16 + name_len + 3);
+    buf[0] = 0; /* Version */
+    assert(name_len < 256);
+    buf[1] = name_len;
+    memcpy(buf+2, name, name_len);
+    tmplen = 2+name_len;
+    buf[tmplen++] = '\0';

whitespace around operators missing

Quoted 6 lines

migration/savevm.c enum LoadVMExitCodes { · line 1184

Quoted 4 lines
migration/savevm.c +3 -0
+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
+                                              uint16_t len)
 
+    len -= 3+strlen(ramid);

ditto

Fixed.

Dave

Quoted 2 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#87 lore raw

PATCH v7 19/42 MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

---
 include/sysemu/sysemu.h |  4 +++
 migration/savevm.c      | 94 +++++++++++++++++++++++++++++++++++++++++++++++++
 trace-events            |  4 +++
 3 files changed, 102 insertions(+)
include/sysemu/sysemu.h +4 -0
index c5738f5..5bf8f80 100644
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */
 
     MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
                                       warn we might want to do PC */
@@ -100,6 +101,8 @@ enum qemu_vm_cmd {
 
 };
 
+#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24)
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -112,6 +115,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+int qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb);
 void qemu_savevm_send_postcopy_advise(QEMUFile *f);
 void qemu_savevm_send_postcopy_listen(QEMUFile *f);
 void qemu_savevm_send_postcopy_run(QEMUFile *f);
migration/savevm.c +94 -0

112 lines not on this page · raw message

trace-events +4 -0
index 44ac831..299805b 100644
@@ -1187,6 +1187,10 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+qemu_savevm_send_packaged(void) ""
+loadvm_handle_cmd_packaged(unsigned int length) "%u"
+loadvm_handle_cmd_packaged_main(int ret) "%d"
+loadvm_handle_cmd_packaged_received(int ret) "%d"
 loadvm_postcopy_handle_advise(void) ""
 loadvm_postcopy_handle_listen(void) ""
 loadvm_postcopy_handle_run(void) ""
Signature · 1 line
2.4.3
#88 lore raw

Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

Quoted 7 lines

+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that

Famous words of advise O:-)

I have to read the rest of series to make my mind on this one.

#89 lore raw

Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

In reply to Dr. David Alan Gilbert (git)
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

Not sure why this is necessary. I suppose I'll have to go read the documentation in patch 1..

However:

Quoted 28 lines
migration/savevm.c +22 -0
@@ -718,6 +718,50 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)
     qemu_savevm_command_send(f, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
 }
 
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
+ *
+ * Returns:
+ *    0 on success
+ *    -ve on error
+ */
+int qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb)
+{
+    size_t cur_iov;
+    size_t len = qsb_get_length(qsb);
+    uint32_t tmp;
+
+    if (len > MAX_VM_CMD_PACKAGED_SIZE) {
+        error_report("%s: Unreasonably large packaged state: %zu",
+                     __func__, len);
+        return -1;
+    }
+
+    tmp = cpu_to_be32(len);

migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 752

Quoted 10 lines
migration/savevm.c +10 -0
+
+    trace_qemu_savevm_send_packaged();
+    qemu_savevm_command_send(f, MIG_CMD_PACKAGED, 4, (uint8_t *)&tmp);
+
+    /* all the data follows (concatinating the iov's) */
+    for (cur_iov = 0; cur_iov < qsb->n_iov; cur_iov++) {
+        /* The iov entries are partially filled */
+        size_t towrite = (qsb->iov[cur_iov].iov_len > len) ?
+                              len :
+                              qsb->iov[cur_iov].iov_len;

If iov_len was > len, we only wrote part of the current buffer, and we skip to the next?

Amit

#90 lore raw

Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 8 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

Quoted 5 lines
Quoted 2 lines

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

Not sure why this is necessary. I suppose I'll have to go read the documentation in patch 1..

Yep - or one of the previous replies where I explained it.

Quoted 35 lines

However:

Quoted 33 lines
migration/savevm.c +27 -0
@@ -718,6 +718,50 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)
     qemu_savevm_command_send(f, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
 }
 
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
+ *
+ * Returns:
+ *    0 on success
+ *    -ve on error
+ */
+int qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb)
+{
+    size_t cur_iov;
+    size_t len = qsb_get_length(qsb);
+    uint32_t tmp;
+
+    if (len > MAX_VM_CMD_PACKAGED_SIZE) {
+        error_report("%s: Unreasonably large packaged state: %zu",
+                     __func__, len);
+        return -1;
+    }
+
+    tmp = cpu_to_be32(len);
+
+    trace_qemu_savevm_send_packaged();
+    qemu_savevm_command_send(f, MIG_CMD_PACKAGED, 4, (uint8_t *)&tmp);
+
+    /* all the data follows (concatinating the iov's) */
Quoted 8 lines

migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 752

Quoted 5 lines
migration/savevm.c +5 -0
+    for (cur_iov = 0; cur_iov < qsb->n_iov; cur_iov++) {
+        /* The iov entries are partially filled */
+        size_t towrite = (qsb->iov[cur_iov].iov_len > len) ?
+                              len :
+                              qsb->iov[cur_iov].iov_len;

If iov_len was > len, we only wrote part of the current buffer, and we skip to the next?

Yes; this is just the end case; the qsb allocates iov entries in 'chunks' but then the data that gets added often doesn't use the whole chunk. 'len' - set above from qsb_get_length - gives the used contents of the qsb, and that's all we want to write. This is normally the case on the last entry in the qsb anyway; however since 'len' gets decremented by the amount written we might go once more around the loop and the 'if (!towrite) { break; }' might break us out of the loop instead.

Dave

Quoted 3 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 19/42
#91 lore raw

Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

In reply to Dr. David Alan Gilbert (git)
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

Amit

#92 lore raw

PATCH v7 20/42 Modify save_live_pending for postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

---
 include/migration/vmstate.h |  5 +++--
 include/sysemu/sysemu.h     |  4 +++-
 migration/block.c           |  7 +++++--
 migration/migration.c       |  9 +++++++--
 migration/ram.c             |  8 ++++++--
 migration/savevm.c          | 21 +++++++++++++++++----
 trace-events                |  2 +-
 7 files changed, 42 insertions(+), 14 deletions(-)
include/migration/vmstate.h +3 -2
index 074747c..7257196 100644
@@ -54,8 +54,9 @@ typedef struct SaveVMHandlers {
 
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
-    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
+    void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
+                              uint64_t *non_postcopiable_pending,
+                              uint64_t *postcopiable_pending);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
include/sysemu/sysemu.h +3 -1
index 5bf8f80..ff6bb2c 100644
@@ -110,7 +110,9 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
migration/block.c +5 -2
index 3005668..4483ce3 100644
@@ -754,7 +754,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;
 }
 
 static int block_load(QEMUFile *f, void *opaque, int version_id)
migration/migration.c +7 -2
index 34cd9a6..e77b8b4 100644
@@ -1024,8 +1024,13 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->file)) {
-            pending_size = qemu_savevm_state_pending(s->file, max_size);
-            trace_migrate_pending(pending_size, max_size);
+            uint64_t pend_post, pend_nonpost;
+
+            qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+                                      &pend_post);
+            pending_size = pend_nonpost + pend_post;
+            trace_migrate_pending(pending_size, max_size,
+                                  pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
migration/ram.c +6 -2
index 492ed8a..fb24954 100644
@@ -1273,7 +1273,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                             uint64_t *non_postcopiable_pending,
+                             uint64_t *postcopiable_pending)
 {
     uint64_t remaining_size;
 
@@ -1287,7 +1289,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
         qemu_mutex_unlock_iothread();
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
     }
-    return remaining_size;
+
+    *non_postcopiable_pending = 0;
+    *postcopiable_pending = remaining_size;
 }
 
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
migration/savevm.c +17 -4
index 2c4cbe1..ebd3d31 100644
@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;
 }
 
 void qemu_savevm_state_cancel(void)
trace-events +1 -1
index 299805b..339eb71 100644
@@ -1419,7 +1419,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
-migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
Signature · 1 line
2.4.3
Reviewed-by on v7 20/42
#93 lore raw

Re: [Qemu-devel] [PATCH v7 20/42] Modify save_live_pending for postcopy

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

I think that if you make a small change of meaning, everything gots easier:

Quoted 8 lines
migration/block.c +3 -1
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;

migration/block.c static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size) · line 779

Quoted 6 lines
migration/block.c +2 -1
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;

Change that two lines to:

       *non_postcopiable_pending += pending;
       *postcopiable_pending += 0; /* ok, equivalent of doing nothing */

This way, chaining gots easier?

Quoted 29 lines
migration/savevm.c +12 -2
index 2c4cbe1..ebd3d31 100644
@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;

migration/savevm.c uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size) · line 1031

Quoted 10 lines
migration/savevm.c +5 -2
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;

With the change, we don't care in the other functions, and this one gets simpler IMHO.

#94 lore raw

Re: [Qemu-devel] [PATCH v7 20/42] Modify save_live_pending for postcopy

Quoted 9 lines · * Juan Quintela ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Thanks,

Quoted 19 lines

I think that if you make a small change of meaning, everything gots easier:

Quoted 8 lines
migration/block.c +3 -1
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;

migration/block.c static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size) · line 779

Quoted 6 lines
migration/block.c +2 -1
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;

Change that two lines to:

Quoted 4 lines
       *non_postcopiable_pending += pending;
       *postcopiable_pending += 0; /* ok, equivalent of doing nothing */

This way, chaining gots easier?

OK, done; I did it as:

+ /* We can do postcopy, and all the data is postcopiable */
+ *postcopiable_pending += remaining_size;

rather than having the odd += 0;

Quoted 34 lines
Quoted 34 lines
migration/savevm.c +14 -3
index 2c4cbe1..ebd3d31 100644
@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
Quoted 8 lines

migration/savevm.c uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size) · line 1031

Quoted 5 lines
migration/savevm.c +3 -1
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;

With the change, we don't care in the other functions, and this one gets simpler IMHO.

Yep,

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#96 lore raw

PATCH v7 21/42 postcopy: OS support test

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 include/migration/postcopy-ram.h |  19 +++++
 migration/Makefile.objs          |   2 +-
 migration/postcopy-ram.c         | 158 +++++++++++++++++++++++++++++++++++++++
 migration/savevm.c               |   5 ++
 4 files changed, 183 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..d81934f
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return true if the host supports everything we need to do postcopy-ram */
+bool postcopy_ram_supported_by_host(void);
+
+#endif
migration/Makefile.objs +1 -1
index d929e96..0cac6d7 100644
@@ -1,7 +1,7 @@
 common-obj-y += migration.o tcp.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
-common-obj-y += xbzrle.o
+common-obj-y += xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +158 -0

158 lines not on this page · raw message

migration/savevm.c +5 -0
index ebd3d31..f324c6e 100644
@@ -37,6 +37,7 @@
 #include "qemu/timer.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/sockets.h"
 #include "qemu/queue.h"
 #include "sysemu/cpus.h"
@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+
     if (remote_hps != getpagesize())  {
         /*
          * Some combinations of mismatch are probably possible but it gets
Signature · 1 line
2.4.3
Reviewed-by on v7 21/42
#97 lore raw

Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

I am guessing that test is ok, but we are doing the test each time that we change the function. We always end calling that kind of functions in several places. Shouldn't be good to rename the function to __postcopy_ram_supported_by_host()

and do a toplevel function that is:

bool postcopy_ram_supported_by_host(void)

{
        static bool first_time = true;
        static supported = false;

        if (firt_time) {
           first_time = false;
           supported = __postcopy_ram_supported_by_host()
        }
        return supported;
}

Notice that I don't know how slow the mmap + usefault thing is, but I guess that the values would not change while running, no?

It has a review-by because I don't see anything wrong with it.

#98 lore raw

Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test

Quoted 31 lines · * Juan Quintela ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

I am guessing that test is ok, but we are doing the test each time that we change the function. We always end calling that kind of functions in several places. Shouldn't be good to rename the function to __postcopy_ram_supported_by_host()

and do a toplevel function that is:

bool postcopy_ram_supported_by_host(void)

{
        static bool first_time = true;
        static supported = false;

        if (firt_time) {
           first_time = false;
           supported = __postcopy_ram_supported_by_host()
        }
        return supported;
}
Quoted 2 lines

Notice that I don't know how slow the mmap + usefault thing is, but I guess that the values would not change while running, no?

Since we only call this once, at the start of an incoming migration, it seems overkill to do that.

Dave

Quoted 2 lines

It has a review-by because I don't see anything wrong with it.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#99 lore raw

Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test

In reply to Dr. David Alan Gilbert (git)
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1172

Quoted 7 lines
migration/savevm.c +4 -0
@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+

So this is just advise: if we receive this, and we can't handle postcopy, we're going to abort migration? Shouldn't we continue and let src know that we can't accept postcopy? ie the problem is currently punted to higher levels, should we try to handle it ourselves?

Amit

#100 lore raw

Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test

Quoted 14 lines · * Amit Shah ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

migration/savevm.c +0 -0
@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
Quoted 10 lines

migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1172

Quoted 4 lines
migration/savevm.c +4 -0
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+

So this is just advise: if we receive this, and we can't handle postcopy, we're going to abort migration? Shouldn't we continue and let src know that we can't accept postcopy? ie the problem is currently punted to higher levels, should we try to handle it ourselves?

We could, although it happens right at the start of migration, and it only happens if you explicitly enabled the postcopy capability, so I'm not sure there's any advantage in trying to fall back when you've been told to use it.

Dave

Quoted 4 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 21/42
#101 lore raw

Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

Amit

#102 lore raw

PATCH v7 22/42 migrate_start_postcopy: Command to trigger transition to postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index 3d7dfcc..5124698 100644
@@ -1006,6 +1006,21 @@ Set the parameter @var{parameter} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index 514f22f..bf1ad53 100644
@@ -1268,6 +1268,13 @@ void hmp_client_migrate_info(Monitor *mon, const QDict *qdict)
     hmp_handle_error(mon, &err);
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index a70ac4f..97901c2 100644
@@ -68,6 +68,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
 void hmp_client_migrate_info(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index a5951ac..e973490 100644
@@ -111,6 +111,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +22 -0
index e77b8b4..6fc47f9 100644
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);
+}
+
 /* shared migration helpers */
 
 static void migrate_set_state(MigrationState *s, int old_state, int new_state)
qapi-schema.json +8 -0
index 0b6fe54..b0177bb 100644
@@ -661,6 +661,14 @@
             '*tls-port': 'int', '*cert-subject': 'str' } }
 
 ##
+# @migrate-start-postcopy
+#
+# Switch migration to postcopy mode
+#
+# Since: 2.4
+{ 'command': 'migrate-start-postcopy' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
qmp-commands.hx +19 -0
index 867a21f..dc63ff3 100644
@@ -713,6 +713,25 @@ Example:
 
 EQMP
     {
+        .name       = "migrate-start-postcopy",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy,
+    },
+
+SQMP
+migrate-start-postcopy
+----------------------
+
+Switch an in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+
+Example:
+-> { "execute": "migrate-start-postcopy" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 1 line
2.4.3
#103 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 40 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

include/migration/migration.h +3 -0
index a5951ac..e973490 100644
@@ -111,6 +111,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +1 -0
index e77b8b4..6fc47f9 100644
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 10 lines
migration/migration.c +10 -0
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

Thanks, Juan.

#104 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

include/migration/migration.h +3 -0
index a5951ac..e973490 100644
@@ -111,6 +111,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +1 -0
index e77b8b4..6fc47f9 100644
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 10 lines
migration/migration.c +10 -0
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

Quoted 4 lines
    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever

which wont work, because migrate_init reinitialises
the flag that start previously set.

However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy

if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.

Dave

Quoted 2 lines

Thanks, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#105 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 73 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

include/migration/migration.h +3 -0
index a5951ac..e973490 100644
@@ -111,6 +111,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +1 -0
index e77b8b4..6fc47f9 100644
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 10 lines
migration/migration.c +10 -0
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

Quoted 4 lines
    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever

which wont work, because migrate_init reinitialises
the flag that start previously set.

However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy

Quoted 4 lines

if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.

Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.

Later, Juan.

Quoted 7 lines

Dave

Quoted 2 lines

Thanks, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#106 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 35 lines · Juan Quintela
Quoted 34 lines · Dr. David Alan Gilbert
Quoted 14 lines · * Juan Quintela ([email protected]) wrote:

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 11 lines
migration/migration.c +11 -0
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

Quoted 4 lines
    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever

which wont work, because migrate_init reinitialises
the flag that start previously set.

However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy

Quoted 8 lines
Quoted 4 lines

if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.

Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.

Yeah, I tend to agree, because in the cases where migration has failed or has been cancelled, we'll end up setting the postcopy bit. Then, upon the next migration, this bit could get reused - resulting in the previous condition of setting postcopy bit before starting migration.

Amit

#107 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 41 lines · * Amit Shah ([email protected]) wrote:
Quoted 40 lines · Juan Quintela
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 14 lines · * Juan Quintela ([email protected]) wrote:

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 11 lines
migration/migration.c +11 -0
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

Quoted 4 lines
    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever

which wont work, because migrate_init reinitialises
the flag that start previously set.

However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy

if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.

Quoted 8 lines
Quoted 3 lines

Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.

Yeah, I tend to agree, because in the cases where migration has failed or has been cancelled, we'll end up setting the postcopy bit. Then, upon the next migration, this bit could get reused - resulting in the previous condition of setting postcopy bit before starting migration.

No, that doesn't happen; the bit is cleared at the start of migration so that race condition doesn't exist.

Dave

Quoted 3 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#108 lore raw

Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy

In reply to Juan Quintela
Quoted 74 lines · * Juan Quintela ([email protected]) wrote:
Quoted 73 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

include/migration/migration.h +3 -0
index a5951ac..e973490 100644
@@ -111,6 +111,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +1 -0
index e77b8b4..6fc47f9 100644
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)

migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478

Quoted 10 lines
migration/migration.c +10 -0
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIGRATION_STATUS_NONE) {

I would claim that this check should be:

Quoted 4 lines
    if (s->state != MIGRATION_STATUS_ACTIVE) {

??

FAILED, COMPLETED, CANCELL* don't make sense, right?

What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever

which wont work, because migrate_init reinitialises
the flag that start previously set.

However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy

Quoted 8 lines
Quoted 4 lines

if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.

Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.

The problem is that produces a race-condition for the command. If you wait too long and the migration finishes before you issue the command you get an error, when the migration has completed perfectly happily.

Dave

Quoted 11 lines

Later, Juan.

Quoted 7 lines

Dave

Quoted 2 lines

Thanks, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#109 lore raw

PATCH v7 23/42 MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

---
 include/migration/migration.h |  2 ++
 migration/migration.c         | 56 ++++++++++++++++++++++++++++++++++++-------
 qapi-schema.json              |  4 +++-
 trace-events                  |  1 +
 4 files changed, 54 insertions(+), 9 deletions(-)
include/migration/migration.h +2 -0
index e973490..2a22381 100644
@@ -154,6 +154,8 @@ MigrationState *migrate_init(const MigrationParams *params);
 bool migration_in_setup(MigrationState *);
 bool migration_has_finished(MigrationState *);
 bool migration_has_failed(MigrationState *);
+/* True if outgoing migration has entered postcopy phase */
+bool migration_postcopy_phase(MigrationState *);
 MigrationState *migrate_get_current(void);
 
 void migrate_compress_threads_create(void);
migration/migration.c +48 -8

104 lines not on this page · raw message

qapi-schema.json +3 -1
index b0177bb..5e2f487 100644
@@ -424,6 +424,8 @@
 #
 # @active: in the process of doing migration.
 #
+# @postcopy-active: like active, but now in postcopy mode. (since 2.4)
+#
 # @completed: migration is finished.
 #
 # @failed: some error occurred during migration process.
@@ -433,7 +435,7 @@
 ##
 { 'enum': 'MigrationStatus',
   'data': [ 'none', 'setup', 'cancelling', 'cancelled',
-            'active', 'completed', 'failed' ] }
+            'active', 'postcopy-active', 'completed', 'failed' ] }
 
 ##
 # @MigrationInfo
trace-events +1 -0
index 339eb71..72e9889 100644
@@ -1421,6 +1421,7 @@ migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+migration_thread_setup_complete(void) ""
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
 source_return_path_thread_bad_end(void) ""
Signature · 1 line
2.4.3
Reviewed-by on v7 23/42
#110 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

But (there is always a but....)

Quoted 24 lines
migration/migration.c +20 -0
@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_POSTCOPY_ACTIVE:
+        /* Mostly the same as active; TODO add some postcopy stats */
+        info->has_status = true;
+        info->has_total_time = true;
+        info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
+            - s->total_time;
+        info->has_expected_downtime = true;
+        info->expected_downtime = s->expected_downtime;
+        info->has_setup_time = true;
+        info->setup_time = s->setup_time;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = ram_bytes_remaining();
+        info->ram->total = ram_bytes_total();
+        info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
+        info->ram->normal = norm_mig_pages_transferred();
+        info->ram->normal_bytes = norm_mig_bytes_transferred();

migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391

Quoted 10 lines
migration/migration.c +10 -0
+        info->ram->dirty_pages_rate = s->dirty_pages_rate;
+        info->ram->mbps = s->mbps;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }

Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?

Once here, I guess we can get the migrate_already_active() bit without problem?

Later, Juan.

#111 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

But (there is always a but....)

Quoted 30 lines
migration/migration.c +26 -0
@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_POSTCOPY_ACTIVE:
+        /* Mostly the same as active; TODO add some postcopy stats */
+        info->has_status = true;
+        info->has_total_time = true;
+        info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
+            - s->total_time;
+        info->has_expected_downtime = true;
+        info->expected_downtime = s->expected_downtime;
+        info->has_setup_time = true;
+        info->setup_time = s->setup_time;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = ram_bytes_remaining();
+        info->ram->total = ram_bytes_total();
+        info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
+        info->ram->normal = norm_mig_pages_transferred();
+        info->ram->normal_bytes = norm_mig_bytes_transferred();
+        info->ram->dirty_pages_rate = s->dirty_pages_rate;
+        info->ram->mbps = s->mbps;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
Quoted 8 lines

migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391

Quoted 4 lines
migration/migration.c +4 -0
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }

Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?

I think you can, I've not tried it; however I also think it should be safe.

migration/block.c's block_save_pending always puts a value in the
non_postcopiable_pending return value (and 0 in the postcopiable_pending); the migrate thread checks the non_postcopiable_pending size to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.

A possibly harder question is what happens if block.c did implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.

Quoted 2 lines

Once here, I guess we can get the migrate_already_active() bit without problem?

I'm not sure of the question here; but the idea of migration_already_active() is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#112 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

But (there is always a but....)

Quoted 30 lines
migration/migration.c +26 -0
@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_POSTCOPY_ACTIVE:
+        /* Mostly the same as active; TODO add some postcopy stats */
+        info->has_status = true;
+        info->has_total_time = true;
+        info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
+            - s->total_time;
+        info->has_expected_downtime = true;
+        info->expected_downtime = s->expected_downtime;
+        info->has_setup_time = true;
+        info->setup_time = s->setup_time;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = ram_bytes_remaining();
+        info->ram->total = ram_bytes_total();
+        info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
+        info->ram->normal = norm_mig_pages_transferred();
+        info->ram->normal_bytes = norm_mig_bytes_transferred();
+        info->ram->dirty_pages_rate = s->dirty_pages_rate;
+        info->ram->mbps = s->mbps;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
Quoted 8 lines

migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391

Quoted 4 lines
migration/migration.c +4 -0
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }

Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?

I think you can, I've not tried it; however I also think it should be safe.

Quoted 7 lines

migration/block.c's block_save_pending always puts a value in the
non_postcopiable_pending return value (and 0 in the postcopiable_pending); the migrate thread checks the non_postcopiable_pending size to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.

I mean that as it is right now, the info under blk_mig_active() check would be zero/the same than before entering postcopy.

Quoted 6 lines

A possibly harder question is what happens if block.c did implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.

Quoted 6 lines
Quoted 2 lines

Once here, I guess we can get the migrate_already_active() bit without problem?

I'm not sure of the question here; but the idea of migration_already_active() is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.

Sorry. I mean that the migrate_already_active() bits can get in without further ado. Don't need to wait for postcopy to be integrated.

Quoted 7 lines

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#113 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 70 lines · * Juan Quintela ([email protected]) wrote:
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

But (there is always a but....)

Quoted 30 lines
migration/migration.c +26 -0
@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_POSTCOPY_ACTIVE:
+        /* Mostly the same as active; TODO add some postcopy stats */
+        info->has_status = true;
+        info->has_total_time = true;
+        info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
+            - s->total_time;
+        info->has_expected_downtime = true;
+        info->expected_downtime = s->expected_downtime;
+        info->has_setup_time = true;
+        info->setup_time = s->setup_time;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = ram_bytes_remaining();
+        info->ram->total = ram_bytes_total();
+        info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
+        info->ram->normal = norm_mig_pages_transferred();
+        info->ram->normal_bytes = norm_mig_bytes_transferred();
+        info->ram->dirty_pages_rate = s->dirty_pages_rate;
+        info->ram->mbps = s->mbps;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
Quoted 8 lines

migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391

Quoted 4 lines
migration/migration.c +4 -0
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }

Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?

I think you can, I've not tried it; however I also think it should be safe.

Quoted 7 lines

migration/block.c's block_save_pending always puts a value in the
non_postcopiable_pending return value (and 0 in the postcopiable_pending); the migrate thread checks the non_postcopiable_pending size to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.

Quoted 2 lines

I mean that as it is right now, the info under blk_mig_active() check would be zero/the same than before entering postcopy.

Ah, yes; would blk_mig_bytes_total/transferred still have valid values you would want to display, even at the end of the block migration phase?

Quoted 9 lines
Quoted 9 lines

A possibly harder question is what happens if block.c did implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.

Quoted 2 lines

Once here, I guess we can get the migrate_already_active() bit without problem?

Quoted 6 lines
Quoted 3 lines

I'm not sure of the question here; but the idea of migration_already_active() is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.

Sorry. I mean that the migrate_already_active() bits can get in without further ado. Don't need to wait for postcopy to be integrated.

Yes; do you want it split out?

Dave

Quoted 8 lines
Quoted 8 lines

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#114 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 86 lines · Dr. David Alan Gilbert
Quoted 70 lines · * Juan Quintela ([email protected]) wrote:
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

But (there is always a but....)

Quoted 30 lines
migration/migration.c +26 -0
@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
         get_xbzrle_cache_stats(info);
         break;
+    case MIGRATION_STATUS_POSTCOPY_ACTIVE:
+        /* Mostly the same as active; TODO add some postcopy stats */
+        info->has_status = true;
+        info->has_total_time = true;
+        info->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME)
+            - s->total_time;
+        info->has_expected_downtime = true;
+        info->expected_downtime = s->expected_downtime;
+        info->has_setup_time = true;
+        info->setup_time = s->setup_time;
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = ram_bytes_remaining();
+        info->ram->total = ram_bytes_total();
+        info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
+        info->ram->normal = norm_mig_pages_transferred();
+        info->ram->normal_bytes = norm_mig_bytes_transferred();
+        info->ram->dirty_pages_rate = s->dirty_pages_rate;
+        info->ram->mbps = s->mbps;
+
+        if (blk_mig_active()) {
+            info->has_disk = true;
+            info->disk = g_malloc0(sizeof(*info->disk));
Quoted 8 lines

migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391

Quoted 4 lines
migration/migration.c +4 -0
+            info->disk->transferred = blk_mig_bytes_transferred();
+            info->disk->remaining = blk_mig_bytes_remaining();
+            info->disk->total = blk_mig_bytes_total();
+        }

Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?

I think you can, I've not tried it; however I also think it should be safe.

Quoted 7 lines

migration/block.c's block_save_pending always puts a value in the
non_postcopiable_pending return value (and 0 in the postcopiable_pending); the migrate thread checks the non_postcopiable_pending size to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.

Quoted 2 lines

I mean that as it is right now, the info under blk_mig_active() check would be zero/the same than before entering postcopy.

Ah, yes; would blk_mig_bytes_total/transferred still have valid values you would want to display, even at the end of the block migration phase?

Quoted 9 lines
Quoted 9 lines

A possibly harder question is what happens if block.c did implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.

Quoted 2 lines

Once here, I guess we can get the migrate_already_active() bit without problem?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I'm not sure of the question here; but the idea of migration_already_active() is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.

Sorry. I mean that the migrate_already_active() bits can get in without further ado. Don't need to wait for postcopy to be integrated.

Yes; do you want it split out?

Whatever is easier for you. I mean that it can be integrated independently of postcopy.

so, it is up to you.

Juan.

Quoted 13 lines

Dave

Quoted 8 lines
Quoted 8 lines

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 23/42
#115 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

In reply to Dr. David Alan Gilbert (git)

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 436

Quoted 3 lines · Dr. David Alan Gilbert (git)
migration/migration.c +1 -2
-    if (s->state == MIGRATION_STATUS_ACTIVE ||
-        s->state == MIGRATION_STATUS_SETUP) {
+    if (migration_already_active(s)) {

(I know, not introduced here, but:)

A better name is migration_is_active()

migration/migration.c bool migration_has_failed(MigrationState *s) · line 650

Quoted 4 lines
migration/migration.c +4 -0
+bool migration_postcopy_phase(MigrationState *s)
+{
+    return (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
+}

And this is better named migration_in_postcopy()

otherwise,

Amit

#116 lore raw

Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state

Quoted 9 lines · * Amit Shah ([email protected]) wrote:

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 436

Quoted 3 lines · Dr. David Alan Gilbert (git)
migration/migration.c +1 -2
-    if (s->state == MIGRATION_STATUS_ACTIVE ||
-        s->state == MIGRATION_STATUS_SETUP) {
+    if (migration_already_active(s)) {

(I know, not introduced here, but:)

A better name is migration_is_active()

Done.

Quoted 7 lines

migration/migration.c bool migration_has_failed(MigrationState *s) · line 650

Quoted 5 lines
migration/migration.c +4 -0
+bool migration_postcopy_phase(MigrationState *s)
+{
+    return (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
+}

And this is better named migration_in_postcopy()

Done

Quoted 8 lines

otherwise,

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#117 lore raw

PATCH v7 24/42 Add qemu_savevm_state_complete_postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.

The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.

The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.

---
 include/migration/vmstate.h |  1 +
 include/sysemu/sysemu.h     |  1 +
 migration/ram.c             |  1 +
 migration/savevm.c          | 51 ++++++++++++++++++++++++++++++++++++++++++---
 4 files changed, 51 insertions(+), 3 deletions(-)
include/migration/vmstate.h +1 -0
index 7257196..dddeadd 100644
@@ -40,6 +40,7 @@ typedef struct SaveVMHandlers {
     SaveStateHandler *save_state;
 
     void (*cancel)(void *opaque);
+    int (*save_live_complete_postcopy)(QEMUFile *f, void *opaque);
     int (*save_live_complete_precopy)(QEMUFile *f, void *opaque);
 
     /* This runs both outside and inside the iothread lock.  */
include/sysemu/sysemu.h +1 -0
index ff6bb2c..1af2ea0 100644
@@ -108,6 +108,7 @@ void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
 void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
+void qemu_savevm_state_complete_postcopy(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
migration/ram.c +1 -0
index fb24954..ff1a2fb 100644
@@ -1608,6 +1608,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
+    .save_live_complete_postcopy = ram_save_complete,
     .save_live_complete_precopy = ram_save_complete,
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
migration/savevm.c +48 -3

75 lines not on this page · raw message

Signature · 1 line
2.4.3
Reviewed-by on v7 24/42
#118 lore raw

Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.

Quoted 8 lines

The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.

The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.

Quoted 6 lines
migration/savevm.c +1 -0
@@ -947,13 +987,15 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)
     int vmdesc_len;
     SaveStateEntry *se;
     int ret;
+    bool in_postcopy = migration_postcopy_phase(migrate_get_current());
 

migration/savevm.c void qemu_savevm_state_complete_precopy(QEMUFile *f) · line 1000

Quoted 10 lines
migration/savevm.c +2 -1
     trace_savevm_state_complete_precopy();
 
     cpu_synchronize_all_states();
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
-        if (!se->ops || !se->ops->save_live_complete_precopy) {
+        if (!se->ops || !se->ops->save_live_complete_precopy ||
+            (in_postcopy && se->ops->save_live_complete_postcopy)) {
             continue;
         }

I would change the formatting to something like:

       if (!se->ops ||
           (in_postcopy && se->ops->save_live_complete_postcopy)
           !se->ops->save_live_complete_precopy) {
              continue
           }

Just to make easier to see when we exit?

Later, Juan.

#119 lore raw

Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy

Quoted 11 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.

The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.

Quoted 6 lines
Quoted 4 lines

The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.

Thanks.

Quoted 20 lines
Quoted 7 lines
migration/savevm.c +1 -0
@@ -947,13 +987,15 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)
     int vmdesc_len;
     SaveStateEntry *se;
     int ret;
+    bool in_postcopy = migration_postcopy_phase(migrate_get_current());
 

migration/savevm.c void qemu_savevm_state_complete_precopy(QEMUFile *f) · line 1000

Quoted 10 lines
migration/savevm.c +2 -1
     trace_savevm_state_complete_precopy();
 
     cpu_synchronize_all_states();
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
-        if (!se->ops || !se->ops->save_live_complete_precopy) {
+        if (!se->ops || !se->ops->save_live_complete_precopy ||
+            (in_postcopy && se->ops->save_live_complete_postcopy)) {
             continue;
         }

I would change the formatting to something like:

Quoted 7 lines
       if (!se->ops ||
           (in_postcopy && se->ops->save_live_complete_postcopy)
           !se->ops->save_live_complete_precopy) {
              continue
           }

Just to make easier to see when we exit?

Done

Dave (Starting with the easy fix first :-)

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 24/42
#120 lore raw

Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy

In reply to Dr. David Alan Gilbert (git)
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.

Quoted 8 lines

The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.

The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.

But:

migration/savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 943

Quoted 10 lines
migration/savevm.c +3 -1
@@ -938,7 +938,47 @@ int qemu_savevm_state_iterate(QEMUFile *f)
 static bool should_send_vmdesc(void)
 {
     MachineState *machine = MACHINE(qdev_get_machine());
-    return !machine->suppress_vmdesc;
+    bool in_postcopy = migration_postcopy_phase(migrate_get_current());
+    return !machine->suppress_vmdesc && !in_postcopy;
+}

This should be split in its own patch.

Amit

#121 lore raw

Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy

Quoted 20 lines · * Amit Shah ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.

Quoted 8 lines

The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.

The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.

But:

Quoted 12 lines

migration/savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 943

Quoted 10 lines
migration/savevm.c +3 -1
@@ -938,7 +938,47 @@ int qemu_savevm_state_iterate(QEMUFile *f)
 static bool should_send_vmdesc(void)
 {
     MachineState *machine = MACHINE(qdev_get_machine());
-    return !machine->suppress_vmdesc;
+    bool in_postcopy = migration_postcopy_phase(migrate_get_current());
+    return !machine->suppress_vmdesc && !in_postcopy;
+}

This should be split in its own patch.

Thanks, split.

Dave

Quoted 3 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#122 lore raw

PATCH v7 25/42 Postcopy: Maintain sentmap and calculate discard

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

---
 include/migration/migration.h    |  12 +++
 include/migration/postcopy-ram.h |  35 +++++++
 include/qemu/typedefs.h          |   1 +
 migration/migration.c            |   1 +
 migration/postcopy-ram.c         | 108 +++++++++++++++++++++
 migration/ram.c                  | 203 ++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |   2 -
 trace-events                     |   5 +
 8 files changed, 363 insertions(+), 4 deletions(-)
include/migration/migration.h +12 -0
index 2a22381..4c6cf95 100644
@@ -114,6 +114,13 @@ struct MigrationState
 
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
+
+    /* bitmap of pages that have been sent at least once
+     * only maintained and used in postcopy at the moment
+     * where it's used to send the dirtymap at the start
+     * of the postcopy phase
+     */
+    unsigned long *sentmap;
 };
 
 void process_incoming_migration(QEMUFile *f);
@@ -183,6 +190,11 @@ double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
 void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
+/* For outgoing discard bitmap */
+int ram_postcopy_send_discard_bitmap(MigrationState *ms);
+/* For incoming postcopy discard */
+int ram_discard_range(MigrationIncomingState *mis, const char *block_name,
+                      uint64_t start, uint64_t end);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +35 -0
index d81934f..1d38f76 100644
@@ -16,4 +16,39 @@
 /* Return true if the host supports everything we need to do postcopy-ram */
 bool postcopy_ram_supported_by_host(void);
 
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end);
+
+
+/*
+ * Called at the start of each RAMBlock by the bitmap code
+ * 'offset' is the bitmap offset of the named RAMBlock in the migration
+ * bitmap.
+ * Returns a new PDS
+ */
+PostcopyDiscardState *postcopy_discard_send_init(MigrationState *ms,
+                                                 unsigned long offset,
+                                                 const char *name);
+
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed to postcopy_discard_send_init
+ */
+void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
+                                unsigned long start, unsigned long end);
+
+/*
+ * Called at the end of each RAMBlock by the bitmap code
+ * Sends any outstanding discard messages, frees the PDS
+ */
+void postcopy_discard_send_finish(MigrationState *ms,
+                                  PostcopyDiscardState *pds);
+
 #endif
include/qemu/typedefs.h +1 -0
index 8403856..61b5b46 100644
@@ -61,6 +61,7 @@ typedef struct PCIExpressHost PCIExpressHost;
 typedef struct PCIHostState PCIHostState;
 typedef struct PCMCIACardState PCMCIACardState;
 typedef struct PixelFormat PixelFormat;
+typedef struct PostcopyDiscardState PostcopyDiscardState;
 typedef struct PropertyInfo PropertyInfo;
 typedef struct Property Property;
 typedef struct QEMUBH QEMUBH;
migration/migration.c +1 -0
index 22be23e..180e8b9 100644
@@ -22,6 +22,7 @@
 #include "block/block.h"
 #include "qemu/sockets.h"
 #include "migration/block.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
migration/postcopy-ram.c +108 -0

125 lines not on this page · raw message

migration/ram.c +201 -2

271 lines not on this page · raw message

migration/savevm.c +0 -2
index b7f17b4..e6398dd 100644
@@ -1295,7 +1295,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
     }
     trace_loadvm_postcopy_ram_handle_discard_header(ramid, len);
     while (len) {
-        /* TODO - ram_discard_range gets added in a later patch
         uint64_t start_addr, end_addr;
         start_addr = qemu_get_be64(mis->file);
         end_addr = qemu_get_be64(mis->file);
@@ -1305,7 +1304,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
         if (ret) {
             return ret;
         }
-        */
     }
     trace_loadvm_postcopy_ram_handle_discard_end();
 
trace-events +5 -0
index 72e9889..5e8a120 100644
@@ -1231,6 +1231,7 @@ qemu_file_fclose(void) ""
 migration_bitmap_sync_start(void) ""
 migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64""
 migration_throttle(void) ""
+ram_postcopy_send_discard_bitmap(void) ""
 
 # hw/display/qxl.c
 disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
@@ -1495,6 +1496,10 @@ rdma_start_incoming_migration_after_rdma_listen(void) ""
 rdma_start_outgoing_migration_after_rdma_connect(void) ""
 rdma_start_outgoing_migration_after_rdma_source_init(void) ""
 
+# migration/postcopy-ram.c
+postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands"
+postcopy_ram_discard_range(void *start, void *end) "%p,%p"
+
 # kvm-all.c
 kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
 kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.4.3
#123 lore raw

Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Quoted 5 lines

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

Not a patch without a suggestion O:-)

Quoted 17 lines
---
 include/migration/migration.h    |  12 +++
 include/migration/postcopy-ram.h |  35 +++++++
 include/qemu/typedefs.h          |   1 +
 migration/migration.c            |   1 +
 migration/postcopy-ram.c         | 108 +++++++++++++++++++++
 migration/ram.c                  | 203 ++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |   2 -
 trace-events                     |   5 +
 8 files changed, 363 insertions(+), 4 deletions(-)
include/migration/migration.h +0 -0
index 2a22381..4c6cf95 100644
@@ -114,6 +114,13 @@ struct MigrationState
 

include/migration/migration.h struct MigrationState · line 124

Quoted 10 lines
include/migration/migration.h +7 -0
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
+
+    /* bitmap of pages that have been sent at least once
+     * only maintained and used in postcopy at the moment
+     * where it's used to send the dirtymap at the start
+     * of the postcopy phase
+     */
+    unsigned long *sentmap;
 };

We can use this sentmap for zero page optimization. If page is on sentmap, we need to sent a zero page, otherwise, just sent sentmap at the end of migration and clean everything not there?

include/migration/migration.h struct MigrationState

Quoted 9 lines
include/migration/migration.h +9 -0
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end)
+{
+    trace_postcopy_ram_discard_range(start, end);
+    if (madvise(start, (end-start)+1, MADV_DONTNEED)) {

Can we s/end/lenght/ and adjust everywhere?

Not here, but putting a comment explaining where magic 12 cames from on definition of constant?

I think that the sentbitmap bits could we used without the rest.

#124 lore raw

Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard

Quoted 33 lines · * Juan Quintela ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Quoted 5 lines

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

Not a patch without a suggestion O:-)

Quoted 17 lines
---
 include/migration/migration.h    |  12 +++
 include/migration/postcopy-ram.h |  35 +++++++
 include/qemu/typedefs.h          |   1 +
 migration/migration.c            |   1 +
 migration/postcopy-ram.c         | 108 +++++++++++++++++++++
 migration/ram.c                  | 203 ++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |   2 -
 trace-events                     |   5 +
 8 files changed, 363 insertions(+), 4 deletions(-)
include/migration/migration.h +0 -0
index 2a22381..4c6cf95 100644
@@ -114,6 +114,13 @@ struct MigrationState
 
Quoted 14 lines

include/migration/migration.h struct MigrationState · line 124

Quoted 10 lines
include/migration/migration.h +7 -0
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
+
+    /* bitmap of pages that have been sent at least once
+     * only maintained and used in postcopy at the moment
+     * where it's used to send the dirtymap at the start
+     * of the postcopy phase
+     */
+    unsigned long *sentmap;
 };

We can use this sentmap for zero page optimization. If page is on sentmap, we need to sent a zero page, otherwise, just sent sentmap at the end of migration and clean everything not there?

Just as a compact way of sending zero pages? I'm not sure it would help.

Quoted 11 lines

include/migration/migration.h struct MigrationState

Quoted 9 lines
include/migration/migration.h +9 -0
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end)
+{
+    trace_postcopy_ram_discard_range(start, end);
+    if (madvise(start, (end-start)+1, MADV_DONTNEED)) {

Can we s/end/lenght/ and adjust everywhere?

Done - partially; everything that works in bytes is now start & length, everything that works in indexes into the RAM bitmap is still start/end, since generally that's what they're working with already.

Quoted 2 lines

Not here, but putting a comment explaining where magic 12 cames from on definition of constant?

Done.

Quoted 1 line

I think that the sentbitmap bits could we used without the rest.

Possibly, I can kind of see it's useful - but I'm not convinced what else for yet.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 25/42
#125 lore raw

Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Quoted 5 lines

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

Some whitespace issues, and some sentences in comments don't have a full-stop:

include/migration/postcopy-ram.h line 45

Quoted 9 lines
include/migration/postcopy-ram.h +9 -0
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed to postcopy_discard_send_init
+ */
+void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
+                                unsigned long start, unsigned long end);

unaligned line; no full-stop in comment above (similar elsewhere, not repeating that).

migration/postcopy-ram.c out: · line 172

Quoted 9 lines
migration/postcopy-ram.c +9 -0
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end)
+{
+    trace_postcopy_ram_discard_range(start, end);
+    if (madvise(start, (end-start)+1, MADV_DONTNEED)) {

whitespace around operators

Quoted 6 lines

+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed to postcopy_discard_send_init

missing punctuation

(also, you had started doing doxygen-style comments, want to keep on following that style?)

Quoted 1 line

+static RAMBlock *ram_find_block(const char *id)

just a suggestion, not very particular about this: rename to

ram_find_block_by_id()

instead, so that it's clear what method of finding we're using; also no name conflicts when there might be other ways of doing a find.

Amit

#126 lore raw

Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard

Quoted 19 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Quoted 5 lines

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

Some whitespace issues, and some sentences in comments don't have a full-stop:

Quoted 12 lines

include/migration/postcopy-ram.h line 45

Quoted 9 lines
include/migration/postcopy-ram.h +9 -0
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed to postcopy_discard_send_init
+ */
+void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
+                                unsigned long start, unsigned long end);

unaligned line; no full-stop in comment above (similar elsewhere, not repeating that).

Fixed.

Quoted 11 lines

migration/postcopy-ram.c out: · line 172

Quoted 9 lines
migration/postcopy-ram.c +9 -0
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end)
+{
+    trace_postcopy_ram_discard_range(start, end);
+    if (madvise(start, (end-start)+1, MADV_DONTNEED)) {

whitespace around operators

Fixed.

Quoted 11 lines
Quoted 6 lines

+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed to postcopy_discard_send_init

missing punctuation

(also, you had started doing doxygen-style comments, want to keep on following that style?)

Fixed (I'm sure there are others, still getting into the hang of that).

Quoted 8 lines
Quoted 1 line

+static RAMBlock *ram_find_block(const char *id)

just a suggestion, not very particular about this: rename to

ram_find_block_by_id()

instead, so that it's clear what method of finding we're using; also no name conflicts when there might be other ways of doing a find.

Done.

Dave

Quoted 2 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#127 lore raw

PATCH v7 26/42 postcopy: Incoming initialisation

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h    |   3 +
 include/migration/postcopy-ram.h |  12 ++++
 migration/postcopy-ram.c         | 116 +++++++++++++++++++++++++++++++++++++++
 migration/ram.c                  |  11 ++++
 migration/savevm.c               |   4 ++
 trace-events                     |   2 +
 6 files changed, 148 insertions(+)
include/migration/migration.h +3 -0
index 4c6cf95..98e2568 100644
@@ -69,6 +69,8 @@ struct MigrationIncomingState {
      */
     QemuEvent      main_thread_load_event;
 
+    /* For the kernel to send us notifications */
+    int            userfault_fd;
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
     PostcopyState postcopy_state;
@@ -195,6 +197,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms);
 /* For incoming postcopy discard */
 int ram_discard_range(MigrationIncomingState *mis, const char *block_name,
                       uint64_t start, uint64_t end);
+int ram_postcopy_incoming_init(MigrationIncomingState *mis);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +12 -0
index 1d38f76..4192108 100644
@@ -17,6 +17,18 @@
 bool postcopy_ram_supported_by_host(void);
 
 /*
+ * Initialise postcopy-ram, setting the RAM to a state where we can go into
+ * postcopy later; must be called prior to any precopy.
+ * called from ram.c's similarly named ram_postcopy_incoming_init
+ */
+int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages);
+
+/*
+ * At the end of a migration where postcopy_ram_incoming_init was called.
+ */
+int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis);
+
+/*
  * Discard the contents of memory start..end inclusive.
  * We can assume that if we've been called postcopy_ram_hosttest returned true
  */
migration/postcopy-ram.c +116 -0

128 lines not on this page · raw message

migration/ram.c +11 -0
index 8e681f2..f7d957e 100644
@@ -1667,6 +1667,17 @@ static void decompress_data_with_multi_threads(uint8_t *compbuf,
     }
 }
 
+/*
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}
+
 static int ram_load(QEMUFile *f, void *opaque, int version_id)
 {
     int flags = 0, ret = 0;
migration/savevm.c +4 -0
index e6398dd..f4de52d 100644
@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (ram_postcopy_incoming_init(mis)) {
+        return -1;
+    }
+
     postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);
 
     return 0;
trace-events +2 -0
index 5e8a120..2ffc1c6 100644
@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
 
 # migration/postcopy-ram.c
 postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands"
+postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"
 postcopy_ram_discard_range(void *start, void *end) "%p,%p"
+postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"
 
 # kvm-all.c
 kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.4.3
#128 lore raw

Re: [Qemu-devel] [PATCH v7 26/42] postcopy: Incoming initialisation

Quoted 4 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Quoted 9 lines
---
 include/migration/migration.h    |   3 +
 include/migration/postcopy-ram.h |  12 ++++
 migration/postcopy-ram.c         | 116 +++++++++++++++++++++++++++++++++++++++
 migration/ram.c                  |  11 ++++
 migration/savevm.c               |   4 ++
 trace-events                     |   2 +
 6 files changed, 148 insertions(+)

qemu_hugepage_enable(host_addr, length)?

migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 214

Quoted 6 lines
migration/postcopy-ram.c +6 -0
+#ifdef MADV_NOHUGEPAGE
+    if (madvise(host_addr, length, MADV_NOHUGEPAGE)) {
+        error_report("%s: NOHUGEPAGE: %s", __func__, strerror(errno));
+        return -1;
+    }
+#endif

qemu_hugepage_disable(host_addr, length)?

migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 245

Quoted 12 lines
migration/postcopy-ram.c +12 -0
+#ifdef MADV_HUGEPAGE
+    if (madvise(host_addr, length, MADV_HUGEPAGE)) {
+        error_report("%s HUGEPAGE: %s", __func__, strerror(errno));
+        return -1;
+    }
+#endif
+
+    /*
+     * We can also turn off userfault now since we should have all the
+     * pages.   It can be useful to leave it on to debug postcopy
+     * if you're not sure it's always getting every page.
+     */

qemu_userfault_unregister(host_addr, length)?

Quoted 11 lines
(quoted diff) +9 -0
+    range_struct.start = (uintptr_t)host_addr;
+    range_struct.len = length;
+
+    if (ioctl(mis->userfault_fd, UFFDIO_UNREGISTER, &range_struct)) {
+        error_report("%s: userfault unregister %s", __func__, strerror(errno));
+
+        return -1;
+    }
 
 
+/*

Review

Quoted 10 lines
(quoted diff) +10 -0
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}
+

ram_postocpy_incoming_init()
and
postcopy_ram_incoming_init()

ouch Thinking about better names ....

Quoted 8 lines

static int ram_load(QEMUFile *f, void *opaque, int version_id)

 {
     int flags = 0, ret = 0;
migration/savevm.c +0 -0
index e6398dd..f4de52d 100644

migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1244

Quoted 7 lines
migration/savevm.c +4 -0
@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (ram_postcopy_incoming_init(mis)) {
+        return -1;
+    }
+

how/where we know that this is called soon enough?

Quoted 8 lines
migration/savevm.c +0 -0
     postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);
 
     return 0;
trace-events +0 -0
index 5e8a120..2ffc1c6 100644

trace-events rdma_start_outgoing_migration_after_rdma_source_init(void) "" · line 1503

Quoted 6 lines
trace-events +2 -0
@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
 
 # migration/postcopy-ram.c
 postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands"
+postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"
 postcopy_ram_discard_range(void *start, void *end) "%p,%p"
+postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"

once here, if we have range names before, what about:

postcopy_ram_cleanup_range()
postcopy_ram_init_range()

And let the ram* functions the same?

#129 lore raw

Re: [Qemu-devel] [PATCH v7 26/42] postcopy: Incoming initialisation

Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 4 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Quoted 9 lines
---
 include/migration/migration.h    |   3 +
 include/migration/postcopy-ram.h |  12 ++++
 migration/postcopy-ram.c         | 116 +++++++++++++++++++++++++++++++++++++++
 migration/ram.c                  |  11 ++++
 migration/savevm.c               |   4 ++
 trace-events                     |   2 +
 6 files changed, 148 insertions(+)

qemu_hugepage_enable(host_addr, length)?

Quoted 8 lines

migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 214

Quoted 6 lines
migration/postcopy-ram.c +6 -0
+#ifdef MADV_NOHUGEPAGE
+    if (madvise(host_addr, length, MADV_NOHUGEPAGE)) {
+        error_report("%s: NOHUGEPAGE: %s", __func__, strerror(errno));
+        return -1;
+    }
+#endif

qemu_hugepage_disable(host_addr, length)?

I've flipped those both to use the qemu_madvise which is how it's done in most other places in the codebase (I added
the QEMU_MADV_NOHUGEPAGE since HUGEPAGE was there but not the opposite).

Quoted 6 lines
Quoted 6 lines
migration/postcopy-ram.c +6 -0
+#ifdef MADV_HUGEPAGE
+    if (madvise(host_addr, length, MADV_HUGEPAGE)) {
+        error_report("%s HUGEPAGE: %s", __func__, strerror(errno));
+        return -1;
+    }
+#endif
Quoted 8 lines

migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 245

Quoted 6 lines
migration/postcopy-ram.c +6 -0
+
+    /*
+     * We can also turn off userfault now since we should have all the
+     * pages.   It can be useful to leave it on to debug postcopy
+     * if you're not sure it's always getting every page.
+     */

qemu_userfault_unregister(host_addr, length)?

Is it worth wrapping that ioctl, when it's already in a function that has to do other calls around it?

Quoted 26 lines
Quoted 11 lines
(quoted diff) +9 -0
+    range_struct.start = (uintptr_t)host_addr;
+    range_struct.len = length;
+
+    if (ioctl(mis->userfault_fd, UFFDIO_UNREGISTER, &range_struct)) {
+        error_report("%s: userfault unregister %s", __func__, strerror(errno));
+
+        return -1;
+    }
 
 
+/*

Review

Quoted 10 lines
(quoted diff) +10 -0
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}
+

ram_postocpy_incoming_init()
and
postcopy_ram_incoming_init()

Quoted 1 line

ouch Thinking about better names ....

Agreed; suggestions welcome. the 'ram' and 'postcopy_ram' are both the convention based on the filename.

Quoted 11 lines
Quoted 11 lines

static int ram_load(QEMUFile *f, void *opaque, int version_id)

 {
     int flags = 0, ret = 0;
migration/savevm.c +0 -0
index e6398dd..f4de52d 100644
@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
Quoted 6 lines

migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1244

Quoted 4 lines
migration/savevm.c +4 -0
+    if (ram_postcopy_incoming_init(mis)) {
+        return -1;
+    }
+

how/where we know that this is called soon enough?

It's driven by the sequence of commands byte that walk it through the state machine.

Quoted 17 lines
Quoted 8 lines
migration/savevm.c +0 -0
     postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);
 
     return 0;
trace-events +0 -0
index 5e8a120..2ffc1c6 100644

trace-events rdma_start_outgoing_migration_after_rdma_source_init(void) "" · line 1503

Quoted 6 lines
trace-events +2 -0
@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
 
 # migration/postcopy-ram.c
 postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands"
+postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"
 postcopy_ram_discard_range(void *start, void *end) "%p,%p"
+postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"

once here, if we have range names before, what about:

Quoted 2 lines

postcopy_ram_cleanup_range()
postcopy_ram_init_range()

Done.

Quoted 1 line

And let the ram* functions the same?

Not sure which ones those refer to?

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#131 lore raw

PATCH v7 27/42 postcopy: ram_enable_notify to switch on userfault

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)

---
 include/migration/migration.h    |  3 ++
 include/migration/postcopy-ram.h |  6 ++++
 migration/postcopy-ram.c         | 69 +++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |  9 ++++++
 4 files changed, 86 insertions(+), 1 deletion(-)
include/migration/migration.h +3 -0
index 98e2568..e6585c5 100644
@@ -69,6 +69,9 @@ struct MigrationIncomingState {
      */
     QemuEvent      main_thread_load_event;
 
+    QemuThread     fault_thread;
+    QemuSemaphore  fault_thread_sem;
+
     /* For the kernel to send us notifications */
     int            userfault_fd;
     QEMUFile *return_path;
include/migration/postcopy-ram.h +6 -0
index 4192108..8a8616b 100644
@@ -17,6 +17,12 @@
 bool postcopy_ram_supported_by_host(void);
 
 /*
+ * Make all of RAM sensitive to accesses to areas that haven't yet been written
+ * and wire up anything necessary to deal with it.
+ */
+int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+
+/*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
  * postcopy later; must be called prior to any precopy.
  * called from ram.c's similarly named ram_postcopy_incoming_init
migration/postcopy-ram.c +68 -1

83 lines not on this page · raw message

migration/savevm.c +9 -0
index f4de52d..b87238a 100644
@@ -1324,6 +1324,15 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
         return -1;
     }
 
+    /*
+     * Sensitise RAM - can now generate requests for blocks that don't exist
+     * However, at this point the CPU shouldn't be running, and the IO
+     * shouldn't be doing anything yet so don't actually expect requests
+     */
+    if (postcopy_ram_enable_notify(mis)) {
+        return -1;
+    }
+
     /* TODO start up the postcopy listening thread */
     return 0;
 }
Signature · 1 line
2.4.3
Reviewed-by on v7 27/42
#132 lore raw

Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault

Quoted 20 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)

---
 include/migration/migration.h    |  3 ++
 include/migration/postcopy-ram.h |  6 ++++
 migration/postcopy-ram.c         | 69 +++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |  9 ++++++
 4 files changed, 86 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index 98e2568..e6585c5 100644

include/migration/migration.h struct MigrationIncomingState { · line 72

Quoted 4 lines
include/migration/migration.h +1 -0
@@ -69,6 +69,9 @@ struct MigrationIncomingState {
      */
     QemuEvent      main_thread_load_event;
 
+    QemuThread     fault_thread;

Best name ever? Well, it could be "wrong_thread" O:-)

include/migration/migration.h struct MigrationIncomingState { · line 75

Quoted 3 lines
include/migration/migration.h +3 -0
+static void *postcopy_ram_fault_thread(void *opaque)
+{
+    MigrationIncomingState *mis = (MigrationIncomingState *)opaque;

Uneeded cast.

Quoted 21 lines
include/migration/migration.h +21 -0
+
+    fprintf(stderr, "postcopy_ram_fault_thread\n");
+    /* TODO: In later patch */
+    qemu_sem_post(&mis->fault_thread_sem);
+    while (1) {
+        /* TODO: In later patch */
+    }
+
+    return NULL;
+}
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    /* Create the fault handler thread and wait for it to be ready */
+    qemu_sem_init(&mis->fault_thread_sem, 0);
+    qemu_thread_create(&mis->fault_thread, "postcopy/fault",
+                       postcopy_ram_fault_thread, mis, QEMU_THREAD_JOINABLE);
+    qemu_sem_wait(&mis->fault_thread_sem);
+    qemu_sem_destroy(&mis->fault_thread_sem);
+
+    /* Mark so that we get notified of accesses to unwritten areas */

include/migration/migration.h struct MigrationIncomingState {

Quoted 10 lines
include/migration/migration.h +7 -1
+    if (qemu_ram_foreach_block(ram_block_enable_notify, mis)) {
+        return -1;
+    }
+
+    return 0;
+}
+
 #else
 /* No target OS support, stubs just fail */
-

This belongs in a different patch O:-)

If you have to resend, just change them.

#133 lore raw

Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault

Quoted 24 lines · * Juan Quintela ([email protected]) wrote:
Quoted 23 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)

---
 include/migration/migration.h    |  3 ++
 include/migration/postcopy-ram.h |  6 ++++
 migration/postcopy-ram.c         | 69 +++++++++++++++++++++++++++++++++++++++-
 migration/savevm.c               |  9 ++++++
 4 files changed, 86 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index 98e2568..e6585c5 100644
@@ -69,6 +69,9 @@ struct MigrationIncomingState {
      */
     QemuEvent      main_thread_load_event;
 
Quoted 3 lines

include/migration/migration.h struct MigrationIncomingState { · line 72

Quoted 1 line
include/migration/migration.h +1 -0
+    QemuThread     fault_thread;

Best name ever? Well, it could be "wrong_thread" O:-)

Well they are 'user faults'! Not that we blame the user for
them personally of course.

Quoted 5 lines

include/migration/migration.h struct MigrationIncomingState { · line 75

Quoted 3 lines
include/migration/migration.h +3 -0
+static void *postcopy_ram_fault_thread(void *opaque)
+{
+    MigrationIncomingState *mis = (MigrationIncomingState *)opaque;

Uneeded cast.

Thanks, gone.

Quoted 25 lines
Quoted 25 lines
include/migration/migration.h +25 -0
+
+    fprintf(stderr, "postcopy_ram_fault_thread\n");
+    /* TODO: In later patch */
+    qemu_sem_post(&mis->fault_thread_sem);
+    while (1) {
+        /* TODO: In later patch */
+    }
+
+    return NULL;
+}
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    /* Create the fault handler thread and wait for it to be ready */
+    qemu_sem_init(&mis->fault_thread_sem, 0);
+    qemu_thread_create(&mis->fault_thread, "postcopy/fault",
+                       postcopy_ram_fault_thread, mis, QEMU_THREAD_JOINABLE);
+    qemu_sem_wait(&mis->fault_thread_sem);
+    qemu_sem_destroy(&mis->fault_thread_sem);
+
+    /* Mark so that we get notified of accesses to unwritten areas */
+    if (qemu_ram_foreach_block(ram_block_enable_notify, mis)) {
+        return -1;
+    }
+
Quoted 8 lines

include/migration/migration.h

Quoted 6 lines
include/migration/migration.h +3 -1
+    return 0;
+}
+
 #else
 /* No target OS support, stubs just fail */
-

This belongs in a different patch O:-)

Thanks, gone.

Quoted 3 lines

If you have to resend, just change them.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 27/42
#134 lore raw

Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault

In reply to Dr. David Alan Gilbert (git)
Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)

Amit

#135 lore raw

PATCH v7 28/42 Postcopy: Postcopy startup in migration thread

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration/migration.c         | 166 ++++++++++++++++++++++++++++++++++++++++--
 trace-events                  |   4 +
 3 files changed, 167 insertions(+), 6 deletions(-)
include/migration/migration.h +3 -0
index e6585c5..68a1731 100644
@@ -120,6 +120,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
 
+    /* Flag set once the migration thread is running (and needs joining) */
+    bool started_migration_thread;
+
     /* bitmap of pages that have been sent at least once
      * only maintained and used in postcopy at the moment
      * where it's used to send the dirtymap at the start
migration/migration.c +160 -6

239 lines not on this page · raw message

trace-events +4 -0
index 2ffc1c6..f096877 100644
@@ -1422,9 +1422,13 @@ migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+migration_thread_after_loop(void) ""
+migration_thread_file_err(void) ""
 migration_thread_setup_complete(void) ""
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
+postcopy_start(void) ""
+postcopy_start_set_run(void) ""
 source_return_path_thread_bad_end(void) ""
 source_return_path_thread_end(void) ""
 source_return_path_thread_entry(void) ""
Signature · 1 line
2.4.3
#136 lore raw

Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread

Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration/migration.c         | 166 ++++++++++++++++++++++++++++++++++++++++--
 trace-events                  |   4 +
 3 files changed, 167 insertions(+), 6 deletions(-)
include/migration/migration.h +0 -0
index e6585c5..68a1731 100644

include/migration/migration.h struct MigrationState · line 125

Quoted 6 lines
include/migration/migration.h +3 -0
@@ -120,6 +120,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
 
+    /* Flag set once the migration thread is running (and needs joining) */
+    bool started_migration_thread;
+

migration_thread_started?

include/migration/migration.h struct MigrationState

Quoted 7 lines
include/migration/migration.h +7 -0
+
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);

Do we undo this? or, are we sure that it is ok to maximize network output?

include/migration/migration.h

Quoted 2 lines
include/migration/migration.h +2 -0
+    /* Ping just for debugging, helps line traces up */
+    qemu_savevm_send_ping(ms->file, 2);

Change the values 1, 2, 3 to constants?

Quoted 13 lines
include/migration/migration.h +13 -0
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        goto fail;
+    }
+
+    qemu_savevm_state_complete_precopy(fb);
+    qemu_savevm_send_ping(fb, 3);
+

include/migration/migration.h

Quoted 10 lines
include/migration/migration.h +10 -0
+    qemu_savevm_send_postcopy_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
+    if (qemu_savevm_send_packaged(ms->file, qsb)) {
+        goto fail_closefb;
+    }
+    qemu_fclose(fb);

Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?

Quoted 1 line

+ ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - time_at_stop;

Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.

Quoted 7 lines
include/migration/migration.h +1 -0
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;

include/migration/migration.h

Quoted 10 lines
include/migration/migration.h +4 -0
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
     bool old_vm_running = false;
+    bool entered_postcopy = false;
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationStatus current_active_type = MIGRATION_STATUS_ACTIVE;

current_active_state?

#137 lore raw

Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread

Quoted 20 lines · * Juan Quintela ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration/migration.c         | 166 ++++++++++++++++++++++++++++++++++++++++--
 trace-events                  |   4 +
 3 files changed, 167 insertions(+), 6 deletions(-)
include/migration/migration.h +0 -0
index e6585c5..68a1731 100644
@@ -120,6 +120,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     bool start_postcopy;
 
Quoted 5 lines

include/migration/migration.h struct MigrationState · line 125

Quoted 3 lines
include/migration/migration.h +3 -0
+    /* Flag set once the migration thread is running (and needs joining) */
+    bool started_migration_thread;
+

migration_thread_started?

Changed.

Quoted 10 lines

include/migration/migration.h struct MigrationState

Quoted 7 lines
include/migration/migration.h +7 -0
+
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);

Do we undo this? or, are we sure that it is ok to maximize network output?

No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.

Quoted 4 lines

include/migration/migration.h

Quoted 2 lines
include/migration/migration.h +2 -0
+    /* Ping just for debugging, helps line traces up */
+    qemu_savevm_send_ping(ms->file, 2);

Change the values 1, 2, 3 to constants?

Suggestions to names? - they purely for debugging so you can match it up on the destination.

Quoted 19 lines
Quoted 19 lines
include/migration/migration.h +19 -0
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        goto fail;
+    }
+
+    qemu_savevm_state_complete_precopy(fb);
+    qemu_savevm_send_ping(fb, 3);
+
+    qemu_savevm_send_postcopy_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
Quoted 8 lines

include/migration/migration.h

Quoted 4 lines
include/migration/migration.h +4 -0
+    if (qemu_savevm_send_packaged(ms->file, qsb)) {
+        goto fail_closefb;
+    }
+    qemu_fclose(fb);

Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?

The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.

Quoted 4 lines
Quoted 1 line

+ ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - time_at_stop;

Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.

No, not separate.

Quoted 18 lines
Quoted 7 lines
include/migration/migration.h +1 -0
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;

include/migration/migration.h

Quoted 10 lines
include/migration/migration.h +4 -0
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
     bool old_vm_running = false;
+    bool entered_postcopy = false;
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationStatus current_active_type = MIGRATION_STATUS_ACTIVE;
Quoted 1 line

current_active_state?

Changed.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#138 lore raw

Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread

Quoted 11 lines · Dr. David Alan Gilbert
Quoted 10 lines

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1105

Quoted 7 lines
migration/migration.c +7 -0
+
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);

Do we undo this? or, are we sure that it is ok to maximize network output?

Quoted 8 lines

No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.

I see.

Quoted 37 lines
Quoted 5 lines

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1107

Quoted 3 lines
migration/migration.c +2 -0
+    /* Ping just for debugging, helps line traces up */
+    qemu_savevm_send_ping(ms->file, 2);

Change the values 1, 2, 3 to constants?

Suggestions to names? - they purely for debugging so you can match it up on the destination.

Quoted 19 lines
Quoted 19 lines
migration/migration.c +19 -0
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        goto fail;
+    }
+
+    qemu_savevm_state_complete_precopy(fb);
+    qemu_savevm_send_ping(fb, 3);
+
+    qemu_savevm_send_postcopy_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
Quoted 8 lines

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1132

Quoted 4 lines
migration/migration.c +4 -0
+    if (qemu_savevm_send_packaged(ms->file, qsb)) {
+        goto fail_closefb;
+    }
+    qemu_fclose(fb);

Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?

Quoted 7 lines

The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.

Arghhh. ok. Comment?

Quoted 34 lines
Quoted 5 lines
Quoted 2 lines

+ ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - time_at_stop;

Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.

No, not separate.

Quoted 18 lines
Quoted 7 lines
migration/migration.c +1 -0
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1177

Quoted 10 lines
migration/migration.c +4 -0
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
     bool old_vm_running = false;
+    bool entered_postcopy = false;
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationStatus current_active_type = MIGRATION_STATUS_ACTIVE;
Quoted 1 line

current_active_state?

Changed.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#139 lore raw

Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread

Quoted 38 lines · * Juan Quintela ([email protected]) wrote:
Quoted 28 lines · Dr. David Alan Gilbert
Quoted 19 lines
Quoted 19 lines
migration/migration.c +19 -0
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        goto fail;
+    }
+
+    qemu_savevm_state_complete_precopy(fb);
+    qemu_savevm_send_ping(fb, 3);
+
+    qemu_savevm_send_postcopy_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
Quoted 8 lines

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1132

Quoted 4 lines
migration/migration.c +4 -0
+    if (qemu_savevm_send_packaged(ms->file, qsb)) {
+        goto fail_closefb;
+    }
+    qemu_fclose(fb);

Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?

Quoted 7 lines

The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.

Quoted 1 line

Arghhh. ok. Comment?

I've changed the comment at the start of that section to:

* While loading the device state we may trigger page transfer
* requests and the fd must be free to process those, and thus
* the destination must read the whole device state off the fd before
* it starts processing it. Unfortunately the ad-hoc migration format
* doesn't allow the destination to know the size to read without fully
* parsing it through each devices load-state code (especially the open
* coded devices that use get/put).
* So we wrap the device state up in a package with a length at the start;
* to do this we use a qemu_buf to hold the whole of the device state.

Dave

Quoted 27 lines
Quoted 27 lines
Quoted 6 lines
Quoted 3 lines

+ ms->downtime = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - time_at_stop;

Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.

No, not separate.

Quoted 18 lines
Quoted 17 lines
migration/migration.c +5 -0
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
     bool old_vm_running = false;
+    bool entered_postcopy = false;
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationStatus current_active_type = MIGRATION_STATUS_ACTIVE;
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

current_active_state?

Changed.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#140 lore raw

Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread

In reply to Dr. David Alan Gilbert
Quoted 12 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Juan Quintela ([email protected]) wrote:

migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1105

Quoted 6 lines
migration/migration.c +6 -0
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);

Do we undo this? or, are we sure that it is ok to maximize network output?

Quoted 8 lines

No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.

Might be a conflict in the knobs we expose (max_bandwidth) and us not adhering to that.

I agree we want this to go full-throttle, so maybe document that postcopy will override that knob? It's tricky to get everyone to understand that postcopy will do that. Plus there'll be other questions like what else does postcopy override? -- not that there's anythign more, but users will wonder.

Amit

#142 lore raw

PATCH v7 29/42 Postcopy end in migration_thread

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.

The end of migration code is getting a bit complciated now, so move out into its own function.

---
 migration/migration.c | 91 +++++++++++++++++++++++++++++++++++++--------------
 trace-events          |  6 ++++
 2 files changed, 72 insertions(+), 25 deletions(-)
migration/migration.c +66 -25
index 8d15f33..3e5a7c8 100644
@@ -1041,7 +1041,6 @@ static int open_return_path_on_source(MigrationState *ms)
     return 0;
 }
 
-__attribute__ (( unused )) /* Until later in patch series */
 /* Returns 0 if the RP was ok, otherwise there was an error on the RP */
 static int await_return_path_close_on_source(MigrationState *ms)
 {
@@ -1159,6 +1158,68 @@ fail:
 }
 
 /*
+ * Used by migration_thread when there's not much left pending.
+ * The caller 'breaks' the loop when this returns.
+ */
+static void migration_thread_end_of_iteration(MigrationState *s,
+                                              int current_active_state,
+                                              bool *old_vm_running,
+                                              int64_t *start_time)
+{
+    int ret;
+    if (s->state == MIGRATION_STATUS_ACTIVE) {
+        qemu_mutex_lock_iothread();
+        *start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+        qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+        *old_vm_running = runstate_is_running();
+
+        ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+        if (ret >= 0) {
+            qemu_file_set_rate_limit(s->file, INT64_MAX);
+            qemu_savevm_state_complete_precopy(s->file);
+        }
+        qemu_mutex_unlock_iothread();
+
+        if (ret < 0) {
+            goto fail;
+        }
+    } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
+        trace_migration_thread_end_of_iteration_postcopy_end();
+
+        qemu_savevm_state_complete_postcopy(s->file);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_complete();
+    }
+
+    /*
+     * If rp was opened we must clean up the thread before
+     * cleaning everything else up (since if there are no failures
+     * it will wait for the destination to send it's status in
+     * a SHUT command).
+     * Postcopy opens rp if enabled (even if it's not avtivated)
+     */
+    if (migrate_postcopy_ram()) {
+        int rp_error;
+        trace_migration_thread_end_of_iteration_postcopy_end_before_rp();
+        rp_error = await_return_path_close_on_source(s);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_rp(rp_error);
+        if (rp_error) {
+            goto fail;
+        }
+    }
+
+    if (qemu_file_get_error(s->file)) {
+        trace_migration_thread_end_of_iteration_file_err();
+        goto fail;
+    }
+
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_COMPLETED);
+    return;
+
+fail:
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
+}
+
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
@@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)
                 /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
-                int ret;
-
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
-
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete_precopy(s->file);
-                }
-                qemu_mutex_unlock_iothread();
+                trace_migration_thread_low_pending(pending_size);
 
-                if (ret < 0) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_FAILED);
-                    break;
-                }
-
-                if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_COMPLETED);
-                    break;
-                }
+                migration_thread_end_of_iteration(s, current_active_type,
+                    &old_vm_running, &start_time);
+                break;
             }
         }
 
trace-events +6 -0
index f096877..528d5a3 100644
@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migration_thread_after_loop(void) ""
 migration_thread_file_err(void) ""
 migration_thread_setup_complete(void) ""
+migration_thread_low_pending(uint64_t pending) "%" PRIu64
+migration_thread_end_of_iteration_file_err(void) ""
+migration_thread_end_of_iteration_postcopy_end(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_complete(void) ""
+migration_thread_end_of_iteration_postcopy_end_before_rp(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d"
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
 postcopy_start(void) ""
Signature · 1 line
2.4.3
#143 lore raw

Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.

The end of migration code is getting a bit complciated now, so move out into its own function.

I think that I would splint the function and then add the postcopy code.

BTW, it is a local function, we can use shorter names:

migration_completion()?

trace names specifically get hugggggggggge.

migration/migration.c fail: · line 1165

Quoted 2 lines
migration/migration.c +2 -0
+static void migration_thread_end_of_iteration(MigrationState *s,
+                                              int current_active_state,

RunState?
And it is not needed as parameter.

migration/migration.c fail: · line 1170

Quoted 5 lines
migration/migration.c +5 -0
+                                              bool *old_vm_running,
+                                              int64_t *start_time)
+{
+    int ret;
+    if (s->state == MIGRATION_STATUS_ACTIVE) {
           current_active_state = s->state;
Quoted 6 lines
migration/migration.c +6 -0
+        qemu_mutex_lock_iothread();
+        *start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+        qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+        *old_vm_running = runstate_is_running();
+
+        ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);

migration/migration.c fail: · line 1186

Quoted 10 lines
migration/migration.c +10 -0
+        if (ret >= 0) {
+            qemu_file_set_rate_limit(s->file, INT64_MAX);
+            qemu_savevm_state_complete_precopy(s->file);
+        }
+        qemu_mutex_unlock_iothread();
+
+        if (ret < 0) {
+            goto fail;
+        }
+    } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
           current_active_state = s->state;
Quoted 92 lines
migration/migration.c +40 -24
+        trace_migration_thread_end_of_iteration_postcopy_end();
+
+        qemu_savevm_state_complete_postcopy(s->file);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_complete();
+    }
+
+    /*
+     * If rp was opened we must clean up the thread before
+     * cleaning everything else up (since if there are no failures
+     * it will wait for the destination to send it's status in
+     * a SHUT command).
+     * Postcopy opens rp if enabled (even if it's not avtivated)
+     */
+    if (migrate_postcopy_ram()) {
+        int rp_error;
+        trace_migration_thread_end_of_iteration_postcopy_end_before_rp();
+        rp_error = await_return_path_close_on_source(s);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_rp(rp_error);
+        if (rp_error) {
+            goto fail;
+        }
+    }
+
+    if (qemu_file_get_error(s->file)) {
+        trace_migration_thread_end_of_iteration_file_err();
+        goto fail;
+    }
+
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_COMPLETED);
+    return;
+
+fail:
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
+}
+
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
@@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)
                 /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
-                int ret;
-
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
-
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete_precopy(s->file);
-                }
-                qemu_mutex_unlock_iothread();
+                trace_migration_thread_low_pending(pending_size);
 
-                if (ret < 0) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_FAILED);
-                    break;
-                }
-
-                if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_COMPLETED);
-                    break;
-                }
+                migration_thread_end_of_iteration(s, current_active_type,
+                    &old_vm_running, &start_time);
+                break;
             }
         }
 
trace-events +6 -0
index f096877..528d5a3 100644
@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migration_thread_after_loop(void) ""
 migration_thread_file_err(void) ""
 migration_thread_setup_complete(void) ""
+migration_thread_low_pending(uint64_t pending) "%" PRIu64
+migration_thread_end_of_iteration_file_err(void) ""
+migration_thread_end_of_iteration_postcopy_end(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_complete(void) ""
+migration_thread_end_of_iteration_postcopy_end_before_rp(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d"
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
 postcopy_start(void) ""
#144 lore raw

Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread

Quoted 7 lines · Juan Quintela
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.

Quoted 6 lines
Quoted 4 lines

The end of migration code is getting a bit complciated now, so move out into its own function.

I think that I would splint the function and then add the postcopy code.

Yeah, esp since this code was added / modified in the previous patch.

Amit

#145 lore raw

Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread

In reply to Juan Quintela
Quoted 7 lines · * Juan Quintela ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.

Quoted 6 lines
Quoted 4 lines

The end of migration code is getting a bit complciated now, so move out into its own function.

I think that I would splint the function and then add the postcopy code.

Done; I now have two patches:
Split out end of migration code from migration_thread
Postcopy: End of iteration

Quoted 5 lines

BTW, it is a local function, we can use shorter names:

migration_completion()?

trace names specifically get hugggggggggge.

Done.

Quoted 5 lines

migration/migration.c fail: · line 1165

Quoted 2 lines
migration/migration.c +2 -0
+static void migration_thread_end_of_iteration(MigrationState *s,
+                                              int current_active_state,

RunState?
And it is not needed as parameter.

No, it's not RunState, it's derived from s->state which is still an int; it's also not the current state, but the current state we're expecting to be in, i.e. one of MIGRATION_STATUS_ACTIVE or MIGRATION_STATE_POSTCOPY_ACTIVE (which is why it's current_*active*_state); and it's only used as the parameter to migrate_set_state - in the same way the current code does:

                   migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
                                     MIGRATION_STATUS_COMPLETED);

to ensure that any failure or cancel occuring at the same time isn't lost.

Dave

Quoted 109 lines

migration/migration.c fail: · line 1170

Quoted 7 lines
migration/migration.c +5 -0
+                                              bool *old_vm_running,
+                                              int64_t *start_time)
+{
+    int ret;
+    if (s->state == MIGRATION_STATUS_ACTIVE) {
           current_active_state = s->state;
Quoted 6 lines
migration/migration.c +6 -0
+        qemu_mutex_lock_iothread();
+        *start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+        qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+        *old_vm_running = runstate_is_running();
+
+        ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);

migration/migration.c fail: · line 1186

Quoted 10 lines
migration/migration.c +10 -0
+        if (ret >= 0) {
+            qemu_file_set_rate_limit(s->file, INT64_MAX);
+            qemu_savevm_state_complete_precopy(s->file);
+        }
+        qemu_mutex_unlock_iothread();
+
+        if (ret < 0) {
+            goto fail;
+        }
+    } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
           current_active_state = s->state;
Quoted 84 lines
migration/migration.c +40 -24
+        trace_migration_thread_end_of_iteration_postcopy_end();
+
+        qemu_savevm_state_complete_postcopy(s->file);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_complete();
+    }
+
+    /*
+     * If rp was opened we must clean up the thread before
+     * cleaning everything else up (since if there are no failures
+     * it will wait for the destination to send it's status in
+     * a SHUT command).
+     * Postcopy opens rp if enabled (even if it's not avtivated)
+     */
+    if (migrate_postcopy_ram()) {
+        int rp_error;
+        trace_migration_thread_end_of_iteration_postcopy_end_before_rp();
+        rp_error = await_return_path_close_on_source(s);
+        trace_migration_thread_end_of_iteration_postcopy_end_after_rp(rp_error);
+        if (rp_error) {
+            goto fail;
+        }
+    }
+
+    if (qemu_file_get_error(s->file)) {
+        trace_migration_thread_end_of_iteration_file_err();
+        goto fail;
+    }
+
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_COMPLETED);
+    return;
+
+fail:
+    migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
+}
+
+/*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
  */
@@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)
                 /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
-                int ret;
-
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
-
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete_precopy(s->file);
-                }
-                qemu_mutex_unlock_iothread();
+                trace_migration_thread_low_pending(pending_size);
 
-                if (ret < 0) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_FAILED);
-                    break;
-                }
-
-                if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
-                                      MIGRATION_STATUS_COMPLETED);
-                    break;
-                }
+                migration_thread_end_of_iteration(s, current_active_type,
+                    &old_vm_running, &start_time);
+                break;
             }
         }
 
trace-events +1 -0
index f096877..528d5a3 100644
@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migration_thread_after_loop(void) ""
 migration_thread_file_err(void) ""
 migration_thread_setup_complete(void) ""
+migration_thread_low_pending(uint64_t pending) "%" PRIu64
Quoted 8 lines

trace-events migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" · line 1436

Quoted 8 lines
trace-events +5 -0
+migration_thread_end_of_iteration_file_err(void) ""
+migration_thread_end_of_iteration_postcopy_end(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_complete(void) ""
+migration_thread_end_of_iteration_postcopy_end_before_rp(void) ""
+migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d"
 open_return_path_on_source(void) ""
 open_return_path_on_source_continue(void) ""
 postcopy_start(void) ""

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by on v7 29/42
#146 lore raw

Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread

In reply to Dr. David Alan Gilbert (git)
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.

The end of migration code is getting a bit complciated now, so move out into its own function.

Amit

#147 lore raw

PATCH v7 30/42 Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.

---
 include/migration/migration.h |  4 +++
 migration/migration.c         | 70 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 3 files changed, 75 insertions(+)
include/migration/migration.h +4 -0
index 68a1731..8742d53 100644
@@ -47,6 +47,8 @@ enum mig_rp_message_type {
     MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
     MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
     MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+
+    MIG_RP_MSG_REQ_PAGES,    /* data (start: be64, len: be64) */
 };
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
@@ -246,6 +248,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis,
                           uint32_t value);
 void migrate_send_rp_pong(MigrationIncomingState *mis,
                           uint32_t value);
+void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
+                              ram_addr_t start, ram_addr_t len);
 
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
migration/migration.c +70 -0

100 lines not on this page · raw message

trace-events +1 -0
index 528d5a3..328d64c 100644
@@ -1422,6 +1422,7 @@ migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
+migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at %zx len %zx"
 migration_thread_after_loop(void) ""
 migration_thread_file_err(void) ""
 migration_thread_setup_complete(void) ""
Signature · 1 line
2.4.3
#148 lore raw

Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

Quoted 17 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.

---
 include/migration/migration.h |  4 +++
 migration/migration.c         | 70 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 3 files changed, 75 insertions(+)
include/migration/migration.h +0 -0
index 68a1731..8742d53 100644

include/migration/migration.h enum mig_rp_message_type { · line 51

Quoted 5 lines
include/migration/migration.h +2 -0
@@ -47,6 +47,8 @@ enum mig_rp_message_type {
     MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
     MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
     MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+
+    MIG_RP_MSG_REQ_PAGES,    /* data (start: be64, len: be64) */

Not that I really care, buht I think that leng could be 32bits. I am not seing networking getting good at multigigabytes transfers soon O:-)

include/migration/migration.h void migrate_send_rp_shut(MigrationIncomingState *mis, · line 252

Quoted 2 lines
include/migration/migration.h +2 -0
+void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
+                              ram_addr_t start, ram_addr_t len);

Shouldn't len be a size_t?
(yes, I know that migration code is not really consistent about that)

Quoted 15 lines
include/migration/migration.h +0 -0
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
migration/migration.c +5 -0
index 3e5a7c8..0373b77 100644
@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp)
     deferred_incoming = true;
 }
 
+/* Request a range of pages from the source VM at the given
+ * start address.
+ *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
+ *           as the last request (a name must have been given previously)
+ *   Start: Address offset within the RB

migration/migration.c static void deferred_incoming_migration(Error **errp) · line 130

Quoted 10 lines
migration/migration.c +10 -0
+ *   Len: Length in bytes required - must be a multiple of pagesize
+ */
+void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname,
+                               ram_addr_t start, ram_addr_t len)
+{
+    uint8_t bufc[16+1+255]; /* start (8 byte), len (8 byte), rbname upto 256 */
+    uint64_t *buf64 = (uint64_t *)bufc;
+    size_t msglen = 16; /* start + len */
+
+    assert(!(len & 1));

ohhhh, why can't we get a real flags field?

Scratch that. Seeing the rest of the code, can't we have two commands:

MIG_RP_MSG_REQ_PAGES
MIG_RP_MSG_REQ_PAGES_WITH_ID

I am not really sure that it makes sense getting a command that can be of two different lengths only for that?

I am not sure, but having a command with two different payloads look strange.

Later, Juan.

#149 lore raw

Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

Quoted 13 lines · * Juan Quintela ([email protected]) wrote:
Quoted 12 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.

---
 include/migration/migration.h |  4 +++
 migration/migration.c         | 70 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 3 files changed, 75 insertions(+)
Quoted 13 lines

include/migration/migration.h enum mig_rp_message_type { · line 51

Quoted 10 lines
include/migration/migration.h +2 -0
index 68a1731..8742d53 100644
@@ -47,6 +47,8 @@ enum mig_rp_message_type {
     MIG_RP_MSG_INVALID = 0,  /* Must be 0 */
     MIG_RP_MSG_SHUT,         /* sibling will not send any more RP messages */
     MIG_RP_MSG_PONG,         /* Response to a PING; data (seq: be32 ) */
+
+    MIG_RP_MSG_REQ_PAGES,    /* data (start: be64, len: be64) */

Not that I really care, buht I think that leng could be 32bits. I am not seing networking getting good at multigigabytes transfers soon O:-)

Done.

Quoted 5 lines

include/migration/migration.h void migrate_send_rp_shut(MigrationIncomingState *mis, · line 252

Quoted 2 lines
include/migration/migration.h +2 -0
+void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
+                              ram_addr_t start, ram_addr_t len);

Shouldn't len be a size_t?
(yes, I know that migration code is not really consistent about that)

Done (fun combination with the change above, but still)

Quoted 30 lines
Quoted 15 lines
include/migration/migration.h +0 -0
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
migration/migration.c +5 -0
index 3e5a7c8..0373b77 100644
@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp)
     deferred_incoming = true;
 }
 
+/* Request a range of pages from the source VM at the given
+ * start address.
+ *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
+ *           as the last request (a name must have been given previously)
+ *   Start: Address offset within the RB

migration/migration.c static void deferred_incoming_migration(Error **errp) · line 130

Quoted 10 lines
migration/migration.c +10 -0
+ *   Len: Length in bytes required - must be a multiple of pagesize
+ */
+void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname,
+                               ram_addr_t start, ram_addr_t len)
+{
+    uint8_t bufc[16+1+255]; /* start (8 byte), len (8 byte), rbname upto 256 */
+    uint64_t *buf64 = (uint64_t *)bufc;
+    size_t msglen = 16; /* start + len */
+
+    assert(!(len & 1));

ohhhh, why can't we get a real flags field?

Scratch that. Seeing the rest of the code, can't we have two commands:

Quoted 8 lines

MIG_RP_MSG_REQ_PAGES
MIG_RP_MSG_REQ_PAGES_WITH_ID

I am not really sure that it makes sense getting a command that can be of two different lengths only for that?

I am not sure, but having a command with two different payloads look strange.

Done (I made it _ID rather than WITH_ID - it was getting a bit long).

Dave

Quoted 2 lines

Later, Juan.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#150 lore raw

Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

In reply to Dr. David Alan Gilbert (git)
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.

migration/migration.c static void deferred_incoming_migration(Error **errp) · line 119

Quoted 7 lines
migration/migration.c +4 -0
@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp)
     deferred_incoming = true;
 }
 
+/* Request a range of pages from the source VM at the given
+ * start address.
+ *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
+ *           as the last request (a name must have been given previously)

Why not just send the name all the time?

Quoted 5 lines
migration/migration.c +1 -0
@@ -1010,6 +1058,28 @@ static void *source_return_path_thread(void *opaque)
             trace_source_return_path_thread_pong(tmp32);
             break;
 
+        case MIG_RP_MSG_REQ_PAGES:

migration/migration.c static void *source_return_path_thread(void *opaque) · line 1071

Quoted 10 lines
migration/migration.c +10 -0
+            start = be64_to_cpup((uint64_t *)buf);
+            len = be64_to_cpup(((uint64_t *)buf)+1);
+            tmpstr = NULL;
+            if (len & 1) {
+                len -= 1; /* Remove the flag */
+                /* Now we expect an idstr */
+                tmp32 = buf[16]; /* Length of the following idstr */
+                tmpstr = (char *)&buf[17];
+                buf[17+tmp32] = '\0';
+                expected_len = 16+1+tmp32;

Whitespace missing around operators

migration/migration.c static void *source_return_path_thread(void *opaque) · line 1074

Quoted 3 lines
migration/migration.c +3 -0
+            } else {
+                expected_len = 16;
+            }

This else can be removed if expected_len is set before the if

Amit

#151 lore raw

Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command

Quoted 14 lines · * Amit Shah ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.

migration/migration.c +0 -0
@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp)
     deferred_incoming = true;
 }
 
Quoted 6 lines

migration/migration.c static void deferred_incoming_migration(Error **errp) · line 119

Quoted 4 lines
migration/migration.c +4 -0
+/* Request a range of pages from the source VM at the given
+ * start address.
+ *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
+ *           as the last request (a name must have been given previously)

Why not just send the name all the time?

It does shrink the messages quite a bit, and you have to do a name lookup on them when you receive it, rather than just knowing it's the same one.

Quoted 4 lines
Quoted 4 lines
migration/migration.c +0 -0
@@ -1010,6 +1058,28 @@ static void *source_return_path_thread(void *opaque)
             trace_source_return_path_thread_pong(tmp32);
             break;
 
Quoted 13 lines

migration/migration.c static void *source_return_path_thread(void *opaque) · line 1071

Quoted 11 lines
migration/migration.c +11 -0
+        case MIG_RP_MSG_REQ_PAGES:
+            start = be64_to_cpup((uint64_t *)buf);
+            len = be64_to_cpup(((uint64_t *)buf)+1);
+            tmpstr = NULL;
+            if (len & 1) {
+                len -= 1; /* Remove the flag */
+                /* Now we expect an idstr */
+                tmp32 = buf[16]; /* Length of the following idstr */
+                tmpstr = (char *)&buf[17];
+                buf[17+tmp32] = '\0';
+                expected_len = 16+1+tmp32;

Whitespace missing around operators

Done.

Quoted 5 lines

migration/migration.c static void *source_return_path_thread(void *opaque) · line 1074

Quoted 3 lines
migration/migration.c +3 -0
+            } else {
+                expected_len = 16;
+            }

This else can be removed if expected_len is set before the if

That's simplified out with the change Juan suggested.

Dave

Quoted 2 lines

Amit

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK