Skip to contentSkip to conversation

Postcopy implementation

PATCH Superseded Dr. David Alan Gilbert (git) 5 participants 181 messages Feb 25, 2015 – Jun 16, 2015 Page 1 of 2

v5 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v5 cover Postcopy implementation

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This is the 5th cut of my version of postcopy; it is designed for use with
the Linux kernel additions posted by Andrea Arcangeli here:

git clone --reference linux -b userfault16 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git

(Note this is a different API from the last version)

This qemu series can be found at:

https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v5 tag.

I believe I've fixed most of the issues from the reviews, but there are probably a few more to cover before the final version:
* I've got a rare (1/1000 migration) bug I'm fighting
* The kernel API is still being discussed

v5
New kernel API
Uses an atomic copy rather than remap to avoid IPIs
Fix builds on older machines
Fix seg if block migration enabled (thanks to Gary Hook for reporting) Fix for mlock (thanks to zhanghailiang for reporting)
Send postcopy-notify before sending the 'begin' part of device state Disable ballooning during postcopy
Don't allocate zero pages as they migrate
Stop using sysconf to get pagesizes
- using getpagesize() and qemu_host_page_size in target dependent parts
Fix for worst case page size ratio (untested)
- ARM is 64k host, 1k target; with 32bit longs
Drop the assertion check on migration_dirty_pages
I don't like killing src vms anyway, and I've not seen it trigger in 9months+
Free up migration_incoming-state at end of postcopy
Calculate downtime on source differently for postcopy
Use trace rather than dprintf
Paolo's comments:
Rename postcopy_ram_sensitise_area
Tidy up of loadvm loop flags
Remove one layer of socket_shutdown abstraction
Remove _RAM_ from many of the postcopy symbols as appropriate
- most messages/states could now be used for other postcopy state.
Used a qemu_event to wait for the end of the main thread finishing rather
than spinning on postcopy state
Rename inward/outward cmd messages for consistency
Fixup old-vm-running check for postcopy

Dave Gibson's comments:
Lots of cleanup

Eric's comments
80col in qapi-schema

Dr. David Alan Gilbert (45):
Start documenting how postcopy works.
Split header writing out of qemu_save_state_begin
qemu_ram_foreach_block: pass up error value, and down the ramblock
name
Add qemu_get_counted_string to read a string prefixed by a count byte Create MigrationIncomingState
Provide runtime Target page information
Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source Return path: Source handling of return path
ram_debug_dump_bitmap: Dump a migration bitmap as text Move loadvm_handlers into MigrationIncomingState
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
migrate_init: Call from savevm
Modify savevm handlers for postcopy
Add Linux userfaultfd header
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
MIG_STATE_POSTCOPY_ACTIVE: Add new migration state qemu_savevm_state_complete: Postcopy changes
Postcopy page-map-incoming (PMI) structure
Postcopy: Maintain sentmap and calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy end in migration_thread
Page request: Add MIG_RP_CMD_REQ_PAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration qemu_ram_block_from_host
Don't sync dirty bitmaps in postcopy
Host page!=target page: Cleanup bitmaps
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_handle_{run,end} commands End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopy

 arch_init.c                       |  908 +++++++++++++++++++++++++++++++--
 balloon.c                         |   11 +
 docs/migration.txt                |  189 +++++++
 exec.c                            |   76 ++-
 hmp-commands.hx                   |   15 +
 hmp.c                             |    7 +
 hmp.h                             |    1 +
 hw/virtio/virtio-balloon.c        |    4 +-
 include/exec/cpu-all.h            |    2 -
 include/exec/cpu-common.h         |    8 +-
 include/migration/migration.h     |  145 +++++-
 include/migration/postcopy-ram.h  |   99 ++++
 include/migration/qemu-file.h     |   10 +
 include/migration/vmstate.h       |   12 +-
 include/qemu/typedefs.h           |    6 +
 include/sysemu/balloon.h          |    2 +
 include/sysemu/sysemu.h           |   46 +-
 linux-headers/linux/userfaultfd.h |  150 ++++++
 migration/Makefile.objs           |    2 +-
 migration/block.c                 |    7 +-
 migration/migration.c             |  739 +++++++++++++++++++++++++--
 migration/postcopy-ram.c          | 1018 +++++++++++++++++++++++++++++++++++++
 migration/qemu-file-internal.h    |    2 +
 migration/qemu-file-unix.c        |   99 +++-
 migration/qemu-file.c             |   28 +
 migration/rdma.c                  |    4 +-
 qapi-schema.json                  |   15 +-
 qmp-commands.hx                   |   19 +
 savevm.c                          |  866 ++++++++++++++++++++++++++++---
 trace-events                      |   80 ++-
 30 files changed, 4381 insertions(+), 189 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 linux-headers/linux/userfaultfd.h
 create mode 100644 migration/postcopy-ram.c
Signature · 1 line
2.1.0
#1 lore raw

PATCH v5 1/45 Start documenting how postcopy works.

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
docs/migration.txt +189 -0
index 0492a45..c6c3798 100644
@@ -294,3 +294,192 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that once in postcopy mode, the sent map is still updated; however,
+its contents are not necessarily consistent with the pages already sent
+due to the masking with the migration bitmap.
+
+=== Destination side page maps ===
+
+(Needs to be changed so we can update both easily - at the moment updates are done
+ with a lock)
+The destination keeps a state for each page which is 'missing', 'received'
+or 'requested'; these three states are encoded in a 2 bit state array.
+Incoming requests from the kernel cause the state to transition from 'missing'
+to 'requested'.   Received pages cause a transition from either 'missing' or
+'requested' to 'received'; the kernel is notified on reception to wake up
+any threads that were waiting for the page.
+If the kernel requests a page that has already been 'received' the kernel is
+notified without re-requesting.
+
+This leads to four valid page states:
+page states:
+    missing        - page not yet received or requested
+    received       - Page received
+    requested      - page requested but not yet received
+
+state transitions:
+      received -> missing   (only during setup/discard)
+      missing -> received   (normal incoming page)
+      requested -> received (incoming page previously requested)
+      missing -> requested  (userfault request)
Signature · 1 line
2.1.0
#2 lore raw

Re: [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works.

Quoted 54 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
docs/migration.txt +39 -0
index 0492a45..c6c3798 100644
@@ -294,3 +294,192 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 345

Quoted 10 lines
docs/migration.txt +10 -0
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.

It's not quite clear to me what this means. Does "migrate_start_postcopy" mean it will immediately transfer execution and transfer any remaining pages postcopy, or does it just mean it will start postcopying once the remaining data to transfer is small enough?

What's the reason for this rather awkward two stage activation of postcopy?

Quoted 15 lines
docs/migration.txt +15 -0
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 371

Quoted 10 lines
docs/migration.txt +10 -0
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.

It seems to me the "ram listen", "ram run" and CMD_PACKAGED really have to be used in conjuction this way, they don't really have any use on their own. So why not make it all CMD_POSTCOPY_TRANSITION and have the "listen" and "run" take effect implicitly at the beginning and end of the device data.

Quoted 115 lines
docs/migration.txt +113 -1
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram
 
- is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that once in postcopy mode, the sent map is still updated; however,
+its contents are not necessarily consistent with the pages already sent
+due to the masking with the migration bitmap.
+
+=== Destination side page maps ===
+
+(Needs to be changed so we can update both easily - at the moment updates are done
+ with a lock)
+The destination keeps a state for each page which is 'missing', 'received'
+or 'requested'; these three states are encoded in a 2 bit state array.
+Incoming requests from the kernel cause the state to transition from 'missing'
+to 'requested'.   Received pages cause a transition from either 'missing' or
+'requested' to 'received'; the kernel is notified on reception to wake up
+any threads that were waiting for the page.
+If the kernel requests a page that has already been 'received' the kernel is
+notified without re-requesting.
+
+This leads to four valid page states:
+page states:
+    missing        - page not yet received or requested
+    received       - Page received
+    requested      - page requested but not yet received
+
+state transitions:
+      received -> missing   (only during setup/discard)
+      missing -> received   (normal incoming page)
+      requested -> received (incoming page previously requested)
+      missing -> requested  (userfault request)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#3 lore raw

Re: [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works.

Quoted 63 lines · * David Gibson ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
docs/migration.txt +47 -0
index 0492a45..c6c3798 100644
@@ -294,3 +294,192 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 345

Quoted 2 lines
docs/migration.txt +2 -0
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.

It's not quite clear to me what this means. Does "migrate_start_postcopy" mean it will immediately transfer execution and transfer any remaining pages postcopy, or does it just mean it will start postcopying once the remaining data to transfer is small enough?

Yes; it will flip into postcopy soon after issuing that command irrespective of the amount of data remaining.

Quoted 2 lines

What's the reason for this rather awkward two stage activation of postcopy?

We need to keep track of the pages that are received during the precopy phase, and do some madvise and other setups on the destination RAM area before precopy starts; and so we need to know we might want to do postcopy - so we need to be told early. In the earliest posted version of my patches I had a time-limit setting and after the time limit expired QEMU would switch into the second phase of postcopy itself, but Paolo suggested the migrate_start_postcopy:

https://lists.nongnu.org/archive/html/qemu-devel/2014-07/msg00943.html

and it works out simpler anyway.

Quoted 23 lines
Quoted 23 lines
docs/migration.txt +23 -0
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 371

Quoted 2 lines
docs/migration.txt +2 -0
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.

It seems to me the "ram listen", "ram run" and CMD_PACKAGED really have to be used in conjuction this way, they don't really have any use on their own. So why not make it all CMD_POSTCOPY_TRANSITION and have the "listen" and "run" take effect implicitly at the beginning and end of the device data.

CMD_PACKAGED seems like something that was generally useful; it's fairly complicated on it's own and so it seemed best to keep it separate.

(Reading your comment here I notice I've still got it as 'postcopy ram listen' when I removed the 'ram' based on previous review comments; I've fixed that locally).

Dave

Quoted 107 lines
Quoted 107 lines
docs/migration.txt +105 -1
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram
 
- is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_state) from
+ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that once in postcopy mode, the sent map is still updated; however,
+its contents are not necessarily consistent with the pages already sent
+due to the masking with the migration bitmap.
+
+=== Destination side page maps ===
+
+(Needs to be changed so we can update both easily - at the moment updates are done
+ with a lock)
+The destination keeps a state for each page which is 'missing', 'received'
+or 'requested'; these three states are encoded in a 2 bit state array.
+Incoming requests from the kernel cause the state to transition from 'missing'
+to 'requested'.   Received pages cause a transition from either 'missing' or
+'requested' to 'received'; the kernel is notified on reception to wake up
+any threads that were waiting for the page.
+If the kernel requests a page that has already been 'received' the kernel is
+notified without re-requesting.
+
+This leads to four valid page states:
+page states:
+    missing        - page not yet received or requested
Quoted 14 lines
Quoted 8 lines
docs/migration.txt +8 -0
+    received       - Page received
+    requested      - page requested but not yet received
+
+state transitions:
+      received -> missing   (only during setup/discard)
+      missing -> received   (normal incoming page)
+      requested -> received (incoming page previously requested)
+      missing -> requested  (userfault request)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw

Re: [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works.

Quoted 3 lines · Dr. David Alan Gilbert
Quoted 1 line · * David Gibson ([email protected]) wrote:

On Wed, Feb 25, 2015 at 04:51:24PM +0000, Dr. David Alan Gilbert

(git) wrote:

[snip]

Quoted 33 lines
Quoted 11 lines
Quoted 11 lines

+=== Enabling postcopy ===

docs/migration.txt +10 -0
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 345

Quoted 2 lines
docs/migration.txt +2 -0
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.

It's not quite clear to me what this means. Does "migrate_start_postcopy" mean it will immediately transfer execution and transfer any remaining pages postcopy, or does it just mean it will start postcopying once the remaining data to transfer is small enough?

Yes; it will flip into postcopy soon after issuing that command irrespective of the amount of data remaining.

Quoted 2 lines

What's the reason for this rather awkward two stage activation of postcopy?

We need to keep track of the pages that are received during the precopy phase, and do some madvise and other setups on the destination RAM area before precopy starts; and so we need to know we might want to do postcopy - so we need to be told early. In the earliest posted version of my patches I had a time-limit setting and after the time limit expired QEMU would switch into the second phase of postcopy itself, but Paolo suggested the migrate_start_postcopy:

Quoted 3 lines

https://lists.nongnu.org/archive/html/qemu-devel/2014-07/msg00943.html

and it works out simpler anyway.

Ok, that makes sense.

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines

+=== Postcopy device transfer ===

docs/migration.txt +22 -0
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 371

Quoted 2 lines
docs/migration.txt +2 -0
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.

It seems to me the "ram listen", "ram run" and CMD_PACKAGED really have to be used in conjuction this way, they don't really have any use on their own. So why not make it all CMD_POSTCOPY_TRANSITION and have the "listen" and "run" take effect implicitly at the beginning and end of the device data.

Quoted 2 lines

CMD_PACKAGED seems like something that was generally useful; it's fairly complicated on it's own and so it seemed best to keep it separate.

And can you actually think of another use case for it?

The thing that bothers me is that the "listen" and "run" operations will not work correctly anywhere other than at the beginning and end of the packaged blob.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#5 lore raw

Re: [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works.

Quoted 76 lines · * David Gibson ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Quoted 1 line · * David Gibson ([email protected]) wrote:

On Wed, Feb 25, 2015 at 04:51:24PM +0000, Dr. David Alan Gilbert

(git) wrote:

[snip]

Quoted 33 lines
Quoted 11 lines
Quoted 11 lines

+=== Enabling postcopy ===

docs/migration.txt +10 -0
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
Quoted 8 lines

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 345

Quoted 2 lines
docs/migration.txt +2 -0
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.

It's not quite clear to me what this means. Does "migrate_start_postcopy" mean it will immediately transfer execution and transfer any remaining pages postcopy, or does it just mean it will start postcopying once the remaining data to transfer is small enough?

Yes; it will flip into postcopy soon after issuing that command irrespective of the amount of data remaining.

Quoted 2 lines

What's the reason for this rather awkward two stage activation of postcopy?

We need to keep track of the pages that are received during the precopy phase, and do some madvise and other setups on the destination RAM area before precopy starts; and so we need to know we might want to do postcopy - so we need to be told early. In the earliest posted version of my patches I had a time-limit setting and after the time limit expired QEMU would switch into the second phase of postcopy itself, but Paolo suggested the migrate_start_postcopy:

Quoted 3 lines

https://lists.nongnu.org/archive/html/qemu-devel/2014-07/msg00943.html

and it works out simpler anyway.

Ok, that makes sense.

Quoted 32 lines
Quoted 31 lines
Quoted 15 lines

+=== Postcopy device transfer ===

docs/migration.txt +14 -0
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then

docs/migration.txt save/send this state when we are in the middle of a pio operation · line 371

Quoted 10 lines
docs/migration.txt +10 -0
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.

It seems to me the "ram listen", "ram run" and CMD_PACKAGED really have to be used in conjuction this way, they don't really have any use on their own. So why not make it all CMD_POSTCOPY_TRANSITION and have the "listen" and "run" take effect implicitly at the beginning and end of the device data.

Quoted 8 lines
Quoted 2 lines

CMD_PACKAGED seems like something that was generally useful; it's fairly complicated on it's own and so it seemed best to keep it separate.

And can you actually think of another use case for it?

The thing that bothers me is that the "listen" and "run" operations will not work correctly anywhere other than at the beginning and end of the packaged blob.

It feels similar to the packaged blobs that checkpointing schemes like COLO and microcheckpointing use; although they seem to craft their own wire protocol rather than sticking with a migration protocol.
What they do in terms of controlling the CPU etc is certainly different (so the RUN/listen stuff is different).

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#6 lore raw

PATCH v5 2/45 Split header writing out of qemu_save_state_begin

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Split qemu_save_state_begin to:
qemu_save_state_header That writes the initial file header.
qemu_save_state_begin That sets up devices and does the first
device pass.

Used later in postcopy.

---
 include/sysemu/sysemu.h |  1 +
 migration/migration.c   |  1 +
 savevm.c                | 11 ++++++++---
 trace-events            |  1 +
 4 files changed, 11 insertions(+), 3 deletions(-)
include/sysemu/sysemu.h +1 -0
index 748d059..de1c885 100644
@@ -84,6 +84,7 @@ void qemu_announce_self(void);
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
+void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
migration/migration.c +1 -0
index b3adbc6..4a06d79 100644
@@ -597,6 +597,7 @@ static void *migration_thread(void *opaque)
     int64_t start_time = initial_time;
     bool old_vm_running = false;
 
+    qemu_savevm_state_header(s->file);
     qemu_savevm_state_begin(s->file, &s->params);
 
     s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
savevm.c +8 -3
index 8040766..192110a 100644
@@ -616,6 +616,13 @@ bool qemu_savevm_state_blocked(Error **errp)
     return false;
 }
 
+void qemu_savevm_state_header(QEMUFile *f)
+{
+    trace_savevm_state_header();
+    qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
+    qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+}
+
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params)
 {
@@ -630,9 +637,6 @@ void qemu_savevm_state_begin(QEMUFile *f,
         se->ops->set_params(params, se->opaque);
     }
 
-    qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
-    qemu_put_be32(f, QEMU_VM_FILE_VERSION);
-
     QTAILQ_FOREACH(se, &savevm_handlers, entry) {
         int len;
 
@@ -834,6 +838,7 @@ static int qemu_savevm_state(QEMUFile *f)
     }
 
     qemu_mutex_unlock_iothread();
+    qemu_savevm_state_header(f);
     qemu_savevm_state_begin(f, &params);
     qemu_mutex_lock_iothread();
 
trace-events +1 -0
index f87b077..83231d7 100644
@@ -1170,6 +1170,7 @@ qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_state_begin(void) ""
+savevm_state_header(void) ""
 savevm_state_iterate(void) ""
 savevm_state_complete(void) ""
 savevm_state_cancel(void) ""
Signature · 1 line
2.1.0
Reviewed-by on v5 2/45
#7 lore raw

Re: [Qemu-devel] [PATCH v5 02/45] Split header writing out of qemu_save_state_begin

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Split qemu_save_state_begin to:
qemu_save_state_header That writes the initial file header.
qemu_save_state_begin That sets up devices and does the first
device pass.

Used later in postcopy.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#8 lore raw

PATCH v5 3/45 qemu_ram_foreach_block: pass up error value, and down the ramblock name

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.

Pass the name of the ramblock to the function; helps in debugging.

---
 exec.c                    | 10 ++++++++--
 include/exec/cpu-common.h |  4 ++--
 migration/rdma.c          |  4 ++--
 3 files changed, 12 insertions(+), 6 deletions(-)
exec.c +8 -2
index 6dff7bc..018b07a 100644
@@ -2944,12 +2944,18 @@ bool cpu_physical_memory_is_io(hwaddr phys_addr)
              memory_region_is_romd(mr));
 }
 
-void qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
+int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
 {
     RAMBlock *block;
+    int ret;
 
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
-        func(block->host, block->offset, block->used_length, opaque);
+        ret = func(block->idstr, block->host, block->offset,
+                   block->used_length, opaque);
+        if (ret) {
+            return ret;
+        }
     }
+    return 0;
 }
 #endif
include/exec/cpu-common.h +2 -2
index 427b851..a31300c 100644
@@ -124,10 +124,10 @@ void cpu_flush_icache_range(hwaddr start, int len);
 extern struct MemoryRegion io_mem_rom;
 extern struct MemoryRegion io_mem_notdirty;
 
-typedef void (RAMBlockIterFunc)(void *host_addr,
+typedef int (RAMBlockIterFunc)(const char *block_name, void *host_addr,
     ram_addr_t offset, ram_addr_t length, void *opaque);
 
-void qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
+int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
 
 #endif
 
migration/rdma.c +2 -2
index 6bee30c..d5cb6b7 100644
@@ -569,10 +569,10 @@ static int __qemu_rdma_add_block(RDMAContext *rdma, void *host_addr,
  * in advanced before the migration starts. This tells us where the RAM blocks
  * are so that we can register them individually.
  */
-static void qemu_rdma_init_one_block(void *host_addr,
+static int qemu_rdma_init_one_block(const char *block_name, void *host_addr,
     ram_addr_t block_offset, ram_addr_t length, void *opaque)
 {
-    __qemu_rdma_add_block(opaque, host_addr, block_offset, length);
+    return __qemu_rdma_add_block(opaque, host_addr, block_offset, length);
 }
 
 /*
Signature · 1 line
2.1.0
#9 lore raw

Re: [Qemu-devel] [PATCH v5 03/45] qemu_ram_foreach_block: pass up error value, and down the ramblock name

Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.

Should we also be changing the name of __qemu_rdma_add_block to get rid of the wrong use of the reserved leading __ namespace?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#10 lore raw

Re: [Qemu-devel] [PATCH v5 03/45] qemu_ram_foreach_block: pass up error value, and down the ramblock name

Quoted 9 lines · * Eric Blake ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.

Should we also be changing the name of __qemu_rdma_add_block to get rid of the wrong use of the reserved leading __ namespace?

I already did that in ba795761857082a9bb8c4be1f28e7cdd82f039b0 that got merged after I cut this patch.

Dave

Quoted 4 lines
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#11 lore raw

PATCH v5 4/45 Add qemu_get_counted_string to read a string prefixed by a count byte

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

and use it in loadvm_state and ram_load.

---
 arch_init.c                   |  5 +----
 include/migration/qemu-file.h |  3 +++
 migration/qemu-file.c         | 16 ++++++++++++++++
 savevm.c                      | 11 ++++++-----
 4 files changed, 26 insertions(+), 9 deletions(-)
arch_init.c +1 -4
index 89c8fa4..91645cc 100644
@@ -1077,13 +1077,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
             total_ram_bytes = addr;
             while (!ret && total_ram_bytes) {
                 RAMBlock *block;
-                uint8_t len;
                 char id[256];
                 ram_addr_t length;
 
-                len = qemu_get_byte(f);
-                qemu_get_buffer(f, (uint8_t *)id, len);
-                id[len] = 0;
+                qemu_get_counted_string(f, id);
                 length = qemu_get_be64(f);
 
                 QTAILQ_FOREACH(block, &ram_list.blocks, next) {
include/migration/qemu-file.h +3 -0
index a923cec..6ae0b03 100644
@@ -310,4 +310,7 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 {
     qemu_get_be64s(f, (uint64_t *)pv);
 }
+
+int qemu_get_counted_string(QEMUFile *f, char buf[256]);
+
 #endif
migration/qemu-file.c +16 -0
index e66e557..57eb868 100644
@@ -545,3 +545,19 @@ uint64_t qemu_get_be64(QEMUFile *f)
     v |= qemu_get_be32(f);
     return v;
 }
+
+/*
+ * Get a string whose length is determined by a single preceding byte
+ * A preallocated 256 byte buffer must be passed in.
+ * Returns: 0 on success and a 0 terminated string in the buffer
+ */
+int qemu_get_counted_string(QEMUFile *f, char buf[256])
+{
+    unsigned int len = qemu_get_byte(f);
+    int res = qemu_get_buffer(f, (uint8_t *)buf, len);
+
+    buf[len] = 0;
+
+    return res != len;
+}
+
savevm.c +6 -5
index 192110a..2f8ef45 100644
@@ -960,8 +960,7 @@ int qemu_loadvm_state(QEMUFile *f)
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
-        char idstr[257];
-        int len;
+        char idstr[256];
 
         trace_qemu_loadvm_state_section(section_type);
         switch (section_type) {
@@ -969,9 +968,11 @@ int qemu_loadvm_state(QEMUFile *f)
         case QEMU_VM_SECTION_FULL:
             /* Read section start */
             section_id = qemu_get_be32(f);
-            len = qemu_get_byte(f);
-            qemu_get_buffer(f, (uint8_t *)idstr, len);
-            idstr[len] = 0;
+            if (qemu_get_counted_string(f, idstr)) {
+                error_report("Unable to read ID string for section %u",
+                            section_id);
+                return -EINVAL;
+            }
             instance_id = qemu_get_be32(f);
             version_id = qemu_get_be32(f);
 
Signature · 1 line
2.1.0
Reviewed-by on v5 4/45
#12 lore raw

Re: [Qemu-devel] [PATCH v5 04/45] Add qemu_get_counted_string to read a string prefixed by a count byte

Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

and use it in loadvm_state and ram_load.

Pure cleanup, no change to migration stream.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#13 lore raw

PATCH v5 5/45 Create MigrationIncomingState

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.

allocate MIS in process_incoming_migration_co

---
 include/migration/migration.h |  9 +++++++++
 include/qemu/typedefs.h       |  1 +
 migration/migration.c         | 28 ++++++++++++++++++++++++++++
 savevm.c                      |  2 ++
 4 files changed, 40 insertions(+)
include/migration/migration.h +9 -0
index f37348b..8505543 100644
@@ -42,6 +42,15 @@ struct MigrationParams {
 
 typedef struct MigrationState MigrationState;
 
+/* State for the incoming migration */
+struct MigrationIncomingState {
+    QEMUFile *file;
+};
+
+MigrationIncomingState *migration_incoming_get_current(void);
+MigrationIncomingState *migration_incoming_state_new(QEMUFile *f);
+void migration_incoming_state_destroy(void);
+
 struct MigrationState
 {
     int64_t bandwidth_limit;
include/qemu/typedefs.h +1 -0
index cde3314..74dfad3 100644
@@ -38,6 +38,7 @@ typedef struct MemoryListener MemoryListener;
 typedef struct MemoryMappingList MemoryMappingList;
 typedef struct MemoryRegion MemoryRegion;
 typedef struct MemoryRegionSection MemoryRegionSection;
+typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct MigrationParams MigrationParams;
 typedef struct Monitor Monitor;
 typedef struct MouseTransformInfo MouseTransformInfo;
migration/migration.c +28 -0
index 4a06d79..a36ea65 100644
@@ -53,6 +53,7 @@ static NotifierList migration_state_notifiers =
    migrations at once.  For now we don't need to add
    dynamic creation of migration */
 
+/* For outgoing */
 MigrationState *migrate_get_current(void)
 {
     static MigrationState current_migration = {
@@ -65,6 +66,28 @@ MigrationState *migrate_get_current(void)
     return &current_migration;
 }
 
+/* For incoming */
+static MigrationIncomingState *mis_current;
+
+MigrationIncomingState *migration_incoming_get_current(void)
+{
+    return mis_current;
+}
+
+MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
+{
+    mis_current = g_malloc0(sizeof(MigrationIncomingState));
+    mis_current->file = f;
+
+    return mis_current;
+}
+
+void migration_incoming_state_destroy(void)
+{
+    g_free(mis_current);
+    mis_current = NULL;
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
@@ -94,9 +117,14 @@ static void process_incoming_migration_co(void *opaque)
     Error *local_err = NULL;
     int ret;
 
+    migration_incoming_state_new(f);
+
     ret = qemu_loadvm_state(f);
+
     qemu_fclose(f);
     free_xbzrle_decoded_buf();
+    migration_incoming_state_destroy();
+
     if (ret < 0) {
         error_report("load of migration failed: %s", strerror(-ret));
         exit(EXIT_FAILURE);
savevm.c +2 -0
index 2f8ef45..cce7ff0 100644
@@ -1290,9 +1290,11 @@ int load_vmstate(const char *name)
     }
 
     qemu_system_reset(VMRESET_SILENT);
+    migration_incoming_state_new(f);
     ret = qemu_loadvm_state(f);
 
     qemu_fclose(f);
+    migration_incoming_state_destroy();
     if (ret < 0) {
         error_report("Error %d while loading VM state", ret);
         return ret;
Signature · 1 line
2.1.0
Reviewed-by on v5 5/45
#14 lore raw

Re: [Qemu-devel] [PATCH v5 05/45] Create MigrationIncomingState

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.

allocate MIS in process_incoming_migration_co

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#15 lore raw

PATCH v5 6/45 Provide runtime Target page information

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init.

Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.

---
 exec.c                  | 10 ++++++++++
 include/sysemu/sysemu.h |  1 +
 2 files changed, 11 insertions(+)
exec.c +10 -0
index 018b07a..eafd964 100644
@@ -2915,6 +2915,16 @@ int cpu_memory_rw_debug(CPUState *cpu, target_ulong addr,
     }
     return 0;
 }
+
+/*
+ * Allows code that needs to deal with migration bitmaps etc to still be built
+ * target independent.
+ */
+size_t qemu_target_page_bits(void)
+{
+    return TARGET_PAGE_BITS;
+}
+
 #endif
 
 /*
include/sysemu/sysemu.h +1 -0
index de1c885..ebab098 100644
@@ -68,6 +68,7 @@ int qemu_reset_requested_get(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_devices_reset(void);
 void qemu_system_reset(bool report);
+size_t qemu_target_page_bits(void);
 
 void qemu_add_exit_notifier(Notifier *notify);
 void qemu_remove_exit_notifier(Notifier *notify);
Signature · 1 line
2.1.0
Reviewed-by on v5 6/45
#16 lore raw

Re: [Qemu-devel] [PATCH v5 06/45] Provide runtime Target page information

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init.

Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#17 lore raw

PATCH v5 7/45 Return path: Open a return path on QEMUFile for sockets

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

---
 include/migration/qemu-file.h  |  7 +++++
 migration/qemu-file-internal.h |  2 ++
 migration/qemu-file-unix.c     | 58 +++++++++++++++++++++++++++++++++++-------
 migration/qemu-file.c          | 12 +++++++++
 4 files changed, 70 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
index 6ae0b03..3c38963 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                int *bytes_sent);
 
 /*
+ * Return a QEMUFile for comms in the opposite direction
+ */
+typedef QEMUFile *(QEMURetPathFunc)(void *opaque);
+
+/*
  * Stop any read or write (depending on flags) on the underlying
  * transport on the QEMUFile.
  * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
@@ -188,6 +194,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
+QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
migration/qemu-file-internal.h +2 -0
index d95e853..a39b8e3 100644
@@ -48,6 +48,8 @@ struct QEMUFile {
     unsigned int iovcnt;
 
     int last_error;
+
+    struct QEMUFile *return_path;
 };
 
 #endif
migration/qemu-file-unix.c +49 -9

73 lines not on this page · raw message

migration/qemu-file.c +12 -0
index 57eb868..02122a5 100644
@@ -42,6 +42,18 @@ int qemu_file_shutdown(QEMUFile *f)
     return f->ops->shut_down(f->opaque, true, true);
 }
 
+/*
+ * Result: QEMUFile* for a 'return path' for comms in the opposite direction
+ *         NULL if not available
+ */
+QEMUFile *qemu_file_get_return_path(QEMUFile *f)
+{
+    if (!f->ops->get_return_path) {
+        return NULL;
+    }
+    return f->ops->get_return_path(f->opaque);
+}
+
 bool qemu_file_mode_is_not_valid(const char *mode)
 {
     if (mode == NULL ||
Signature · 1 line
2.1.0
#18 lore raw

Re: [Qemu-devel] [PATCH v5 07/45] Return path: Open a return path on QEMUFile for sockets

Quoted 52 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

---
 include/migration/qemu-file.h  |  7 +++++
 migration/qemu-file-internal.h |  2 ++
 migration/qemu-file-unix.c     | 58 +++++++++++++++++++++++++++++++++++-------
 migration/qemu-file.c          | 12 +++++++++
 4 files changed, 70 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
index 6ae0b03..3c38963 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                int *bytes_sent);
 
 /*
+ * Return a QEMUFile for comms in the opposite direction
+ */
+typedef QEMUFile *(QEMURetPathFunc)(void *opaque);
+
+/*
  * Stop any read or write (depending on flags) on the underlying
  * transport on the QEMUFile.
  * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
@@ -188,6 +194,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
+QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
migration/qemu-file-internal.h +0 -0
index d95e853..a39b8e3 100644

migration/qemu-file-internal.h struct QEMUFile { · line 52

Quoted 5 lines
migration/qemu-file-internal.h +2 -0
@@ -48,6 +48,8 @@ struct QEMUFile {
     unsigned int iovcnt;
 
     int last_error;
+
+    struct QEMUFile *return_path;

AFAICT, the only thing this field is used for is an assert, which seems a bit pointless. I'd suggest either getting rid of it, or make qemu_file_get_return_path() safely idempotent by having it only call the FileOps pointer if QEMUFile::return_path is non-NULL, otherwise just return the existing return_path.

Setting the field probably belongs better in the wrapper than in the socket specific callback, too, since there's nothing inherently related to the socket implementation about it.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#19 lore raw

Re: [Qemu-devel] [PATCH v5 07/45] Return path: Open a return path on QEMUFile for sockets

Quoted 55 lines · * David Gibson ([email protected]) wrote:
Quoted 54 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

---
 include/migration/qemu-file.h  |  7 +++++
 migration/qemu-file-internal.h |  2 ++
 migration/qemu-file-unix.c     | 58 +++++++++++++++++++++++++++++++++++-------
 migration/qemu-file.c          | 12 +++++++++
 4 files changed, 70 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
index 6ae0b03..3c38963 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                int *bytes_sent);
 
 /*
+ * Return a QEMUFile for comms in the opposite direction
+ */
+typedef QEMUFile *(QEMURetPathFunc)(void *opaque);
+
+/*
  * Stop any read or write (depending on flags) on the underlying
  * transport on the QEMUFile.
  * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
@@ -188,6 +194,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 int qemu_file_shutdown(QEMUFile *f);
+QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
migration/qemu-file-internal.h +0 -0
index d95e853..a39b8e3 100644
@@ -48,6 +48,8 @@ struct QEMUFile {
     unsigned int iovcnt;
 
Quoted 6 lines

migration/qemu-file-internal.h struct QEMUFile { · line 52

Quoted 3 lines
migration/qemu-file-internal.h +2 -0
     int last_error;
+
+    struct QEMUFile *return_path;

AFAICT, the only thing this field is used for is an assert, which seems a bit pointless. I'd suggest either getting rid of it, or

Done; it's gone.

Dave

Quoted 4 lines

make qemu_file_get_return_path() safely idempotent by having it only call the FileOps pointer if QEMUFile::return_path is non-NULL, otherwise just return the existing return_path.

Quoted 9 lines

Setting the field probably belongs better in the wrapper than in the socket specific callback, too, since there's nothing inherently related to the socket implementation about it.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#20 lore raw

PATCH v5 8/45 Return path: socket_writev_buffer: Block even on non-blocking fd's

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

---
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +36 -5
index 50291cf..218dbd0 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
Signature · 1 line
2.1.0
#21 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

It's not clear to me from this description exactly where the situation is that you need to write to the non-blocking socket. Is it on the source or the destination? If the source, why are you writing to the return path? If the destination, why are you marking the outgoing return path as non-blocking?

Quoted 58 lines
---
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +36 -5
index 50291cf..218dbd0 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#22 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 7 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

Quoted 5 lines

It's not clear to me from this description exactly where the situation is that you need to write to the non-blocking socket. Is it on the source or the destination? If the source, why are you writing to the return path? If the destination, why are you marking the outgoing return path as non-blocking?

My understanding is that the semantics of set_nonblock() are to set non-blocking on all operations on the transport associated with the fd - and that it's true even if you dup() the fd; and so if you set non-blocking in one direction you get it in the other direction as well.

The (existing) destination side sets non-block (see process_incoming_migration in migration.c), and so it gets non-blocking on the incoming data stream, but that has the side effect that it's also going to be non-blocking on the destinations writes to the reverse-path; thus we need to be able to safely do writes from the destination reverse-path.

The text is out of date, back in v2 the source used to use non-blocking for the return-path, but we managed to kill that off by using a thread for the return path in the source.

How about changing the text to:
--------
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

Dave

Quoted 58 lines
Quoted 58 lines
---
 migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +36 -5
index 50291cf..218dbd0 100644
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
-    }
-    return len;
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
+     }
+
+    return offset;
 }
 
Quoted 7 lines
Quoted 1 line
migration/qemu-file-unix.c +0 -0
 static int socket_get_fd(void *opaque)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#23 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 14 lines · Dr. David Alan Gilbert
Quoted 7 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

Quoted 5 lines

It's not clear to me from this description exactly where the situation is that you need to write to the non-blocking socket. Is it on the source or the destination? If the source, why are you writing to the return path? If the destination, why are you marking the outgoing return path as non-blocking?

Quoted 4 lines

My understanding is that the semantics of set_nonblock() are to set non-blocking on all operations on the transport associated with the fd - and that it's true even if you dup() the fd; and so if you set non-blocking in one direction you get it in the other direction as well.

Ah.. yes, I think you're right.

Quoted 10 lines

The (existing) destination side sets non-block (see process_incoming_migration in migration.c), and so it gets non-blocking on the incoming data stream, but that has the side effect that it's also going to be non-blocking on the destinations writes to the reverse-path; thus we need to be able to safely do writes from the destination reverse-path.

The text is out of date, back in v2 the source used to use non-blocking for the return-path, but we managed to kill that off by using a thread for the return path in the source.

Quoted 5 lines

How about changing the text to:
--------
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.

Yes, I think that makes it clearer.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#25 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 6 lines · Paolo Bonzini
Quoted 1 line · Dr. David Alan Gilbert (git)

+ if (err != EAGAIN) {

if (err != EAGAIN && err != EWOULDBLOCK)

I assume that's for the benefit of non-Linux hosts? On Linux EAGAIN == EWOULDBLOCK.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#26 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 10 lines · David Gibson
Quoted 6 lines · Paolo Bonzini
Quoted 1 line · Dr. David Alan Gilbert (git)

+ if (err != EAGAIN) {

if (err != EAGAIN && err != EWOULDBLOCK)

I assume that's for the benefit of non-Linux hosts? On Linux EAGAIN == EWOULDBLOCK.

Yes, that's just the standard idiom in QEMU. This is generic code, so assumption based on the host platform are not wise. :)

Paolo

#27 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 9 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 6 lines · David Gibson
Quoted 5 lines · Paolo Bonzini
Quoted 1 line · Dr. David Alan Gilbert (git)

+ if (err != EAGAIN) {

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

if (err != EAGAIN && err != EWOULDBLOCK)

I assume that's for the benefit of non-Linux hosts? On Linux EAGAIN == EWOULDBLOCK.

Yes, that's just the standard idiom in QEMU. This is generic code, so assumption based on the host platform are not wise. :)

Done; I didn't know of EWOULDBLOCK - and indeed as far as I can tell most places we only test for EAGAIN.

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#28 lore raw

Re: [Qemu-devel] [PATCH v5 08/45] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 12 lines · Dr. David Alan Gilbert
Quoted 11 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 8 lines · David Gibson
Quoted 6 lines · Paolo Bonzini
Quoted 1 line · Dr. David Alan Gilbert (git)

+ if (err != EAGAIN) {

if (err != EAGAIN && err != EWOULDBLOCK)

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I assume that's for the benefit of non-Linux hosts? On Linux EAGAIN == EWOULDBLOCK.

Yes, that's just the standard idiom in QEMU. This is generic code, so assumption based on the host platform are not wise. :)

Done; I didn't know of EWOULDBLOCK - and indeed as far as I can tell most places we only test for EAGAIN.

Bug unless the place in question is effectively #ifdef __GNU_LIBRARY__ or similar.

https://www.gnu.org/software/libc/manual/html_node/Error-Codes.html#Error-Codes

-- Macro: int EAGAIN
Resource temporarily unavailable; the call might work if you try again later. The macro 'EWOULDBLOCK' is another name for 'EAGAIN'; they are always the same in the GNU C Library.

This error can happen in a few different situations:

* An operation that would block was attempted on an object that has non-blocking mode selected. Trying the same operation again will block until some external condition makes it possible to read, write, or connect (whatever the operation). You can use 'select' to find out when the operation will be possible; *note Waiting for I/O::.

*Portability Note:* In many older Unix systems, this condition was indicated by 'EWOULDBLOCK', which was a distinct error code different from 'EAGAIN'. To make your program portable, you should check for both codes and treat them the same.

* A temporary resource shortage made an operation impossible. 'fork' can return this error. It indicates that the shortage is expected to pass, so your program can try the call again later and it may succeed. It is probably a good idea to delay for a few seconds before trying it again, to allow time for other processes to release scarce resources. Such shortages are usually fairly serious and affect the whole system, so usually an interactive program should report the error to the user and return to its command loop.

-- Macro: int EWOULDBLOCK
In the GNU C Library, this is another name for 'EAGAIN' (above). The values are always the same, on every operating system.

C libraries in many older Unix systems have 'EWOULDBLOCK' as a separate error code.

#29 lore raw

PATCH v5 9/45 Migration commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

---
 include/migration/migration.h |  1 +
 include/sysemu/sysemu.h       |  7 +++++++
 savevm.c                      | 48 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  1 +
 4 files changed, 57 insertions(+)
include/migration/migration.h +1 -0
index 8505543..1b1dc34 100644
@@ -34,6 +34,7 @@
 #define QEMU_VM_SECTION_FULL         0x04
 #define QEMU_VM_SUBSECTION           0x05
 #define QEMU_VM_VMDESCRIPTION        0x06
+#define QEMU_VM_COMMAND              0x07
 
 struct MigrationParams {
     bool blk;
include/sysemu/sysemu.h +7 -0
index ebab098..88e5e76 100644
@@ -82,6 +82,11 @@ void do_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+/* Subcommands for QEMU_VM_COMMAND */
+enum qemu_vm_cmd {
+    MIG_CMD_INVALID = 0,   /* Must be 0 */
+};
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
+                              uint16_t len, uint8_t *data);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +48 -0

66 lines not on this page · raw message

trace-events +1 -0
index 83231d7..4e2fbc8 100644
@@ -1167,6 +1167,7 @@ vmware_setmode(uint32_t w, uint32_t h, uint32_t bpp) "%dx%d @ %d bpp"
 qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_state_begin(void) ""
Signature · 1 line
2.1.0
#30 lore raw

Re: [Qemu-devel] [PATCH v5 09/45] Migration commands

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

[snip]

savevm.c static void vmstate_save(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc) · line 614

Quoted 9 lines
savevm.c +9 -0
+/* Send a 'QEMU_VM_COMMAND' type element with the command
+ * and associated data.
+ */
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    uint32_t tmp = (uint16_t)command;

Erm.. cast to u16, assign to u32, then send as u16? What's up with that?

Quoted 9 lines
savevm.c +9 -0
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, tmp);
+    qemu_put_be16(f, len);
+    if (len) {
+        qemu_put_buffer(f, data, len);
+    }
+    qemu_fflush(f);
+}
+
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#31 lore raw

Re: [Qemu-devel] [PATCH v5 09/45] Migration commands

Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

[snip]

Quoted 12 lines

savevm.c static void vmstate_save(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc) · line 614

Quoted 9 lines
savevm.c +9 -0
+/* Send a 'QEMU_VM_COMMAND' type element with the command
+ * and associated data.
+ */
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    uint32_t tmp = (uint16_t)command;

Erm.. cast to u16, assign to u32, then send as u16? What's up with that?

Hmm yes, that is insane; now changed to:

+void qemu_savevm_command_send(QEMUFile *f,

+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, (uint16_t)command);
+    qemu_put_be16(f, len);
+    if (len) {

+ qemu_put_buffer(f, data, len);
+ }
+ qemu_fflush(f);
+}

Thanks,

Dave

Quoted 16 lines
Quoted 10 lines
savevm.c +9 -0
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, tmp);
+    qemu_put_be16(f, len);
+    if (len) {
+        qemu_put_buffer(f, data, len);
+    }
+    qemu_fflush(f);
+}
+
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#32 lore raw

Re: [Qemu-devel] [PATCH v5 09/45] Migration commands

Quoted 31 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

[snip]

Quoted 12 lines

savevm.c static void vmstate_save(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc) · line 614

Quoted 9 lines
savevm.c +9 -0
+/* Send a 'QEMU_VM_COMMAND' type element with the command
+ * and associated data.
+ */
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,
+                              uint8_t *data)
+{
+    uint32_t tmp = (uint16_t)command;

Erm.. cast to u16, assign to u32, then send as u16? What's up with that?

Hmm yes, that is insane; now changed to:

savevm.c +3 -0
+void qemu_savevm_command_send(QEMUFile *f,
+                              enum qemu_vm_cmd command,
+                              uint16_t len,

savevm.c static void vmstate_save(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc) · line 622

Quoted 10 lines
savevm.c +10 -0
+                              uint8_t *data)
+{
+    qemu_put_byte(f, QEMU_VM_COMMAND);
+    qemu_put_be16(f, (uint16_t)command);
+    qemu_put_be16(f, len);
+    if (len) {
+        qemu_put_buffer(f, data, len);
+    }
+    qemu_fflush(f);
+}

That looks better.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#33 lore raw

PATCH v5 10/45 Return path: Control commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* SEND_PING - Request an acknowledge from the destination

---
 include/migration/migration.h |  2 ++
 include/sysemu/sysemu.h       |  6 ++++-
 savevm.c                      | 59 +++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  2 ++
 4 files changed, 68 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index 1b1dc34..c514dd4 100644
@@ -46,6 +46,8 @@ typedef struct MigrationState MigrationState;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
+
+    QEMUFile *return_path;
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
include/sysemu/sysemu.h +5 -1
index 88e5e76..8da879f 100644
@@ -84,7 +84,9 @@ void qemu_announce_self(void);
 
 /* Subcommands for QEMU_VM_COMMAND */
 enum qemu_vm_cmd {
-    MIG_CMD_INVALID = 0,   /* Must be 0 */
+    MIG_CMD_INVALID = 0,       /* Must be 0 */
+    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
+    MIG_CMD_PING,              /* Request a PONG on the RP */
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -97,6 +99,8 @@ void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
+void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
+void qemu_savevm_send_open_return_path(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +59 -0

85 lines not on this page · raw message

trace-events +2 -0
index 4e2fbc8..99e00b5 100644
@@ -1168,8 +1168,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
+loadvm_process_command_ping(uint32_t val) "%x"
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
+savevm_send_ping(uint32_t val) "%x"
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
Signature · 1 line
2.1.0
Reviewed-by on v5 10/45
#34 lore raw

Re: [Qemu-devel] [PATCH v5 10/45] Return path: Control commands

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* SEND_PING - Request an acknowledge from the destination

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#35 lore raw

Re: [Qemu-devel] [PATCH v5 10/45] Return path: Control commands

In reply to Dr. David Alan Gilbert (git)
Quoted 4 lines · Dr. David Alan Gilbert (git)

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* SEND_PING - Request an acknowledge from the destination

It's just PING, not SEND_PING.

Paolo

#36 lore raw

Re: [Qemu-devel] [PATCH v5 10/45] Return path: Control commands

Quoted 9 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 4 lines · Dr. David Alan Gilbert (git)

Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* SEND_PING - Request an acknowledge from the destination

It's just PING, not SEND_PING.

Fixed, thanks.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#37 lore raw

PATCH v5 11/45 Return path: Send responses from destination to source

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the CMD_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
index c514dd4..6775747 100644
@@ -41,6 +41,13 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {
+    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
+    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_CMD_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -48,6 +55,7 @@ struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
@@ -169,6 +177,15 @@ int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +45 -0

57 lines not on this page · raw message

savevm.c +1 -1
index d082738..7084d07 100644
@@ -1008,7 +1008,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 99e00b5..4f3eff8 100644
@@ -1379,6 +1379,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 1 line
2.1.0
#38 lore raw

Re: [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

Quoted 22 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the CMD_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index c514dd4..6775747 100644

include/migration/migration.h struct MigrationParams { · line 45

Quoted 5 lines
include/migration/migration.h +2 -0
@@ -41,6 +41,13 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {

"command" doesn't seem like quite the right description for these rp messages.

Quoted 45 lines
include/migration/migration.h +15 -0
+    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
+    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_CMD_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -48,6 +55,7 @@ struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
@@ -169,6 +177,15 @@ int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +1 -0
index a36ea65..80d234c 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }

migration/migration.c void migration_incoming_state_destroy(void) · line 98

Quoted 10 lines
migration/migration.c +7 -0
@@ -88,6 +89,50 @@ void migration_incoming_state_destroy(void)
     mis_current = NULL;
 }
 
+/*
+ * Send a message on the return channel back to the source
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data)

Using (void *) for data would avoid casts in a bunch of the callers.

Quoted 12 lines
migration/migration.c +12 -0
+{
+    trace_migrate_send_rp_message((int)cmd, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)cmd);
+    qemu_put_be16(mis->return_path, len);
+    qemu_put_buffer(mis->return_path, data, len);
+    qemu_fflush(mis->return_path);
+    qemu_mutex_unlock(&mis->rp_mutex);
+}
+
+/*
+ * Send a 'SHUT' message on the return channel with the given value

migration/migration.c void migration_incoming_state_destroy(void) · line 120

Quoted 10 lines
migration/migration.c +10 -0
+ * to indicate that we've finished with the RP.  None-0 value indicates
+ * error.
+ */
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_SHUT, 4, (uint8_t *)&buf);

^ sizeof(buf) would be safer

migration/migration.c void migration_incoming_state_destroy(void) · line 133

Quoted 13 lines
migration/migration.c +13 -0
+}
+
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_PONG, 4, (uint8_t *)&buf);

It occurs to me that you could define PONG as returning the whole buffer that PING sends, instead of just 4-bytes. Might allow for some more testing of variable sized messages.

Quoted 30 lines
migration/migration.c +2 -0
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
savevm.c +1 -1
index d082738..7084d07 100644
@@ -1008,7 +1008,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 99e00b5..4f3eff8 100644
@@ -1379,6 +1379,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#39 lore raw

Re: [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

Quoted 26 lines · * David Gibson ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the CMD_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index c514dd4..6775747 100644
@@ -41,6 +41,13 @@ struct MigrationParams {
     bool shared;
 };
 
Quoted 5 lines

include/migration/migration.h struct MigrationParams { · line 45

Quoted 2 lines
include/migration/migration.h +2 -0
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {

"command" doesn't seem like quite the right description for these rp messages.

Would you prefer 'message' ?

Quoted 48 lines
Quoted 48 lines
include/migration/migration.h +15 -0
+    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
+    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_CMD_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -48,6 +55,7 @@ struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
@@ -169,6 +177,15 @@ int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +1 -0
index a36ea65..80d234c 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -88,6 +89,50 @@ void migration_incoming_state_destroy(void)
     mis_current = NULL;
 }
 
Quoted 9 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 98

Quoted 7 lines
migration/migration.c +7 -0
+/*
+ * Send a message on the return channel back to the source
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data)

Using (void *) for data would avoid casts in a bunch of the callers.

Fixed; thanks.

Quoted 17 lines
Quoted 17 lines
migration/migration.c +17 -0
+{
+    trace_migrate_send_rp_message((int)cmd, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)cmd);
+    qemu_put_be16(mis->return_path, len);
+    qemu_put_buffer(mis->return_path, data, len);
+    qemu_fflush(mis->return_path);
+    qemu_mutex_unlock(&mis->rp_mutex);
+}
+
+/*
+ * Send a 'SHUT' message on the return channel with the given value
+ * to indicate that we've finished with the RP.  None-0 value indicates
+ * error.
+ */
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value)
Quoted 8 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 120

Quoted 5 lines
migration/migration.c +5 -0
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_SHUT, 4, (uint8_t *)&buf);

^ sizeof(buf) would be safer

Done.

Quoted 9 lines
Quoted 9 lines
migration/migration.c +9 -0
+}
+
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
Quoted 8 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 133

Quoted 4 lines
migration/migration.c +4 -0
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_PONG, 4, (uint8_t *)&buf);

It occurs to me that you could define PONG as returning the whole buffer that PING sends, instead of just 4-bytes. Might allow for some more testing of variable sized messages.

Yes; although it would complicate things a lot if I made it fully generic because I'd have to worry about allocating a buffer etc and I'm not making vast use of the 4 bytes I've already got.

Dave

Quoted 30 lines
Quoted 30 lines
migration/migration.c +2 -0
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
savevm.c +1 -1
index d082738..7084d07 100644
@@ -1008,7 +1008,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_pong(mis, tmp32);
         break;
 
     default:
trace-events +1 -0
index 99e00b5..4f3eff8 100644
@@ -1379,6 +1379,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
Quoted 7 lines
Quoted 1 line
trace-events +0 -0
 # migration/rdma.c
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#40 lore raw

Re: [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * David Gibson ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the CMD_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index c514dd4..6775747 100644
@@ -41,6 +41,13 @@ struct MigrationParams {
     bool shared;
 };
 
Quoted 7 lines
Quoted 5 lines

include/migration/migration.h struct MigrationParams { · line 45

Quoted 2 lines
include/migration/migration.h +2 -0
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {

"command" doesn't seem like quite the right description for these rp messages.

Would you prefer 'message' ?

Perhaps "message type" to distinguish from the the blob including both tag and data.

Quoted 106 lines
Quoted 48 lines
Quoted 48 lines
include/migration/migration.h +15 -0
+    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
+    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RP_CMD_PONG,         /* Response to a PING; data (seq: be32 ) */
+};
+
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -48,6 +55,7 @@ struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
@@ -169,6 +177,15 @@ int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value);
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration/migration.c +1 -0
index a36ea65..80d234c 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
 }
@@ -88,6 +89,50 @@ void migration_incoming_state_destroy(void)
     mis_current = NULL;
 }
 
Quoted 9 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 98

Quoted 7 lines
migration/migration.c +7 -0
+/*
+ * Send a message on the return channel back to the source
+ * of the migration.
+ */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data)

Using (void *) for data would avoid casts in a bunch of the callers.

Fixed; thanks.

Quoted 17 lines
Quoted 17 lines
migration/migration.c +17 -0
+{
+    trace_migrate_send_rp_message((int)cmd, len);
+    qemu_mutex_lock(&mis->rp_mutex);
+    qemu_put_be16(mis->return_path, (unsigned int)cmd);
+    qemu_put_be16(mis->return_path, len);
+    qemu_put_buffer(mis->return_path, data, len);
+    qemu_fflush(mis->return_path);
+    qemu_mutex_unlock(&mis->rp_mutex);
+}
+
+/*
+ * Send a 'SHUT' message on the return channel with the given value
+ * to indicate that we've finished with the RP.  None-0 value indicates
+ * error.
+ */
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value)
Quoted 8 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 120

Quoted 5 lines
migration/migration.c +5 -0
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_SHUT, 4, (uint8_t *)&buf);

^ sizeof(buf) would be safer

Done.

Quoted 9 lines
Quoted 9 lines
migration/migration.c +9 -0
+}
+
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
Quoted 8 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 133

Quoted 4 lines
migration/migration.c +4 -0
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_PONG, 4, (uint8_t *)&buf);

It occurs to me that you could define PONG as returning the whole buffer that PING sends, instead of just 4-bytes. Might allow for some more testing of variable sized messages.

Quoted 3 lines

Yes; although it would complicate things a lot if I made it fully generic because I'd have to worry about allocating a buffer etc and I'm not making vast use of the 4 bytes I've already got.

Couldn't migrate_send_rp_pong just take a buf pointer and length, then you can point that directly at the buffer in the ping message you've received.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#41 lore raw

Re: [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

Quoted 34 lines · * David Gibson ([email protected]) wrote:
Quoted 33 lines · Dr. David Alan Gilbert
Quoted 31 lines · * David Gibson ([email protected]) wrote:
Quoted 22 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the CMD_PING handler

---
 include/migration/migration.h | 17 ++++++++++++++++
 migration/migration.c         | 45 +++++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 trace-events                  |  1 +
 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +0 -0
index c514dd4..6775747 100644

include/migration/migration.h struct MigrationParams { · line 45

Quoted 5 lines
include/migration/migration.h +2 -0
@@ -41,6 +41,13 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {

"command" doesn't seem like quite the right description for these rp messages.

Quoted 4 lines
Quoted 1 line

Would you prefer 'message' ?

Perhaps "message type" to distinguish from the the blob including both tag and data.

OK, done:

/* Messages sent on the return path from destination to source */

enum mig_rp_message_type {
    MIG_RP_MSG_INVALID = 0,  /* Must be 0 */

<snip>

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

include/migration/migration.h

Quoted 11 lines
include/migration/migration.h +11 -0
+/*
+ * Send a 'PONG' message on the return channel with the given value
+ * (normally in response to a 'PING')
+ */
+void migrate_send_rp_pong(MigrationIncomingState *mis,
+                          uint32_t value)
+{
+    uint32_t buf;
+
+    buf = cpu_to_be32(value);
+    migrate_send_rp_message(mis, MIG_RP_CMD_PONG, 4, (uint8_t *)&buf);

It occurs to me that you could define PONG as returning the whole buffer that PING sends, instead of just 4-bytes. Might allow for some more testing of variable sized messages.

Quoted 7 lines
Quoted 3 lines

Yes; although it would complicate things a lot if I made it fully generic because I'd have to worry about allocating a buffer etc and I'm not making vast use of the 4 bytes I've already got.

Couldn't migrate_send_rp_pong just take a buf pointer and length, then you can point that directly at the buffer in the ping message you've received.

The buffer is a few levels down at that point, so it's non-trivial to do that, where as it's currently very very simple to just pass that fixed length value; and it's only a bit of debug.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#42 lore raw

Re: [Qemu-devel] [PATCH v5 11/45] Return path: Send responses from destination to source

In reply to David Gibson
Quoted 4 lines · David Gibson
Quoted 3 lines

migration/migration.c void migration_incoming_state_destroy(void) · line 98

Quoted 3 lines
migration/migration.c +3 -0
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data)

Using (void *) for data would avoid casts in a bunch of the callers.

Could also use uint8_t in the callers and replace cpu_to_be32 with stl_be_p. This makes it harder to send stuff with the wrong endianness.

Paolo

#43 lore raw

PATCH v5 12/45 Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +177 -1

209 lines not on this page · raw message

trace-events +13 -0
index 4f3eff8..1951b25 100644
@@ -1374,12 +1374,25 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
 flic_reset_failed(int err) "flic: reset failed %d"
 
 # migration.c
+await_outgoing_return_path_close(void) ""
+await_outgoing_return_path_joining(void) ""
 migrate_set_state(int new_state) "new state %d"
 migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
+open_outgoing_return_path(void) ""
+open_outgoing_return_path_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_bad_header_com(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 1 line
2.1.0
#44 lore raw

Re: [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path

Quoted 61 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +17 -0
index 80d234c..34cd4fe 100644
@@ -237,6 +237,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     return head;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -362,6 +379,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }

migration/migration.c static void migrate_set_state(MigrationState *s, int old_state, int new_state) · line 390

Quoted 10 lines
migration/migration.c +9 -0
 
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);

A cmpxchg seems dangerously subtle for such a basic and infrequent operation, but ok.

Quoted 19 lines
migration/migration.c +8 -0
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -369,6 +401,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();

migration/migration.c static void migrate_fd_cancel(MigrationState *s) · line 445

Quoted 6 lines
migration/migration.c +3 -0
@@ -406,6 +440,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);

I missed where qemu_file_shutdown() was implemented. Does this introduce a leftover socket dependency?

Quoted 35 lines
migration/migration.c +27 -1
+    }
+
     do {
         old_state = s->state;
         if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {
@@ -658,8 +697,145 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_com;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 734

Quoted 10 lines
migration/migration.c +10 -0
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_com = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+        case MIG_RP_CMD_PONG:
+            expected_len = 4;

Could the knowledge of expected lengths be folded into the switch below? Switching twice on the same thing is a bit icky.

Quoted 43 lines
migration/migration.c +42 -0
+            break;
+
+        default:
+            error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
+                    header_com, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received command 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_com, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the command and the data */
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_CMD_PONG:

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 787

Quoted 10 lines
migration/migration.c +10 -0
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
+        default:
+            /* This shouldn't happen because we should catch this above */
+            trace_source_return_path_bad_header_com();
+        }
+        /* Latest command processed, now leave a gap for the next one */
+        header_com = MIG_RP_CMD_INVALID;

This assignment will always get overwritten.

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 800

Quoted 13 lines
migration/migration.c +13 -0
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)

Uh.. surely this should be open_incoming_return_path(); it's designed to be used on the source side, AFAICT.

Quoted 8 lines
migration/migration.c +8 -0
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_outgoing_return_path();

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 818

Quoted 10 lines
migration/migration.c +10 -0
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_outgoing_return_path_continue();
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static void await_outgoing_return_path_close(MigrationState *ms)

Likewise "incoming" here, surely.

Quoted 53 lines
migration/migration.c +20 -0
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->rp_state.file) {
+        qemu_file_shutdown(ms->rp_state.file);
+    }
+    trace_await_outgoing_return_path_joining();
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    trace_await_outgoing_return_path_close();
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
trace-events +13 -0
index 4f3eff8..1951b25 100644
@@ -1374,12 +1374,25 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
 flic_reset_failed(int err) "flic: reset failed %d"
 
 # migration.c
+await_outgoing_return_path_close(void) ""
+await_outgoing_return_path_joining(void) ""
 migrate_set_state(int new_state) "new state %d"
 migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
+open_outgoing_return_path(void) ""
+open_outgoing_return_path_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_bad_header_com(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
 # migration/rdma.c
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#45 lore raw

Re: [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path

Quoted 63 lines · * David Gibson ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +17 -0
index 80d234c..34cd4fe 100644
@@ -237,6 +237,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     return head;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -362,6 +379,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }
 
Quoted 12 lines

migration/migration.c static void migrate_set_state(MigrationState *s, int old_state, int new_state) · line 390

Quoted 9 lines
migration/migration.c +9 -0
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);

A cmpxchg seems dangerously subtle for such a basic and infrequent operation, but ok.

I'll take other suggestions; but I'm trying to just do 'if the qemu_file still exists close it', and it didn't seem worth introducing another state variable to atomically update when we've already got the file pointer itself.

Quoted 22 lines
Quoted 22 lines
migration/migration.c +8 -0
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -369,6 +401,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -406,6 +440,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
Quoted 6 lines

migration/migration.c static void migrate_fd_cancel(MigrationState *s) · line 445

Quoted 3 lines
migration/migration.c +3 -0
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);

I missed where qemu_file_shutdown() was implemented. Does this introduce a leftover socket dependency?

No, it shouldn't. The shutdown() causes a shutdown(2) syscall to be issued on the socket stopping anything blocking on it; it then gets closed at the end after the rp thread has exited.

Quoted 40 lines
Quoted 40 lines
migration/migration.c +32 -1
+    }
+
     do {
         old_state = s->state;
         if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {
@@ -658,8 +697,145 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_com;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_com = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 734

Quoted 5 lines
migration/migration.c +5 -0
+
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+        case MIG_RP_CMD_PONG:
+            expected_len = 4;

Could the knowledge of expected lengths be folded into the switch below? Switching twice on the same thing is a bit icky.

No, because the length at this point is used to valdiate the length field in the header prior to reading the body. The other switch processes the contents of the body that have been read.

Quoted 47 lines
Quoted 47 lines
migration/migration.c +46 -0
+            break;
+
+        default:
+            error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
+                    header_com, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received command 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_com, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the command and the data */
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_CMD_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 787

Quoted 6 lines
migration/migration.c +6 -0
+        default:
+            /* This shouldn't happen because we should catch this above */
+            trace_source_return_path_bad_header_com();
+        }
+        /* Latest command processed, now leave a gap for the next one */
+        header_com = MIG_RP_CMD_INVALID;

This assignment will always get overwritten.

Thanks; gone - it's a left over from an old version.

Quoted 29 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 800

Quoted 13 lines
migration/migration.c +13 -0
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)

Uh.. surely this should be open_incoming_return_path(); it's designed to be used on the source side, AFAICT.

Quoted 12 lines
migration/migration.c +12 -0
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_outgoing_return_path();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_outgoing_return_path_continue();
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 818

Quoted 6 lines
migration/migration.c +6 -0
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static void await_outgoing_return_path_close(MigrationState *ms)

Likewise "incoming" here, surely.

I've changed those two to open_source_return_path() which seems less ambiguous; that OK?

Dave

Quoted 53 lines
Quoted 53 lines
migration/migration.c +20 -0
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->rp_state.file) {
+        qemu_file_shutdown(ms->rp_state.file);
+    }
+    trace_await_outgoing_return_path_joining();
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    trace_await_outgoing_return_path_close();
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
trace-events +13 -0
index 4f3eff8..1951b25 100644
@@ -1374,12 +1374,25 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d"
 flic_reset_failed(int err) "flic: reset failed %d"
 
 # migration.c
+await_outgoing_return_path_close(void) ""
+await_outgoing_return_path_joining(void) ""
 migrate_set_state(int new_state) "new state %d"
 migrate_fd_cleanup(void) ""
+migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
 migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
+open_outgoing_return_path(void) ""
+open_outgoing_return_path_continue(void) ""
+source_return_path_thread_bad_end(void) ""
+source_return_path_bad_header_com(void) ""
+source_return_path_thread_end(void) ""
+source_return_path_thread_entry(void) ""
+source_return_path_thread_failed_read_cmd_data(void) ""
+source_return_path_thread_loop_top(void) ""
+source_return_path_thread_pong(uint32_t val) "%x"
+source_return_path_thread_shut(uint32_t val) "%x"
 migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
 
Quoted 7 lines
Quoted 1 line
trace-events +0 -0
 # migration/rdma.c
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#46 lore raw

Re: [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path

Quoted 77 lines · Dr. David Alan Gilbert
Quoted 63 lines · * David Gibson ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +17 -0
index 80d234c..34cd4fe 100644
@@ -237,6 +237,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     return head;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -362,6 +379,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }
 
Quoted 12 lines

migration/migration.c static void migrate_set_state(MigrationState *s, int old_state, int new_state) · line 390

Quoted 9 lines
migration/migration.c +9 -0
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);

A cmpxchg seems dangerously subtle for such a basic and infrequent operation, but ok.

Quoted 4 lines

I'll take other suggestions; but I'm trying to just do 'if the qemu_file still exists close it', and it didn't seem worth introducing another state variable to atomically update when we've already got the file pointer itself.

Yes, I see the rationale. My concern is just that the more atomicity mechanisms are scattered through the code, the harder it is to analyze and be sure you haven't missed race cases (or introduced then with a future change).

In short, I prefer to see a simple-as-possible, and preferably documented, consistent overall concurrency scheme for a data structure, rather than scattered atomic ops for various variable where it's difficult to see how all the pieces might relate together.

Quoted 29 lines
Quoted 22 lines
Quoted 22 lines
migration/migration.c +8 -0
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -369,6 +401,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -406,6 +440,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
Quoted 6 lines

migration/migration.c static void migrate_fd_cancel(MigrationState *s) · line 445

Quoted 3 lines
migration/migration.c +3 -0
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);

I missed where qemu_file_shutdown() was implemented. Does this introduce a leftover socket dependency?

Quoted 3 lines

No, it shouldn't. The shutdown() causes a shutdown(2) syscall to be issued on the socket stopping anything blocking on it; it then gets closed at the end after the rp thread has exited.

Sorry, that's not what I meant. I mean is this a hole in the abstraction of the QemuFile, because it assumes that what you're dealing with here is indeed a socket, rather than something else?

Quoted 49 lines
Quoted 40 lines
Quoted 40 lines
migration/migration.c +32 -1
+    }
+
     do {
         old_state = s->state;
         if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {
@@ -658,8 +697,145 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_com;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_com = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 734

Quoted 5 lines
migration/migration.c +5 -0
+
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+        case MIG_RP_CMD_PONG:
+            expected_len = 4;

Could the knowledge of expected lengths be folded into the switch below? Switching twice on the same thing is a bit icky.

Quoted 4 lines

No, because the length at this point is used to valdiate the length field in the header prior to reading the body. The other switch processes the contents of the body that have been read.

Ok.

Quoted 96 lines
Quoted 47 lines
Quoted 47 lines
migration/migration.c +46 -0
+            break;
+
+        default:
+            error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
+                    header_com, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received command 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_com, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the command and the data */
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_CMD_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 787

Quoted 6 lines
migration/migration.c +6 -0
+        default:
+            /* This shouldn't happen because we should catch this above */
+            trace_source_return_path_bad_header_com();
+        }
+        /* Latest command processed, now leave a gap for the next one */
+        header_com = MIG_RP_CMD_INVALID;

This assignment will always get overwritten.

Thanks; gone - it's a left over from an old version.

Quoted 29 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 800

Quoted 13 lines
migration/migration.c +13 -0
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)

Uh.. surely this should be open_incoming_return_path(); it's designed to be used on the source side, AFAICT.

Quoted 12 lines
migration/migration.c +12 -0
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_outgoing_return_path();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_outgoing_return_path_continue();
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 818

Quoted 6 lines
migration/migration.c +6 -0
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static void await_outgoing_return_path_close(MigrationState *ms)

Likewise "incoming" here, surely.

Quoted 2 lines

I've changed those two to open_source_return_path() which seems less ambiguous; that OK?

Uh.. not really, it just moves the ambiguity to a different place (is "source return path" the return path *on* the source or *to* the source).

Perhaps "open_return_path_on_source" and "await_return_path_close_on_source"? I'm not particularly fond of those, but they're the best I've come up with yet.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#47 lore raw

Re: [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path

Quoted 122 lines · * David Gibson ([email protected]) wrote:
Quoted 77 lines · Dr. David Alan Gilbert
Quoted 63 lines · * David Gibson ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +17 -0
index 80d234c..34cd4fe 100644
@@ -237,6 +237,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     return head;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -362,6 +379,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }
 
Quoted 12 lines

migration/migration.c static void migrate_set_state(MigrationState *s, int old_state, int new_state) · line 390

Quoted 9 lines
migration/migration.c +9 -0
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);

A cmpxchg seems dangerously subtle for such a basic and infrequent operation, but ok.

Quoted 4 lines

I'll take other suggestions; but I'm trying to just do 'if the qemu_file still exists close it', and it didn't seem worth introducing another state variable to atomically update when we've already got the file pointer itself.

Yes, I see the rationale. My concern is just that the more atomicity mechanisms are scattered through the code, the harder it is to analyze and be sure you haven't missed race cases (or introduced then with a future change).

In short, I prefer to see a simple-as-possible, and preferably documented, consistent overall concurrency scheme for a data structure, rather than scattered atomic ops for various variable where it's difficult to see how all the pieces might relate together.

Quoted 29 lines
Quoted 28 lines
Quoted 19 lines
migration/migration.c +8 -0
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -369,6 +401,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();

migration/migration.c static void migrate_fd_cancel(MigrationState *s) · line 445

Quoted 6 lines
migration/migration.c +3 -0
@@ -406,6 +440,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);

I missed where qemu_file_shutdown() was implemented. Does this introduce a leftover socket dependency?

Quoted 8 lines
Quoted 3 lines

No, it shouldn't. The shutdown() causes a shutdown(2) syscall to be issued on the socket stopping anything blocking on it; it then gets closed at the end after the rp thread has exited.

Sorry, that's not what I meant. I mean is this a hole in the abstraction of the QemuFile, because it assumes that what you're dealing with here is indeed a socket, rather than something else?

It's just a dependency that we have a shutdown method on the qemu_file we're using; if it's not a socket then whatever it is, if we're going to use it for a rp then it needs to implement something equivalent.

Quoted 155 lines
Quoted 49 lines
Quoted 40 lines
Quoted 40 lines
migration/migration.c +32 -1
+    }
+
     do {
         old_state = s->state;
         if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {
@@ -658,8 +697,145 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
+
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->rp_state.file;
+    uint16_t expected_len, header_len, header_com;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    trace_source_return_path_thread_entry();
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        trace_source_return_path_thread_loop_top();
+        header_com = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 734

Quoted 5 lines
migration/migration.c +5 -0
+
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+        case MIG_RP_CMD_PONG:
+            expected_len = 4;

Could the knowledge of expected lengths be folded into the switch below? Switching twice on the same thing is a bit icky.

Quoted 4 lines

No, because the length at this point is used to valdiate the length field in the header prior to reading the body. The other switch processes the contents of the body that have been read.

Ok.

Quoted 96 lines
Quoted 47 lines
Quoted 47 lines
migration/migration.c +46 -0
+            break;
+
+        default:
+            error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
+                    header_com, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
 
+        if (header_len > expected_len) {
+            error_report("RP: Received command 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_com, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            trace_source_return_path_thread_failed_read_cmd_data();
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the command and the data */
+        switch (header_com) {
+        case MIG_RP_CMD_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_shut(tmp32);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            }
+            /*
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RP_CMD_PONG:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            trace_source_return_path_thread_pong(tmp32);
+            break;
+
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 787

Quoted 6 lines
migration/migration.c +6 -0
+        default:
+            /* This shouldn't happen because we should catch this above */
+            trace_source_return_path_bad_header_com();
+        }
+        /* Latest command processed, now leave a gap for the next one */
+        header_com = MIG_RP_CMD_INVALID;

This assignment will always get overwritten.

Thanks; gone - it's a left over from an old version.

Quoted 29 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 800

Quoted 13 lines
migration/migration.c +13 -0
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        trace_source_return_path_thread_bad_end();
+        source_return_path_bad(ms);
+    }
+
+    trace_source_return_path_thread_end();
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)

Uh.. surely this should be open_incoming_return_path(); it's designed to be used on the source side, AFAICT.

Quoted 12 lines
migration/migration.c +12 -0
+{
+
+    ms->rp_state.file = qemu_file_get_return_path(ms->file);
+    if (!ms->rp_state.file) {
+        return -1;
+    }
+
+    trace_open_outgoing_return_path();
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    trace_open_outgoing_return_path_continue();
Quoted 8 lines

migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 818

Quoted 6 lines
migration/migration.c +6 -0
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static void await_outgoing_return_path_close(MigrationState *ms)

Likewise "incoming" here, surely.

Quoted 2 lines

I've changed those two to open_source_return_path() which seems less ambiguous; that OK?

Quoted 7 lines

Uh.. not really, it just moves the ambiguity to a different place (is "source return path" the return path *on* the source or *to* the source).

Perhaps "open_return_path_on_source" and "await_return_path_close_on_source"? I'm not particularly fond of those, but they're the best I've come up with yet.

Done.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#48 lore raw

Re: [Qemu-devel] [PATCH v5 12/45] Return path: Source handling of return path

Quoted 128 lines · Dr. David Alan Gilbert
Quoted 127 lines · * David Gibson ([email protected]) wrote:
Quoted 77 lines · Dr. David Alan Gilbert
Quoted 63 lines · * David Gibson ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |   8 ++
 migration/migration.c         | 178 +++++++++++++++++++++++++++++++++++++++++-
 trace-events                  |  13 +++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
index 6775747..5242ead 100644
@@ -73,6 +73,14 @@ struct MigrationState
 
     int state;
     MigrationParams params;
+
+    /* State related to return path */
+    struct {
+        QEMUFile     *file;
+        QemuThread    rp_thread;
+        bool          error;
+    } rp_state;
+
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration/migration.c +17 -0
index 80d234c..34cd4fe 100644
@@ -237,6 +237,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     return head;
 }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
 static void get_xbzrle_cache_stats(MigrationInfo *info)
 {
     if (migrate_use_xbzrle()) {
@@ -362,6 +379,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
     }
 }
 
Quoted 12 lines

migration/migration.c static void migrate_set_state(MigrationState *s, int old_state, int new_state) · line 390

Quoted 9 lines
migration/migration.c +9 -0
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->rp_state.file;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->rp_state.file, rp, NULL);

A cmpxchg seems dangerously subtle for such a basic and infrequent operation, but ok.

Quoted 4 lines

I'll take other suggestions; but I'm trying to just do 'if the qemu_file still exists close it', and it didn't seem worth introducing another state variable to atomically update when we've already got the file pointer itself.

Yes, I see the rationale. My concern is just that the more atomicity mechanisms are scattered through the code, the harder it is to analyze and be sure you haven't missed race cases (or introduced then with a future change).

In short, I prefer to see a simple-as-possible, and preferably documented, consistent overall concurrency scheme for a data structure, rather than scattered atomic ops for various variable where it's difficult to see how all the pieces might relate together.

Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
migration/migration.c +8 -0
+    if (rp) {
+        trace_migrate_fd_cleanup_src_rp();
+        qemu_fclose(rp);
+    }
+}
+
 static void migrate_fd_cleanup(void *opaque)
 {
     MigrationState *s = opaque;
@@ -369,6 +401,8 @@ static void migrate_fd_cleanup(void *opaque)
     qemu_bh_delete(s->cleanup_bh);
     s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -406,6 +440,11 @@ static void migrate_fd_cancel(MigrationState *s)
     QEMUFile *f = migrate_get_current()->file;
     trace_migrate_fd_cancel();
 
Quoted 6 lines

migration/migration.c static void migrate_fd_cancel(MigrationState *s) · line 445

Quoted 3 lines
migration/migration.c +3 -0
+    if (s->rp_state.file) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->rp_state.file);

I missed where qemu_file_shutdown() was implemented. Does this introduce a leftover socket dependency?

No, it shouldn't. The shutdown() causes a shutdown(2) syscall to be issued on the socket stopping anything blocking on it; it then gets closed at the end after the rp thread has exited.

Quoted 7 lines
Quoted 3 lines

Sorry, that's not what I meant. I mean is this a hole in the abstraction of the QemuFile, because it assumes that what you're dealing with here is indeed a socket, rather than something else?

It's just a dependency that we have a shutdown method on the qemu_file we're using; if it's not a socket then whatever it is, if we're going to use it for a rp then it needs to implement something equivalent.

Um, yeah, except I don't think most file types really have an operation that's semantically similar to socket shutdown.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#49 lore raw

PATCH v5 13/45 ram_debug_dump_bitmap: Dump a migration bitmap as text

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +39 -0
index 91645cc..fe0df0d 100644
@@ -776,6 +776,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}
+
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
     RAMBlock *block;
include/migration/migration.h +1 -0
index 5242ead..3776e86 100644
@@ -156,6 +156,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
Signature · 1 line
2.1.0
#50 lore raw

Re: [Qemu-devel] [PATCH v5 13/45] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 24 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +5 -0
index 91645cc..fe0df0d 100644
@@ -776,6 +776,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */

arch_init.c static void reset_ram_globals(void) · line 793

Quoted 10 lines
arch_init.c +10 -0
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;

Any reason not to just have the caller pass migration_bitmap, if that's what they want?

Quoted 10 lines
arch_init.c +10 -0
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {

arch_init.c static void reset_ram_globals(void) · line 813

Quoted 10 lines
arch_init.c +10 -0
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);

Might be slightly more readable if it printed GPAs instead of page numbers.

Quoted 19 lines
arch_init.c +4 -0
+        }
+    }
+}
+
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
     RAMBlock *block;
include/migration/migration.h +1 -0
index 5242ead..3776e86 100644
@@ -156,6 +156,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#51 lore raw

Re: [Qemu-devel] [PATCH v5 13/45] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 30 lines · * David Gibson ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +10 -0
index 91645cc..fe0df0d 100644
@@ -776,6 +776,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
Quoted 8 lines

arch_init.c static void reset_ram_globals(void) · line 793

Quoted 5 lines
arch_init.c +5 -0
+    int64_t linelen = 128;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;

Any reason not to just have the caller pass migration_bitmap, if that's what they want?

migration_bitmap is a static in this file; I wanted to be able to add calls to this debug function from elsewhere, e.g. before aborts/crashes whereever I was testing things that were going wrong.

Quoted 21 lines
Quoted 10 lines
arch_init.c +10 -0
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {

arch_init.c static void reset_ram_globals(void) · line 813

Quoted 10 lines
arch_init.c +10 -0
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found = found || (thisbit != expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
Quoted 2 lines

Might be slightly more readable if it printed GPAs instead of page numbers.

Indeed it would; but translating bitmap position into GPA is non-trivial anyway; RAM blocks aren't necessarily in order within the bitmap and indeed may be mixed on any one line.

Dave

Quoted 15 lines
Quoted 15 lines
arch_init.c +4 -0
+        }
+    }
+}
+
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
     RAMBlock *block;
include/migration/migration.h +0 -0
index 5242ead..3776e86 100644
@@ -156,6 +156,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
Quoted 11 lines
Quoted 5 lines
include/migration/migration.h +1 -0
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#52 lore raw

PATCH v5 14/45 Move loadvm_handlers into MigrationIncomingState

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more.

---
 include/migration/migration.h |  5 +++++
 include/migration/vmstate.h   |  2 ++
 include/qemu/typedefs.h       |  1 +
 migration/migration.c         |  2 ++
 savevm.c                      | 28 ++++++++++++++++------------
 5 files changed, 26 insertions(+), 12 deletions(-)
include/migration/migration.h +5 -0
index 3776e86..751caa0 100644
@@ -50,10 +50,15 @@ enum mig_rpcomm_cmd {
 
 typedef struct MigrationState MigrationState;
 
+typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    /* See savevm.c */
+    LoadStateEntry_Head loadvm_handlers;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/migration/vmstate.h +2 -0
index c20f2d1..18da207 100644
@@ -797,6 +797,8 @@ extern const VMStateInfo vmstate_info_bitmap;
 
 #define SELF_ANNOUNCE_ROUNDS 5
 
+void loadvm_free_handlers(MigrationIncomingState *mis);
+
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                        void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
include/qemu/typedefs.h +1 -0
index 74dfad3..6fdcbcd 100644
@@ -31,6 +31,7 @@ typedef struct I2CBus I2CBus;
 typedef struct I2SCodec I2SCodec;
 typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
+typedef struct LoadStateEntry LoadStateEntry;
 typedef struct MACAddr MACAddr;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
migration/migration.c +2 -0
index 34cd4fe..4592060 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    QLIST_INIT(&mis_current->loadvm_handlers);
     qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
@@ -85,6 +86,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 
 void migration_incoming_state_destroy(void)
 {
+    loadvm_free_handlers(mis_current);
     g_free(mis_current);
     mis_current = NULL;
 }
savevm.c +16 -12
index 7084d07..f42713d 100644
@@ -1019,18 +1019,26 @@ static int loadvm_process_command(QEMUFile *f)
     return 0;
 }
 
-typedef struct LoadStateEntry {
+struct LoadStateEntry {
     QLIST_ENTRY(LoadStateEntry) entry;
     SaveStateEntry *se;
     int section_id;
     int version_id;
-} LoadStateEntry;
+};
 
-int qemu_loadvm_state(QEMUFile *f)
+void loadvm_free_handlers(MigrationIncomingState *mis)
 {
-    QLIST_HEAD(, LoadStateEntry) loadvm_handlers =
-        QLIST_HEAD_INITIALIZER(loadvm_handlers);
     LoadStateEntry *le, *new_le;
+
+    QLIST_FOREACH_SAFE(le, &mis->loadvm_handlers, entry, new_le) {
+        QLIST_REMOVE(le, entry);
+        g_free(le);
+    }
+}
+
+int qemu_loadvm_state(QEMUFile *f)
+{
+    MigrationIncomingState *mis = migration_incoming_get_current();
     Error *local_err = NULL;
     uint8_t section_type;
     unsigned int v;
@@ -1061,6 +1069,7 @@ int qemu_loadvm_state(QEMUFile *f)
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
+        LoadStateEntry *le;
         char idstr[256];
 
         trace_qemu_loadvm_state_section(section_type);
@@ -1102,7 +1111,7 @@ int qemu_loadvm_state(QEMUFile *f)
             le->se = se;
             le->section_id = section_id;
             le->version_id = version_id;
-            QLIST_INSERT_HEAD(&loadvm_handlers, le, entry);
+            QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
@@ -1116,7 +1125,7 @@ int qemu_loadvm_state(QEMUFile *f)
             section_id = qemu_get_be32(f);
 
             trace_qemu_loadvm_state_section_partend(section_id);
-            QLIST_FOREACH(le, &loadvm_handlers, entry) {
+            QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
                 if (le->section_id == section_id) {
                     break;
                 }
@@ -1152,11 +1161,6 @@ int qemu_loadvm_state(QEMUFile *f)
     ret = 0;
 
 out:
-    QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) {
-        QLIST_REMOVE(le, entry);
-        g_free(le);
-    }
-
     if (ret == 0) {
         ret = qemu_file_get_error(f);
     }
Signature · 1 line
2.1.0
#53 lore raw

Re: [Qemu-devel] [PATCH v5 14/45] Move loadvm_handlers into MigrationIncomingState

Quoted 73 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more.

---
 include/migration/migration.h |  5 +++++
 include/migration/vmstate.h   |  2 ++
 include/qemu/typedefs.h       |  1 +
 migration/migration.c         |  2 ++
 savevm.c                      | 28 ++++++++++++++++------------
 5 files changed, 26 insertions(+), 12 deletions(-)
include/migration/migration.h +5 -0
index 3776e86..751caa0 100644
@@ -50,10 +50,15 @@ enum mig_rpcomm_cmd {
 
 typedef struct MigrationState MigrationState;
 
+typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    /* See savevm.c */
+    LoadStateEntry_Head loadvm_handlers;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/migration/vmstate.h +2 -0
index c20f2d1..18da207 100644
@@ -797,6 +797,8 @@ extern const VMStateInfo vmstate_info_bitmap;
 
 #define SELF_ANNOUNCE_ROUNDS 5
 
+void loadvm_free_handlers(MigrationIncomingState *mis);
+
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                        void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
include/qemu/typedefs.h +1 -0
index 74dfad3..6fdcbcd 100644
@@ -31,6 +31,7 @@ typedef struct I2CBus I2CBus;
 typedef struct I2SCodec I2SCodec;
 typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
+typedef struct LoadStateEntry LoadStateEntry;
 typedef struct MACAddr MACAddr;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
migration/migration.c +1 -0
index 34cd4fe..4592060 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    QLIST_INIT(&mis_current->loadvm_handlers);
     qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;

migration/migration.c MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) · line 89

Quoted 4 lines
migration/migration.c +1 -0
@@ -85,6 +86,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 
 void migration_incoming_state_destroy(void)
 {
+    loadvm_free_handlers(mis_current);

AFAICT this is the only caler of loadvm_free_handlers(), so why not just open-code it here?

savevm.c static int loadvm_process_command(QEMUFile *f) · line 1022

Quoted 13 lines
migration/migration.c +0 -0
     g_free(mis_current);
     mis_current = NULL;
 }
savevm.c +1 -1
index 7084d07..f42713d 100644
@@ -1019,18 +1019,26 @@ static int loadvm_process_command(QEMUFile *f)
     return 0;
 }
 
-typedef struct LoadStateEntry {
+struct LoadStateEntry {

Why remove the typedef?

Quoted 64 lines
savevm.c +15 -11
     QLIST_ENTRY(LoadStateEntry) entry;
     SaveStateEntry *se;
     int section_id;
     int version_id;
-} LoadStateEntry;
+};
 
-int qemu_loadvm_state(QEMUFile *f)
+void loadvm_free_handlers(MigrationIncomingState *mis)
 {
-    QLIST_HEAD(, LoadStateEntry) loadvm_handlers =
-        QLIST_HEAD_INITIALIZER(loadvm_handlers);
     LoadStateEntry *le, *new_le;
+
+    QLIST_FOREACH_SAFE(le, &mis->loadvm_handlers, entry, new_le) {
+        QLIST_REMOVE(le, entry);
+        g_free(le);
+    }
+}
+
+int qemu_loadvm_state(QEMUFile *f)
+{
+    MigrationIncomingState *mis = migration_incoming_get_current();
     Error *local_err = NULL;
     uint8_t section_type;
     unsigned int v;
@@ -1061,6 +1069,7 @@ int qemu_loadvm_state(QEMUFile *f)
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
+        LoadStateEntry *le;
         char idstr[256];
 
         trace_qemu_loadvm_state_section(section_type);
@@ -1102,7 +1111,7 @@ int qemu_loadvm_state(QEMUFile *f)
             le->se = se;
             le->section_id = section_id;
             le->version_id = version_id;
-            QLIST_INSERT_HEAD(&loadvm_handlers, le, entry);
+            QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
@@ -1116,7 +1125,7 @@ int qemu_loadvm_state(QEMUFile *f)
             section_id = qemu_get_be32(f);
 
             trace_qemu_loadvm_state_section_partend(section_id);
-            QLIST_FOREACH(le, &loadvm_handlers, entry) {
+            QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
                 if (le->section_id == section_id) {
                     break;
                 }
@@ -1152,11 +1161,6 @@ int qemu_loadvm_state(QEMUFile *f)
     ret = 0;
 
 out:
-    QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) {
-        QLIST_REMOVE(le, entry);
-        g_free(le);
-    }
-
     if (ret == 0) {
         ret = qemu_file_get_error(f);
     }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#54 lore raw

Re: [Qemu-devel] [PATCH v5 14/45] Move loadvm_handlers into MigrationIncomingState

Quoted 75 lines · * David Gibson ([email protected]) wrote:
Quoted 74 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more.

---
 include/migration/migration.h |  5 +++++
 include/migration/vmstate.h   |  2 ++
 include/qemu/typedefs.h       |  1 +
 migration/migration.c         |  2 ++
 savevm.c                      | 28 ++++++++++++++++------------
 5 files changed, 26 insertions(+), 12 deletions(-)
include/migration/migration.h +5 -0
index 3776e86..751caa0 100644
@@ -50,10 +50,15 @@ enum mig_rpcomm_cmd {
 
 typedef struct MigrationState MigrationState;
 
+typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    /* See savevm.c */
+    LoadStateEntry_Head loadvm_handlers;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/migration/vmstate.h +2 -0
index c20f2d1..18da207 100644
@@ -797,6 +797,8 @@ extern const VMStateInfo vmstate_info_bitmap;
 
 #define SELF_ANNOUNCE_ROUNDS 5
 
+void loadvm_free_handlers(MigrationIncomingState *mis);
+
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                        void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
include/qemu/typedefs.h +1 -0
index 74dfad3..6fdcbcd 100644
@@ -31,6 +31,7 @@ typedef struct I2CBus I2CBus;
 typedef struct I2SCodec I2SCodec;
 typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
+typedef struct LoadStateEntry LoadStateEntry;
 typedef struct MACAddr MACAddr;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
migration/migration.c +1 -0
index 34cd4fe..4592060 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    QLIST_INIT(&mis_current->loadvm_handlers);
     qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
@@ -85,6 +86,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 
Quoted 6 lines

migration/migration.c MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) · line 89

Quoted 3 lines
migration/migration.c +1 -0
 void migration_incoming_state_destroy(void)
 {
+    loadvm_free_handlers(mis_current);

AFAICT this is the only caler of loadvm_free_handlers(), so why not just open-code it here?

I was keeping this handler list as owned by savevm.c; all it's allocation and it's use are done in savevm.c, so it would seem odd to open code freeing the list in a separate file.

Quoted 11 lines
Quoted 11 lines
migration/migration.c +0 -0
     g_free(mis_current);
     mis_current = NULL;
 }
savevm.c +0 -0
index 7084d07..f42713d 100644
@@ -1019,18 +1019,26 @@ static int loadvm_process_command(QEMUFile *f)
     return 0;
 }
 
Quoted 4 lines

savevm.c static int loadvm_process_command(QEMUFile *f) · line 1022

Quoted 2 lines
savevm.c +1 -1
-typedef struct LoadStateEntry {
+struct LoadStateEntry {

Why remove the typedef?

Because it is now typedef'd in typedefs.h, and older gcc (RHEL6) object to two typedefs even if they're of the same thing.

Dave

Quoted 55 lines
Quoted 55 lines
savevm.c +15 -6
     QLIST_ENTRY(LoadStateEntry) entry;
     SaveStateEntry *se;
     int section_id;
     int version_id;
-} LoadStateEntry;
+};
 
-int qemu_loadvm_state(QEMUFile *f)
+void loadvm_free_handlers(MigrationIncomingState *mis)
 {
-    QLIST_HEAD(, LoadStateEntry) loadvm_handlers =
-        QLIST_HEAD_INITIALIZER(loadvm_handlers);
     LoadStateEntry *le, *new_le;
+
+    QLIST_FOREACH_SAFE(le, &mis->loadvm_handlers, entry, new_le) {
+        QLIST_REMOVE(le, entry);
+        g_free(le);
+    }
+}
+
+int qemu_loadvm_state(QEMUFile *f)
+{
+    MigrationIncomingState *mis = migration_incoming_get_current();
     Error *local_err = NULL;
     uint8_t section_type;
     unsigned int v;
@@ -1061,6 +1069,7 @@ int qemu_loadvm_state(QEMUFile *f)
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
+        LoadStateEntry *le;
         char idstr[256];
 
         trace_qemu_loadvm_state_section(section_type);
@@ -1102,7 +1111,7 @@ int qemu_loadvm_state(QEMUFile *f)
             le->se = se;
             le->section_id = section_id;
             le->version_id = version_id;
-            QLIST_INSERT_HEAD(&loadvm_handlers, le, entry);
+            QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
@@ -1116,7 +1125,7 @@ int qemu_loadvm_state(QEMUFile *f)
             section_id = qemu_get_be32(f);
 
             trace_qemu_loadvm_state_section_partend(section_id);
-            QLIST_FOREACH(le, &loadvm_handlers, entry) {
+            QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
                 if (le->section_id == section_id) {
                     break;
                 }
@@ -1152,11 +1161,6 @@ int qemu_loadvm_state(QEMUFile *f)
     ret = 0;
 
Quoted 15 lines
Quoted 9 lines
savevm.c +0 -5
 out:
-    QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) {
-        QLIST_REMOVE(le, entry);
-        g_free(le);
-    }
-
     if (ret == 0) {
         ret = qemu_file_get_error(f);
     }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Reviewed-by
#55 lore raw

Re: [Qemu-devel] [PATCH v5 14/45] Move loadvm_handlers into MigrationIncomingState

Quoted 98 lines · Dr. David Alan Gilbert
Quoted 75 lines · * David Gibson ([email protected]) wrote:
Quoted 74 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

In postcopy we need the loadvm_handlers to be used in a couple of different instances of the loadvm loop/routine, and thus it can't be local any more.

---
 include/migration/migration.h |  5 +++++
 include/migration/vmstate.h   |  2 ++
 include/qemu/typedefs.h       |  1 +
 migration/migration.c         |  2 ++
 savevm.c                      | 28 ++++++++++++++++------------
 5 files changed, 26 insertions(+), 12 deletions(-)
include/migration/migration.h +5 -0
index 3776e86..751caa0 100644
@@ -50,10 +50,15 @@ enum mig_rpcomm_cmd {
 
 typedef struct MigrationState MigrationState;
 
+typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    /* See savevm.c */
+    LoadStateEntry_Head loadvm_handlers;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/migration/vmstate.h +2 -0
index c20f2d1..18da207 100644
@@ -797,6 +797,8 @@ extern const VMStateInfo vmstate_info_bitmap;
 
 #define SELF_ANNOUNCE_ROUNDS 5
 
+void loadvm_free_handlers(MigrationIncomingState *mis);
+
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                        void *opaque, int version_id);
 void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
include/qemu/typedefs.h +1 -0
index 74dfad3..6fdcbcd 100644
@@ -31,6 +31,7 @@ typedef struct I2CBus I2CBus;
 typedef struct I2SCodec I2SCodec;
 typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
+typedef struct LoadStateEntry LoadStateEntry;
 typedef struct MACAddr MACAddr;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
migration/migration.c +1 -0
index 34cd4fe..4592060 100644
@@ -78,6 +78,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 {
     mis_current = g_malloc0(sizeof(MigrationIncomingState));
     mis_current->file = f;
+    QLIST_INIT(&mis_current->loadvm_handlers);
     qemu_mutex_init(&mis_current->rp_mutex);
 
     return mis_current;
@@ -85,6 +86,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
 
Quoted 6 lines

migration/migration.c MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) · line 89

Quoted 3 lines
migration/migration.c +1 -0
 void migration_incoming_state_destroy(void)
 {
+    loadvm_free_handlers(mis_current);

AFAICT this is the only caler of loadvm_free_handlers(), so why not just open-code it here?

I was keeping this handler list as owned by savevm.c; all it's allocation and it's use are done in savevm.c, so it would seem odd to open code freeing the list in a separate file.

Quoted 11 lines
Quoted 11 lines
migration/migration.c +0 -0
     g_free(mis_current);
     mis_current = NULL;
 }
savevm.c +0 -0
index 7084d07..f42713d 100644
@@ -1019,18 +1019,26 @@ static int loadvm_process_command(QEMUFile *f)
     return 0;
 }
 
Quoted 7 lines
Quoted 4 lines

savevm.c static int loadvm_process_command(QEMUFile *f) · line 1022

Quoted 2 lines
savevm.c +1 -1
-typedef struct LoadStateEntry {
+struct LoadStateEntry {

Why remove the typedef?

Because it is now typedef'd in typedefs.h, and older gcc (RHEL6) object to two typedefs even if they're of the same thing.

Ok, makes sense.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#56 lore raw

PATCH v5 15/45 Rework loadvm path for subloops

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

---
 savevm.c     | 106 ++++++++++++++++++++++++++++++++++++-----------------------
 trace-events |   4 +++
 2 files changed, 69 insertions(+), 41 deletions(-)
savevm.c +65 -41
index f42713d..4b619da 100644
@@ -951,6 +951,16 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
+/* ORable flags that control the (potentially nested) loadvm_state loops */
+enum LoadVMExitCodes {
+    /* Quit the loop level that received this command */
+    LOADVM_QUIT_LOOP     =  1,
+    /* Quit this loop and our parent */
+    LOADVM_QUIT_PARENT   =  2,
+};
+
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
+
 static int loadvm_process_command_simple_lencheck(const char *name,
                                                   unsigned int actual,
                                                   unsigned int expected)
@@ -967,6 +977,8 @@ static int loadvm_process_command_simple_lencheck(const char *name,
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
+ * 0   just a normal return
+ * 1   All good, but exit the loop
  */
 static int loadvm_process_command(QEMUFile *f)
 {
@@ -1036,36 +1048,13 @@ void loadvm_free_handlers(MigrationIncomingState *mis)
     }
 }
 
-int qemu_loadvm_state(QEMUFile *f)
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
 {
-    MigrationIncomingState *mis = migration_incoming_get_current();
-    Error *local_err = NULL;
     uint8_t section_type;
-    unsigned int v;
     int ret;
+    int exitcode = 0;
 
-    if (qemu_savevm_state_blocked(&local_err)) {
-        error_report("%s", error_get_pretty(local_err));
-        error_free(local_err);
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v != QEMU_VM_FILE_MAGIC) {
-        error_report("Not a migration stream");
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        error_report("SaveVM v2 format is obsolete and don't work anymore");
-        return -ENOTSUP;
-    }
-    if (v != QEMU_VM_FILE_VERSION) {
-        error_report("Unsupported migration stream version");
-        return -ENOTSUP;
-    }
-
+    trace_qemu_loadvm_state_main();
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
@@ -1093,16 +1082,14 @@ int qemu_loadvm_state(QEMUFile *f)
             if (se == NULL) {
                 error_report("Unknown savevm section or instance '%s' %d",
                              idstr, instance_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Validate version */
             if (version_id > se->version_id) {
                 error_report("savevm: unsupported version %d for '%s' v%d",
                              version_id, idstr, se->version_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Add entry */
@@ -1117,7 +1104,7 @@ int qemu_loadvm_state(QEMUFile *f)
             if (ret < 0) {
                 error_report("error while loading state for instance 0x%x of"
                              " device '%s'", instance_id, idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_SECTION_PART:
@@ -1132,36 +1119,73 @@ int qemu_loadvm_state(QEMUFile *f)
             }
             if (le == NULL) {
                 error_report("Unknown savevm section %d", section_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
                 error_report("error while loading state section id %d(%s)",
                              section_id, le->se->idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_COMMAND:
             ret = loadvm_process_command(f);
-            if (ret < 0) {
-                goto out;
+            trace_qemu_loadvm_state_section_command(ret);
+            if ((ret < 0) || (ret & LOADVM_QUIT_LOOP)) {
+                return ret;
             }
+            exitcode |= ret; /* Lets us pass flags up to the parent */
             break;
         default:
             error_report("Unknown savevm section type %d", section_type);
-            ret = -EINVAL;
-            goto out;
+            return -EINVAL;
         }
     }
 
-    cpu_synchronize_all_post_init();
+    if (exitcode & LOADVM_QUIT_PARENT) {
+        trace_qemu_loadvm_state_main_quit_parent();
+        exitcode &= ~LOADVM_QUIT_PARENT;
+        exitcode |= LOADVM_QUIT_LOOP;
+    }
+
+    return exitcode;
+}
+
+int qemu_loadvm_state(QEMUFile *f)
+{
+    MigrationIncomingState *mis = migration_incoming_get_current();
+    Error *local_err = NULL;
+    unsigned int v;
+    int ret;
+
+    if (qemu_savevm_state_blocked(&local_err)) {
+        error_report("%s", error_get_pretty(local_err));
+        error_free(local_err);
+        return -EINVAL;
+    }
+
+    v = qemu_get_be32(f);
+    if (v != QEMU_VM_FILE_MAGIC) {
+        error_report("Not a migration stream");
+        return -EINVAL;
+    }
+
+    v = qemu_get_be32(f);
+    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
+        error_report("SaveVM v2 format is obsolete and don't work anymore");
+        return -ENOTSUP;
+    }
+    if (v != QEMU_VM_FILE_VERSION) {
+        error_report("Unsupported migration stream version");
+        return -ENOTSUP;
+    }
 
-    ret = 0;
+    ret = qemu_loadvm_state_main(f, mis);
 
-out:
+    trace_qemu_loadvm_state_post_main(ret);
     if (ret == 0) {
+        cpu_synchronize_all_post_init();
         ret = qemu_file_get_error(f);
     }
 
trace-events +4 -0
index 1951b25..4ff55fe 100644
@@ -1165,7 +1165,11 @@ vmware_setmode(uint32_t w, uint32_t h, uint32_t bpp) "%dx%d @ %d bpp"
 
 # savevm.c
 qemu_loadvm_state_section(unsigned int section_type) "%d"
+qemu_loadvm_state_section_command(int ret) "%d"
 qemu_loadvm_state_section_partend(uint32_t section_id) "%u"
+qemu_loadvm_state_main(void) ""
+qemu_loadvm_state_main_quit_parent(void) ""
+qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 loadvm_process_command_ping(uint32_t val) "%x"
Signature · 1 line
2.1.0
#57 lore raw

Re: [Qemu-devel] [PATCH v5 15/45] Rework loadvm path for subloops

Quoted 22 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

---
 savevm.c     | 106 ++++++++++++++++++++++++++++++++++++-----------------------
 trace-events |   4 +++
 2 files changed, 69 insertions(+), 41 deletions(-)
savevm.c +0 -0
index f42713d..4b619da 100644

savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 960

Quoted 10 lines
savevm.c +7 -0
@@ -951,6 +951,16 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
+/* ORable flags that control the (potentially nested) loadvm_state loops */
+enum LoadVMExitCodes {
+    /* Quit the loop level that received this command */
+    LOADVM_QUIT_LOOP     =  1,
+    /* Quit this loop and our parent */
+    LOADVM_QUIT_PARENT   =  2,
+};

The semantics of all the exit code stuff is doing my head in; I'm not sure how to make it more comprehensible.

savevm.c static int loadvm_process_command_simple_lencheck(const char *name, · line 981

Quoted 11 lines
savevm.c +4 -0
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
+
 static int loadvm_process_command_simple_lencheck(const char *name,
                                                   unsigned int actual,
                                                   unsigned int expected)
@@ -967,6 +977,8 @@ static int loadvm_process_command_simple_lencheck(const char *name,
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
+ * 0   just a normal return
+ * 1   All good, but exit the loop

This should probably also mention the possibility of negative returns for errors.

Am I correct in thinking that at this point the function never returns 1? I'm assuming later patches in the series change that.

Maybe I'm missing something in my mental model here, but tying the duration of the containing loop to execution of specific commands seems problematic. What's the circumstance in which it makes sense for a command to indicate that the rest of the packaged data should be essentially ignored

Quoted 102 lines
savevm.c +12 -38
  */
 static int loadvm_process_command(QEMUFile *f)
 {
@@ -1036,36 +1048,13 @@ void loadvm_free_handlers(MigrationIncomingState *mis)
     }
 }
 
-int qemu_loadvm_state(QEMUFile *f)
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
 {
-    MigrationIncomingState *mis = migration_incoming_get_current();
-    Error *local_err = NULL;
     uint8_t section_type;
-    unsigned int v;
     int ret;
+    int exitcode = 0;
 
-    if (qemu_savevm_state_blocked(&local_err)) {
-        error_report("%s", error_get_pretty(local_err));
-        error_free(local_err);
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v != QEMU_VM_FILE_MAGIC) {
-        error_report("Not a migration stream");
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        error_report("SaveVM v2 format is obsolete and don't work anymore");
-        return -ENOTSUP;
-    }
-    if (v != QEMU_VM_FILE_VERSION) {
-        error_report("Unsupported migration stream version");
-        return -ENOTSUP;
-    }
-
+    trace_qemu_loadvm_state_main();
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
@@ -1093,16 +1082,14 @@ int qemu_loadvm_state(QEMUFile *f)
             if (se == NULL) {
                 error_report("Unknown savevm section or instance '%s' %d",
                              idstr, instance_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Validate version */
             if (version_id > se->version_id) {
                 error_report("savevm: unsupported version %d for '%s' v%d",
                              version_id, idstr, se->version_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Add entry */
@@ -1117,7 +1104,7 @@ int qemu_loadvm_state(QEMUFile *f)
             if (ret < 0) {
                 error_report("error while loading state for instance 0x%x of"
                              " device '%s'", instance_id, idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_SECTION_PART:
@@ -1132,36 +1119,73 @@ int qemu_loadvm_state(QEMUFile *f)
             }
             if (le == NULL) {
                 error_report("Unknown savevm section %d", section_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
                 error_report("error while loading state section id %d(%s)",
                              section_id, le->se->idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_COMMAND:
             ret = loadvm_process_command(f);
-            if (ret < 0) {
-                goto out;
+            trace_qemu_loadvm_state_section_command(ret);
+            if ((ret < 0) || (ret & LOADVM_QUIT_LOOP)) {
+                return ret;
             }
+            exitcode |= ret; /* Lets us pass flags up to the parent */
             break;
         default:
             error_report("Unknown savevm section type %d", section_type);
-            ret = -EINVAL;
-            goto out;

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1150

Quoted 10 lines
savevm.c +6 -1
+            return -EINVAL;
         }
     }
 
-    cpu_synchronize_all_post_init();
+    if (exitcode & LOADVM_QUIT_PARENT) {
+        trace_qemu_loadvm_state_main_quit_parent();
+        exitcode &= ~LOADVM_QUIT_PARENT;
+        exitcode |= LOADVM_QUIT_LOOP;
+    }

So, if I'm following properly putting a QUIT_PARENT will cause this loop to exit, also returning QUIT_LOOP, so the next loop out also quits. If there was a third lood beyond that it wouldn't quit.

But are those really the semantics you want; or do you want the options to be "quit one level" and "quit all levels", which seems a little bit simpler. In the current plans you only have the two levels so they're equivalent.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#58 lore raw

Re: [Qemu-devel] [PATCH v5 15/45] Rework loadvm path for subloops

Quoted 26 lines · * David Gibson ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

---
 savevm.c     | 106 ++++++++++++++++++++++++++++++++++++-----------------------
 trace-events |   4 +++
 2 files changed, 69 insertions(+), 41 deletions(-)
savevm.c +0 -0
index f42713d..4b619da 100644
@@ -951,6 +951,16 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
Quoted 10 lines

savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 960

Quoted 7 lines
savevm.c +7 -0
+/* ORable flags that control the (potentially nested) loadvm_state loops */
+enum LoadVMExitCodes {
+    /* Quit the loop level that received this command */
+    LOADVM_QUIT_LOOP     =  1,
+    /* Quit this loop and our parent */
+    LOADVM_QUIT_PARENT   =  2,
+};

The semantics of all the exit code stuff is doing my head in; I'm not sure how to make it more comprehensible.

I've managed to kill one of the states off; it's now a single flag that quits all levels; much easier to understand.

Quoted 6 lines
Quoted 6 lines
savevm.c +2 -0
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
+
 static int loadvm_process_command_simple_lencheck(const char *name,
                                                   unsigned int actual,
                                                   unsigned int expected)
@@ -967,6 +977,8 @@ static int loadvm_process_command_simple_lencheck(const char *name,
Quoted 8 lines

savevm.c static int loadvm_process_command_simple_lencheck(const char *name, · line 981

Quoted 5 lines
savevm.c +2 -0
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
+ * 0   just a normal return
+ * 1   All good, but exit the loop

This should probably also mention the possibility of negative returns for errors.

'negative return on error' it says there; I've made that clearer now so it now has a '<0' line.

Quoted 2 lines

Am I correct in thinking that at this point the function never returns 1? I'm assuming later patches in the series change that.

Yes, it's one of the commands that now returns that flag. (Actually it was the LOADVM_QUIT_* enum, not '1')

Quoted 5 lines

Maybe I'm missing something in my mental model here, but tying the duration of the containing loop to execution of specific commands seems problematic. What's the circumstance in which it makes sense for a command to indicate that the rest of the packaged data should be essentially ignored

Yes, the only case we actually care about here in the postcopy case is that when we are reading from the package we stop the main loops ever reading from the main fd, since the listen thread takes that over.

Quoted 117 lines
Quoted 102 lines
savevm.c +12 -38
  */
 static int loadvm_process_command(QEMUFile *f)
 {
@@ -1036,36 +1048,13 @@ void loadvm_free_handlers(MigrationIncomingState *mis)
     }
 }
 
-int qemu_loadvm_state(QEMUFile *f)
+static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
 {
-    MigrationIncomingState *mis = migration_incoming_get_current();
-    Error *local_err = NULL;
     uint8_t section_type;
-    unsigned int v;
     int ret;
+    int exitcode = 0;
 
-    if (qemu_savevm_state_blocked(&local_err)) {
-        error_report("%s", error_get_pretty(local_err));
-        error_free(local_err);
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v != QEMU_VM_FILE_MAGIC) {
-        error_report("Not a migration stream");
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        error_report("SaveVM v2 format is obsolete and don't work anymore");
-        return -ENOTSUP;
-    }
-    if (v != QEMU_VM_FILE_VERSION) {
-        error_report("Unsupported migration stream version");
-        return -ENOTSUP;
-    }
-
+    trace_qemu_loadvm_state_main();
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
@@ -1093,16 +1082,14 @@ int qemu_loadvm_state(QEMUFile *f)
             if (se == NULL) {
                 error_report("Unknown savevm section or instance '%s' %d",
                              idstr, instance_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Validate version */
             if (version_id > se->version_id) {
                 error_report("savevm: unsupported version %d for '%s' v%d",
                              version_id, idstr, se->version_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Add entry */
@@ -1117,7 +1104,7 @@ int qemu_loadvm_state(QEMUFile *f)
             if (ret < 0) {
                 error_report("error while loading state for instance 0x%x of"
                              " device '%s'", instance_id, idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_SECTION_PART:
@@ -1132,36 +1119,73 @@ int qemu_loadvm_state(QEMUFile *f)
             }
             if (le == NULL) {
                 error_report("Unknown savevm section %d", section_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
                 error_report("error while loading state section id %d(%s)",
                              section_id, le->se->idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_COMMAND:
             ret = loadvm_process_command(f);
-            if (ret < 0) {
-                goto out;
+            trace_qemu_loadvm_state_section_command(ret);
+            if ((ret < 0) || (ret & LOADVM_QUIT_LOOP)) {
+                return ret;
             }
+            exitcode |= ret; /* Lets us pass flags up to the parent */
             break;
         default:
             error_report("Unknown savevm section type %d", section_type);
-            ret = -EINVAL;
-            goto out;

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1150

Quoted 10 lines
savevm.c +6 -1
+            return -EINVAL;
         }
     }
 
-    cpu_synchronize_all_post_init();
+    if (exitcode & LOADVM_QUIT_PARENT) {
+        trace_qemu_loadvm_state_main_quit_parent();
+        exitcode &= ~LOADVM_QUIT_PARENT;
+        exitcode |= LOADVM_QUIT_LOOP;
+    }

So, if I'm following properly putting a QUIT_PARENT will cause this loop to exit, also returning QUIT_LOOP, so the next loop out also quits. If there was a third lood beyond that it wouldn't quit.

Quoted 4 lines

But are those really the semantics you want; or do you want the options to be "quit one level" and "quit all levels", which seems a little bit simpler. In the current plans you only have the two levels so they're equivalent.

Right, that's all gone - now have a 'quit all levels..

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#59 lore raw

PATCH v5 16/45 Add migration-capability boolean for postcopy-ram.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h | 1 +
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 7 ++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 751caa0..f94af5b 100644
@@ -177,6 +177,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy_ram(void);
 bool migrate_rdma_pin_all(void);
 bool migrate_zero_blocks(void);
 
migration/migration.c +9 -0
index 4592060..434864a 100644
@@ -663,6 +663,15 @@ bool migrate_rdma_pin_all(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
 }
 
+bool migrate_postcopy_ram(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM];
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
qapi-schema.json +6 -1
index e16f8eb..a8af1cb 100644
@@ -494,10 +494,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.3)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
+           'x-postcopy-ram'] }
 
 ##
 # @MigrationCapabilityStatus
Signature · 1 line
2.1.0
#60 lore raw

Re: [Qemu-devel] [PATCH v5 16/45] Add migration-capability boolean for postcopy-ram.

Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This absolutely needs a commit message. I shouldn't have to look at the code to find out what the presence of this capability asserts, and from where to where it's communicating that information.

Quoted 25 lines
---
 include/migration/migration.h | 1 +
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 7 ++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 751caa0..f94af5b 100644
@@ -177,6 +177,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy_ram(void);
 bool migrate_rdma_pin_all(void);
 bool migrate_zero_blocks(void);
 
migration/migration.c +0 -0
index 4592060..434864a 100644

migration/migration.c bool migrate_rdma_pin_all(void) · line 672

Quoted 10 lines
migration/migration.c +7 -0
@@ -663,6 +663,15 @@ bool migrate_rdma_pin_all(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
 }
 
+bool migrate_postcopy_ram(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM];

As an asside, I'm assuming you'll get rid of these "x-" prefixes before you post a series intended for final inclusion?

Quoted 26 lines
migration/migration.c +2 -0
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
qapi-schema.json +6 -1
index e16f8eb..a8af1cb 100644
@@ -494,10 +494,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.3)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
+           'x-postcopy-ram'] }
 
 ##
 # @MigrationCapabilityStatus
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#61 lore raw

Re: [Qemu-devel] [PATCH v5 16/45] Add migration-capability boolean for postcopy-ram.

Quoted 6 lines · * David Gibson ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This absolutely needs a commit message. I shouldn't have to look at the code to find out what the presence of this capability asserts, and from where to where it's communicating that information.

OK, how about:

The 'postcopy ram' flag allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).

Quoted 28 lines
Quoted 28 lines
---
 include/migration/migration.h | 1 +
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 7 ++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 751caa0..f94af5b 100644
@@ -177,6 +177,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy_ram(void);
 bool migrate_rdma_pin_all(void);
 bool migrate_zero_blocks(void);
 
migration/migration.c +0 -0
index 4592060..434864a 100644
@@ -663,6 +663,15 @@ bool migrate_rdma_pin_all(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
 }
 
Quoted 10 lines

migration/migration.c bool migrate_rdma_pin_all(void) · line 672

Quoted 7 lines
migration/migration.c +7 -0
+bool migrate_postcopy_ram(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM];

As an asside, I'm assuming you'll get rid of these "x-" prefixes before you post a series intended for final inclusion?

I was going to do that as a final patch that removed the x-.

Dave

Quoted 25 lines
Quoted 25 lines
migration/migration.c +2 -0
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
qapi-schema.json +6 -1
index e16f8eb..a8af1cb 100644
@@ -494,10 +494,15 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has
+#          been migrated, pulling the remaining pages along as needed. NOTE: If
+#          the migration fails during postcopy the VM will fail.  (since 2.3)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
+           'x-postcopy-ram'] }
 
Quoted 8 lines
Quoted 2 lines
qapi-schema.json +0 -0
 ##
 # @MigrationCapabilityStatus
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#62 lore raw

PATCH v5 17/45 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  15 ++
 include/sysemu/sysemu.h       |  23 +++
 migration/migration.c         |  13 ++
 savevm.c                      | 325 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  11 ++
 5 files changed, 387 insertions(+)
include/migration/migration.h +15 -0
index f94af5b..81cd1f2 100644
@@ -52,6 +52,14 @@ typedef struct MigrationState MigrationState;
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -59,6 +67,8 @@ struct MigrationIncomingState {
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 
+    PostcopyState postcopy_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
@@ -219,4 +229,9 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +23 -0
index 8da879f..d6a6d51 100644
@@ -87,6 +87,18 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+    MIG_CMD_POSTCOPY_END,          /* Postcopy is finished. */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +113,17 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
migration/migration.c +13 -0
index 434864a..957115a 100644
@@ -971,3 +971,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}
+
savevm.c +325 -0

355 lines not on this page · raw message

trace-events +11 -0
index 4ff55fe..050f553 100644
@@ -1171,11 +1171,22 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_postcopy_handle_advise(void) ""
+loadvm_postcopy_handle_end(void) ""
+loadvm_postcopy_handle_listen(void) ""
+loadvm_postcopy_handle_run(void) ""
+loadvm_postcopy_ram_handle_discard(void) ""
+loadvm_postcopy_ram_handle_discard_end(void) ""
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 loadvm_process_command_ping(uint32_t val) "%x"
+qemu_savevm_send_postcopy_advise(void) ""
+qemu_savevm_send_postcopy_ram_discard(void) ""
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_send_ping(uint32_t val) "%x"
+savevm_send_postcopy_end(void) ""
+savevm_send_postcopy_listen(void) ""
+savevm_send_postcopy_run(void) ""
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
Signature · 1 line
2.1.0
#63 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 103 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  15 ++
 include/sysemu/sysemu.h       |  23 +++
 migration/migration.c         |  13 ++
 savevm.c                      | 325 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  11 ++
 5 files changed, 387 insertions(+)
include/migration/migration.h +15 -0
index f94af5b..81cd1f2 100644
@@ -52,6 +52,14 @@ typedef struct MigrationState MigrationState;
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -59,6 +67,8 @@ struct MigrationIncomingState {
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 
+    PostcopyState postcopy_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
@@ -219,4 +229,9 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +23 -0
index 8da879f..d6a6d51 100644
@@ -87,6 +87,18 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+    MIG_CMD_POSTCOPY_END,          /* Postcopy is finished. */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +113,17 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
migration/migration.c +1 -0
index 434864a..957115a 100644
@@ -971,3 +971,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 984

Quoted 10 lines
migration/migration.c +10 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);

Is there anything explaining what the overall atomicity requirements are for this state variable? It's a bit hard to tell if an atomic xchg is necessary or sufficient without a description of what the overall concurrency scheme is with regards to this variable.

Quoted 15 lines
migration/migration.c +2 -0
+}
+
savevm.c +1 -0
index 4b619da..e31ccb0 100644
@@ -39,6 +39,7 @@
 #include "exec/memory.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "qemu/bitops.h"
 #include "qemu/iov.h"
 #include "block/snapshot.h"
 #include "block/qapi.h"

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 644

Quoted 9 lines
savevm.c +6 -0
@@ -635,6 +636,90 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)
     qemu_savevm_command_send(f, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
 }
 
+/* Send prior to any postcopy transfer */
+void qemu_savevm_send_postcopy_advise(QEMUFile *f)
+{
+    uint64_t tmp[2];
+    tmp[0] = cpu_to_be64(getpagesize());
+    tmp[1] = cpu_to_be64(1ul << qemu_target_page_bits());

I wonder if using a structure for the tmp buffer might be an odea, as a form of documentation of the data expected in the ADVISE command.

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 656

Quoted 11 lines
savevm.c +11 -0
+    trace_qemu_savevm_send_postcopy_advise();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_ADVISE, 16, (uint8_t *)tmp);
+}
+
+/* Prior to running, to cause pages that have been dirtied after precopy
+ * started to be discarded on the destination.
+ * CMD_POSTCOPY_RAM_DISCARD consist of:
+ *  3 byte header (filled in by qemu_savevm_send_postcopy_ram_discard)
+ *      byte   version (0)
+ *      byte   offset to be subtracted from each page address to deal with
+ *             RAMBlocks that don't start on a mask word boundary.

I think this needs more explanation. Why can't this be folded into the page addresses sent over the wire?

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 658

Quoted 2 lines
savevm.c +2 -0
+ *      byte   Length of name field
+ *  n x byte   RAM block name (NOT 0 terminated)

I think \0 terminating would probably be safer, even if it's technically redundant.

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 661

Quoted 3 lines
savevm.c +3 -0
+ *  n x
+ *      be64   Page addresses for start of an invalidation range
+ *      be32   mask of 32 pages, '1' to discard'

Is the extra compactness from this semi-sparse bitmap encoding actually worth it? A simple list of page addresses, or address ranges to discard would be substantially simpler to get one's head around, and also seems like it might be more robust against future implementation changes as a wire format.

Quoted 18 lines
savevm.c +18 -0
+ *  Hopefully this is pretty sparse so we don't get too many entries,
+ *  and using the mask should deal with most pagesize differences
+ *  just ending up as a single full mask
+ *
+ * The mask is always 32bits irrespective of the long size
+ *
+ *  name:  RAMBlock name that these entries are part of
+ *  len: Number of page entries
+ *  addrlist: 'len' addresses
+ *  masklist: 'len' masks (corresponding to the addresses)
+ */
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist)
+{
+    uint8_t *buf;
+    uint16_t tmplen;

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 690

Quoted 10 lines
savevm.c +10 -0
+    uint16_t t;
+
+    trace_qemu_savevm_send_postcopy_ram_discard();
+    buf = g_malloc0(len*12 + strlen(name) + 3);
+    buf[0] = 0; /* Version */
+    buf[1] = offset;
+    assert(strlen(name) < 256);
+    buf[2] = strlen(name);
+    memcpy(buf+3, name, strlen(name));
+    tmplen = 3+strlen(name);

Repeated calls to strlen() always seem icky to me, although I guess it's all gcc builtins here, so they are probably optimized out by CSE.

Quoted 13 lines
savevm.c +13 -0
+    for (t = 0; t < len; t++) {
+        cpu_to_be64w((uint64_t *)(buf + tmplen), addrlist[t]);
+        tmplen += 8;
+        cpu_to_be32w((uint32_t *)(buf + tmplen), masklist[t]);
+        tmplen += 4;
+    }
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_RAM_DISCARD, tmplen, buf);
+    g_free(buf);
+}
+
+/* Get the destination into a state where it can receive postcopy data. */
+void qemu_savevm_send_postcopy_listen(QEMUFile *f)
+{

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 714

Quoted 10 lines
savevm.c +10 -0
+    trace_savevm_send_postcopy_listen();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_LISTEN, 0, NULL);
+}
+
+/* Kick the destination into running */
+void qemu_savevm_send_postcopy_run(QEMUFile *f)
+{
+    trace_savevm_send_postcopy_run();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_RUN, 0, NULL);
+}

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 721

Quoted 7 lines
savevm.c +7 -0
+
+/* End of postcopy - with a status byte; 0 is good, anything else is a fail */
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status)
+{
+    trace_savevm_send_postcopy_end();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_END, 1, &status);
+}

What's the distinction between the postcopy END command and the normal end of the migration stream? Is there already a way to detect the end of stream normally?

Quoted 37 lines
savevm.c +30 -0
 bool qemu_savevm_state_blocked(Error **errp)
 {
     SaveStateEntry *se;
@@ -961,6 +1046,212 @@ enum LoadVMExitCodes {
 
 static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
 
+/* ------ incoming postcopy messages ------ */
+/* 'advise' arrives before any transfers just to tell us that a postcopy
+ * *might* happen - it might be skipped if precopy transferred everything
+ * quickly.
+ */
+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
+                                         uint64_t remote_hps,
+                                         uint64_t remote_tps)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_advise();
+    if (ps != POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (remote_hps != getpagesize())  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, getpagesize());
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {

savevm.c enum LoadVMExitCodes { · line 1088

Quoted 10 lines
savevm.c +10 -0
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());
+        return -1;
+    }
+
+    postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

savevm.c enum LoadVMExitCodes { · line 1099

Quoted 10 lines
savevm.c +10 -0
+    return 0;
+}
+
+/* After postcopy we will be told to throw some pages away since they're
+ * dirty and will have to be demand fetched.  Must happen before CPU is
+ * started.
+ * There can be 0..many of these messages, each encoding multiple pages.
+ * Bits set in the message represent a page in the source VMs bitmap, but
+ * since the guest/target page sizes can be different on s/d then we have
+ * to convert.

Uh.. I thought the checks in the ADVISE processing eliminated that possibility.

savevm.c enum LoadVMExitCodes { · line 1111

Quoted 12 lines
savevm.c +12 -0
+ */
+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
+                                              uint16_t len)
+{
+    int tmp;
+    unsigned int first_bit_offset;
+    char ramid[256];
+    PostcopyState ps = postcopy_state_get(mis);
+
+    trace_loadvm_postcopy_ram_handle_discard();
+
+    if (ps != POSTCOPY_INCOMING_ADVISE) {

Could you theoretically also get these in LISTEN state? I realise the current implementation doesn't do that.

Quoted 80 lines
savevm.c +80 -0
+        error_report("CMD_POSTCOPY_RAM_DISCARD in wrong postcopy state (%d)",
+                     ps);
+        return -1;
+    }
+    /* We're expecting a
+     *    3 byte header,
+     *    a RAM ID string
+     *    then at least 1 12 byte chunks
+    */
+    if (len < 16) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+
+    tmp = qemu_get_byte(mis->file);
+    if (tmp != 0) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid version (%d)", tmp);
+        return -1;
+    }
+    first_bit_offset = qemu_get_byte(mis->file);
+
+    if (qemu_get_counted_string(mis->file, ramid)) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD Failed to read RAMBlock ID");
+        return -1;
+    }
+
+    len -= 3+strlen(ramid);
+    if (len % 12) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+    while (len) {
+        uint64_t startaddr;
+        uint32_t mask;
+        /*
+         * We now have pairs of address, mask
+         *   Each word of mask is 32 bits, where each bit corresponds to one
+         *   target page.
+         *   RAMBlocks don't necessarily start on word boundaries,
+         *   and the offset in the header indicates the offset into the 1st
+         *   mask word that corresponds to the 1st page of the RAMBlock.
+         */
+        startaddr = qemu_get_be64(mis->file);
+        mask = qemu_get_be32(mis->file);
+
+        len -= 12;
+
+        while (mask) {
+            /* mask= .....?10...0 */
+            /*             ^fs    */
+            int firstset = ctz32(mask);
+
+            /* tmp32=.....?11...1 */
+            /*             ^fs    */
+            uint32_t tmp32 = mask | ((((uint32_t)1)<<firstset)-1);
+
+            /* mask= .?01..10...0 */
+            /*         ^fz ^fs    */
+            int firstzero = cto32(tmp32);
+
+            if ((startaddr == 0) && (firstset < first_bit_offset)) {
+                error_report("CMD_POSTCOPY_RAM_DISCARD bad data; bit set"
+                               " prior to block; block=%s offset=%d"
+                               " firstset=%d\n", ramid, first_bit_offset,
+                               firstzero);
+                return -1;
+            }
+
+            /*
+             * we know there must be at least 1 bit set due to the loop entry
+             * If there is no 0 firstzero will be 32
+             */
+            /* TODO - ram_discard_range gets added in a later patch
+            int ret = ram_discard_range(mis, ramid,
+                                startaddr + firstset - first_bit_offset,
+                                startaddr + (firstzero - 1) - first_bit_offset);
+            ret = -1;
+            if (ret) {
+                return ret;
+            }

savevm.c enum LoadVMExitCodes { · line 1201

Quoted 10 lines
savevm.c +10 -0
+            */
+
+            /* mask= .?0000000000 */
+            /*         ^fz ^fs    */
+            if (firstzero != 32) {
+                mask &= (((uint32_t)-1) << firstzero);
+            } else {
+                mask = 0;
+            }
+        }

Ugh. Again I ask, are you really sure the semi-sparse wire representation is worth all this hassle?

savevm.c enum LoadVMExitCodes { · line 1208

Quoted 7 lines
savevm.c +7 -0
+    }
+    trace_loadvm_postcopy_ram_handle_discard_end();
+
+    return 0;
+}
+
+/* After this message we must be able to immediately receive postcopy data */

This doesn't quite make sense to me. AFAICT, this is executed on the destination side, so this isn't so much a command to start listening as an assertion that the source side is already listening and ready to receive page requests on the return path.

Quoted 124 lines
savevm.c +80 -0
+static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
+{
+    PostcopyState ps = postcopy_state_set(mis, POSTCOPY_INCOMING_LISTENING);
+    trace_loadvm_postcopy_handle_listen();
+    if (ps != POSTCOPY_INCOMING_ADVISE) {
+        error_report("CMD_POSTCOPY_LISTEN in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    /* TODO start up the postcopy listening thread */
+    return 0;
+}
+
+/* After all discards we can start running and asking for pages */
+static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
+{
+    PostcopyState ps = postcopy_state_set(mis, POSTCOPY_INCOMING_RUNNING);
+    trace_loadvm_postcopy_handle_run();
+    if (ps != POSTCOPY_INCOMING_LISTENING) {
+        error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (autostart) {
+        /* Hold onto your hats, starting the CPU */
+        vm_start();
+    } else {
+        /* leave it paused and let management decide when to start the CPU */
+        runstate_set(RUN_STATE_PAUSED);
+    }
+
+    return 0;
+}
+
+/* The end - with a byte from the source which can tell us to fail. */
+static int loadvm_postcopy_handle_end(MigrationIncomingState *mis)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_end();
+    if (ps == POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_END in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+    return -1; /* TODO - expecting 1 byte good/fail */
+}
+
 static int loadvm_process_command_simple_lencheck(const char *name,
                                                   unsigned int actual,
                                                   unsigned int expected)
@@ -986,6 +1277,7 @@ static int loadvm_process_command(QEMUFile *f)
     uint16_t com;
     uint16_t len;
     uint32_t tmp32;
+    uint64_t tmp64a, tmp64b;
 
     com = qemu_get_be16(f);
     len = qemu_get_be16(f);
@@ -1023,6 +1315,39 @@ static int loadvm_process_command(QEMUFile *f)
         migrate_send_rp_pong(mis, tmp32);
         break;
 
+    case MIG_CMD_POSTCOPY_ADVISE:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_ADVISE",
+                                                   len, 16)) {
+            return -1;
+        }
+        tmp64a = qemu_get_be64(f); /* hps */
+        tmp64b = qemu_get_be64(f); /* tps */
+        return loadvm_postcopy_handle_advise(mis, tmp64a, tmp64b);
+
+    case MIG_CMD_POSTCOPY_LISTEN:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_LISTEN",
+                                                   len, 0)) {
+            return -1;
+        }
+        return loadvm_postcopy_handle_listen(mis);
+
+    case MIG_CMD_POSTCOPY_RUN:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_RUN",
+                                                   len, 0)) {
+            return -1;
+        }
+        return loadvm_postcopy_handle_run(mis);
+
+    case MIG_CMD_POSTCOPY_END:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_END",
+                                                   len, 1)) {
+            return -1;
+        }
+        return loadvm_postcopy_handle_end(mis);
+
+    case MIG_CMD_POSTCOPY_RAM_DISCARD:
+        return loadvm_postcopy_ram_handle_discard(mis, len);
+
     default:
         error_report("VM_COMMAND 0x%x unknown (len 0x%x)", com, len);
         return -1;
trace-events +11 -0
index 4ff55fe..050f553 100644
@@ -1171,11 +1171,22 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+loadvm_postcopy_handle_advise(void) ""
+loadvm_postcopy_handle_end(void) ""
+loadvm_postcopy_handle_listen(void) ""
+loadvm_postcopy_handle_run(void) ""
+loadvm_postcopy_ram_handle_discard(void) ""
+loadvm_postcopy_ram_handle_discard_end(void) ""
 loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d"
 loadvm_process_command_ping(uint32_t val) "%x"
+qemu_savevm_send_postcopy_advise(void) ""
+qemu_savevm_send_postcopy_ram_discard(void) ""
 savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"
 savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d"
 savevm_send_ping(uint32_t val) "%x"
+savevm_send_postcopy_end(void) ""
+savevm_send_postcopy_listen(void) ""
+savevm_send_postcopy_run(void) ""
 savevm_state_begin(void) ""
 savevm_state_header(void) ""
 savevm_state_iterate(void) ""
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#64 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

(Only replying to some of the items in this mail - the others I'll get to another time).

Quoted 111 lines · * David Gibson ([email protected]) wrote:
Quoted 110 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  15 ++
 include/sysemu/sysemu.h       |  23 +++
 migration/migration.c         |  13 ++
 savevm.c                      | 325 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  11 ++
 5 files changed, 387 insertions(+)
include/migration/migration.h +15 -0
index f94af5b..81cd1f2 100644
@@ -52,6 +52,14 @@ typedef struct MigrationState MigrationState;
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -59,6 +67,8 @@ struct MigrationIncomingState {
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 
+    PostcopyState postcopy_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
@@ -219,4 +229,9 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +23 -0
index 8da879f..d6a6d51 100644
@@ -87,6 +87,18 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+    MIG_CMD_POSTCOPY_END,          /* Postcopy is finished. */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +113,17 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
migration/migration.c +8 -0
index 434864a..957115a 100644
@@ -971,3 +971,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
Quoted 8 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 984

Quoted 3 lines
migration/migration.c +3 -0
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);

Is there anything explaining what the overall atomicity requirements are for this state variable? It's a bit hard to tell if an atomic xchg is necessary or sufficient without a description of what the overall concurrency scheme is with regards to this variable.

Can you tell me how to define the requirements?
It's a state variable tested and changed by at least two threads and it's got to go through a correct sequence of states.
So generally you're doing a 'I expect to be in .... now change to ....' so the exchange works well for that.

Quoted 9 lines

migration/migration.c void migrate_fd_connect(MigrationState *s)

Quoted 3 lines
migration/migration.c +3 -0
+ *  n x
+ *      be64   Page addresses for start of an invalidation range
+ *      be32   mask of 32 pages, '1' to discard'

Is the extra compactness from this semi-sparse bitmap encoding actually worth it? A simple list of page addresses, or address ranges to discard would be substantially simpler to get one's head around, and also seems like it might be more robust against future implementation changes as a wire format.

As previously discussed I really think it is; what I'm tending to see when I've been looking at these in debug is something that's sparse but tends to be blobby with sets of pages discarded near by. However you do this you're going to have to walk this bitmap and format out some sort of set of messages.

Quoted 24 lines
Quoted 24 lines
+ *  Hopefully this is pretty sparse so we don't get too many entries,
+ *  and using the mask should deal with most pagesize differences
+ *  just ending up as a single full mask
+ *
+ * The mask is always 32bits irrespective of the long size
+ *
+ *  name:  RAMBlock name that these entries are part of
+ *  len: Number of page entries
+ *  addrlist: 'len' addresses
+ *  masklist: 'len' masks (corresponding to the addresses)

+ */

migration/migration.c +13 -0
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist)
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;
+
+    trace_qemu_savevm_send_postcopy_ram_discard();
+    buf = g_malloc0(len*12 + strlen(name) + 3);
+    buf[0] = 0; /* Version */
+    buf[1] = offset;
Quoted 8 lines

migration/migration.c

Quoted 4 lines
migration/migration.c +4 -0
+    assert(strlen(name) < 256);
+    buf[2] = strlen(name);
+    memcpy(buf+3, name, strlen(name));
+    tmplen = 3+strlen(name);

Repeated calls to strlen() always seem icky to me, although I guess it's all gcc builtins here, so they are probably optimized out by CSE.

Yeh, those are now gone. Thanks.

Quoted 18 lines
Quoted 18 lines
migration/migration.c +18 -0
+    for (t = 0; t < len; t++) {
+        cpu_to_be64w((uint64_t *)(buf + tmplen), addrlist[t]);
+        tmplen += 8;
+        cpu_to_be32w((uint32_t *)(buf + tmplen), masklist[t]);
+        tmplen += 4;
+    }
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_RAM_DISCARD, tmplen, buf);
+    g_free(buf);
+}
+
+/* Get the destination into a state where it can receive postcopy data. */
+void qemu_savevm_send_postcopy_listen(QEMUFile *f)
+{
+    trace_savevm_send_postcopy_listen();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_LISTEN, 0, NULL);
+}
+
+/* Kick the destination into running */
Quoted 8 lines

migration/migration.c

Quoted 5 lines
migration/migration.c +5 -0
+void qemu_savevm_send_postcopy_run(QEMUFile *f)
+{
+    trace_savevm_send_postcopy_run();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_RUN, 0, NULL);
+}

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

Discard data can be quite large, so I potentially send multiple discard commands.
(Also as you can tell generally I've got a preference for one message does one thing, and thus I have tried to keep them separate).

Quoted 11 lines

migration/migration.c

Quoted 7 lines
migration/migration.c +7 -0
+
+/* End of postcopy - with a status byte; 0 is good, anything else is a fail */
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status)
+{
+    trace_savevm_send_postcopy_end();
+    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_END, 1, &status);
+}

What's the distinction between the postcopy END command and the normal end of the migration stream? Is there already a way to detect the end of stream normally?

OK, thanks I've killed that off.
The short answer is that the 'end of migration stream' is just a terminating byte and I was hoping to put something better on there with a status from the source side; but that fix can be a general fix and doesn't need to be postcopy.

Quoted 53 lines
Quoted 38 lines

bool qemu_savevm_state_blocked(Error **errp)

 {
     SaveStateEntry *se;
migration/migration.c +30 -0
@@ -961,6 +1046,212 @@ enum LoadVMExitCodes {
 
 static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
 
+/* ------ incoming postcopy messages ------ */
+/* 'advise' arrives before any transfers just to tell us that a postcopy
+ * *might* happen - it might be skipped if precopy transferred everything
+ * quickly.
+ */
+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
+                                         uint64_t remote_hps,
+                                         uint64_t remote_tps)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_advise();
+    if (ps != POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (remote_hps != getpagesize())  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, getpagesize());
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {

migration/migration.c enum LoadVMExitCodes { · line 1088

Quoted 10 lines
migration/migration.c +10 -0
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());
+        return -1;
+    }
+
+    postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Quoted 2 lines

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

There's no race at this point yet; going from None->advise we still only have one thread. The check at the top is a check against protocol violations (e.g. getting two advise or something like that).

Quoted 4 lines
Quoted 4 lines
migration/migration.c +4 -0
+    return 0;
+}
+
+/* After postcopy we will be told to throw some pages away since they're
Quoted 8 lines

migration/migration.c enum LoadVMExitCodes { · line 1098

Quoted 6 lines
migration/migration.c +6 -0
+ * dirty and will have to be demand fetched.  Must happen before CPU is
+ * started.
+ * There can be 0..many of these messages, each encoding multiple pages.
+ * Bits set in the message represent a page in the source VMs bitmap, but
+ * since the guest/target page sizes can be different on s/d then we have
+ * to convert.

Uh.. I thought the checks in the ADVISE processing eliminated that possibility.

Old message; I was originally trying to keep it more general, but ripped the conversions out when it became just too messy for now. Gone.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#65 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 124 lines · Dr. David Alan Gilbert

(Only replying to some of the items in this mail - the others I'll get to another time).

Quoted 111 lines · * David Gibson ([email protected]) wrote:
Quoted 110 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  15 ++
 include/sysemu/sysemu.h       |  23 +++
 migration/migration.c         |  13 ++
 savevm.c                      | 325 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  11 ++
 5 files changed, 387 insertions(+)
include/migration/migration.h +15 -0
index f94af5b..81cd1f2 100644
@@ -52,6 +52,14 @@ typedef struct MigrationState MigrationState;
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -59,6 +67,8 @@ struct MigrationIncomingState {
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 
+    PostcopyState postcopy_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
@@ -219,4 +229,9 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +23 -0
index 8da879f..d6a6d51 100644
@@ -87,6 +87,18 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+    MIG_CMD_POSTCOPY_END,          /* Postcopy is finished. */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +113,17 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
migration/migration.c +8 -0
index 434864a..957115a 100644
@@ -971,3 +971,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
Quoted 8 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 984

Quoted 3 lines
migration/migration.c +3 -0
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);

Is there anything explaining what the overall atomicity requirements are for this state variable? It's a bit hard to tell if an atomic xchg is necessary or sufficient without a description of what the overall concurrency scheme is with regards to this variable.

Quoted 1 line

Can you tell me how to define the requirements?

Well, that is always the tricky question.

Quoted 4 lines

It's a state variable tested and changed by at least two threads and it's got to go through a correct sequence of states.
So generally you're doing a 'I expect to be in .... now change to ....' so the exchange works well for that.

So.. in this case. It seems what might make sense as a basic atomicity option here is a cmpxchg. So your state_set function takes old_state and new_state. It will atomically update old_state to new_state, returning an error if somebody else changed the state away from old_state in the meantime.

Then you can have a blurb next to the state_set helper explaining why you need this atomic op for updates to the state variable from multiple threads.

Quoted 10 lines
Quoted 9 lines

migration/migration.c void migrate_fd_connect(MigrationState *s)

Quoted 3 lines
migration/migration.c +3 -0
+ *  n x
+ *      be64   Page addresses for start of an invalidation range
+ *      be32   mask of 32 pages, '1' to discard'

Is the extra compactness from this semi-sparse bitmap encoding actually worth it? A simple list of page addresses, or address ranges to discard would be substantially simpler to get one's head around, and also seems like it might be more robust against future implementation changes as a wire format.

Quoted 5 lines

As previously discussed I really think it is; what I'm tending to see when I've been looking at these in debug is something that's sparse but tends to be blobby with sets of pages discarded near by. However you do this you're going to have to walk this bitmap and format out some sort of set of messages.

I'd really need to see some numbers to be convinced. The semi-sparse bitmap introduces a huge amount of code and complexity at both ends.

Remember, using ranges, you can still coalesce adjacent discarded pages. Even using some compat differential encodings of the range endpoints might work out simpler than the bitmap fragments.

Plus, ranges handle the host versus target page size distinctions very naturally.

[snip]

Quoted 7 lines
Quoted 2 lines

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

Discard data can be quite large, so I potentially send multiple discard commands.
(Also as you can tell generally I've got a preference for one message does one thing, and thus I have tried to keep them separate).

So, I like the idea of one message per action in principle - but only if that action really is well-defined without reference to what operations come before and after it. If there are hidden dependencies about what actions have to come in what order, I'd rather bake that into the command structure.

Quoted 40 lines
Quoted 40 lines
Quoted 25 lines
migration/migration.c +25 -0
+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
+                                         uint64_t remote_hps,
+                                         uint64_t remote_tps)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_advise();
+    if (ps != POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (remote_hps != getpagesize())  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, getpagesize());
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {

migration/migration.c

Quoted 10 lines
migration/migration.c +10 -0
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());
+        return -1;
+    }
+
+    postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Quoted 6 lines
Quoted 2 lines

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

There's no race at this point yet; going from None->advise we still only have one thread. The check at the top is a check against protocol violations (e.g. getting two advise or something like that).

Yeah.. and having to have intimate knowledge of the thread structure at each point in order to analyze the atomic operation correctness is exactly what bothers me about it.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#66 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 140 lines · * David Gibson ([email protected]) wrote:
Quoted 124 lines · Dr. David Alan Gilbert

(Only replying to some of the items in this mail - the others I'll get to another time).

Quoted 111 lines · * David Gibson ([email protected]) wrote:
Quoted 110 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

---
 include/migration/migration.h |  15 ++
 include/sysemu/sysemu.h       |  23 +++
 migration/migration.c         |  13 ++
 savevm.c                      | 325 ++++++++++++++++++++++++++++++++++++++++++
 trace-events                  |  11 ++
 5 files changed, 387 insertions(+)
include/migration/migration.h +15 -0
index f94af5b..81cd1f2 100644
@@ -52,6 +52,14 @@ typedef struct MigrationState MigrationState;
 
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
+typedef enum {
+    POSTCOPY_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+    POSTCOPY_INCOMING_ADVISE,
+    POSTCOPY_INCOMING_LISTENING,
+    POSTCOPY_INCOMING_RUNNING,
+    POSTCOPY_INCOMING_END
+} PostcopyState;
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -59,6 +67,8 @@ struct MigrationIncomingState {
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
 
+    PostcopyState postcopy_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
@@ -219,4 +229,9 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+PostcopyState postcopy_state_get(MigrationIncomingState *mis);
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state);
 #endif
include/sysemu/sysemu.h +23 -0
index 8da879f..d6a6d51 100644
@@ -87,6 +87,18 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+
+    MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
+                                      warn we might want to do PC */
+    MIG_CMD_POSTCOPY_LISTEN,       /* Start listening for incoming
+                                      pages as it's running. */
+    MIG_CMD_POSTCOPY_RUN,          /* Start execution */
+    MIG_CMD_POSTCOPY_END,          /* Postcopy is finished. */
+
+    MIG_CMD_POSTCOPY_RAM_DISCARD,  /* A list of pages to discard that
+                                      were previously sent during
+                                      precopy but are dirty. */
+
 };
 
 bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +113,17 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_postcopy_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_end(QEMUFile *f, uint8_t status);
+
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
migration/migration.c +8 -0
index 434864a..957115a 100644
@@ -971,3 +971,16 @@ void migrate_fd_connect(MigrationState *s)
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
 }
+
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
Quoted 8 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 984

Quoted 3 lines
migration/migration.c +3 -0
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);

Is there anything explaining what the overall atomicity requirements are for this state variable? It's a bit hard to tell if an atomic xchg is necessary or sufficient without a description of what the overall concurrency scheme is with regards to this variable.

Quoted 1 line

Can you tell me how to define the requirements?

Well, that is always the tricky question.

Quoted 4 lines

It's a state variable tested and changed by at least two threads and it's got to go through a correct sequence of states.
So generally you're doing a 'I expect to be in .... now change to ....' so the exchange works well for that.

So.. in this case. It seems what might make sense as a basic atomicity option here is a cmpxchg. So your state_set function takes old_state and new_state. It will atomically update old_state to new_state, returning an error if somebody else changed the state away from old_state in the meantime.

Quoted 3 lines

Then you can have a blurb next to the state_set helper explaining why you need this atomic op for updates to the state variable from multiple threads.

I can document that next to state_set, and I'll look at whether it's possible to move the error check down to there.

Quoted 19 lines
Quoted 10 lines
Quoted 9 lines

migration/migration.c void migrate_fd_connect(MigrationState *s)

Quoted 3 lines
migration/migration.c +3 -0
+ *  n x
+ *      be64   Page addresses for start of an invalidation range
+ *      be32   mask of 32 pages, '1' to discard'

Is the extra compactness from this semi-sparse bitmap encoding actually worth it? A simple list of page addresses, or address ranges to discard would be substantially simpler to get one's head around, and also seems like it might be more robust against future implementation changes as a wire format.

Quoted 5 lines

As previously discussed I really think it is; what I'm tending to see when I've been looking at these in debug is something that's sparse but tends to be blobby with sets of pages discarded near by. However you do this you're going to have to walk this bitmap and format out some sort of set of messages.

I'd really need to see some numbers to be convinced. The semi-sparse bitmap introduces a huge amount of code and complexity at both ends.

Quoted 6 lines

Remember, using ranges, you can still coalesce adjacent discarded pages. Even using some compat differential encodings of the range endpoints might work out simpler than the bitmap fragments.

Plus, ranges handle the host versus target page size distinctions very naturally.

I guess I'm going to have to implement this and compare then; what format do you want?

Quoted 9 lines

[snip]

Quoted 7 lines
Quoted 2 lines

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

Discard data can be quite large, so I potentially send multiple discard commands.
(Also as you can tell generally I've got a preference for one message does one thing, and thus I have tried to keep them separate).

Quoted 5 lines

So, I like the idea of one message per action in principle - but only if that action really is well-defined without reference to what operations come before and after it. If there are hidden dependencies about what actions have to come in what order, I'd rather bake that into the command structure.

In no way is it hidden; the commands match the state transitions.

Quoted 43 lines
Quoted 43 lines
Quoted 42 lines
Quoted 25 lines
migration/migration.c +25 -0
+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
+                                         uint64_t remote_hps,
+                                         uint64_t remote_tps)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_advise();
+    if (ps != POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (remote_hps != getpagesize())  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, getpagesize());
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {

migration/migration.c

Quoted 10 lines
migration/migration.c +10 -0
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());
+        return -1;
+    }
+
+    postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

Quoted 7 lines
Quoted 3 lines

There's no race at this point yet; going from None->advise we still only have one thread. The check at the top is a check against protocol violations (e.g. getting two advise or something like that).

Yeah.. and having to have intimate knowledge of the thread structure at each point in order to analyze the atomic operation correctness is exactly what bothers me about it.

No, you don't; By always using the postcopy_state_set you don't need to worry about the thread structure or protocol to understand the atomic operation correctness. The only reason you got into that comparison is because you worried that it might be overkill in this case; by keeping it consistent like it already is then you don't need to understand the thread structure.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#67 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 7 lines · Dr. David Alan Gilbert
Quoted 5 lines

So, I like the idea of one message per action in principle - but only if that action really is well-defined without reference to what operations come before and after it. If there are hidden dependencies about what actions have to come in what order, I'd rather bake that into the command structure.

In no way is it hidden; the commands match the state transitions.

So is there only one writer thread? In which states can the other threads read the postcopy state, and why is sequential consistency important? That is, in threads other than the writer, how does reading the state interact with other reads and writes?

Paolo

#68 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert
Quoted 4 lines · Dr. David Alan Gilbert
Quoted 3 lines · * David Gibson ([email protected]) wrote:
Quoted 2 lines · Dr. David Alan Gilbert

(Only replying to some of the items in this mail - the others I'll get to another time).

[snip]

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

Discard data can be quite large, so I potentially send multiple discard commands.
(Also as you can tell generally I've got a preference for one message does one thing, and thus I have tried to keep them separate).

Quoted 7 lines
Quoted 5 lines

So, I like the idea of one message per action in principle - but only if that action really is well-defined without reference to what operations come before and after it. If there are hidden dependencies about what actions have to come in what order, I'd rather bake that into the command structure.

In no way is it hidden; the commands match the state transitions.

Not all of them. In particular DISCARD's relation to state transitions is not at all obvious. Likewise I don't think the connection of LISTEN to the internal state machines at each end is terribly clear.

Quoted 51 lines
Quoted 43 lines
Quoted 43 lines
Quoted 42 lines
Quoted 25 lines
savevm.c +25 -0
+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
+                                         uint64_t remote_hps,
+                                         uint64_t remote_tps)
+{
+    PostcopyState ps = postcopy_state_get(mis);
+    trace_loadvm_postcopy_handle_advise();
+    if (ps != POSTCOPY_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    if (remote_hps != getpagesize())  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, getpagesize());
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {

savevm.c enum LoadVMExitCodes { · line 1088

Quoted 10 lines
savevm.c +10 -0
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());
+        return -1;
+    }
+
+    postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

Quoted 7 lines
Quoted 3 lines

There's no race at this point yet; going from None->advise we still only have one thread. The check at the top is a check against protocol violations (e.g. getting two advise or something like that).

Yeah.. and having to have intimate knowledge of the thread structure at each point in order to analyze the atomic operation correctness is exactly what bothers me about it.

Quoted 5 lines

No, you don't; By always using the postcopy_state_set you don't need to worry about the thread structure or protocol to understand the atomic operation correctness. The only reason you got into that comparison is because you worried that it might be overkill in this case; by keeping it consistent like it already is then you don't need to understand the thread structure.

You really do, though. You may be using the same function in the original, but not the same idiom: in this case you don't check the return value and deal with it accordingly.

Because of that, this looks pretty much exactly like the classic "didn't understand what was atomic in the atomic" race condition, and you need to know that there's only one thread at this point to realize it's correct after all.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#69 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to David Gibson
Quoted 11 lines · David Gibson
Quoted 4 lines

It's a state variable tested and changed by at least two threads and it's got to go through a correct sequence of states. So generally you're doing a 'I expect to be in .... now change to ....' so the exchange works well for that.

So.. in this case. It seems what might make sense as a basic atomicity option here is a cmpxchg. So your state_set function takes old_state and new_state. It will atomically update old_state to new_state, returning an error if somebody else changed the state away from old_state in the meantime.

Quoted 3 lines

Then you can have a blurb next to the state_set helper explaining why you need this atomic op for updates to the state variable from multiple threads.

I agree.

My humble suggestion is:

- document the transitions, see this existing example in thread-pool.c:

    /* Moving state out of THREAD_QUEUED is protected by lock.  After
     * that, only the worker thread can write to it.  Reads and writes
     * of state and ret are ordered with memory barriers.
     */
    enum ThreadState state;

The outgoing migration state fails to document this. My fault.

- perhaps you can use a pattern similar to what is used for the outgoing migration state. There one thread does the work, and the second (could be "the others", e.g. all VCPU threads) affect the first just by triggering changes in the migration state. Transitions in the first thread use cmpxchg without a loop---if it fails it means the second thread intervened, and the thread moves on to do whatever the second thread requested. Transition in the second thread use cmpxchg within a loop, in case there was a concurrent change.

Paolo

Quoted 87 lines
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

+ * n x + * be64 Page addresses for start of an
invalidation range + * be32 mask of 32 pages, '1' to
discard'

Is the extra compactness from this semi-sparse bitmap encoding actually worth it? A simple list of page addresses, or address ranges to discard would be substantially simpler to get one's head around, and also seems like it might be more robust against future implementation changes as a wire format.

Quoted 5 lines

As previously discussed I really think it is; what I'm tending to see when I've been looking at these in debug is something that's sparse but tends to be blobby with sets of pages discarded near by. However you do this you're going to have to walk this bitmap and format out some sort of set of messages.

I'd really need to see some numbers to be convinced. The semi-sparse bitmap introduces a huge amount of code and complexity at both ends.

Remember, using ranges, you can still coalesce adjacent discarded pages. Even using some compat differential encodings of the range endpoints might work out simpler than the bitmap fragments.

Plus, ranges handle the host versus target page size distinctions very naturally.

[snip]

Quoted 8 lines
Quoted 3 lines

DISCARD will typically immediately precede LISTEN, won't it? Is there a reason not to put the discard data into the LISTEN command?

Discard data can be quite large, so I potentially send multiple discard commands. (Also as you can tell generally I've got a preference for one message does one thing, and thus I have tried to keep them separate).

So, I like the idea of one message per action in principle - but only if that action really is well-defined without reference to what operations come before and after it. If there are hidden dependencies about what actions have to come in what order, I'd rather bake that into the command structure.

Quoted 31 lines
Quoted 31 lines
Quoted 17 lines

+static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, + uint64_t remote_hps, +
uint64_t remote_tps) +{ + PostcopyState ps =
postcopy_state_get(mis); +
trace_loadvm_postcopy_handle_advise(); + if (ps !=
POSTCOPY_INCOMING_NONE) { + error_report("CMD_POSTCOPY_ADVISE in wrong postcopy state
(%d)", ps); + return -1; + } + + if (remote_hps
!= getpagesize()) { + /* + * Some
combinations of mismatch are probably possible but it gets +
* a bit more complicated. In particular we need to place
whole + * host pages on the dest at once, and we need
to ensure that we + * handle dirtying to make sure we
never end up sending part of + * a hostpage on it's
own. + */ + error_report("Postcopy needs
matching host page sizes (s=%d d=%d)", +

Quoted 8 lines

(int)remote_hps, getpagesize()); + return -1; + }
+ + if (remote_tps != (1ul << qemu_target_page_bits())) {
+ /* + * Again, some differences could be
dealt with, but for now keep it + * simple. +
*/ + error_report("Postcopy needs matching target page
sizes (s=%d d=%d)", + (int)remote_tps, 1
<< qemu_target_page_bits()); + return -1; + } + +
postcopy_state_set(mis, POSTCOPY_INCOMING_ADVISE);

Should you be checking the return value here to make sure it's still POSTCOPY_INCOMING_NONE? Atomic xchgs seem overkill if you still have a race between the fetch at the top and the set here.

Quoted 8 lines
Quoted 3 lines

Or, in fact, should you be just doing an atomic exchange-then-check at the top, rather than checking at the top, then changing at the bottom.

There's no race at this point yet; going from None->advise we still only have one thread. The check at the top is a check against protocol violations (e.g. getting two advise or something like that).

Yeah.. and having to have intimate knowledge of the thread structure at each point in order to analyze the atomic operation correctness is exactly what bothers me about it.

#70 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert (git)

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 985

Quoted 11 lines · Dr. David Alan Gilbert (git)
migration/migration.c +11 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}

Which are the (multiple) threads are calling these functions?

Paolo

#71 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 16 lines · * Paolo Bonzini ([email protected]) wrote:

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 985

Quoted 11 lines · Dr. David Alan Gilbert (git)
migration/migration.c +11 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}

Which are the (multiple) threads are calling these functions?

The main thread receiving the migration and the postcopy ram_listen_thread receiving the RAM pages.
It's not actually racy between multiple threads updating it, it's sequenced so that the main thread initialises it and then hands over to the listen thread that takes it over from that point.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#72 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 13 lines · Dr. David Alan Gilbert
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
migration/migration.c +5 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 985

Quoted 6 lines
migration/migration.c +6 -0
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}

Which are the (multiple) threads are calling these functions?

Quoted 5 lines

The main thread receiving the migration and the postcopy ram_listen_thread receiving the RAM pages.
It's not actually racy between multiple threads updating it, it's sequenced so that the main thread initialises it and then hands over to the listen thread that takes it over from that point.

I would use atomic_mb_read/atomic_mb_set here.

Paolo

#73 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 16 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 985

Quoted 11 lines
migration/migration.c +11 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}

Which are the (multiple) threads are calling these functions?

Quoted 7 lines
Quoted 5 lines

The main thread receiving the migration and the postcopy ram_listen_thread receiving the RAM pages.
It's not actually racy between multiple threads updating it, it's sequenced so that the main thread initialises it and then hands over to the listen thread that takes it over from that point.

I would use atomic_mb_read/atomic_mb_set here.

The benefit of the atomic_xchg is that I get the old value back so that I can sanity check I was in the state I thought I should have been.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#74 lore raw

Re: [Qemu-devel] [PATCH v5 17/45] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 23 lines · Dr. David Alan Gilbert
Quoted 22 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
migration/migration.c +5 -0
+PostcopyState  postcopy_state_get(MigrationIncomingState *mis)
+{
+    return atomic_fetch_add(&mis->postcopy_state, 0);
+}
+
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

migration/migration.c void migrate_fd_connect(MigrationState *s) · line 985

Quoted 6 lines
migration/migration.c +6 -0
+/* Set the state and return the old state */
+PostcopyState postcopy_state_set(MigrationIncomingState *mis,
+                                 PostcopyState new_state)
+{
+    return atomic_xchg(&mis->postcopy_state, new_state);
+}

Which are the (multiple) threads are calling these functions?

The main thread receiving the migration and the postcopy ram_listen_thread receiving the RAM pages.
It's not actually racy between multiple threads updating it, it's sequenced so that the main thread initialises it and then hands over to the listen thread that takes it over from that point.

Quoted 5 lines
Quoted 1 line

I would use atomic_mb_read/atomic_mb_set here.

The benefit of the atomic_xchg is that I get the old value back so that I can sanity check I was in the state I thought I should have been.

Ok, you can still pair atomic_xchg with atomic_mb_read, it's a bit cheaper.

Paolo

#75 lore raw

PATCH v5 18/45 MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

---
 include/sysemu/sysemu.h |  4 +++
 savevm.c                | 82 +++++++++++++++++++++++++++++++++++++++++++++++++
 trace-events            |  4 +++
 3 files changed, 90 insertions(+)
include/sysemu/sysemu.h +4 -0
index d6a6d51..e83bf80 100644
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */
 
     MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
                                       warn we might want to do PC */
@@ -101,6 +102,8 @@ enum qemu_vm_cmd {
 
 };
 
+#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24)
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -113,6 +116,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb);
 void qemu_savevm_send_postcopy_advise(QEMUFile *f);
 void qemu_savevm_send_postcopy_listen(QEMUFile *f);
 void qemu_savevm_send_postcopy_run(QEMUFile *f);
savevm.c +82 -0

100 lines not on this page · raw message

trace-events +4 -0
index 050f553..cbf995c 100644
@@ -1171,6 +1171,10 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+qemu_savevm_send_packaged(void) ""
+loadvm_handle_cmd_packaged(unsigned int length) "%u"
+loadvm_handle_cmd_packaged_main(int ret) "%d"
+loadvm_handle_cmd_packaged_received(int ret) "%d"
 loadvm_postcopy_handle_advise(void) ""
 loadvm_postcopy_handle_end(void) ""
 loadvm_postcopy_handle_listen(void) ""
Signature · 1 line
2.1.0
#76 lore raw

Re: [Qemu-devel] [PATCH v5 18/45] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

Hrm. I'd be more comfortable if the semantics of CMD_PACKAGED were defined in terms of visible effects on the other end, rather than in terms of how it's implemented internally.

Quoted 2 lines

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Which makes the above paragraph a bit misleading - the whole point here is that loading the package data *does* interact with the migration stream - just that it's the migration stream after the end of the package.

Quoted 88 lines
---
 include/sysemu/sysemu.h |  4 +++
 savevm.c                | 82 +++++++++++++++++++++++++++++++++++++++++++++++++
 trace-events            |  4 +++
 3 files changed, 90 insertions(+)
include/sysemu/sysemu.h +4 -0
index d6a6d51..e83bf80 100644
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */
 
     MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
                                       warn we might want to do PC */
@@ -101,6 +102,8 @@ enum qemu_vm_cmd {
 
 };
 
+#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24)
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -113,6 +116,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb);
 void qemu_savevm_send_postcopy_advise(QEMUFile *f);
 void qemu_savevm_send_postcopy_listen(QEMUFile *f);
 void qemu_savevm_send_postcopy_run(QEMUFile *f);
savevm.c +37 -0
index e31ccb0..f65bff3 100644
@@ -636,6 +636,38 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)
     qemu_savevm_command_send(f, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
 }
 
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
+ */
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb)
+{
+    size_t cur_iov;
+    size_t len = qsb_get_length(qsb);
+    uint32_t tmp;
+
+    tmp = cpu_to_be32(len);
+
+    trace_qemu_savevm_send_packaged();
+    qemu_savevm_command_send(f, MIG_CMD_PACKAGED, 4, (uint8_t *)&tmp);
+
+    /* all the data follows (concatinating the iov's) */
+    for (cur_iov = 0; cur_iov < qsb->n_iov; cur_iov++) {
+        /* The iov entries are partially filled */
+        size_t towrite = (qsb->iov[cur_iov].iov_len > len) ?
+                              len :
+                              qsb->iov[cur_iov].iov_len;
+        len -= towrite;
+
+        if (!towrite) {
+            break;
+        }
+
+        qemu_put_buffer(f, qsb->iov[cur_iov].iov_base, towrite);
+    }
+}
+
 /* Send prior to any postcopy transfer */
 void qemu_savevm_send_postcopy_advise(QEMUFile *f)
 {
@@ -1265,6 +1297,48 @@ static int loadvm_process_command_simple_lencheck(const char *name,
     return 0;
 }
 
+/* Immediately following this command is a blob of data containing an embedded
+ * chunk of migration stream; read it and load it.
+ */
+static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis,
+                                      uint32_t length)

savevm.c static int loadvm_process_command_simple_lencheck(const char *name, · line 1314

Quoted 10 lines
savevm.c +10 -0
+{
+    int ret;
+    uint8_t *buffer;
+    QEMUSizedBuffer *qsb;
+
+    trace_loadvm_handle_cmd_packaged(length);
+
+    if (length > MAX_VM_CMD_PACKAGED_SIZE) {
+        error_report("Unreasonably large packaged state: %u", length);
+        return -1;

It would be a good idea to check this on the send side as well as receive, wouldn't it?

Quoted 60 lines
savevm.c +35 -0
+    }
+    buffer = g_malloc0(length);
+    ret = qemu_get_buffer(mis->file, buffer, (int)length);
+    if (ret != length) {
+        g_free(buffer);
+        error_report("CMD_PACKAGED: Buffer receive fail ret=%d length=%d\n",
+                ret, length);
+        return (re/t < 0) ? ret : -EAGAIN;
+    }
+    trace_loadvm_handle_cmd_packaged_received(ret);
+
+    /* Setup a dummy QEMUFile that actually reads from the buffer */
+    qsb = qsb_create(buffer, length);
+    g_free(buffer); /* Because qsb_create copies */
+    if (!qsb) {
+        error_report("Unable to create qsb");
+    }
+    QEMUFile *packf = qemu_bufopen("r", qsb);
+
+    ret = qemu_loadvm_state_main(packf, mis);
+    trace_loadvm_handle_cmd_packaged_main(ret);
+    qemu_fclose(packf);
+    qsb_free(qsb);
+
+    return ret;
+}
+
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
@@ -1315,6 +1389,14 @@ static int loadvm_process_command(QEMUFile *f)
         migrate_send_rp_pong(mis, tmp32);
         break;
 
+    case MIG_CMD_PACKAGED:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_PACKAGED",
+            len, 4)) {
+            return -1;
+         }
+        tmp32 = qemu_get_be32(f);
+        return loadvm_handle_cmd_packaged(mis, tmp32);
+
     case MIG_CMD_POSTCOPY_ADVISE:
         if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_ADVISE",
                                                    len, 16)) {
trace-events +4 -0
index 050f553..cbf995c 100644
@@ -1171,6 +1171,10 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+qemu_savevm_send_packaged(void) ""
+loadvm_handle_cmd_packaged(unsigned int length) "%u"
+loadvm_handle_cmd_packaged_main(int ret) "%d"
+loadvm_handle_cmd_packaged_received(int ret) "%d"
 loadvm_postcopy_handle_advise(void) ""
 loadvm_postcopy_handle_end(void) ""
 loadvm_postcopy_handle_listen(void) ""
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#77 lore raw

Re: [Qemu-devel] [PATCH v5 18/45] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

Hrm. I'd be more comfortable if the semantics of CMD_PACKAGED were defined in terms of visible effects on the other end, rather than in terms of how it's implemented internally.

Quoted 7 lines
Quoted 2 lines

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Which makes the above paragraph a bit misleading - the whole point here is that loading the package data *does* interact with the migration stream - just that it's the migration stream after the end of the package.

Hmm, how about:

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

Quoted 93 lines
Quoted 93 lines
---
 include/sysemu/sysemu.h |  4 +++
 savevm.c                | 82 +++++++++++++++++++++++++++++++++++++++++++++++++
 trace-events            |  4 +++
 3 files changed, 90 insertions(+)
include/sysemu/sysemu.h +4 -0
index d6a6d51..e83bf80 100644
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {
     MIG_CMD_INVALID = 0,       /* Must be 0 */
     MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
     MIG_CMD_PING,              /* Request a PONG on the RP */
+    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */
 
     MIG_CMD_POSTCOPY_ADVISE = 20,  /* Prior to any page transfers, just
                                       warn we might want to do PC */
@@ -101,6 +102,8 @@ enum qemu_vm_cmd {
 
 };
 
+#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24)
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -113,6 +116,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_open_return_path(QEMUFile *f);
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb);
 void qemu_savevm_send_postcopy_advise(QEMUFile *f);
 void qemu_savevm_send_postcopy_listen(QEMUFile *f);
 void qemu_savevm_send_postcopy_run(QEMUFile *f);
savevm.c +42 -0
index e31ccb0..f65bff3 100644
@@ -636,6 +636,38 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)
     qemu_savevm_command_send(f, MIG_CMD_OPEN_RETURN_PATH, 0, NULL);
 }
 
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
+ */
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb)
+{
+    size_t cur_iov;
+    size_t len = qsb_get_length(qsb);
+    uint32_t tmp;
+
+    tmp = cpu_to_be32(len);
+
+    trace_qemu_savevm_send_packaged();
+    qemu_savevm_command_send(f, MIG_CMD_PACKAGED, 4, (uint8_t *)&tmp);
+
+    /* all the data follows (concatinating the iov's) */
+    for (cur_iov = 0; cur_iov < qsb->n_iov; cur_iov++) {
+        /* The iov entries are partially filled */
+        size_t towrite = (qsb->iov[cur_iov].iov_len > len) ?
+                              len :
+                              qsb->iov[cur_iov].iov_len;
+        len -= towrite;
+
+        if (!towrite) {
+            break;
+        }
+
+        qemu_put_buffer(f, qsb->iov[cur_iov].iov_base, towrite);
+    }
+}
+
 /* Send prior to any postcopy transfer */
 void qemu_savevm_send_postcopy_advise(QEMUFile *f)
 {
@@ -1265,6 +1297,48 @@ static int loadvm_process_command_simple_lencheck(const char *name,
     return 0;
 }
 
+/* Immediately following this command is a blob of data containing an embedded
+ * chunk of migration stream; read it and load it.
+ */
+static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis,
+                                      uint32_t length)
+{
+    int ret;
+    uint8_t *buffer;
+    QEMUSizedBuffer *qsb;
+
Quoted 8 lines

savevm.c static int loadvm_process_command_simple_lencheck(const char *name, · line 1314

Quoted 5 lines
savevm.c +5 -0
+    trace_loadvm_handle_cmd_packaged(length);
+
+    if (length > MAX_VM_CMD_PACKAGED_SIZE) {
+        error_report("Unreasonably large packaged state: %u", length);
+        return -1;

It would be a good idea to check this on the send side as well as receive, wouldn't it?

Yes, I had been doing that in the postcopy code that called the send code; but I've now moved it down into savevm_send_packaged. Thanks.

Dave

Quoted 52 lines
Quoted 52 lines
savevm.c +35 -0
+    }
+    buffer = g_malloc0(length);
+    ret = qemu_get_buffer(mis->file, buffer, (int)length);
+    if (ret != length) {
+        g_free(buffer);
+        error_report("CMD_PACKAGED: Buffer receive fail ret=%d length=%d\n",
+                ret, length);
+        return (re/t < 0) ? ret : -EAGAIN;
+    }
+    trace_loadvm_handle_cmd_packaged_received(ret);
+
+    /* Setup a dummy QEMUFile that actually reads from the buffer */
+    qsb = qsb_create(buffer, length);
+    g_free(buffer); /* Because qsb_create copies */
+    if (!qsb) {
+        error_report("Unable to create qsb");
+    }
+    QEMUFile *packf = qemu_bufopen("r", qsb);
+
+    ret = qemu_loadvm_state_main(packf, mis);
+    trace_loadvm_handle_cmd_packaged_main(ret);
+    qemu_fclose(packf);
+    qsb_free(qsb);
+
+    return ret;
+}
+
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
@@ -1315,6 +1389,14 @@ static int loadvm_process_command(QEMUFile *f)
         migrate_send_rp_pong(mis, tmp32);
         break;
 
+    case MIG_CMD_PACKAGED:
+        if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_PACKAGED",
+            len, 4)) {
+            return -1;
+         }
+        tmp32 = qemu_get_be32(f);
+        return loadvm_handle_cmd_packaged(mis, tmp32);
+
     case MIG_CMD_POSTCOPY_ADVISE:
         if (loadvm_process_command_simple_lencheck("CMD_POSTCOPY_ADVISE",
                                                    len, 16)) {
trace-events +0 -0
index 050f553..cbf995c 100644
@@ -1171,6 +1171,10 @@ qemu_loadvm_state_main(void) ""
 qemu_loadvm_state_main_quit_parent(void) ""
 qemu_loadvm_state_post_main(int ret) "%d"
Quoted 14 lines
Quoted 8 lines
trace-events +4 -0
 qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
+qemu_savevm_send_packaged(void) ""
+loadvm_handle_cmd_packaged(unsigned int length) "%u"
+loadvm_handle_cmd_packaged_main(int ret) "%d"
+loadvm_handle_cmd_packaged_received(int ret) "%d"
 loadvm_postcopy_handle_advise(void) ""
 loadvm_postcopy_handle_end(void) ""
 loadvm_postcopy_handle_listen(void) ""
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#78 lore raw

Re: [Qemu-devel] [PATCH v5 18/45] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 24 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

MIG_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

Hrm. I'd be more comfortable if the semantics of CMD_PACKAGED were defined in terms of visible effects on the other end, rather than in terms of how it's implemented internally.

Quoted 7 lines
Quoted 2 lines

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Which makes the above paragraph a bit misleading - the whole point here is that loading the package data *does* interact with the migration stream - just that it's the migration stream after the end of the package.

Hmm, how about:

Quoted 7 lines

MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.

This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.

It's an improvement.

I'm still a bit concerned that the semantics of CMD_PACKAGED are unhealthily bound up with hw the current implementation works.

[snip]

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
savevm.c +10 -0
+/* Immediately following this command is a blob of data containing an embedded
+ * chunk of migration stream; read it and load it.
+ */
+static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis,
+                                      uint32_t length)
+{
+    int ret;
+    uint8_t *buffer;
+    QEMUSizedBuffer *qsb;
+
Quoted 8 lines

savevm.c static int loadvm_process_command_simple_lencheck(const char *name, · line 1314

Quoted 5 lines
savevm.c +5 -0
+    trace_loadvm_handle_cmd_packaged(length);
+
+    if (length > MAX_VM_CMD_PACKAGED_SIZE) {
+        error_report("Unreasonably large packaged state: %u", length);
+        return -1;

It would be a good idea to check this on the send side as well as receive, wouldn't it?

Quoted 2 lines

Yes, I had been doing that in the postcopy code that called the send code; but I've now moved it down into savevm_send_packaged.

Good, better symmetry that way.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#79 lore raw

PATCH v5 19/45 migrate_init: Call from savevm

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

---
 include/migration/migration.h | 3 +--
 include/qemu/typedefs.h       | 1 +
 migration/migration.c         | 2 +-
 savevm.c                      | 2 ++
 4 files changed, 5 insertions(+), 3 deletions(-)
include/migration/migration.h +1 -2
index 81cd1f2..e6a814a 100644
@@ -48,8 +48,6 @@ enum mig_rpcomm_cmd {
     MIG_RP_CMD_PONG,         /* Response to a PING; data (seq: be32 ) */
 };
 
-typedef struct MigrationState MigrationState;
-
 typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
 
 typedef enum {
@@ -146,6 +144,7 @@ int migrate_fd_close(MigrationState *s);
 
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
+MigrationState *migrate_init(const MigrationParams *params);
 bool migration_in_setup(MigrationState *);
 bool migration_has_finished(MigrationState *);
 bool migration_has_failed(MigrationState *);
include/qemu/typedefs.h +1 -0
index 6fdcbcd..611db46 100644
@@ -41,6 +41,7 @@ typedef struct MemoryRegion MemoryRegion;
 typedef struct MemoryRegionSection MemoryRegionSection;
 typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct MigrationParams MigrationParams;
+typedef struct MigrationState MigrationState;
 typedef struct Monitor Monitor;
 typedef struct MouseTransformInfo MouseTransformInfo;
 typedef struct MSIMessage MSIMessage;
migration/migration.c +1 -1
index 957115a..2e6adca 100644
@@ -493,7 +493,7 @@ bool migration_has_failed(MigrationState *s)
             s->state == MIG_STATE_ERROR);
 }
 
-static MigrationState *migrate_init(const MigrationParams *params)
+MigrationState *migrate_init(const MigrationParams *params)
 {
     MigrationState *s = migrate_get_current();
     int64_t bandwidth_limit = s->bandwidth_limit;
savevm.c +2 -0
index f65bff3..df48ba8 100644
@@ -982,6 +982,8 @@ static int qemu_savevm_state(QEMUFile *f)
         .blk = 0,
         .shared = 0
     };
+    MigrationState *ms = migrate_init(&params);
+    ms->file = f;
 
     if (qemu_savevm_state_blocked(NULL)) {
         return -EINVAL;
Signature · 1 line
2.1.0
#80 lore raw

PATCH v5 20/45 Modify savevm handlers for postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

---
 arch_init.c                 | 15 +++++++++++++--
 include/migration/vmstate.h | 10 ++++++++--
 include/sysemu/sysemu.h     |  4 +++-
 migration/block.c           |  7 +++++--
 migration/migration.c       |  9 +++++++--
 savevm.c                    | 21 +++++++++++++++++----
 trace-events                |  2 +-
 7 files changed, 54 insertions(+), 14 deletions(-)
arch_init.c +13 -2
index fe0df0d..7bc5fa6 100644
@@ -997,7 +997,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                             uint64_t *non_postcopiable_pending,
+                             uint64_t *postcopiable_pending)
 {
     uint64_t remaining_size;
 
@@ -1009,7 +1011,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
         qemu_mutex_unlock_iothread();
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
     }
-    return remaining_size;
+
+    *non_postcopiable_pending = 0;
+    *postcopiable_pending = remaining_size;
 }
 
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
@@ -1204,6 +1208,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
@@ -1211,6 +1221,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,
 };
 
 void ram_mig_init(void)
include/migration/vmstate.h +8 -2
index 18da207..c9ec74a 100644
@@ -54,8 +54,14 @@ typedef struct SaveVMHandlers {
 
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
-    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
+    /*
+     * postcopiable_pending must return 0 unless the can_postcopy
+     * handler returns true.
+     */
+    void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
+                              uint64_t *non_postcopiable_pending,
+                              uint64_t *postcopiable_pending);
+    bool (*can_postcopy)(void *opaque);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
include/sysemu/sysemu.h +3 -1
index e83bf80..5f518b3 100644
@@ -111,7 +111,9 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
migration/block.c +5 -2
index 0c76106..0f6f209 100644
@@ -754,7 +754,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;
 }
 
 static int block_load(QEMUFile *f, void *opaque, int version_id)
migration/migration.c +7 -2
index 2e6adca..a4fc7d7 100644
@@ -868,8 +868,13 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->file)) {
-            pending_size = qemu_savevm_state_pending(s->file, max_size);
-            trace_migrate_pending(pending_size, max_size);
+            uint64_t pend_post, pend_nonpost;
+
+            qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+                                      &pend_post);
+            pending_size = pend_nonpost + pend_post;
+            trace_migrate_pending(pending_size, max_size,
+                                  pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
savevm.c +17 -4
index df48ba8..e301a0a 100644
@@ -944,10 +944,20 @@ void qemu_savevm_state_complete(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -958,9 +968,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;
 }
 
 void qemu_savevm_state_cancel(void)
trace-events +1 -1
index cbf995c..83312b6 100644
@@ -1400,7 +1400,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
-migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 open_outgoing_return_path(void) ""
 open_outgoing_return_path_continue(void) ""
Signature · 1 line
2.1.0
#81 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 195 lines
---
 arch_init.c                 | 15 +++++++++++++--
 include/migration/vmstate.h | 10 ++++++++--
 include/sysemu/sysemu.h     |  4 +++-
 migration/block.c           |  7 +++++--
 migration/migration.c       |  9 +++++++--
 savevm.c                    | 21 +++++++++++++++++----
 trace-events                |  2 +-
 7 files changed, 54 insertions(+), 14 deletions(-)
arch_init.c +13 -2
index fe0df0d..7bc5fa6 100644
@@ -997,7 +997,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                             uint64_t *non_postcopiable_pending,
+                             uint64_t *postcopiable_pending)
 {
     uint64_t remaining_size;
 
@@ -1009,7 +1011,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
         qemu_mutex_unlock_iothread();
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
     }
-    return remaining_size;
+
+    *non_postcopiable_pending = 0;
+    *postcopiable_pending = remaining_size;
 }
 
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
@@ -1204,6 +1208,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
@@ -1211,6 +1221,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,
 };
 
 void ram_mig_init(void)
include/migration/vmstate.h +8 -2
index 18da207..c9ec74a 100644
@@ -54,8 +54,14 @@ typedef struct SaveVMHandlers {
 
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
-    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
+    /*
+     * postcopiable_pending must return 0 unless the can_postcopy
+     * handler returns true.
+     */
+    void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
+                              uint64_t *non_postcopiable_pending,
+                              uint64_t *postcopiable_pending);
+    bool (*can_postcopy)(void *opaque);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
include/sysemu/sysemu.h +3 -1
index e83bf80..5f518b3 100644
@@ -111,7 +111,9 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
migration/block.c +5 -2
index 0c76106..0f6f209 100644
@@ -754,7 +754,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;
 }
 
 static int block_load(QEMUFile *f, void *opaque, int version_id)
migration/migration.c +7 -2
index 2e6adca..a4fc7d7 100644
@@ -868,8 +868,13 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->file)) {
-            pending_size = qemu_savevm_state_pending(s->file, max_size);
-            trace_migrate_pending(pending_size, max_size);
+            uint64_t pend_post, pend_nonpost;
+
+            qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+                                      &pend_post);
+            pending_size = pend_nonpost + pend_post;
+            trace_migrate_pending(pending_size, max_size,
+                                  pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
savevm.c +17 -4
index df48ba8..e301a0a 100644
@@ -944,10 +944,20 @@ void qemu_savevm_state_complete(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -958,9 +968,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;
 }
 
 void qemu_savevm_state_cancel(void)
trace-events +1 -1
index cbf995c..83312b6 100644
@@ -1400,7 +1400,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
-migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 open_outgoing_return_path(void) ""
 open_outgoing_return_path_continue(void) ""
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#82 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Dave

Quoted 182 lines
Quoted 182 lines
---
 arch_init.c                 | 15 +++++++++++++--
 include/migration/vmstate.h | 10 ++++++++--
 include/sysemu/sysemu.h     |  4 +++-
 migration/block.c           |  7 +++++--
 migration/migration.c       |  9 +++++++--
 savevm.c                    | 21 +++++++++++++++++----
 trace-events                |  2 +-
 7 files changed, 54 insertions(+), 14 deletions(-)
arch_init.c +13 -2
index fe0df0d..7bc5fa6 100644
@@ -997,7 +997,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                             uint64_t *non_postcopiable_pending,
+                             uint64_t *postcopiable_pending)
 {
     uint64_t remaining_size;
 
@@ -1009,7 +1011,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
         qemu_mutex_unlock_iothread();
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
     }
-    return remaining_size;
+
+    *non_postcopiable_pending = 0;
+    *postcopiable_pending = remaining_size;
 }
 
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
@@ -1204,6 +1208,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
@@ -1211,6 +1221,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,
 };
 
 void ram_mig_init(void)
include/migration/vmstate.h +8 -2
index 18da207..c9ec74a 100644
@@ -54,8 +54,14 @@ typedef struct SaveVMHandlers {
 
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
-    uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
+    /*
+     * postcopiable_pending must return 0 unless the can_postcopy
+     * handler returns true.
+     */
+    void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
+                              uint64_t *non_postcopiable_pending,
+                              uint64_t *postcopiable_pending);
+    bool (*can_postcopy)(void *opaque);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
include/sysemu/sysemu.h +3 -1
index e83bf80..5f518b3 100644
@@ -111,7 +111,9 @@ void qemu_savevm_state_header(QEMUFile *f);
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
migration/block.c +5 -2
index 0c76106..0f6f209 100644
@@ -754,7 +754,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
     return 0;
 }
 
-static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
+static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                               uint64_t *non_postcopiable_pending,
+                               uint64_t *postcopiable_pending)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     qemu_mutex_unlock_iothread();
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
-    return pending;
+    *non_postcopiable_pending = pending;
+    *postcopiable_pending = 0;
 }
 
 static int block_load(QEMUFile *f, void *opaque, int version_id)
migration/migration.c +7 -2
index 2e6adca..a4fc7d7 100644
@@ -868,8 +868,13 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->file)) {
-            pending_size = qemu_savevm_state_pending(s->file, max_size);
-            trace_migrate_pending(pending_size, max_size);
+            uint64_t pend_post, pend_nonpost;
+
+            qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+                                      &pend_post);
+            pending_size = pend_nonpost + pend_post;
+            trace_migrate_pending(pending_size, max_size,
+                                  pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
savevm.c +17 -4
index df48ba8..e301a0a 100644
@@ -944,10 +944,20 @@ void qemu_savevm_state_complete(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t tmp_non_postcopiable, tmp_postcopiable;
+
+    *res_non_postcopiable = 0;
+    *res_postcopiable = 0;
+
 
     QTAILQ_FOREACH(se, &savevm_handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -958,9 +968,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        se->ops->save_live_pending(f, se->opaque, max_size,
+                                   &tmp_non_postcopiable, &tmp_postcopiable);
+
+        *res_postcopiable += tmp_postcopiable;
+        *res_non_postcopiable += tmp_non_postcopiable;
     }
-    return ret;
 }
 
Quoted 20 lines
Quoted 14 lines
savevm.c +0 -0
 void qemu_savevm_state_cancel(void)
trace-events +1 -1
index cbf995c..83312b6 100644
@@ -1400,7 +1400,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_cleanup_src_rp(void) ""
 migrate_fd_error(void) ""
 migrate_fd_cancel(void) ""
-migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64
+migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
 migrate_send_rp_message(int cmd, uint16_t len) "cmd=%d, len=%d"
 open_outgoing_return_path(void) ""
 open_outgoing_return_path_continue(void) ""
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#83 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 9 lines · Dr. David Alan Gilbert
Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

Quoted 7 lines
Quoted 3 lines

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#84 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#85 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#86 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy

?

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#87 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 57 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

Quoted 8 lines

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#88 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#89 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 73 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

Quoted 6 lines

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

Actually, I spoke too soon; testing this found a bad breakage.

the functions in savevm.c add the per-section headers, and then call the _complete methods on the devices. Those _complete methods can't elect to do nothing, because a header has already been planted.

I've ended up with something between the two; we still have a complete_precopy and complete_postcopy method on the devices; if the complete_postcopy method exists and we're in postcopy mode, the complete_precopy method isn't called at all.
A device could decide to do something different in complete_postcopy from complete_precopy but it must do something to complete the section.
Effectively the presence of the complete_postcopy is now doing what can_postcopy() used to do.

Dave

Quoted 11 lines

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
Quoted 2 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#90 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 81 lines · Dr. David Alan Gilbert
Quoted 73 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

Quoted 6 lines

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

Quoted 5 lines

Actually, I spoke too soon; testing this found a bad breakage.

the functions in savevm.c add the per-section headers, and then call the _complete methods on the devices. Those _complete methods can't elect to do nothing, because a header has already been planted.

Hrm.. couldn't you move the test for presence of the hook earlier so you don't sent the header if the hook is NULL?

Quoted 7 lines

I've ended up with something between the two; we still have a complete_precopy and complete_postcopy method on the devices; if the complete_postcopy method exists and we're in postcopy mode, the complete_precopy method isn't called at all.
A device could decide to do something different in complete_postcopy from complete_precopy but it must do something to complete the section.
Effectively the presence of the complete_postcopy is now doing what can_postcopy() used to do.

Hmm.. but it means there's no per-device hook for the precopy to postcopy transition point. I'm not sure if that might matter.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#91 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 82 lines · * David Gibson ([email protected]) wrote:
Quoted 81 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

Quoted 8 lines
Quoted 5 lines

Actually, I spoke too soon; testing this found a bad breakage.

the functions in savevm.c add the per-section headers, and then call the _complete methods on the devices. Those _complete methods can't elect to do nothing, because a header has already been planted.

Hrm.. couldn't you move the test for presence of the hook earlier so you don't sent the header if the hook is NULL?

There's two tests that you have to make:
a) in qemu_savevm_state_complete_precopy do you call save_live_complete_precopy
b) in qemu_savevm_state_complete_postcopy do you call save_live_complete_postcopy

The obvious case is if either hook is NULL you don't call it.
(a) is the harder cases, if we're doing postcopy then we don't want to call the save_live_complete_precopy method on a device which isn't expecting to complete until postcopy.
The code in qemu_savevm_state_complete_precopy checks for the presence of the *postcopy* hook, and doesn't emit the header or call the precopy commit if the postcopy hook is present and we're in postcopy.

Quoted 10 lines
Quoted 7 lines

I've ended up with something between the two; we still have a complete_precopy and complete_postcopy method on the devices; if the complete_postcopy method exists and we're in postcopy mode, the complete_precopy method isn't called at all.
A device could decide to do something different in complete_postcopy from complete_precopy but it must do something to complete the section.
Effectively the presence of the complete_postcopy is now doing what can_postcopy() used to do.

Hmm.. but it means there's no per-device hook for the precopy to postcopy transition point. I'm not sure if that might matter.

This is true, but if we needed a generic hook for that (which might be useful) it probably shouldn't be 'complete'.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#92 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 92 lines · Dr. David Alan Gilbert
Quoted 82 lines · * David Gibson ([email protected]) wrote:
Quoted 81 lines · Dr. David Alan Gilbert
Quoted 79 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

Quoted 8 lines
Quoted 5 lines

Actually, I spoke too soon; testing this found a bad breakage.

the functions in savevm.c add the per-section headers, and then call the _complete methods on the devices. Those _complete methods can't elect to do nothing, because a header has already been planted.

Hrm.. couldn't you move the test for presence of the hook earlier so you don't sent the header if the hook is NULL?

Quoted 8 lines

There's two tests that you have to make:
a) in qemu_savevm_state_complete_precopy do you call save_live_complete_precopy
b) in qemu_savevm_state_complete_postcopy do you call save_live_complete_postcopy

The obvious case is if either hook is NULL you don't call it.
(a) is the harder cases, if we're doing postcopy then we don't want to call the save_live_complete_precopy method on a device which isn't expecting to complete until postcopy.

Uh.. no, I'm expecting the complete_precopy hook to be called every time if it's non-NULL. If it's a postcopy item that doesn't expect to finish at the end of precopy, it should put its code in the complete_postcopy hook instead.

That should be fine for the non-postcopy case too, because the postcopy_complete hook will be called momentarily.

Quoted 12 lines

The code in qemu_savevm_state_complete_precopy checks for the presence of the *postcopy* hook, and doesn't emit the header or call the precopy commit if the postcopy hook is present and we're in postcopy.

Quoted 8 lines
Quoted 7 lines

I've ended up with something between the two; we still have a complete_precopy and complete_postcopy method on the devices; if the complete_postcopy method exists and we're in postcopy mode, the complete_precopy method isn't called at all.
A device could decide to do something different in complete_postcopy from complete_precopy but it must do something to complete the section.
Effectively the presence of the complete_postcopy is now doing what can_postcopy() used to do.

Quoted 5 lines
Quoted 2 lines

Hmm.. but it means there's no per-device hook for the precopy to postcopy transition point. I'm not sure if that might matter.

This is true, but if we needed a generic hook for that (which might be useful) it probably shouldn't be 'complete'.

Well, I'm thinking of these as "state (complete precopy)" rather than "(state complete) precopy". But yeah, a different name might be less ambiguous.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#93 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

In reply to Dr. David Alan Gilbert
Quoted 73 lines · Dr. David Alan Gilbert
Quoted 65 lines · * David Gibson ([email protected]) wrote:
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 49 lines · * David Gibson ([email protected]) wrote:
Quoted 35 lines · Dr. David Alan Gilbert
Quoted 14 lines · * David Gibson ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify save_live_pending to return separate postcopiable and non-postcopiable counts.

Add 'can_postcopy' to allow a device to state if it can postcopy

What's the purpose of the can_postcopy callback? There are no callers in this patch - is it still necessary with the change to save_live_pending?

Quoted 6 lines
Quoted 3 lines

The patch 'qemu_savevm_state_complete: Postcopy changes' uses it in qemu_savevm_state_postcopy_complete and qemu_savevm_state_complete to decide which devices must be completed at that point.

Couldn't they check for non-zero postcopiable state from save_live_pending instead?

That would be a bit weird.

At the moment for each device we call the:
save_live_setup method (from qemu_savevm_state_begin)

0...multiple times we call:
save_live_pending
save_live_iterate

and then we always call
save_live_complete

Quoted 7 lines

To my mind we have to call save_live_complete for any device that we've called save_live_setup on (maybe it allocated something in _setup that it clears up in _complete).

save_live_pending could perfectly well return 0 remaining at the end of the migrate for our device, and thus if we used that then we wouldn't call save_live_complete.

Um.. I don't follow. I was suggesting that at the precopy->postcopy transition point you call save_live_complete for everything that reports 0 post-copiable state.

Quoted 6 lines

Then again, a different approach would be to split the save_live_complete hook into (possibly NULL) "complete precopy" and "complete postcopy" hooks. The core would ensure that every chunk of state has both completion hooks called (unless NULL). That might also address my concerns about the no longer entirely accurate save_live_complete function name.

OK, that one I prefer. Are you OK with:

    qemu_savevm_state_complete_precopy
       calls -> save_live_complete_precopy

    qemu_savevm_state_complete_postcopy
       calls -> save_live_complete_postcopy
Quoted 6 lines
Quoted 1 line

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

Quoted 6 lines

OK, I've made the change as described in my last mail; but I haven't called the complete_postcopy hook in the precopy case. If it was as simple as making all devices use one or the other then it would work, however there are existing (precopy) assumptions about ordering of device state on the wire that I want to be careful not to alter; for example RAM must come first is the one I know.

It's not obvious to me why that matters to the hook scheme.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#94 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

In reply to David Gibson
Quoted 4 lines · David Gibson
Quoted 4 lines
Quoted 4 lines

OK, that one I prefer. Are you OK with:
qemu_savevm_state_complete_precopy calls ->
save_live_complete_precopy

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

qemu_savevm_state_complete_postcopy calls ->
save_live_complete_postcopy

?

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

What about then calling them save_live_after_precopy and save_live_complete, or having save_live_before_postcopy and save_live_complete?

Paolo

#95 lore raw

Re: [Qemu-devel] [PATCH v5 20/45] Modify savevm handlers for postcopy

Quoted 11 lines · Paolo Bonzini
Quoted 8 lines · David Gibson
Quoted 8 lines
Quoted 8 lines

OK, that one I prefer. Are you OK with:
qemu_savevm_state_complete_precopy calls ->
save_live_complete_precopy

qemu_savevm_state_complete_postcopy calls ->
save_live_complete_postcopy

?

Quoted 8 lines
Quoted 4 lines

Sounds ok to me. Fwiw, I was thinking that both the complete_precopy and complete_postcopy hooks should always be called. For a non-postcopy migration, the postcopy hooks would just be called immediately after the precopy hooks.

What about then calling them save_live_after_precopy and save_live_complete, or having save_live_before_postcopy and save_live_complete?

Good idea.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#96 lore raw

PATCH v5 21/45 Add Linux userfaultfd header

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

TODO: Update when kernel interface settles
Update update-linux-headers.sh

---
 linux-headers/linux/userfaultfd.h | 150 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 150 insertions(+)
 create mode 100644 linux-headers/linux/userfaultfd.h
linux-headers/linux/userfaultfd.h new file +150 -0
new file mode 100644
index 0000000..db6e99a
@@ -0,0 +1,150 @@
+/*
+ *  include/linux/userfaultfd.h
+ *
+ *  Copyright (C) 2007  Davide Libenzi <[email protected]>
+ *  Copyright (C) 2015  Red Hat, Inc.
+ *
+ */
+
+#ifndef _LINUX_USERFAULTFD_H
+#define _LINUX_USERFAULTFD_H
+
+#define UFFD_API ((__u64)0xAA)
+/* FIXME: add "|UFFD_BIT_WP" to UFFD_API_BITS after implementing it */
+#define UFFD_API_BITS (UFFD_BIT_WRITE)
+#define UFFD_API_IOCTLS				\
+	((__u64)1 << _UFFDIO_REGISTER |		\
+	 (__u64)1 << _UFFDIO_UNREGISTER |	\
+	 (__u64)1 << _UFFDIO_API)
+#define UFFD_API_RANGE_IOCTLS			\
+	((__u64)1 << _UFFDIO_WAKE |		\
+	 (__u64)1 << _UFFDIO_COPY |		\
+	 (__u64)1 << _UFFDIO_ZEROPAGE |		\
+	 (__u64)1 << _UFFDIO_REMAP)
+
+/*
+ * Valid ioctl command number range with this API is from 0x00 to
+ * 0x3F.  UFFDIO_API is the fixed number, everything else can be
+ * changed by implementing a different UFFD_API. If sticking to the
+ * same UFFD_API more ioctl can be added and userland will be aware of
+ * which ioctl the running kernel implements through the ioctl command
+ * bitmask written by the UFFDIO_API.
+ */
+#define _UFFDIO_REGISTER		(0x00)
+#define _UFFDIO_UNREGISTER		(0x01)
+#define _UFFDIO_WAKE			(0x02)
+#define _UFFDIO_COPY			(0x03)
+#define _UFFDIO_ZEROPAGE		(0x04)
+#define _UFFDIO_REMAP			(0x05)
+#define _UFFDIO_API			(0x3F)
+
+/* userfaultfd ioctl ids */
+#define UFFDIO 0xAA
+#define UFFDIO_API		_IOWR(UFFDIO, _UFFDIO_API,	\
+				      struct uffdio_api)
+#define UFFDIO_REGISTER		_IOWR(UFFDIO, _UFFDIO_REGISTER, \
+				      struct uffdio_register)
+#define UFFDIO_UNREGISTER	_IOR(UFFDIO, _UFFDIO_UNREGISTER,	\
+				     struct uffdio_range)
+#define UFFDIO_WAKE		_IOR(UFFDIO, _UFFDIO_WAKE,	\
+				     struct uffdio_range)
+#define UFFDIO_COPY		_IOWR(UFFDIO, _UFFDIO_COPY,	\
+				      struct uffdio_copy)
+#define UFFDIO_ZEROPAGE		_IOWR(UFFDIO, _UFFDIO_ZEROPAGE,	\
+				      struct uffdio_zeropage)
+#define UFFDIO_REMAP		_IOWR(UFFDIO, _UFFDIO_REMAP,	\
+				      struct uffdio_remap)
+
+/*
+ * Valid bits below PAGE_SHIFT in the userfault address read through
+ * the read() syscall.
+ */
+#define UFFD_BIT_WRITE	(1<<0)	/* this was a write fault, MISSING or WP */
+#define UFFD_BIT_WP	(1<<1)	/* handle_userfault() reason VM_UFFD_WP */
+#define UFFD_BITS	2	/* two above bits used for UFFD_BIT_* mask */
+
+struct uffdio_api {
+	/* userland asks for an API number */
+	__u64 api;
+
+	/* kernel answers below with the available features for the API */
+	__u64 bits;
+	__u64 ioctls;
+};
+
+struct uffdio_range {
+	__u64 start;
+	__u64 len;
+};
+
+struct uffdio_register {
+	struct uffdio_range range;
+#define UFFDIO_REGISTER_MODE_MISSING	((__u64)1<<0)
+#define UFFDIO_REGISTER_MODE_WP		((__u64)1<<1)
+	__u64 mode;
+
+	/*
+	 * kernel answers which ioctl commands are available for the
+	 * range, keep at the end as the last 8 bytes aren't read.
+	 */
+	__u64 ioctls;
+};
+
+struct uffdio_copy {
+	__u64 dst;
+	__u64 src;
+	__u64 len;
+	/*
+	 * There will be a wrprotection flag later that allows to map
+	 * pages wrprotected on the fly. And such a flag will be
+	 * available if the wrprotection ioctl are implemented for the
+	 * range according to the uffdio_register.ioctls.
+	 */
+#define UFFDIO_COPY_MODE_DONTWAKE		((__u64)1<<0)
+	__u64 mode;
+
+	/*
+	 * "copy" and "wake" are written by the ioctl and must be at
+	 * the end: the copy_from_user will not read the last 16
+	 * bytes.
+	 */
+	__s64 copy;
+	__s64 wake;
+};
+
+struct uffdio_zeropage {
+	struct uffdio_range range;
+#define UFFDIO_ZEROPAGE_MODE_DONTWAKE		((__u64)1<<0)
+	__u64 mode;
+
+	/*
+	 * "zeropage" and "wake" are written by the ioctl and must be
+	 * at the end: the copy_from_user will not read the last 16
+	 * bytes.
+	 */
+	__s64 zeropage;
+	__s64 wake;
+};
+
+struct uffdio_remap {
+	__u64 dst;
+	__u64 src;
+	__u64 len;
+	/*
+	 * Especially if used to atomically remove memory from the
+	 * address space the wake on the dst range is not needed.
+	 */
+#define UFFDIO_REMAP_MODE_DONTWAKE		((__u64)1<<0)
+#define UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES	((__u64)1<<1)
+	__u64 mode;
+
+	/*
+	 * "remap" and "wake" are written by the ioctl and must be at
+	 * the end: the copy_from_user will not read the last 16
+	 * bytes.
+	 */
+	__s64 remap;
+	__s64 wake;
+};
+
+#endif /* _LINUX_USERFAULTFD_H */
Signature · 1 line
2.1.0
#97 lore raw

PATCH v5 22/45 postcopy: OS support test

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 include/migration/postcopy-ram.h |  19 +++++
 migration/Makefile.objs          |   2 +-
 migration/postcopy-ram.c         | 161 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   5 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..d81934f
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return true if the host supports everything we need to do postcopy-ram */
+bool postcopy_ram_supported_by_host(void);
+
+#endif
migration/Makefile.objs +1 -1
index d929e96..0cac6d7 100644
@@ -1,7 +1,7 @@
 common-obj-y += migration.o tcp.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
-common-obj-y += xbzrle.o
+common-obj-y += xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +161 -0

161 lines not on this page · raw message

savevm.c +5 -0
index e301a0a..2ea4c76 100644
@@ -33,6 +33,7 @@
 #include "qemu/timer.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/sockets.h"
 #include "qemu/queue.h"
 #include "sysemu/cpus.h"
@@ -1109,6 +1110,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+
     if (remote_hps != getpagesize())  {
         /*
          * Some combinations of mismatch are probably possible but it gets
Signature · 1 line
2.1.0
#98 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 95 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 include/migration/postcopy-ram.h |  19 +++++
 migration/Makefile.objs          |   2 +-
 migration/postcopy-ram.c         | 161 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   5 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..d81934f
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return true if the host supports everything we need to do postcopy-ram */
+bool postcopy_ram_supported_by_host(void);
+
+#endif
migration/Makefile.objs +1 -1
index d929e96..0cac6d7 100644
@@ -1,7 +1,7 @@
 common-obj-y += migration.o tcp.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
-common-obj-y += xbzrle.o
+common-obj-y += xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +34 -0
new file mode 100644
index 0000000..a0e20b2
@@ -0,0 +1,161 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013-2014 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/*
+ * Postcopy is a migration technique where the execution flips from the
+ * source to the destination before all the data has been copied.
+ */
+
+#include <glib.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "migration/postcopy-ram.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+
+/* Postcopy needs to detect accesses to pages that haven't yet been copied
+ * across, and efficiently map new pages in, the techniques for doing this
+ * are target OS specific.
+ */
+#if defined(__linux__)

migration/postcopy-ram.c line 44

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+
+#include <sys/mman.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <asm/types.h> /* for __u64 */
+#include <linux/userfaultfd.h>
+
+#ifdef HOST_X86_64
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 323

Sholdn't this come from the kernel headers imported in the previous patch? Rather than having an arch-specific hack.

Quoted 5 lines
migration/postcopy-ram.c new file +5 -0
+#endif
+#endif
+
+#endif
+

migration/postcopy-ram.c line 59

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+#if defined(__linux__) && defined(__NR_userfaultfd)
+
+static bool ufd_version_check(int ufd)
+{
+    struct uffdio_api api_struct;
+    uint64_t feature_mask;
+
+    api_struct.api = UFFD_API;
+    if (ioctl(ufd, UFFDIO_API, &api_struct)) {
+        perror("postcopy_ram_supported_by_host: UFFDIO_API failed");

This should be error_report() not, perror(), to match qemu conventions, shouldn't it?

Quoted 23 lines
migration/postcopy-ram.c new file +23 -0
+        return false;
+    }
+
+    feature_mask = (__u64)1 << _UFFDIO_REGISTER |
+                   (__u64)1 << _UFFDIO_UNREGISTER;
+    if ((api_struct.ioctls & feature_mask) != feature_mask) {
+        error_report("Missing userfault features: %" PRIu64,
+                     (uint64_t)(~api_struct.ioctls & feature_mask));
+        return false;
+    }
+
+    return true;
+}
+
+bool postcopy_ram_supported_by_host(void)
+{
+    long pagesize = getpagesize();
+    int ufd = -1;
+    bool ret = false; /* Error unless we change it */
+    void *testarea = NULL;
+    struct uffdio_register reg_struct;
+    struct uffdio_range range_struct;
+    uint64_t feature_mask;

migration/postcopy-ram.c line 92

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+
+    if ((1ul << qemu_target_page_bits()) > pagesize) {
+        /* The PMI code doesn't yet deal with TPS>HPS */
+        error_report("Target page size bigger than host page size");
+        goto out;
+    }
+
+    ufd = syscall(__NR_userfaultfd, O_CLOEXEC);
+    if (ufd == -1) {
+        perror("postcopy_ram_supported_by_host: userfaultfd not available");

And here as well? And several places below.

Quoted 6 lines
migration/postcopy-ram.c new file +6 -0
+        goto out;
+    }
+
+    /* Version and features check */
+    if (!ufd_version_check(ufd)) {
+        goto out;

migration/postcopy-ram.c line 108

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+    }
+
+    /*
+     *  We need to check that the ops we need are supported on anon memory
+     *  To do that we need to register a chunk and see the flags that
+     *  are returned.
+     */
+    testarea = mmap(NULL, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE |
+                                    MAP_ANONYMOUS, -1, 0);
+    if (!testarea) {

This should be (testarea == MAP_FAILED). Otherwise mmap() failures will always trip the assert below.

Quoted 6 lines
migration/postcopy-ram.c new file +6 -0
+        perror("postcopy_ram_supported_by_host: Failed to map test area");
+        goto out;
+    }
+    g_assert(((size_t)testarea & (pagesize-1)) == 0);
+
+    reg_struct.range.start = (uint64_t)(uintptr_t)testarea;

migration/postcopy-ram.c line 124

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+    reg_struct.range.len = (uint64_t)pagesize;
+    reg_struct.mode = UFFDIO_REGISTER_MODE_MISSING;
+
+    if (ioctl(ufd, UFFDIO_REGISTER, &reg_struct)) {
+        perror("postcopy_ram_supported_by_host userfault register");
+        goto out;
+    }
+
+    range_struct.start = (uint64_t)(uintptr_t)testarea;
+    range_struct.len = (uint64_t)pagesize;

I don't think you need the (uint64_t) casts (though you do need the uintptr_t cast). I think the assignment will do an implicit conversion without probvlems.

migration/postcopy-ram.c line 134

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+    if (ioctl(ufd, UFFDIO_UNREGISTER, &range_struct)) {
+        perror("postcopy_ram_supported_by_host userfault unregister");
+        goto out;
+    }
+
+    feature_mask = (__u64)1 << _UFFDIO_WAKE |
+                   (__u64)1 << _UFFDIO_COPY |
+                   (__u64)1 << _UFFDIO_ZEROPAGE;
+    if ((reg_struct.ioctls & feature_mask) != feature_mask) {
+        error_report("Missing userfault map features: %" PRIu64,

I'm guessing you want PRIx64, in order to make the feature mask at least semi-readable.

Quoted 50 lines
migration/postcopy-ram.c new file +27 -0
+                     (uint64_t)(~reg_struct.ioctls & feature_mask));
+        goto out;
+    }
+
+    /* Success! */
+    ret = true;
+out:
+    if (testarea) {
+        munmap(testarea, pagesize);
+    }
+    if (ufd != -1) {
+        close(ufd);
+    }
+    return ret;
+}
+
+#else
+/* No target OS support, stubs just fail */
+
+bool postcopy_ram_supported_by_host(void)
+{
+    error_report("%s: No OS support", __func__);
+    return false;
+}
+
+#endif
+
savevm.c +5 -0
index e301a0a..2ea4c76 100644
@@ -33,6 +33,7 @@
 #include "qemu/timer.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/sockets.h"
 #include "qemu/queue.h"
 #include "sysemu/cpus.h"
@@ -1109,6 +1110,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+
     if (remote_hps != getpagesize())  {
         /*
          * Some combinations of mismatch are probably possible but it gets
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#99 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 101 lines · * David Gibson ([email protected]) wrote:
Quoted 100 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 include/migration/postcopy-ram.h |  19 +++++
 migration/Makefile.objs          |   2 +-
 migration/postcopy-ram.c         | 161 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   5 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..d81934f
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return true if the host supports everything we need to do postcopy-ram */
+bool postcopy_ram_supported_by_host(void);
+
+#endif
migration/Makefile.objs +1 -1
index d929e96..0cac6d7 100644
@@ -1,7 +1,7 @@
 common-obj-y += migration.o tcp.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
-common-obj-y += xbzrle.o
+common-obj-y += xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +39 -0
new file mode 100644
index 0000000..a0e20b2
@@ -0,0 +1,161 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013-2014 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/*
+ * Postcopy is a migration technique where the execution flips from the
+ * source to the destination before all the data has been copied.
+ */
+
+#include <glib.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "migration/postcopy-ram.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+
+/* Postcopy needs to detect accesses to pages that haven't yet been copied
+ * across, and efficiently map new pages in, the techniques for doing this
+ * are target OS specific.
+ */
+#if defined(__linux__)
+
+#include <sys/mman.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <asm/types.h> /* for __u64 */
Quoted 8 lines

migration/postcopy-ram.c line 44

Quoted 5 lines
migration/postcopy-ram.c new file +5 -0
+#include <linux/userfaultfd.h>
+
+#ifdef HOST_X86_64
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 323

Sholdn't this come from the kernel headers imported in the previous patch? Rather than having an arch-specific hack.

The header, like the rest of the kernel headers, just provides the constant and structure definitions for the call; the syscall numbers come from arch specific headers. I guess in the final world I wouldn't need this at all since it'll come from the system headers; but what's the right way to put this in for new syscalls?

Quoted 10 lines
Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+#endif
+#endif
+
+#endif
+
+#if defined(__linux__) && defined(__NR_userfaultfd)
+
+static bool ufd_version_check(int ufd)
+{
+    struct uffdio_api api_struct;
Quoted 8 lines

migration/postcopy-ram.c line 59

Quoted 5 lines
migration/postcopy-ram.c new file +5 -0
+    uint64_t feature_mask;
+
+    api_struct.api = UFFD_API;
+    if (ioctl(ufd, UFFDIO_API, &api_struct)) {
+        perror("postcopy_ram_supported_by_host: UFFDIO_API failed");

This should be error_report() not, perror(), to match qemu conventions, shouldn't it?

Thanks; I've done all of the perrors.

Quoted 47 lines
Quoted 23 lines
migration/postcopy-ram.c new file +23 -0
+        return false;
+    }
+
+    feature_mask = (__u64)1 << _UFFDIO_REGISTER |
+                   (__u64)1 << _UFFDIO_UNREGISTER;
+    if ((api_struct.ioctls & feature_mask) != feature_mask) {
+        error_report("Missing userfault features: %" PRIu64,
+                     (uint64_t)(~api_struct.ioctls & feature_mask));
+        return false;
+    }
+
+    return true;
+}
+
+bool postcopy_ram_supported_by_host(void)
+{
+    long pagesize = getpagesize();
+    int ufd = -1;
+    bool ret = false; /* Error unless we change it */
+    void *testarea = NULL;
+    struct uffdio_register reg_struct;
+    struct uffdio_range range_struct;
+    uint64_t feature_mask;

migration/postcopy-ram.c line 92

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+
+    if ((1ul << qemu_target_page_bits()) > pagesize) {
+        /* The PMI code doesn't yet deal with TPS>HPS */
+        error_report("Target page size bigger than host page size");
+        goto out;
+    }
+
+    ufd = syscall(__NR_userfaultfd, O_CLOEXEC);
+    if (ufd == -1) {
+        perror("postcopy_ram_supported_by_host: userfaultfd not available");

And here as well? And several places below.

Quoted 11 lines
migration/postcopy-ram.c new file +11 -0
+        goto out;
+    }
+
+    /* Version and features check */
+    if (!ufd_version_check(ufd)) {
+        goto out;
+    }
+
+    /*
+     *  We need to check that the ops we need are supported on anon memory
+     *  To do that we need to register a chunk and see the flags that
Quoted 8 lines

migration/postcopy-ram.c line 108

Quoted 5 lines
migration/postcopy-ram.c new file +5 -0
+     *  are returned.
+     */
+    testarea = mmap(NULL, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE |
+                                    MAP_ANONYMOUS, -1, 0);
+    if (!testarea) {

This should be (testarea == MAP_FAILED). Otherwise mmap() failures will always trip the assert below.

Thanks; fixed.

Quoted 12 lines
Quoted 12 lines
migration/postcopy-ram.c new file +12 -0
+        perror("postcopy_ram_supported_by_host: Failed to map test area");
+        goto out;
+    }
+    g_assert(((size_t)testarea & (pagesize-1)) == 0);
+
+    reg_struct.range.start = (uint64_t)(uintptr_t)testarea;
+    reg_struct.range.len = (uint64_t)pagesize;
+    reg_struct.mode = UFFDIO_REGISTER_MODE_MISSING;
+
+    if (ioctl(ufd, UFFDIO_REGISTER, &reg_struct)) {
+        perror("postcopy_ram_supported_by_host userfault register");
+        goto out;
Quoted 8 lines

migration/postcopy-ram.c line 124

Quoted 4 lines
migration/postcopy-ram.c new file +4 -0
+    }
+
+    range_struct.start = (uint64_t)(uintptr_t)testarea;
+    range_struct.len = (uint64_t)pagesize;

I don't think you need the (uint64_t) casts (though you do need the uintptr_t cast). I think the assignment will do an implicit conversion without probvlems.

Yes, you're right - they're gone.

Quoted 11 lines

migration/postcopy-ram.c line 134

Quoted 10 lines
migration/postcopy-ram.c new file +10 -0
+    if (ioctl(ufd, UFFDIO_UNREGISTER, &range_struct)) {
+        perror("postcopy_ram_supported_by_host userfault unregister");
+        goto out;
+    }
+
+    feature_mask = (__u64)1 << _UFFDIO_WAKE |
+                   (__u64)1 << _UFFDIO_COPY |
+                   (__u64)1 << _UFFDIO_ZEROPAGE;
+    if ((reg_struct.ioctls & feature_mask) != feature_mask) {
+        error_report("Missing userfault map features: %" PRIu64,
Quoted 2 lines

I'm guessing you want PRIx64, in order to make the feature mask at least semi-readable.

Yes, thanks - done.

Quoted 43 lines
Quoted 43 lines
migration/postcopy-ram.c new file +27 -0
+                     (uint64_t)(~reg_struct.ioctls & feature_mask));
+        goto out;
+    }
+
+    /* Success! */
+    ret = true;
+out:
+    if (testarea) {
+        munmap(testarea, pagesize);
+    }
+    if (ufd != -1) {
+        close(ufd);
+    }
+    return ret;
+}
+
+#else
+/* No target OS support, stubs just fail */
+
+bool postcopy_ram_supported_by_host(void)
+{
+    error_report("%s: No OS support", __func__);
+    return false;
+}
+
+#endif
+
savevm.c +1 -0
index e301a0a..2ea4c76 100644
@@ -33,6 +33,7 @@
 #include "qemu/timer.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/sockets.h"
 #include "qemu/queue.h"
 #include "sysemu/cpus.h"
@@ -1109,6 +1110,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
Quoted 13 lines
Quoted 7 lines
savevm.c +4 -0
+    if (!postcopy_ram_supported_by_host()) {
+        return -1;
+    }
+
     if (remote_hps != getpagesize())  {
         /*
          * Some combinations of mismatch are probably possible but it gets
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#100 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 111 lines · Dr. David Alan Gilbert
Quoted 101 lines · * David Gibson ([email protected]) wrote:
Quoted 100 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 include/migration/postcopy-ram.h |  19 +++++
 migration/Makefile.objs          |   2 +-
 migration/postcopy-ram.c         | 161 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   5 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..d81934f
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return true if the host supports everything we need to do postcopy-ram */
+bool postcopy_ram_supported_by_host(void);
+
+#endif
migration/Makefile.objs +1 -1
index d929e96..0cac6d7 100644
@@ -1,7 +1,7 @@
 common-obj-y += migration.o tcp.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
-common-obj-y += xbzrle.o
+common-obj-y += xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +39 -0
new file mode 100644
index 0000000..a0e20b2
@@ -0,0 +1,161 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013-2014 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/*
+ * Postcopy is a migration technique where the execution flips from the
+ * source to the destination before all the data has been copied.
+ */
+
+#include <glib.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "migration/postcopy-ram.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+
+/* Postcopy needs to detect accesses to pages that haven't yet been copied
+ * across, and efficiently map new pages in, the techniques for doing this
+ * are target OS specific.
+ */
+#if defined(__linux__)
+
+#include <sys/mman.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <asm/types.h> /* for __u64 */
Quoted 8 lines

migration/postcopy-ram.c line 44

Quoted 5 lines
migration/postcopy-ram.c new file +5 -0
+#include <linux/userfaultfd.h>
+
+#ifdef HOST_X86_64
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 323

Sholdn't this come from the kernel headers imported in the previous patch? Rather than having an arch-specific hack.

Quoted 5 lines

The header, like the rest of the kernel headers, just provides the constant and structure definitions for the call; the syscall numbers come from arch specific headers. I guess in the final world I wouldn't need this at all since it'll come from the system headers; but what's the right way to put this in for new syscalls?

Uh.. yeah.. I'm not sure actually. I was assuming the imported kernel headers, because part of the reason they're there is for getting defines that aren't in the system headers (e.g. ioctl numbers and structures). But without asm/unistd.h it's not clear to be how a new system call should be handled.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#101 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

In reply to Dr. David Alan Gilbert
Quoted 6 lines · Dr. David Alan Gilbert
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

migration/postcopy-ram.c line 44

Quoted 3 lines
migration/postcopy-ram.c new file +3 -0
+#ifdef HOST_X86_64
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 323

Sholdn't this come from the kernel headers imported in the previous patch? Rather than having an arch-specific hack.

Quoted 6 lines

The header, like the rest of the kernel headers, just provides the constant and structure definitions for the call; the syscall numbers come from arch specific headers. I guess in the final world I wouldn't need this at all since it'll come from the system headers; but what's the right way to put this in for new syscalls?

You would just require new _installed_ kernel headers. Then you can use linux/userfaultfd.h and syscall.h (the latter from glibc, includes asm/unistd.h to get syscall numbers).

linux-headers/ is useful for APIs that do not require system calls, or for APIs that are extensible. However, if a system call is required (and mandatory) it's simpler to just use installed headers.

Paolo

#102 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 16 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

migration/postcopy-ram.c line 44

Quoted 3 lines
migration/postcopy-ram.c new file +3 -0
+#ifdef HOST_X86_64
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 323

Sholdn't this come from the kernel headers imported in the previous patch? Rather than having an arch-specific hack.

Quoted 6 lines

The header, like the rest of the kernel headers, just provides the constant and structure definitions for the call; the syscall numbers come from arch specific headers. I guess in the final world I wouldn't need this at all since it'll come from the system headers; but what's the right way to put this in for new syscalls?

Quoted 7 lines

You would just require new _installed_ kernel headers. Then you can use linux/userfaultfd.h and syscall.h (the latter from glibc, includes asm/unistd.h to get syscall numbers).

linux-headers/ is useful for APIs that do not require system calls, or for APIs that are extensible. However, if a system call is required (and mandatory) it's simpler to just use installed headers.

OK, so then I could check for ifdef __NR_userfault and then do the include and I think that would be safe. Although then what's the best way to tell people to try it out without an updated libc?

Or is it best to modify ./configure to detect it?

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#103 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 11 lines · Dr. David Alan Gilbert
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line

You would just require new _installed_ kernel headers. Then you can use linux/userfaultfd.h and syscall.h (the latter from glibc, includes asm/unistd.h to get syscall numbers).

linux-headers/ is useful for APIs that do not require system calls, or for APIs that are extensible. However, if a system call is required (and mandatory) it's simpler to just use installed headers.

OK, so then I could check for ifdef __NR_userfault and then do the include and I think that would be safe.

I think it's okay. First include syscall.h, then include linux/userfaultfd.h under #ifdef.

Quoted 2 lines

Although then what's the best way to tell people to try it out without an updated libc?

They don't need an updated libc, just an updated kernel. syscall.h is just a wrapper around Linux headers.

Paolo

Quoted 1 line

Or is it best to modify ./configure to detect it?

#104 lore raw

Re: [Qemu-devel] [PATCH v5 22/45] postcopy: OS support test

Quoted 15 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line

You would just require new _installed_ kernel headers. Then you can use linux/userfaultfd.h and syscall.h (the latter from glibc, includes asm/unistd.h to get syscall numbers).

linux-headers/ is useful for APIs that do not require system calls, or for APIs that are extensible. However, if a system call is required (and mandatory) it's simpler to just use installed headers.

OK, so then I could check for ifdef __NR_userfault and then do the include and I think that would be safe.

Quoted 8 lines

I think it's okay. First include syscall.h, then include linux/userfaultfd.h under #ifdef.

Quoted 2 lines

Although then what's the best way to tell people to try it out without an updated libc?

They don't need an updated libc, just an updated kernel. syscall.h is just a wrapper around Linux headers.

That's what I've implemented in the v6 and the v7 I've just posted.

Dave

Quoted 4 lines

Paolo

Quoted 1 line

Or is it best to modify ./configure to detect it?

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#105 lore raw

PATCH v5 23/45 migrate_start_postcopy: Command to trigger transition to postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +22 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);
+}
+
 /* shared migration helpers */
 
 static void migrate_set_state(MigrationState *s, int old_state, int new_state)
qapi-schema.json +8 -0
index a8af1cb..7ff61e9 100644
@@ -542,6 +542,14 @@
 { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
 
 ##
+# @migrate-start-postcopy
+#
+# Switch migration to postcopy mode
+#
+# Since: 2.3
+{ 'command': 'migrate-start-postcopy' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
qmp-commands.hx +19 -0
index a85d847..25d2208 100644
@@ -685,6 +685,25 @@ Example:
 
 EQMP
     {
+        .name       = "migrate-start-postcopy",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy,
+    },
+
+SQMP
+migrate-start-postcopy
+----------------------
+
+Switch an in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+
+Example:
+-> { "execute": "migrate-start-postcopy" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 1 line
2.1.0
#106 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 114 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +10 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 10 lines
migration/migration.c +10 -0
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 54 lines
migration/migration.c +2 -0
+}
+
 /* shared migration helpers */
 
 static void migrate_set_state(MigrationState *s, int old_state, int new_state)
qapi-schema.json +8 -0
index a8af1cb..7ff61e9 100644
@@ -542,6 +542,14 @@
 { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
 
 ##
+# @migrate-start-postcopy
+#
+# Switch migration to postcopy mode
+#
+# Since: 2.3
+{ 'command': 'migrate-start-postcopy' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
qmp-commands.hx +19 -0
index a85d847..25d2208 100644
@@ -685,6 +685,25 @@ Example:
 
 EQMP
     {
+        .name       = "migrate-start-postcopy",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy,
+    },
+
+SQMP
+migrate-start-postcopy
+----------------------
+
+Switch an in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+
+Example:
+-> { "execute": "migrate-start-postcopy" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#107 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 119 lines · * David Gibson ([email protected]) wrote:
Quoted 118 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +14 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
Quoted 8 lines

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 6 lines
migration/migration.c +6 -0
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

Dave

Quoted 47 lines
Quoted 47 lines
migration/migration.c +2 -0
+}
+
 /* shared migration helpers */
 
 static void migrate_set_state(MigrationState *s, int old_state, int new_state)
qapi-schema.json +8 -0
index a8af1cb..7ff61e9 100644
@@ -542,6 +542,14 @@
 { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
 
 ##
+# @migrate-start-postcopy
+#
+# Switch migration to postcopy mode
+#
+# Since: 2.3
+{ 'command': 'migrate-start-postcopy' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
qmp-commands.hx +14 -0
index a85d847..25d2208 100644
@@ -685,6 +685,25 @@ Example:
 
 EQMP
     {
+        .name       = "migrate-start-postcopy",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy,
+    },
+
+SQMP
+migrate-start-postcopy
+----------------------
+
+Switch an in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+
+Example:
+-> { "execute": "migrate-start-postcopy" }
Quoted 14 lines
Quoted 8 lines
qmp-commands.hx +5 -0
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#108 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 129 lines · Dr. David Alan Gilbert
Quoted 119 lines · * David Gibson ([email protected]) wrote:
Quoted 118 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +14 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
Quoted 8 lines

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 6 lines
migration/migration.c +6 -0
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 3 lines

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

I don't think you need an atomic then. AFAIK an atomic_set() in isolation without some sort of atomic on the other side is pretty much meaningless.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#109 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 130 lines · * David Gibson ([email protected]) wrote:
Quoted 129 lines · Dr. David Alan Gilbert
Quoted 127 lines · * David Gibson ([email protected]) wrote:
Quoted 114 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +10 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 10 lines
migration/migration.c +10 -0
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 7 lines
Quoted 3 lines

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

I don't think you need an atomic then. AFAIK an atomic_set() in isolation without some sort of atomic on the other side is pretty much meaningless.

The other side has an atomic_read:

                if (migrate_postcopy_ram() &&
                    s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
                    pend_nonpost <= max_size &&
                    atomic_read(&s->start_postcopy)) {

                    if (!postcopy_start(s, &old_vm_running)) {
                        current_active_type = MIGRATION_STATUS_POSTCOPY_ACTIVE;
                        entered_postcopy = true;
                    }

so it is at least symmetric.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#110 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 151 lines · Dr. David Alan Gilbert
Quoted 130 lines · * David Gibson ([email protected]) wrote:
Quoted 129 lines · Dr. David Alan Gilbert
Quoted 127 lines · * David Gibson ([email protected]) wrote:
Quoted 114 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +10 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 10 lines
migration/migration.c +10 -0
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 7 lines
Quoted 3 lines

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

I don't think you need an atomic then. AFAIK an atomic_set() in isolation without some sort of atomic on the other side is pretty much meaningless.

The other side has an atomic_read:

                if (migrate_postcopy_ram() &&
                    s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
                    pend_nonpost <= max_size &&
                    atomic_read(&s->start_postcopy)) {

                    if (!postcopy_start(s, &old_vm_running)) {
                        current_active_type = MIGRATION_STATUS_POSTCOPY_ACTIVE;
                        entered_postcopy = true;
                    }
Quoted 1 line

so it is at least symmetric.

But still pointless. Atomicity isn't magic pixie dust; it only makes sense if you're making atomic specific operations that need to be. Simple integer loads and stores are already atomic. Unless at least some of the atomic operations are something more complex, there's really no point to atomically marked operations.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#111 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 152 lines · * David Gibson ([email protected]) wrote:
Quoted 151 lines · Dr. David Alan Gilbert
Quoted 130 lines · * David Gibson ([email protected]) wrote:
Quoted 129 lines · Dr. David Alan Gilbert
Quoted 127 lines · * David Gibson ([email protected]) wrote:
Quoted 114 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +10 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 10 lines
migration/migration.c +10 -0
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 7 lines
Quoted 3 lines

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

I don't think you need an atomic then. AFAIK an atomic_set() in isolation without some sort of atomic on the other side is pretty much meaningless.

The other side has an atomic_read:

                if (migrate_postcopy_ram() &&
                    s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
                    pend_nonpost <= max_size &&
                    atomic_read(&s->start_postcopy)) {

                    if (!postcopy_start(s, &old_vm_running)) {
                        current_active_type = MIGRATION_STATUS_POSTCOPY_ACTIVE;
                        entered_postcopy = true;
                    }
Quoted 7 lines
Quoted 1 line

so it is at least symmetric.

But still pointless. Atomicity isn't magic pixie dust; it only makes sense if you're making atomic specific operations that need to be. Simple integer loads and stores are already atomic. Unless at least some of the atomic operations are something more complex, there's really no point to atomically marked operations.

OK, I'll kill it off.

It'll work in practice, but I still believe that what you're saying isn't safe C:
1) There's no barrier after the write, so there's no guarantee the other thread will eventually see it (in practice we've got other pthread ops we take so we will get a barrier somewhere, and most CPUs eventually do propagate the store).

2) The read side could legally be optimised out of the loop by the compiler. (but in practice wont be because compilers won't optimise that far).

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#112 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 161 lines · Dr. David Alan Gilbert
Quoted 152 lines · * David Gibson ([email protected]) wrote:
Quoted 151 lines · Dr. David Alan Gilbert
Quoted 130 lines · * David Gibson ([email protected]) wrote:
Quoted 129 lines · Dr. David Alan Gilbert
Quoted 127 lines · * David Gibson ([email protected]) wrote:
Quoted 114 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration/migration.c         | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index e37bc8b..03b8b78 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index b47f331..df9736c 100644
@@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index e6a814a..293c83e 100644
@@ -104,6 +104,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration/migration.c +10 -0
index a4fc7d7..43ca656 100644
@@ -372,6 +372,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+

migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 394

Quoted 10 lines
migration/migration.c +10 -0
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    atomic_set(&s->start_postcopy, true);

Why atomic_set?

Quoted 7 lines
Quoted 3 lines

It's being read by the migration thread, this is happening in the main thread.

There's no strict ordering requirement or anything.

I don't think you need an atomic then. AFAIK an atomic_set() in isolation without some sort of atomic on the other side is pretty much meaningless.

The other side has an atomic_read:

                if (migrate_postcopy_ram() &&
                    s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
                    pend_nonpost <= max_size &&
                    atomic_read(&s->start_postcopy)) {

                    if (!postcopy_start(s, &old_vm_running)) {
                        current_active_type = MIGRATION_STATUS_POSTCOPY_ACTIVE;
                        entered_postcopy = true;
                    }
Quoted 7 lines
Quoted 1 line

so it is at least symmetric.

But still pointless. Atomicity isn't magic pixie dust; it only makes sense if you're making atomic specific operations that need to be. Simple integer loads and stores are already atomic. Unless at least some of the atomic operations are something more complex, there's really no point to atomically marked operations.

Quoted 8 lines

OK, I'll kill it off.

It'll work in practice, but I still believe that what you're saying isn't safe C:
1) There's no barrier after the write, so there's no guarantee the other thread will eventually see it (in practice we've got other pthread ops we take so we will get a barrier somewhere, and most CPUs eventually do propagate the store).

Sorry, I should have been clearer. If you need a memory barrier, by all means include a memory barrier. But that should be explicit:
atomic set/read operations often include barriers, but it's not obvious which side will include what barrier.

Quoted 2 lines

2) The read side could legally be optimised out of the loop by the compiler. (but in practice wont be because compilers won't optimise that far).

That one's a trickier question. Compilers are absolutely capable of optimizing that far, *but* the C rules about when it's allowed to assume in-memory values remain unchanged are pretty conservative. I think any function call in the loop will require it to reload the value, for example. That said, a (compiler only) memory barrier might be appropriate to ensure that reload.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#113 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 8 lines · David Gibson
Quoted 4 lines
Quoted 4 lines

1) There's no barrier after the write, so there's no guarantee the other thread will eventually see it (in practice we've got other pthread ops we take so we will get a barrier somewhere, and most CPUs eventually do propagate the store).

Sorry, I should have been clearer. If you need a memory barrier, by all means include a memory barrier. But that should be
explicit: atomic set/read operations often include barriers, but it's not obvious which side will include what barrier.

Memory barriers are not needed here. The variable is set independently from every other set. There's no ordering.

atomic_read/atomic_set do not provide sequential consistency. That's ensured instead by atomic_mb_read/atomic_mb_set (and you're right that it's not obvious which side will include barriers, so you have to use the two together).

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

2) The read side could legally be optimised out of the loop by the compiler. (but in practice wont be because compilers won't optimise that far).

That one's a trickier question. Compilers are absolutely capable of optimizing that far, *but* the C rules about when it's allowed to assume in-memory values remain unchanged are pretty conservative. I think any function call in the loop will require it to reload the value, for example. That said, a (compiler only) memory barrier might be appropriate to ensure that reload.

That's exactly what atomic_read provides.

Paolo

#114 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 23 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 8 lines · David Gibson
Quoted 4 lines
Quoted 4 lines

1) There's no barrier after the write, so there's no guarantee the other thread will eventually see it (in practice we've got other pthread ops we take so we will get a barrier somewhere, and most CPUs eventually do propagate the store).

Sorry, I should have been clearer. If you need a memory barrier, by all means include a memory barrier. But that should be
explicit: atomic set/read operations often include barriers, but it's not obvious which side will include what barrier.

Memory barriers are not needed here. The variable is set independently from every other set. There's no ordering.

atomic_read/atomic_set do not provide sequential consistency. That's ensured instead by atomic_mb_read/atomic_mb_set (and you're right that it's not obvious which side will include barriers, so you have to use the two together).

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

2) The read side could legally be optimised out of the loop by the compiler. (but in practice wont be because compilers won't optimise that far).

Quoted 8 lines
Quoted 6 lines

That one's a trickier question. Compilers are absolutely capable of optimizing that far, *but* the C rules about when it's allowed to assume in-memory values remain unchanged are pretty conservative. I think any function call in the loop will require it to reload the value, for example. That said, a (compiler only) memory barrier might be appropriate to ensure that reload.

That's exactly what atomic_read provides.

So does that say I need the atomic_read but not the atomic_write - which seems a bit weird, but I think only due to the naming.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#115 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 10 lines · Dr. David Alan Gilbert
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

That one's a trickier question. Compilers are absolutely capable of optimizing that far, *but* the C rules about when it's allowed to assume in-memory values remain unchanged are pretty conservative. I think any function call in the loop will require it to reload the value, for example. That said, a (compiler only) memory barrier might be appropriate to ensure that reload.

That's exactly what atomic_read provides.

So does that say I need the atomic_read but not the atomic_write - which seems a bit weird, but I think only due to the naming.

No, you need both even though it's even more far-fetched that the compiler will do something bad with the set.

Paolo

#116 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

That one's a trickier question. Compilers are absolutely capable of optimizing that far, *but* the C rules about when it's allowed to assume in-memory values remain unchanged are pretty conservative. I think any function call in the loop will require it to reload the value, for example. That said, a (compiler only) memory barrier might be appropriate to ensure that reload.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

That's exactly what atomic_read provides.

So does that say I need the atomic_read but not the atomic_write - which seems a bit weird, but I think only due to the naming.

No, you need both even though it's even more far-fetched that the compiler will do something bad with the set.

OK, done - it's back to where it was with atomic_set/atomic_read.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#117 lore raw

Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

In reply to Dr. David Alan Gilbert
Quoted 6 lines · Dr. David Alan Gilbert
Quoted 5 lines
Quoted 5 lines

But still pointless. Atomicity isn't magic pixie dust; it only makes sense if you're making atomic specific operations that need to be. Simple integer loads and stores are already atomic. Unless at least some of the atomic operations are something more complex, there's really no point to atomically marked operations.

OK, I'll kill it off.

No, don't.

And both of you, read docs/atomics.txt.

"atomic_read() and atomic_set() prevents the compiler from using optimizations that might otherwise optimize accesses out of existence on the one hand, or that might create unsolicited accesses on the other. [...] it tells readers which variables are shared with
other threads, and which are local to the current thread or protected by other, more mundane means."

atomic_read() and atomic_set() provide the same guarantees as ACCESS_ONCE in the Linux kernel.

Paolo