Postcopy implementation
v7 cover Postcopy implementation
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 7th cut of my version of postcopy; it is designed for use with
the Linux kernel additions posted by Andrea Arcangeli here:
git clone --reference linux -b userfault21git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git
Note this API is slightly different from the last version; but the code is now in the linux-mm tree, the API is getting stable and the kernel code is just getting fixes now.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v7 tag.
It addresses most of the previous review comments, but there are still one or two I'm working.
As with v6, the userfaultfd.h isn't included in the tree, so you'll need a userfaultfd.h header (and syscall define) from Andrea's kernel.
This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/
v7
updated to Andrea's userfault 21 interface
Don't restart the source after an error if we entered postcopy (thanks to Li Liang for reporting) Killed off dead local_tmp_page (thanks to Christoph Seifert for spotting that)
Made sure the incoming page mutex is only initialised once
Moved request alignment from destination to source; destination now just verifies the request is aligned Got rid of the 'dup' on the return-fd
Just use the same fd and make the forward path own the fd
Reworked the consumption of request pages off the queue to handle hosts with larger
pages more cleanly
Disallowed enabling postcopy+compression
This is probably fixable, but it needs reworking of the decompression threads
to place the pages atomically
Disallow postcopy+RDMA
This is probably trickier to fix; RDMA drops the received data straight into memory using DMA; we would need to find a way to stop that. I can see a way to make it so that the precopy phase still uses RDMA and then stops using it later, but that would
make it harder when a real fix came along.
Minor fixups from review
6 of the smaller patches from v6 are now already in head
TODO
Testing on machines with hps!=tps
Tidy up the hps!=tps code on the receive side
Dr. David Alan Gilbert (42):
Start documenting how postcopy works.
Provide runtime Target page information
Init page sizes in qtest
qemu_ram_block_from_host
Add qemu_get_buffer_less_copy to avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile
ram_debug_dump_bitmap: Dump a migration bitmap as text
migrate_init: Call from savevm
Rename save_live_complete to save_live_complete_precopy Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modify save_live_pending for postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Add qemu_savevm_state_complete_postcopy
Postcopy: Maintain sentmap and calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy end in migration_thread
Page request: Add MIG_RP_MSG_REQ_PAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration Don't sync dirty bitmaps in postcopy
Host page!=target page: Cleanup bitmaps
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_handle_ commands
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopy
balloon.c | 11 + docs/migration.txt | 167 +++++++ exec.c | 64 ++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/ppc/spapr.c | 2 +- hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-all.h | 2 - include/exec/cpu-common.h | 3 + include/migration/migration.h | 112 ++++- include/migration/postcopy-ram.h | 88 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 8 +- include/qemu/typedefs.h | 3 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 44 +- migration/Makefile.objs | 2 +- migration/block.c | 9 +- migration/migration.c | 736 ++++++++++++++++++++++++++++-- migration/postcopy-ram.c | 706 +++++++++++++++++++++++++++++ migration/qemu-file-unix.c | 110 ++++- migration/qemu-file.c | 74 ++++ migration/ram.c | 933 ++++++++++++++++++++++++++++++++++++--- migration/savevm.c | 776 +++++++++++++++++++++++++++++--- qapi-schema.json | 18 +- qmp-commands.hx | 19 + qtest.c | 1 + trace-events | 76 +++- 29 files changed, 3788 insertions(+), 215 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 migration/postcopy-ram.c
Signature · 1 line
2.4.3
PATCH v7 1/42 Start documenting how postcopy works.
From: "Dr. David Alan Gilbert" <[email protected]>
--- docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)
docs/migration.txt +167 -0
@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. + += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the Postcopy destination +needs to be able to request pages on demand from the source. + +For these scenarios there is a 'return path' from the destination to the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return +path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread AND postcopy + thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU. + +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it.
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 24 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)docs/migration.txt +0
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middleof a pio operation
(that is whatide_drive_pio_state_needed()checks). IfDRQ_STATis not enabled, the values on that fields are garbage and don't need to be sent.docs/migration.txt +8
-0+ += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the
Quoted 1 line
Postcopy destination
This line is a bit long O:-)
In general, we are too near to the 80 columns limit.
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 29 lines · * Juan Quintela ([email protected]) wrote:
Quoted 24 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)docs/migration.txt +0
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middleof a pio operation
(that is whatide_drive_pio_state_needed()checks). IfDRQ_STATis not enabled, the values on that fields are garbage and don't need to be sent.docs/migration.txt +8
-0+ += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular theQuoted 1 line
Postcopy destination
This line is a bit long O:-)
Quoted 1 line
In general, we are too near to the 80 columns limit.
Thanks, Fixed (interesting, the check-patch doesn't seem to moan about text files).
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
In reply to Dr. David Alan Gilbert (git)Quoted 33 lines
docs/migration.txt index f6df4be..b4b93d1 renamed from docs/migration.txt +0
-0100644
docs/migration.txt +0
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middle of apio operation (that is what
ide_drive_pio_state_needed()checks). IfDRQ_STATis not enabled, the values on that fields are garbage and don't need to be sent.docs/migration.txt +6
-0+ += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd +(although possibly with another fd or similar for some fast way of throwingpages across).
docs/migration.txt +3
-0+ +However, some uses need two way communication; in particular the +Postcopy destination needs to be able to request pages on demand fromthe source.
docs/migration.txt +13
-0+ +For these scenarios there is a 'return path' from the destination to +the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for +the return path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread ANDpostcopy
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 321
Quoted 8 lines
docs/migration.txt +8
-0+ thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to +converge; its plus side is that there is an upper bound on the amount +of migration traffic and time it takes, the down side is that during +the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost.
Hi David,
Do you have any idea or plan to deal with the failure happened during the postcopy phase?
Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.
Liang
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 44 lines · * Li, Liang Z ([email protected]) wrote:
Quoted 33 lines
docs/migration.txt index f6df4be..b4b93d1 renamed from docs/migration.txt +0
-0100644
docs/migration.txt +0
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middle of apio operation (that is what
ide_drive_pio_state_needed()checks). IfDRQ_STATis not enabled, the values on that fields are garbage and don't need to be sent.docs/migration.txt +6
-0+ += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd +(although possibly with another fd or similar for some fast way of throwingpages across).
docs/migration.txt +3
-0+ +However, some uses need two way communication; in particular the +Postcopy destination needs to be able to request pages on demand fromthe source.
docs/migration.txt +13
-0+ +For these scenarios there is a 'return path' from the destination to +the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for +the return path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread ANDpostcopy
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 321
Quoted 8 lines
docs/migration.txt +8
-0+ thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to +converge; its plus side is that there is an upper bound on the amount +of migration traffic and time it takes, the down side is that during +the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost.Hi David,
Quoted 5 lines
Do you have any idea or plan to deal with the failure happened during the postcopy phase?
Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.
The VM memory image is still on the source VM, so you can restart the source, however that's not safe, because once the destination has started running it is sending out packets and also modifying the block storage. If you restarted the source at that point what block and net state can you accept being visible?
Dave
Quoted 2 lines
Liang
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
In reply to Li, Liang ZQuoted 6 lines · Liang Z
Do you have any idea or plan to deal with the failure happened during the postcopy phase?
Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.
There's no solution to this problem, except for rollback to a previous snapshot.
To give an idea, an example of an intended usecase for postcopy is datacenter evacuation in 30 minutes after a tsunami alert. That's not a case where you care much about losing guests to network failures.
Why is there no solution? Let's look at one of the best surveys on migration,
http://courses.cs.vt.edu/~cs5204/fall05-kafura/Papers/Migration/ProcessMigration.pdf
(warning, 59 pages!):
[3.2] If only part of the task state is transferred to another node, the task can start executing sooner, and the initial migration costs are lower.
[3.4] Fault resilience can be improved in several ways. The impact of failures during migration can be reduced by maintaining process state on both the source and destination sites until the destination site instance is successfully promoted to a regular process and the source node is informed about this.
[3.5] Migration algorithms should avoid linear dependencies on the amount of state to be transferred. For example, the eager data transfer strategy has costs proportional to the address space size
"Pre"copy means "start copying *before* promoting the destination to be the primary host" and it has such a linear dependency on the amount of state to be transferred. "Post"copy means "delay some copying to *after* promoting the destination to be the primary host".
So we have:
Precopy Postcopy
3.2 Performance - (1) - (2) 3.4 Fault resilience + - 3.5 Scalability - +
(1) smaller impact, longer freeze time
(2) larger impact, extremely short freeze time
Postcopy can also limit the length of the non-resilient phase, by starting with a precopy phase and only switching to postcopy after some time. Then you have:
Precopy Hybrid Postcopy
3.2 Performance - (1) + (3) - (2) 3.4 Fault resilience + - -- 3.5 Scalability - + +
(3) intermediate impact, extremely short freeze time
but there is still going to be a phase where migration is not resilient to network faults.
Cloud operators can use a combination of precopy and postcopy. For example, I would not use postcopy for mass migration when doing host updates, but it can be used as a last resort before a scheduled downtime.
For example, say you're doing a rolling update and you want it complete by next Sunday. 90% of the guests are shut down by the customers or can be migrated successfully with precopy. The others do not converge and their SLA does not let you throttle them to complete precopy migration.
You then tell your customers that either they shutdown and restart their instances before Saturday 8:00 PM, or they might be shut down forcibly.
Then for customers who haven't rebooted you can do
postcopy---you have alerted them that something might go wrong. So even though postcopy would not be a first choice, it can still help cloud operators.
Paolo
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 6 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines · Liang Z
Do you have any idea or plan to deal with the failure happened during the postcopy phase?
Quoted 6 lines
Quoted 3 lines
Lost the guest is too frightening for a cloud provider, we have a discussion with Alibaba, they said that they can't use the postcopy feature unless there is a mechanism to find the guest back.
There's no solution to this problem, except for rollback to a previous snapshot.
Yes, and you might be able to avoid some of the pain if you COWd the disk data on the destination until the migration was finished; that would allow you to restart the source VM in the state prior to postcopy starting; although the network's view of it is going to be very messy.
Quoted 3 lines
To give an idea, an example of an intended usecase for postcopy is datacenter evacuation in 30 minutes after a tsunami alert. That's not a case where you care much about losing guests to network failures.
Well; you have to make a call as to what your best option is; you could always shut the VM down and boot it up fresh in your new safe data centre. Your preference is determined by your confidence that your VM would boot back up safely and how long it would take and the confidence in that network during the migration period and the pain of knowing what will happen if you explicitly shut the VM down.
Quoted 9 lines
Cloud operators can use a combination of precopy and postcopy. For example, I would not use postcopy for mass migration when doing host updates, but it can be used as a last resort before a scheduled downtime.
For example, say you're doing a rolling update and you want it complete by next Sunday. 90% of the guests are shut down by the customers or can be migrated successfully with precopy. The others do not converge and their SLA does not let you throttle them to complete precopy migration.
Indeed the interface lets you do that pretty easily; since as long as you have enabled postcopy, it starts in precopy mode and is fully recoverable until you issue the 'migrate_start_postcopy' which might be when it's tried 'n' times and you can see that the workload you have isn't going to converge.
Dave
Quoted 8 lines
You then tell your customers that either they shutdown and restart their instances before Saturday 8:00 PM, or they might be shut down forcibly.
Then for customers who haven't rebooted you can do
postcopy---you have alerted them that something might go wrong. So even though postcopy would not be a first choice, it can still help cloud operators.Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
In reply to Dr. David Alan Gilbert (git)Hi Dave,
Quoted 2 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[...]
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325
Quoted 9 lines
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;docs/migration.txt +7
-0+its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU.
I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- When start_postcopy issued, source will paused, and __open another socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.
The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.
I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?
Quoted 136 lines
docs/migration.txt +135
-0+ +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it.
Signature · 2 lines
Thanks, Yang.
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 16 lines · Yang Hongyang
Hi Dave,
Quoted 2 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[...]
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325
Quoted 9 lines
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;docs/migration.txt +7
-0+its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU.
Quoted 7 lines
I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- Whenstart_postcopyissued, source will paused, and__openanother socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.
Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy. Here it does not need any cache at all, it just places the dirty pages where it will be accessed.
Quoted 147 lines
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.
I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?
Quoted 136 lines
docs/migration.txt +135
-0+ +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it.
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 27 lines · zhanghailiang
Quoted 18 lines · Yang Hongyang
Hi Dave,
Quoted 2 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[...]
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325
Quoted 11 lines
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure ofdocs/migration.txt +5
-0+*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU.Quoted 7 lines
I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- Whenstart_postcopyissued, source will paused, and__openanother socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.
Quoted 6 lines
Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.
I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]
Quoted 157 lines
Quoted 144 lines
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.
I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?
Quoted 133 lines
(quoted diff) +2
-0+ +Postcopy can be combined with precopy (i.e. normal migration) so that ifprecopy
docs/migration.txt +22
-0+doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the streamcompletely
docs/migration.txt +14
-0+before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device statestream
docs/migration.txt +39
-0+ containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting thedestination
(quoted diff) +2
-0+CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal runningbehaviour
docs/migration.txt +49
-0+and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs.Quoted 9 lines
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 460
Quoted 8 lines
docs/migration.txt +7
-0+ +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it..
Signature · 2 lines
Thanks, Yang.
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 30 lines · * Yang Hongyang ([email protected]) wrote:
Quoted 27 lines · zhanghailiang
Quoted 25 lines · Yang Hongyang
Hi Dave,
Quoted 2 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[...]
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325
Quoted 11 lines
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure ofdocs/migration.txt +5
-0+*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU.I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- Whenstart_postcopyissued, source will paused, and__openanother socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.
Quoted 6 lines
Quoted 6 lines
Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.
Yes, zhanghailiang is correct; the source keeps sending other pages without being asked, however when asked it sends requested pages immediately. and the 'cache' is just the main memory from which the destination is working.
However, the idea of using a separate socket is one that we have been thinking about; one of the problems is that the urgent requested pages get delayed behind the background page transfer and that increases the latency; a separate socket should fix that.
Quoted 2 lines
I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]
Yes, I can add to the documentation; I've added the following text:
During postcopy the source scans the list of dirty pages and sends them to the destination without being requested (in much the same way as precopy), however when a page request is received from the destination the dirty page scanning restarts from the requested location. This causes requested pages to be sent quickly, and also causes pages directly after the requested page to be sent quickly in the hope that those pages are likely to be requested by the destination soon.
Dave
Quoted 155 lines
Quoted 155 lines
Quoted 153 lines
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.
I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?
Quoted 141 lines
(quoted diff) +2
-0+ +Postcopy can be combined with precopy (i.e. normal migration) so that ifprecopy
docs/migration.txt +22
-0+doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the streamcompletely
docs/migration.txt +14
-0+before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device statestream
docs/migration.txt +39
-0+ containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting thedestination
(quoted diff) +2
-0+CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal runningbehaviour
docs/migration.txt +56
-0+and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it.
Quoted 6 lines
Quoted 2 lines
.
Signature · 2 lines
Thanks, Yang.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 30 lines · * Yang Hongyang ([email protected]) wrote:
Quoted 27 lines · zhanghailiang
Quoted 25 lines · Yang Hongyang
Hi Dave,
Quoted 2 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[...]
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 325
Quoted 11 lines
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure ofdocs/migration.txt +5
-0+*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU.I have a immature idea,
Can we keep a source RAM cache on destination QEMU, instead of request to the source QEMU, that is:
- Whenstart_postcopyissued, source will paused, and__openanother socket (maybe another migration thread)__ to send the remaining dirty pages to destination, at the same time, destination will start, and cache the remaining pages.Quoted 6 lines
Quoted 6 lines
Er, it seems that current implementation is just like what you described except the ram cache:
After switch to post-copy mode, the source side will send the remaining dirty pages as pre-copy.
Here it does not need any cache at all, it just places the dirty pages where it will be accessed.
Quoted 8 lines
Yes, zhanghailiang is correct; the source keeps sending other pages without being asked, however when asked it sends requested pages immediately. and the 'cache' is just the main memory from which the destination is working.
However, the idea of using a separate socket is one that we have been thinking about; one of the problems is that the urgent requested pages get delayed behind the background page transfer and that increases the latency; a separate socket should fix that.
That would be better.
Quoted 6 lines
Quoted 3 lines
I haven't look into the implementation in detail, but if it is, I think it should be documented here...or in the below section [Source behaviour]
Yes, I can add to the documentation; I've added the following text:
Quoted 7 lines
During postcopy the source scans the list of dirty pages and sends them to the destination without being requested (in much the same way as precopy), however when a page request is received from the destination the dirty page scanning restarts from the requested location. This causes requested pages to be sent quickly, and also causes pages directly after the requested page to be sent quickly in the hope that those pages are likely to be requested by the destination soon.
Looks clearer for me now :)
Quoted 168 lines
Dave
Quoted 155 lines
Quoted 155 lines
Quoted 153 lines
- When the page fault occured, first lookup the page in the CACHE, if it is not yet received, request to the source QEMU.
- Once the remaining dirty pages are transfered, the source QEMU can go now.The existing postcopy mechanism does not need to be changed, just add the remaining page transfer mechanism, and the RAM cache.
I don't know if it is feasible and whether it will bring improvement to the postcopy, what do you think?
Quoted 141 lines
(quoted diff) +2
-0+ +Postcopy can be combined with precopy (i.e. normal migration) so that ifprecopy
docs/migration.txt +22
-0+doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the streamcompletely
docs/migration.txt +14
-0+before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device statestream
docs/migration.txt +39
-0+ containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting thedestination
(quoted diff) +2
-0+CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal runningbehaviour
docs/migration.txt +56
-0+and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command) + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that the contents of the sentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it.Quoted 6 lines
Quoted 2 lines
.
--
Thanks,
Yang.--
Dr. David Alan Gilbert / [email protected] / Manchester, UK .
Signature · 2 lines
Thanks, Yang.
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
In reply to Dr. David Alan Gilbert (git)Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>
A few minor comments:
Quoted 27 lines
--- docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)docs/migration.txt +15
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. + += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the Postcopy destination +needs to be able to request pages on demand from the source. + +For these scenarios there is a 'return path' from the destination to the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return +path. + + Source side
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 318
Quoted 10 lines
docs/migration.txt +10
-0+ Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread AND postcopy + thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to converge;
(or take too long to converge)
Quoted 4 lines
docs/migration.txt +4
-0+its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. +
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 332
Quoted 10 lines
docs/migration.txt +10
-0+In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU. + +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration):
How about this instead:
"To enable postcopy, issue this command ont he monitor prior to the start of migration:"
Otherwise, there's ambiguity that there is some way to enable this after a precopy migration has started.
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 336
Quoted 4 lines
docs/migration.txt +4
-0+ +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing:
"A future migration will then start in precopy mode. However, issuing:"
?
Quoted 71 lines
docs/migration.txt +71
-0+ +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. +
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 417
Quoted 10 lines
docs/migration.txt +10
-0+=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command)
Adding:
"This gives the destination a chance to fail early if postcopy is not possible."
?
Quoted 6 lines
docs/migration.txt +6
-0+ + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 433
Quoted 10 lines
docs/migration.txt +10
-0+ reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration).
indentation went off a bit
Amit
Re: [Qemu-devel] [PATCH v7 01/42] Start documenting how postcopy works.
Quoted 40 lines · * Amit Shah ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>A few minor comments:
Quoted 31 lines
--- docs/migration.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)docs/migration.txt +19
-0@@ -291,3 +291,170 @@ save/send this state when we are in the middle of a pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. + += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the Postcopy destination +needs to be able to request pages on demand from the source. + +For these scenarios there is a 'return path' from the destination to the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return +path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side
Quoted 8 lines
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 318
Quoted 6 lines
docs/migration.txt +6
-0+ Forward path - read by main thread + Return path - opened by main thread, written by main thread AND postcopy + thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to converge;(or take too long to converge)
Added.
Quoted 18 lines
Quoted 5 lines
docs/migration.txt +4
-0+its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. +docs/migration.txt save/send this state when we are in the middle of a pio operation · line 332
Quoted 10 lines
docs/migration.txt +10
-0+In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU. + +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration):How about this instead:
Quoted 5 lines
"To enable postcopy, issue this command ont he monitor prior to the start of migration:"
Otherwise, there's ambiguity that there is some way to enable this after a precopy migration has started.
Done.
Quoted 9 lines
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 336
Quoted 4 lines
docs/migration.txt +4
-0+ +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing:"A future migration will then start in precopy mode. However, issuing:"
?
Ah yes, I see it's ambiguous because it doesn't say you still need to do the normal migration stuff to start migration;
I've changed it to:
The normal commands are then used to start a migration, which is still started in precopy mode. Issuing:
migrate_start_postcopy
will now cause the transition from precopy to postcopy.
Quoted 84 lines
Quoted 71 lines
docs/migration.txt +71
-0+ +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. +docs/migration.txt save/send this state when we are in the middle of a pio operation · line 417
Quoted 10 lines
docs/migration.txt +10
-0+=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_ADVISE command)Adding:
Quoted 4 lines
"This gives the destination a chance to fail early if postcopy is not possible."
?
I added:
"The destination will fail early in migration at this point if the
required OS support is not present. "
Quoted 10 lines
Quoted 10 lines
docs/migration.txt +10
-0+ + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). +
Quoted 8 lines
docs/migration.txt save/send this state when we are in the middle of a pio operation · line 433
Quoted 6 lines
docs/migration.txt +6
-0+ Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration).indentation went off a bit
Fixed.
Thanks,
Dave
Quoted 3 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 2/42 Provide runtime Target page information
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into migration/ram.c.
Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.
--- exec.c | 10 ++++++++++ include/sysemu/sysemu.h | 1 + 2 files changed, 11 insertions(+)
exec.c +10 -0
@@ -3313,6 +3313,16 @@ int cpu_memory_rw_debug(CPUState *cpu, target_ulong addr, }return 0 ; } + +/* +* Allows code that needs to deal with migration bitmaps etc to still be built +* target independent. +*/ +size_t qemu_target_page_bits (void ) +{ +return TARGET_PAGE_BITS ; +} +#endif /*
include/sysemu/sysemu.h +1 -0
@@ -68,6 +68,7 @@ int qemu_reset_requested_get(void);void qemu_system_killed (int signal,pid_t pid);void qemu_devices_reset (void );void qemu_system_reset (bool report); +size_t qemu_target_page_bits (void );void qemu_add_exit_notifier (Notifier *notify);void qemu_remove_exit_notifier (Notifier *notify);
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 02/42] Provide runtime Target page information
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into
migration/ram.c.Provide '
qemu_target_page_bits()' that returnsTARGET_PAGE_BITSto other bits of code so that they can stay target-independent.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>
PATCH v7 3/42 Init page sizes in qtest
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
One of my patches used a loop that was based on host page size; it dies in qtest since qtest hadn't bothered init'ing it.
--- qtest.c | 1 + 1 file changed, 1 insertion(+)
qtest.c +1 -0
@@ -657,6 +657,7 @@ void qtest_init(const char *qtest_chrdev, const char *qtest_log, Error **errp) inbuf =g_string_new ("" ); qtest_chr = chr; +page_size_init (); }bool qtest_driver (void )
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 03/42] Init page sizes in qtest
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
One of my patches used a loop that was based on host page size; it dies in qtest since qtest hadn't bothered init'ing it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Re: [Qemu-devel] [PATCH v7 03/42] Init page sizes in qtest
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
One of my patches used a loop that was based on host page size; it dies in qtest since qtest hadn't bothered init'ing it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
Re: [Qemu-devel] [PATCH v7 03/42] Init page sizes in qtest
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
One of my patches used a loop that was based on host page size; it dies in qtest since qtest hadn't bothered init'ing it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 4/42 qemu_ram_block_from_host
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.
qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock and the global ram_addr_t value.
Rewrite qemu_ram_addr_from_host to use qemu_ram_block_from_host.
Provide qemu_ram_get_idstr since its the actual name text sent on the wire.
--- exec.c | 54 +++++++++++++++++++++++++++++++++++++++-------- include/exec/cpu-all.h | 2 -- include/exec/cpu-common.h | 3 +++ include/qemu/typedefs.h | 1 + 4 files changed, 49 insertions(+), 11 deletions(-)
exec.c +45 -9
@@ -1265,6 +1265,11 @@ static RAMBlock *find_ram_block(ram_addr_t addr)return NULL ; } +const char *qemu_ram_get_idstr (RAMBlock *rb) +{ +return rb->idstr; +} +/* Called with iothread lock held. */ void qemu_ram_set_idstr (ram_addr_t addr,const char *name,DeviceState *dev) {
@@ -1755,8 +1760,16 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size) } } -/* Some of the softmmu routines need to translate from a host pointer -* (typically a TLB entry) back to a ram offset. +/* +* Translates a host ptr back to a RAMBlock, a ram_addr and an offset +* in that RAMBlock. +* +* ptr: Host pointer to look up +* round_offset: If true round the result offset down to a page boundary +* *ram_addr: set to result ram_addr +* *offset: set to result offset within the RAMBlock +* +* Returns: RAMBlock (or NULL if not found) * * By the time this function returns, the returned pointer is not protected * by RCU anymore. If the caller is not within an RCU critical section and
@@ -1764,18 +1777,22 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size)* pointer, such as a reference to the region that includes the incoming * ram_addr_t. */ -MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr) +RAMBlock *qemu_ram_block_from_host (void *ptr,bool round_offset, +ram_addr_t *ram_addr, +ram_addr_t *offset) {RAMBlock *block;uint8_t *host = ptr; -MemoryRegion *mr;if (xen_enabled ()) {rcu_read_lock (); *ram_addr =xen_ram_addr_from_mapcache (ptr); - mr =qemu_get_ram_block (*ram_addr)->mr; + block =qemu_get_ram_block (*ram_addr); +if (block) { + *offset = (host - block->host); + }rcu_read_unlock (); -return mr; +return block; }rcu_read_lock ();
@@ -1798,10 +1815,29 @@ MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)return NULL ; found: - *ram_addr = block->offset + (host - block->host); - mr = block->mr; + *offset = (host - block->host); +if (round_offset) { + *offset &=TARGET_PAGE_MASK ; + } + *ram_addr = block->offset + *offset;rcu_read_unlock (); -return mr; +return block; +} + +/* Some of the softmmu routines need to translate from a host pointer +(typically a TLB entry) back to a ram offset. */ +MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr) +{ +RAMBlock *block; +ram_addr_t offset;/* Not used */ + + block =qemu_ram_block_from_host (ptr, false, ram_addr, &offset); + +if (!block) { +return NULL ; + } + +return block->mr; } static void notdirty_mem_write(void *opaque, hwaddr ram_addr,
include/exec/cpu-all.h +0 -2
@@ -266,8 +266,6 @@ CPUArchState *cpu_copy(CPUArchState *env);/* memory API */ -typedef struct RAMBlock RAMBlock ; -struct RAMBlock {struct rcu_head rcu;struct MemoryRegion *mr;
include/exec/cpu-common.h +3 -0
@@ -62,8 +62,11 @@ typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);void qemu_ram_remap (ram_addr_t addr,ram_addr_t length);/* This should not be used by devices. */ MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr); +RAMBlock *qemu_ram_block_from_host (void *ptr,bool round_offset, +ram_addr_t *ram_addr,ram_addr_t *offset);void qemu_ram_set_idstr (ram_addr_t addr,const char *name,DeviceState *dev);void qemu_ram_unset_idstr (ram_addr_t addr); +const char *qemu_ram_get_idstr (RAMBlock *rb);void cpu_physical_memory_rw (hwaddr addr,uint8_t *buf,int len,int is_write);
include/qemu/typedefs.h +1 -0
@@ -70,6 +70,7 @@ typedef struct QEMUSGList QEMUSGList;typedef struct QEMUSizedBuffer QEMUSizedBuffer ;typedef struct QEMUTimerListGroup QEMUTimerListGroup ;typedef struct QEMUTimer QEMUTimer ; +typedef struct RAMBlock RAMBlock ;typedef struct Range Range ;typedef struct SerialState SerialState ;typedef struct SHPCDevice SHPCDevice ;
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 04/42] qemu_ram_block_from_host
Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.
qemu_ram_block_from_hosttranslates a HVA into a RAMBlock, an offset in the RAMBlock and the globalram_addr_tvalue.
Quoted 7 lines
Rewrite
qemu_ram_addr_from_hostto useqemu_ram_block_from_host.Provide
qemu_ram_get_idstrsince its the actual name text sent on the wire.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Re: [Qemu-devel] [PATCH v7 04/42] qemu_ram_block_from_host
In reply to Dr. David Alan Gilbert (git)Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.
qemu_ram_block_from_hosttranslates a HVA into a RAMBlock, an offset in the RAMBlock and the globalram_addr_tvalue.
Quoted 7 lines
Rewrite
qemu_ram_addr_from_hostto useqemu_ram_block_from_host.Provide
qemu_ram_get_idstrsince its the actual name text sent on the wire.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Amit
PATCH v7 5/42 Add qemu_get_buffer_less_copy to avoid copies some of the time
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.
Thus 'qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.
--- include/migration/qemu-file.h | 2 ++ migration/qemu-file.c | 47 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)
include/migration/qemu-file.h +2 -0
@@ -162,6 +162,8 @@ int qemu_get_buffer(QEMUFile *f, uint8_t *buf, int size);ssize_t qemu_put_compression_data (QEMUFile *f,const uint8_t *p,size_t size,int level);int qemu_put_qemu_file (QEMUFile *f_des,QEMUFile *f_src); +int qemu_get_buffer_less_copy (QEMUFile *f,uint8_t **buf,int size); +/* * Note that you can only peek continuous bytes from where the current pointer * is; you aren't guaranteed to be able to peak to +n bytes unless you've
migration/qemu-file.c +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_bufferalways copies the data it reads to a users buffer, however in many cases the file buffer insideqemu_filecould be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.Thus '
qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.Signed-off-by Dr. David Alan Gilbert <[email protected]>
I don't know still where this function is used, but function is correct.
Can I suggest to change th ename to:
qemu_get_buffer_in_place()?
less_copy sounds ambigous to me :p
Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time
Quoted 15 lines · * Juan Quintela ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_bufferalways copies the data it reads to a users buffer, however in many cases the file buffer insideqemu_filecould be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.Thus '
qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>
Quoted 7 lines
I don't know still where this function is used, but function is correct.
Can I suggest to change th ename to:
qemu_get_buffer_in_place()?
less_copysounds ambigous to me :p
Changed.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 05/42] Add qemu_get_buffer_less_copy to avoid copies some of the time
In reply to Dr. David Alan Gilbert (git)Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_bufferalways copies the data it reads to a users buffer, however in many cases the file buffer insideqemu_filecould be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.Thus '
qemu_get_buffer_less_copy' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 6/42 Add wrapper for setting blocking status on a QEMUFile
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.
--- include/migration/qemu-file.h | 1 + migration/qemu-file.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f); +void qemu_file_change_blocking (QEMUFile *f,bool block);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv) {
migration/qemu-file.c +15 -0
@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])return res == len ? res :0 ; } + +/* +* Change the blocking state of the QEMUFile. +* Note: On some transports the OS only keeps a single blocking state for +* both directions, and thus changing the blocking on the main +* QEMUFile can also affect the return path. +*/ +void qemu_file_change_blocking (QEMUFile *f,bool block) +{ +if (block) { +qemu_set_block (qemu_get_fd (f)); + }else { +qemu_set_nonblock (qemu_get_fd (f)); + } +}
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>
Can we improve naming?
Quoted 23 lines
--- include/migration/qemu-file.h | 1 + migration/qemu-file.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+)include/migration/qemu-file.h +1
-0@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f); +void qemu_file_change_blocking (QEMUFile *f,bool block);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv) {migration/qemu-file.c +0
-0@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661
Quoted 10 lines
migration/qemu-file.c +8
-0return res == len ? res :0 ; } + +/* +* Change the blocking state of the QEMUFile. +* Note: On some transports the OS only keeps a single blocking state for +* both directions, and thus changing the blocking on the main +* QEMUFile can also affect the return path. +*/ +void qemu_file_change_blocking (QEMUFile *f,bool block)
qemu_file_set_blocking?
It don't change the blocking, it just do whatever block says?
Quoted 7 lines
migration/qemu-file.c +7
-0+{ +if (block) { +qemu_set_block (qemu_get_fd (f)); + }else { +qemu_set_nonblock (qemu_get_fd (f)); + } +}
Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>Reviewed-by Juan Quintela <[email protected]>Can we improve naming?
Quoted 23 lines
--- include/migration/qemu-file.h | 1 + migration/qemu-file.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+)include/migration/qemu-file.h +1
-0@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f); +void qemu_file_change_blocking (QEMUFile *f,bool block);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv) {migration/qemu-file.c +0
-0@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661
Quoted 10 lines
migration/qemu-file.c +8
-0return res == len ? res :0 ; } + +/* +* Change the blocking state of the QEMUFile. +* Note: On some transports the OS only keeps a single blocking state for +* both directions, and thus changing the blocking on the main +* QEMUFile can also affect the return path. +*/ +void qemu_file_change_blocking (QEMUFile *f,bool block)
qemu_file_set_blocking?It don't change the blocking, it just do whatever block says?
Quoted 7 lines
migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 668
Quoted 7 lines
migration/qemu-file.c +7
-0+{ +if (block) { +qemu_set_block (qemu_get_fd (f)); + }else { +qemu_set_nonblock (qemu_get_fd (f)); + } +}
I worry about having a:
qemu_file_set_blocking
and a
qemu_set_block
it sounds a bit similar when one always 'sets' (i.e. turns on) and the other either turns on or off.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 06/42] Add wrapper for setting blocking status on a QEMUFile
Quoted 68 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>Reviewed-by Juan Quintela <[email protected]>Can we improve naming?
Quoted 23 lines
--- include/migration/qemu-file.h | 1 + migration/qemu-file.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+)include/migration/qemu-file.h +1
-0@@ -193,6 +193,7 @@ int qemu_file_get_error(QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f); +void qemu_file_change_blocking (QEMUFile *f,bool block);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv) {migration/qemu-file.c +0
-0@@ -651,3 +651,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 661
Quoted 10 lines
migration/qemu-file.c +8
-0return res == len ? res :0 ; } + +/* +* Change the blocking state of the QEMUFile. +* Note: On some transports the OS only keeps a single blocking state for +* both directions, and thus changing the blocking on the main +* QEMUFile can also affect the return path. +*/ +void qemu_file_change_blocking (QEMUFile *f,bool block)
qemu_file_set_blocking?It don't change the blocking, it just do whatever block says?
Quoted 7 lines
migration/qemu-file.c size_t qemu_get_counted_string(QEMUFile *f, char buf[256]) · line 668
Quoted 7 lines
migration/qemu-file.c +7
-0+{ +if (block) { +qemu_set_block (qemu_get_fd (f)); + }else { +qemu_set_nonblock (qemu_get_fd (f)); + } +}I worry about having a:
qemu_file_set_blocking
and a
qemu_set_block
Quoted 2 lines
it sounds a bit similar when one always 'sets' (i.e. turns on) and the other either turns on or off.
There is a parameter difference, but I am not writting the code, and don't care so much. I would expect a function with change in its name to change to the other plocking, whatever that is :P
Later, Juan.
PATCH v7 7/42 ram_debug_dump_bitmap: Dump a migration bitmap as text
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.
--- include/migration/migration.h | 1 + migration/ram.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/migration/migration.h +1 -0
@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +38 -0
@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of; it won't bother printing lines that are all this value. +* If 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128 ; +char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + } + +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur + linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur + curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ; + found = found || (thisbit != expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +}/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has * long-running RCU critical section. When rcu-reclaims in the code
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text
Quoted 41 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/ram.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)include/migration/migration.h +1
-0@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding migration/ram.c +6
-0@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of; it won't bother printing lines that are all this value. +* If 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected)
migration/ram.c static void reset_ram_globals(void) · line 1069
Quoted 10 lines
migration/ram.c +10
-0+{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128 ; +char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + }
Why? Just alssert that todump!= NULL?
Quoted 4 lines
migration/ram.c +4
-0+ +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false;
migration/ram.c static void reset_ram_globals(void) · line 1083
Quoted 10 lines
migration/ram.c +10
-0+/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur + linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur + curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ;
Put 1 and 0? Why the dot?
migration/ram.c static void reset_ram_globals(void) · line 1091
Quoted 8 lines
migration/ram.c +8
-0+ found = found || (thisbit != expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +}
And once here, why are we doing it this way? We have
find_first_bit(addr, nbits) and find_first_zero_bit(addr, nbits) and friends?
Doiwg the walk by hand looks weird, no?
Later, Juan.
Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text
Quoted 46 lines · * Juan Quintela ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/ram.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)include/migration/migration.h +1
-0@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding migration/ram.c +10
-0@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of; it won't bother printing lines that are all this value. +* If 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur;
Quoted 8 lines
migration/ram.c static void reset_ram_globals(void) · line 1069
Quoted 6 lines
migration/ram.c +6
-0+int64_t linelen =128 ; +char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + }Why? Just alssert that todump!= NULL?
'migration_bitmap' is static to ram.c, so allowing NULL to get you a dump of the migration_bitmap means that if you call this dump routine from any error path you're debugging anywhere in qemu then you can dump the migration bitmap. e.g. I was adding calls to this in migration.c and migration/postcopy-ram.c in error paths I was trying to debug.
Quoted 14 lines
migration/ram.c static void reset_ram_globals(void) · line 1083
Quoted 13 lines
migration/ram.c +13
-0+for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur + linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur + curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ;
Quoted 1 line
Put 1 and 0? Why the dot?
It's easier to see an occasional '1' in a big field of .'s.
Quoted 10 lines
migration/ram.c static void reset_ram_globals(void) · line 1091
Quoted 8 lines
migration/ram.c +8
-0+ found = found || (thisbit != expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +}
Quoted 6 lines
And once here, why are we doing it this way? We have
find_first_bit(addr, nbits)andfind_first_zero_bit(addr, nbits)and friends?Doiwg the walk by hand looks weird, no?
Here's a compile-tested-only version using find_ - it's bigger, if you think it's better I can use this instead:
void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
{
int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
int64_t cur;
int64_t linelen = 128;
if (!todump) {
todump = migration_bitmap;
}
for (cur = 0; cur < ram_pages; cur += linelen) {
int64_t curb;
unsigned long next_bit;
/*
* Last line; catch the case where the line length
* is longer than remaining ram
*/
if (cur + linelen > ram_pages) {
linelen = ram_pages - cur;
}
if (expected) {
next_bit = find_next_bit(todump, cur + linelen, cur);
} else {
next_bit = find_next_zero_bit(todump, cur + linelen, cur);
}
if (next_bit >= (cur + linelen)) {
continue;
}
for (curb = 0; curb < linelen; curb++) {
bool thisbit = test_bit(cur + curb, todump);
fputc(thisbit ? '1' : '.', stderr);
}
fputc('\n', stderr);
}
}Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Juan Quintela
Reviewed-byRe: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text
Quoted 56 lines · Dr. David Alan Gilbert
Quoted 46 lines · * Juan Quintela ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/ram.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)include/migration/migration.h +1
-0@@ -144,6 +144,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding migration/ram.c +10
-0@@ -1051,6 +1051,44 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of; it won't bother printing lines that are all this value. +* If 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur;Quoted 8 lines
migration/ram.c static void reset_ram_globals(void) · line 1069
Quoted 6 lines
migration/ram.c +6
-0+int64_t linelen =128 ; +char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + }Why? Just alssert that todump!= NULL?
Quoted 6 lines
'
migration_bitmap' is static toram.c, so allowing NULL to get you a dump of themigration_bitmapmeans that if you call this dump routine from any error path you're debugging anywhere in qemu then you can dump the migration bitmap. e.g. I was adding calls to this inmigration.candmigration/postcopy-ram.cin error paths I was trying to debug.
ok.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
migration/ram.c +13
-0+for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur + linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur + curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ;
Quoted 3 lines
Quoted 1 line
Put 1 and 0? Why the dot?
It's easier to see an occasional '1' in a big field of .'s.
ok.
Quoted 60 lines
Quoted 11 lines
migration/ram.c static void reset_ram_globals(void) · line 1091
Quoted 9 lines
migration/ram.c +8
-0+ found = found || (thisbit != expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +}Quoted 6 lines
And once here, why are we doing it this way? We have
find_first_bit(addr, nbits)andfind_first_zero_bit(addr, nbits)and friends?Doiwg the walk by hand looks weird, no?
Here's a compile-tested-only version using
find_- it's bigger, if you think it's better I can use this instead:void
ram_debug_dump_bitmap(unsigned long *todump, bool expected){ int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS; int64_t cur; int64_t linelen = 128; if (!todump) { todump = migration_bitmap; } for (cur = 0; cur < ram_pages; cur += linelen) { int64_t curb; unsigned long next_bit; /* * Last line; catch the case where the line length * is longer than remaining ram */ if (cur + linelen > ram_pages) { linelen = ram_pages - cur; } if (expected) { next_bit = find_next_bit(todump, cur + linelen, cur); } else { next_bit = find_next_zero_bit(todump, cur + linelen, cur); } if (next_bit >= (cur + linelen)) { continue; } for (curb = 0; curb < linelen; curb++) { bool thisbit = test_bit(cur + curb, todump); fputc(thisbit ? '1' : '.', stderr); } fputc('\n', stderr); } }
Quoted 1 line
Dave
Quoted 6 lines
Quoted 4 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 07/42] ram_debug_dump_bitmap: Dump a migration bitmap as text
In reply to Dr. David Alan Gilbert (git)Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 8/42 migrate_init: Call from savevm
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.
--- include/migration/migration.h | 4 +--- include/qemu/typedefs.h | 1 + migration/migration.c | 2 +- migration/savevm.c | 2 ++ 4 files changed, 5 insertions(+), 4 deletions(-)
include/migration/migration.h +1 -3
@@ -41,10 +41,7 @@ struct MigrationParams {bool shared; }; -typedef struct MigrationState MigrationState ; - typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head; -/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;
@@ -115,6 +112,7 @@ int migrate_fd_close(MigrationState *s);void add_migration_state_change_notifier (Notifier *notify);void remove_migration_state_change_notifier (Notifier *notify); +MigrationState *migrate_init (const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *);
include/qemu/typedefs.h +1 -0
@@ -41,6 +41,7 @@ typedef struct MemoryRegion MemoryRegion;typedef struct MemoryRegionSection MemoryRegionSection ;typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct MigrationParams MigrationParams ; +typedef struct MigrationState MigrationState ;typedef struct Monitor Monitor ;typedef struct MouseTransformInfo MouseTransformInfo ;typedef struct MSIMessage MSIMessage ;
migration/migration.c +1 -1
@@ -489,7 +489,7 @@ bool migration_has_failed(MigrationState *s) s->state == MIGRATION_STATUS_FAILED); } -static MigrationState *migrate_init (const MigrationParams *params) +MigrationState *migrate_init (const MigrationParams *params) {MigrationState *s =migrate_get_current ();int64_t bandwidth_limit = s->bandwidth_limit;
migration/savevm.c +2 -0
@@ -903,6 +903,8 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) .blk = 0, .shared = 0 }; +MigrationState *ms =migrate_init (¶ms); + ms->file = f;if (qemu_savevm_state_blocked (errp)) {return -EINVAL ;
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 08/42] migrate_init: Call from savevm
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the
savevm.ccode for suspending.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Re: [Qemu-devel] [PATCH v7 08/42] migrate_init: Call from savevm
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the
savevm.ccode for suspending.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Amit
PATCH v7 9/42 Rename save_live_complete to save_live_complete_precopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious.
--- hw/ppc/spapr.c | 2 +- include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 2 +- migration/block.c | 2 +- migration/migration.c | 2 +- migration/ram.c | 2 +- migration/savevm.c | 10 +++++----- trace-events | 2 +- 8 files changed, 12 insertions(+), 12 deletions(-)
hw/ppc/spapr.c +1 -1
@@ -1375,7 +1375,7 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_htab_handlers = { .save_live_setup = htab_save_setup, .save_live_iterate = htab_save_iterate, - .save_live_complete = htab_save_complete, + .save_live_complete_precopy = htab_save_complete, .load_state = htab_load, };
include/migration/vmstate.h +1 -1
@@ -40,7 +40,7 @@ typedef struct SaveVMHandlers {SaveStateHandler *save_state;void (*cancel)(void *opaque); -int (*save_live_complete)(QEMUFile *f,void *opaque); +int (*save_live_complete_precopy)(QEMUFile *f,void *opaque);/* This runs both outside and inside the iothread lock. */ bool (*is_active)(void *opaque);
include/sysemu/sysemu.h +1 -1
@@ -87,7 +87,7 @@ void qemu_savevm_state_begin(QEMUFile *f, const MigrationParams *params);void qemu_savevm_state_header (QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f); -void qemu_savevm_state_complete (QEMUFile *f); +void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void );uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size);int qemu_loadvm_state (QEMUFile *f);
migration/block.c +1 -1
@@ -877,7 +877,7 @@ static SaveVMHandlers savevm_block_handlers = { .set_params = block_set_params, .save_live_setup = block_save_setup, .save_live_iterate = block_save_iterate, - .save_live_complete = block_save_complete, + .save_live_complete_precopy = block_save_complete, .save_live_pending = block_save_pending, .load_state = block_load, .cancel = block_migration_cancel,
migration/migration.c +1 -1
@@ -794,7 +794,7 @@ static void *migration_thread(void *opaque) ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE );if (ret >=0 ) {qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete (s->file); +qemu_savevm_state_complete_precopy (s->file); }qemu_mutex_unlock_iothread ();
migration/ram.c +1 -1
@@ -1604,7 +1604,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate, - .save_live_complete = ram_save_complete, + .save_live_complete_precopy = ram_save_complete, .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel,
migration/savevm.c +5 -5
@@ -793,19 +793,19 @@ static bool should_send_vmdesc(void)return !machine->suppress_vmdesc; } -void qemu_savevm_state_complete (QEMUFile *f) +void qemu_savevm_state_complete_precopy (QEMUFile *f) {QJSON *vmdesc;int vmdesc_len;SaveStateEntry *se;int ret; -trace_savevm_state_complete (); +trace_savevm_state_complete_precopy ();cpu_synchronize_all_states (); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { -if (!se->ops || !se->ops->save_live_complete) { +if (!se->ops || !se->ops->save_live_complete_precopy) {continue ; }if (se->ops && se->ops->is_active) {
@@ -817,7 +817,7 @@ void qemu_savevm_state_complete(QEMUFile *f)save_section_header (f, se,QEMU_VM_SECTION_END ); - ret = se->ops->save_live_complete (f, se->opaque); + ret = se->ops->save_live_complete_precopy (f, se->opaque);trace_savevm_section_end (se->idstr, se->section_id, ret);save_section_footer (f, se);if (ret <0 ) {
@@ -923,7 +923,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) ret =qemu_file_get_error (f);if (ret ==0 ) { -qemu_savevm_state_complete (f); +qemu_savevm_state_complete_precopy (f); ret =qemu_file_get_error (f); }if (ret !=0 ) {
trace-events +1 -1
@@ -1188,7 +1188,7 @@ savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, sectio savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" -savevm_state_complete(void) "" +savevm_state_complete_precopy(void) "" savevm_state_cancel(void) "" vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s"
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 09/42] Rename save_live_complete to save_live_complete_precopy
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>
I will have called it save_live_precopy() and just drop the complete to make names shorter, but as I am not doing the coding ....
PATCH v7 10/42 Return path: Open a return path on QEMUFile for sockets
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's.
--- include/migration/qemu-file.h | 7 +++++ migration/qemu-file-unix.c | 69 +++++++++++++++++++++++++++++++++++++------ migration/qemu-file.c | 12 ++++++++ 3 files changed, 79 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, uint64_t *bytes_sent);/* +* Return a QEMUFile for comms in the opposite direction +*/ +typedef QEMUFile *(QEMURetPathFunc )(void *opaque); + +/* * Stop any read or write (depending on flags) on the underlying * transport on the QEMUFile. * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {QEMURamHookFunc *after_ram_iterate;QEMURamHookFunc *hook_ram_load;QEMURamSaveFunc *save_page; +QEMURetPathFunc *get_return_path;QEMUFileShutdownFunc *shut_down; } QEMUFileOps;
@@ -192,6 +198,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);int qemu_file_get_error (QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f); +QEMUFile *qemu_file_get_return_path (QEMUFile *f);void qemu_fflush (QEMUFile *f);void qemu_file_change_blocking (QEMUFile *f,bool block);
migration/qemu-file-unix.c +60 -9
84 lines not on this page · raw message
migration/qemu-file.c +12 -0
@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)return f->ops->shut_down (f->opaque, true, true); } +/* +* Result: QEMUFile* for a 'return path' for comms in the opposite direction +* NULL if not available +*/ +QEMUFile *qemu_file_get_return_path (QEMUFile *f) +{ +if (!f->ops->get_return_path) { +return NULL ; + } +return f->ops->get_return_path (f->opaque); +} +bool qemu_file_mode_is_not_valid (const char *mode) { if (mode == NULL ||
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
Quoted 19 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/qemu-file.h | 7 +++++ migration/qemu-file-unix.c | 69 +++++++++++++++++++++++++++++++++++++------ migration/qemu-file.c | 12 ++++++++ 3 files changed, 79 insertions(+), 9 deletions(-)include/migration/qemu-file.h +0
-0@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
Hi
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, · line 89
Quoted 5 lines
include/migration/qemu-file.h +5
-0+/* +* Give a QEMUFile* off the same socket but data in the opposite +* direction. +*/ +static QEMUFile *socket_dup_return_path (void *opaque)
We call it dup
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
Quoted 11 lines
include/migration/qemu-file.h +11
-0+{ +QEMUFileSocket *forward = opaque; +QEMUFileSocket *reverse; + +if (qemu_file_get_error (forward->file)) { +/* If the forward file is in error, don't try and open a return */ +return NULL ; + } + + reverse =g_malloc0 (sizeof (QEMUFileSocket)); + reverse->fd = forward->fd;
But we don't dup it :p
For the cest, I am ok with the patch.
Quoted 64 lines
include/migration/qemu-file.h +19
-9+/* I don't think there's a better way to tell which direction 'this' is */ +if (forward->file->ops->get_buffer !=NULL ) { +/* being called from the read side, so we need to be able to write */ +return qemu_fopen_ops (reverse, &socket_return_write_ops); + }else { +return qemu_fopen_ops (reverse, &socket_return_read_ops); + } +} +static ssize_t unix_writev_buffer (void *opaque,struct iovec *iov,int iovcnt,int64_t pos) {@@ -204,18 +254,19 @@ QEMUFile *qemu_fdopen(int fd, const char *mode) }static const QEMUFileOps socket_read_ops = { - .get_fd = socket_get_fd, - .get_buffer = socket_get_buffer, - .close = socket_close, - .shut_down = socket_shutdown - + .get_fd = socket_get_fd, + .get_buffer = socket_get_buffer, + .close = socket_close, + .shut_down = socket_shutdown, + .get_return_path = socket_dup_return_path };static const QEMUFileOps socket_write_ops = { - .get_fd = socket_get_fd, - .writev_buffer = socket_writev_buffer, - .close = socket_close, - .shut_down = socket_shutdown + .get_fd = socket_get_fd, + .writev_buffer = socket_writev_buffer, + .close = socket_close, + .shut_down = socket_shutdown, + .get_return_path = socket_dup_return_path };QEMUFile *qemu_fopen_socket (int fd,const char *mode)migration/qemu-file.c +12
-0@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)return f->ops->shut_down (f->opaque, true, true); } +/* +* Result: QEMUFile* for a 'return path' for comms in the opposite direction +* NULL if not available +*/ +QEMUFile *qemu_file_get_return_path (QEMUFile *f) +{ +if (!f->ops->get_return_path) { +return NULL ; + } +return f->ops->get_return_path (f->opaque); +} +bool qemu_file_mode_is_not_valid (const char *mode) { if (mode == NULL ||
Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets
Quoted 31 lines · * Juan Quintela ([email protected]) wrote:
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
Quoted 19 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/qemu-file.h | 7 +++++ migration/qemu-file-unix.c | 69 +++++++++++++++++++++++++++++++++++++------ migration/qemu-file.c | 12 ++++++++ 3 files changed, 79 insertions(+), 9 deletions(-)include/migration/qemu-file.h +0
-0@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,Hi
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, · line 89
Quoted 5 lines
include/migration/qemu-file.h +5
-0+/* +* Give a QEMUFile* off the same socket but data in the opposite +* direction. +*/ +static QEMUFile *socket_dup_return_path (void *opaque)We call it dup
Quoted 13 lines
include/migration/qemu-file.h typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
Quoted 11 lines
include/migration/qemu-file.h +11
-0+{ +QEMUFileSocket *forward = opaque; +QEMUFileSocket *reverse; + +if (qemu_file_get_error (forward->file)) { +/* If the forward file is in error, don't try and open a return */ +return NULL ; + } + + reverse =g_malloc0 (sizeof (QEMUFileSocket)); + reverse->fd = forward->fd;But we don't dup it :p
Oh yeh, we used to :-) I've replaced _dup with _get
Quoted 3 lines
For the cest, I am ok with the patch.
Reviewed-by Juan Quintela <[email protected]>
Thanks.
Dave
Quoted 51 lines
Quoted 51 lines
include/migration/qemu-file.h +19
-9+/* I don't think there's a better way to tell which direction 'this' is */ +if (forward->file->ops->get_buffer !=NULL ) { +/* being called from the read side, so we need to be able to write */ +return qemu_fopen_ops (reverse, &socket_return_write_ops); + }else { +return qemu_fopen_ops (reverse, &socket_return_read_ops); + } +} +static ssize_t unix_writev_buffer (void *opaque,struct iovec *iov,int iovcnt,int64_t pos) {@@ -204,18 +254,19 @@ QEMUFile *qemu_fdopen(int fd, const char *mode) }static const QEMUFileOps socket_read_ops = { - .get_fd = socket_get_fd, - .get_buffer = socket_get_buffer, - .close = socket_close, - .shut_down = socket_shutdown - + .get_fd = socket_get_fd, + .get_buffer = socket_get_buffer, + .close = socket_close, + .shut_down = socket_shutdown, + .get_return_path = socket_dup_return_path };static const QEMUFileOps socket_write_ops = { - .get_fd = socket_get_fd, - .writev_buffer = socket_writev_buffer, - .close = socket_close, - .shut_down = socket_shutdown + .get_fd = socket_get_fd, + .writev_buffer = socket_writev_buffer, + .close = socket_close, + .shut_down = socket_shutdown, + .get_return_path = socket_dup_return_path };QEMUFile *qemu_fopen_socket (int fd,const char *mode)migration/qemu-file.c +0
-0@@ -43,6 +43,18 @@ int qemu_file_shutdown(QEMUFile *f)return f->ops->shut_down (f->opaque, true, true); }
Quoted 15 lines
Quoted 5 lines
migration/qemu-file.c +5
-0+/* +* Result: QEMUFile* for a 'return path' for comms in the opposite direction +* NULL if not available +*/ +QEMUFile *qemu_file_get_return_path (QEMUFile *f)migration/qemu-file.c int qemu_file_shutdown(QEMUFile *f) · line 60
Quoted 10 lines
migration/qemu-file.c +7
-0+{ +if (!f->ops->get_return_path) { +return NULL ; + } +return f->ops->get_return_path (f->opaque); +} +bool qemu_file_mode_is_not_valid (const char *mode) { if (mode == NULL ||
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 10/42] Return path: Open a return path on QEMUFile for sockets
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Thanks, this looks better than the dup way of doing it.
Amit
PATCH v7 11/42 Return path: socket_writev_buffer: Block even on non-blocking fd's
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.
--- migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +36 -5
@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); - } -return len; +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN && err !=EWOULDBLOCK ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. +*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } + } + +return offset; }static int socket_get_fd (void *opaque)
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Quoted 16 lines
migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-)migration/qemu-file-unix.c +2
-2@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) {
migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 51
Quoted 10 lines
migration/qemu-file-unix.c +7
-3- len = -socket_error (); - } -return len; +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + }
ion_send() can return -1 on error
This looks a "hacky way" to look for it, althoght my understanding is that it is correct, so the review-by
Quoted 30 lines
migration/qemu-file-unix.c +27
-0+ +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN && err !=EWOULDBLOCK ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. +*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } + } + +return offset; }static int socket_get_fd (void *opaque)
Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's
Quoted 10 lines · * Juan Quintela ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>
Thanks,
Quoted 29 lines
Quoted 16 lines
migration/qemu-file-unix.c | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-)migration/qemu-file-unix.c +2
-2@@ -39,12 +39,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) {migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 51
Quoted 10 lines
migration/qemu-file-unix.c +7
-3- len = -socket_error (); - } -return len; +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + }
ion_send()can return -1 on error
Quoted 2 lines
This looks a "hacky way" to look for it, althoght my understanding is that it is correct, so the review-by
I have to check the socket_error in two cases, one where len == -1 , and the other where it's only sent some of the data. Given I have to do it in both cases it doesn't seme worth explicitly checking for the -1.
Dave
Quoted 29 lines
Quoted 29 lines
migration/qemu-file-unix.c +27
-0+ +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN && err !=EWOULDBLOCK ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. +*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } + } + +return offset; }
Quoted 1 line
migration/qemu-file-unix.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 81
Quoted 1 line
migration/qemu-file-unix.c +0
-0static int socket_get_fd (void *opaque)
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 11/42] Return path: socket_writev_buffer: Block even on non-blocking fd's
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 12/42 Migration commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.
For use in postcopy.
--- include/migration/migration.h | 1 + include/sysemu/sysemu.h | 7 +++++++ migration/savevm.c | 46 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 4 files changed, 55 insertions(+)
include/migration/migration.h +1 -0
@@ -34,6 +34,7 @@#define QEMU_VM_SECTION_FULL 0x04#define QEMU_VM_SUBSECTION 0x05#define QEMU_VM_VMDESCRIPTION 0x06 +#define QEMU_VM_COMMAND 0x07#define QEMU_VM_SECTION_FOOTER 0x7estruct MigrationParams {
include/sysemu/sysemu.h +7 -0
@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);void qemu_announce_self (void ); +/* Subcommands for QEMU_VM_COMMAND */ +enum qemu_vm_cmd { +MIG_CMD_INVALID =0 ,/* Must be 0 */ +}; +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void );uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command, +uint16_t len,uint8_t *data);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/savevm.c +46 -0
64 lines not on this page · raw message
trace-events +1 -0
@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_state_begin(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 12/42] Migration commands
Quoted 21 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Create
QEMU_VM_COMMANDsection type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.For use in postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + include/sysemu/sysemu.h | 7 +++++++ migration/savevm.c | 46 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 4 files changed, 55 insertions(+)include/migration/migration.h +0
-0
include/migration/migration.h line 37
Quoted 4 lines
include/migration/migration.h +1
-0@@ -34,6 +34,7 @@#define QEMU_VM_SECTION_FULL 0x04#define QEMU_VM_SUBSECTION 0x05#define QEMU_VM_VMDESCRIPTION 0x06 +#define QEMU_VM_COMMAND 0x07
conflicts with configuration section just sent
Quoted 33 lines
include/migration/migration.h +0
-0#define QEMU_VM_SECTION_FOOTER 0x7estruct MigrationParams {include/sysemu/sysemu.h +7
-0@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);void qemu_announce_self (void ); +/* Subcommands for QEMU_VM_COMMAND */ +enum qemu_vm_cmd { +MIG_CMD_INVALID =0 ,/* Must be 0 */ +}; +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void );uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command, +uint16_t len,uint8_t *data);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayTypemigration/savevm.c +0
-0
migration/savevm.c static bool check_section_footer(QEMUFile *f, SaveStateEntry *se) · line 690
Quoted 5 lines
migration/savevm.c +2
-0@@ -686,6 +686,23 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)return true; } +/* Send a 'QEMU_VM_COMMAND' type element with the command +* and associated data.
Please, use doxygen comments for new functions :p
Quoted 25 lines
migration/savevm.c +18
-0+*/ +void qemu_savevm_command_send (QEMUFile *f, +enum qemu_vm_cmd command, +uint16_t len, +uint8_t *data) +{ +qemu_put_byte (f,QEMU_VM_COMMAND ); +qemu_put_be16 (f, (uint16_t )command); +qemu_put_be16 (f, len); +if (len) { +qemu_put_buffer (f, data, len); + } +qemu_fflush (f); +} +bool qemu_savevm_state_blocked (Error **errp) {SaveStateEntry *se;@@ -982,6 +999,29 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)return NULL ; } +/* +* Process an incoming 'QEMU_VM_COMMAND' +* negative return on error (will issue error message)
migration/savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 1014
Quoted 10 lines
migration/savevm.c +10
-0+*/ +static int loadvm_process_command (QEMUFile *f) +{ +uint16_t cmd; +uint16_t len; + + cmd =qemu_get_be16 (f); + len =qemu_get_be16 (f); + +trace_loadvm_process_command (cmd, len);
trace for load but not for sending?
Quoted 38 lines
migration/savevm.c +16
-0+switch (cmd) { + +default : +error_report ("VM_COMMAND 0x%x unknown (len 0x%x)" , cmd, len); +return -1 ; + } + +return 0 ; +} +struct LoadStateEntry {QLIST_ENTRY (LoadStateEntry) entry;SaveStateEntry *se;@@ -1114,6 +1154,12 @@ int qemu_loadvm_state(QEMUFile *f) goto out; }break ; +case QEMU_VM_COMMAND : + ret =loadvm_process_command (f); +if (ret <0 ) { + goto out; + } +break ;default :error_report ("Unknown savevm section type %d" , section_type); ret = -EINVAL ;trace-events +1
-0@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_state_begin(void) ""
Re: [Qemu-devel] [PATCH v7 12/42] Migration commands
Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Create
QEMU_VM_COMMANDsection type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.For use in postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + include/sysemu/sysemu.h | 7 +++++++ migration/savevm.c | 46 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 4 files changed, 55 insertions(+)
Quoted 11 lines
include/migration/migration.h line 37
Quoted 9 lines
include/migration/migration.h +1
-0@@ -34,6 +34,7 @@#define QEMU_VM_SECTION_FULL 0x04#define QEMU_VM_SUBSECTION 0x05#define QEMU_VM_VMDESCRIPTION 0x06 +#define QEMU_VM_COMMAND 0x07conflicts with configuration section just sent
Inc'd to 8.
Quoted 36 lines
Quoted 36 lines
include/migration/migration.h +0
-0#define QEMU_VM_SECTION_FOOTER 0x7estruct MigrationParams {include/sysemu/sysemu.h +7
-0@@ -82,6 +82,11 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);void qemu_announce_self (void ); +/* Subcommands for QEMU_VM_COMMAND */ +enum qemu_vm_cmd { +MIG_CMD_INVALID =0 ,/* Must be 0 */ +}; +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);@@ -90,6 +95,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void );uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command, +uint16_t len,uint8_t *data);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayTypemigration/savevm.c +0
-0@@ -686,6 +686,23 @@ static bool check_section_footer(QEMUFile *f, SaveStateEntry *se)return true; }
Quoted 4 lines
migration/savevm.c static bool check_section_footer(QEMUFile *f, SaveStateEntry *se) · line 690
Quoted 2 lines
migration/savevm.c +2
-0+/* Send a 'QEMU_VM_COMMAND' type element with the command +* and associated data. Please, use doxygen comments for new functions :p
Done.
Quoted 23 lines
Quoted 23 lines
migration/savevm.c +15
-0+*/ +void qemu_savevm_command_send (QEMUFile *f, +enum qemu_vm_cmd command, +uint16_t len, +uint8_t *data) +{ +qemu_put_byte (f,QEMU_VM_COMMAND ); +qemu_put_be16 (f, (uint16_t )command); +qemu_put_be16 (f, len); +if (len) { +qemu_put_buffer (f, data, len); + } +qemu_fflush (f); +} +bool qemu_savevm_state_blocked (Error **errp) {SaveStateEntry *se;@@ -982,6 +999,29 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)return NULL ; }
Quoted 15 lines
migration/savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 1014
Quoted 13 lines
migration/savevm.c +13
-0+/* +* Process an incoming 'QEMU_VM_COMMAND' +* negative return on error (will issue error message) +*/ +static int loadvm_process_command (QEMUFile *f) +{ +uint16_t cmd; +uint16_t len; + + cmd =qemu_get_be16 (f); + len =qemu_get_be16 (f); + +trace_loadvm_process_command (cmd, len);trace for load but not for sending?
Added.
Dave
Quoted 30 lines
Quoted 30 lines
migration/savevm.c +16
-0+switch (cmd) { + +default : +error_report ("VM_COMMAND 0x%x unknown (len 0x%x)" , cmd, len); +return -1 ; + } + +return 0 ; +} +struct LoadStateEntry {QLIST_ENTRY (LoadStateEntry) entry;SaveStateEntry *se;@@ -1114,6 +1154,12 @@ int qemu_loadvm_state(QEMUFile *f) goto out; }break ; +case QEMU_VM_COMMAND : + ret =loadvm_process_command (f); +if (ret <0 ) { + goto out; + } +break ;default :error_report ("Unknown savevm section type %d" , section_type); ret = -EINVAL ;trace-events +0
-0
Quoted 8 lines
trace-events virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 · line 1189
Quoted 8 lines
trace-events +1
-0@@ -1183,6 +1183,7 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_state_begin(void) ""
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 12/42] Migration commands
In reply to Dr. David Alan Gilbert (git)Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Create
QEMU_VM_COMMANDsection type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.For use in postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 13/42 Return path: Control commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination
--- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h | 6 ++++- migration/savevm.c | 60 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 2 ++ 4 files changed, 69 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *file; +QEMUFile *return_path; +/* See savevm.c */ LoadStateEntry_Head loadvm_handlers; };
include/sysemu/sysemu.h +5 -1
@@ -84,7 +84,9 @@ void qemu_announce_self(void);/* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd { -MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ +MIG_CMD_PING ,/* Request a PONG on the RP */ };bool qemu_savevm_state_blocked (Error **errp);
@@ -97,6 +99,8 @@ void qemu_savevm_state_cancel(void);uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data); +void qemu_savevm_send_ping (QEMUFile *f,uint32_t value); +void qemu_savevm_send_open_return_path (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/savevm.c +60 -0
86 lines not on this page · raw message
trace-events +2 -0
@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" +loadvm_process_command_ping(uint32_t val) "%x" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" +savevm_send_ping(uint32_t val) "%x" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands
Quoted 33 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add two
src->destcommands:
*OPEN_RETURN_PATH- To request that the destination open the return path
* PING - Request an acknowledge from the destinationSigned-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>--- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h | 6 ++++- migration/savevm.c | 60 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 2 ++ 4 files changed, 69 insertions(+), 1 deletion(-)include/migration/migration.h +2
-0@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *file; +QEMUFile *return_path; +/* See savevm.c */ LoadStateEntry_Head loadvm_handlers; };include/sysemu/sysemu.h +0
-0
include/sysemu/sysemu.h void qemu_announce_self(void); · line 89
Quoted 7 lines
include/sysemu/sysemu.h +3
-1@@ -84,7 +84,9 @@ void qemu_announce_self(void);/* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd { -MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ +MIG_CMD_PING ,/* Request a PONG on the RP */
Add MIG_CMD_MAX
struct cmd_args {
int len;
char *name;
} cmd_args[] ={.[MIG_CMD_INVALID] = {
.len = 0,
.name = "CMD_INVALID"},
.[MIG_CMD_OPEN_RETURN_PATH] = {
.len = 0,
.name = "CMD_OPEN_RETURN_PATH"},
.....
}
}
I have done the initialization by hand, not sure if syntax is ok.
include/sysemu/sysemu.h void qemu_announce_self(void);
Quoted 11 lines
include/sysemu/sysemu.h +2
-0static int loadvm_process_command (QEMUFile *f) { +MigrationIncomingState *mis =migration_incoming_get_current ();uint16_t cmd;uint16_t len; +uint32_t tmp32; cmd =qemu_get_be16 (f); len =qemu_get_be16 (f);trace_loadvm_process_command (cmd, len);
/* yes, this should go in previous patch */
if (cmd > MIG_CMD_MAX ) {
error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len);
.....
}if (loadvm_process_command_simple_lencheck(cmd_args[cmd].name, len,
cmd_args[cmd].len) {
return -1;
}switch (cmd) {
case MIG_CMD_OPEN_RETURN_PATH:
if (mis->return_path) {
error_report("CMD_OPEN_RETURN_PATH called when RP already open");
/* Not really a problem, so don't give up */
return 0;
}
mis->return_path = qemu_file_get_return_path(f);
if (!mis->return_path) {
error_report("CMD_OPEN_RETURN_PATH failed");
return -1;
}
break;.....
}
You get the idea. I normally even put the code from the command in a function pointer, but I know that other people don't like it. This way you factor the common code at the beginning of the function.
What do you think?
Quoted 50 lines
switch (cmd) {include/sysemu/sysemu.h +31
-0+case MIG_CMD_OPEN_RETURN_PATH : +if (loadvm_process_command_simple_lencheck ("CMD_OPEN_RETURN_PATH" , + len,0 )) { +return -1 ; + } +if (mis->return_path) { +error_report ("CMD_OPEN_RETURN_PATH called when RP already open" ); +/* Not really a problem, so don't give up */ +return 0 ; + } + mis->return_path =qemu_file_get_return_path (f); +if (!mis->return_path) { +error_report ("CMD_OPEN_RETURN_PATH failed" ); +return -1 ; + } +break ; + +case MIG_CMD_PING : +if (loadvm_process_command_simple_lencheck ("CMD_PING" , len, +sizeof (tmp32))) { +return -1 ; + } + tmp32 =qemu_get_be32 (f); +trace_loadvm_process_command_ping (tmp32); +if (!mis->return_path) { +error_report ("CMD_PING (0x%x) received with no return path" , + tmp32); +return -1 ; + } +/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +break ;default :error_report ("VM_COMMAND 0x%x unknown (len 0x%x)" , cmd, len);trace-events +2
-0@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" +loadvm_process_command_ping(uint32_t val) "%x" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" +savevm_send_ping(uint32_t val) "%x" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands
Quoted 55 lines · * Juan Quintela ([email protected]) wrote:
Quoted 33 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add two
src->destcommands:
*OPEN_RETURN_PATH- To request that the destination open the return path
* PING - Request an acknowledge from the destinationSigned-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>--- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h | 6 ++++- migration/savevm.c | 60 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 2 ++ 4 files changed, 69 insertions(+), 1 deletion(-)include/migration/migration.h +2
-0@@ -47,6 +47,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *file; +QEMUFile *return_path; +/* See savevm.c */ LoadStateEntry_Head loadvm_handlers; };include/sysemu/sysemu.h +0
-0include/sysemu/sysemu.h void qemu_announce_self(void); · line 89
Quoted 7 lines
include/sysemu/sysemu.h +3
-1@@ -84,7 +84,9 @@ void qemu_announce_self(void);/* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd { -MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ +MIG_CMD_PING ,/* Request a PONG on the RP */ Add
MIG_CMD_MAXstruct cmd_args { int len; char *name; } cmd_args[] ={ .[MIG_CMD_INVALID] = { .len = 0, .name = "CMD_INVALID"}, .[MIG_CMD_OPEN_RETURN_PATH] = { .len = 0, .name = "CMD_OPEN_RETURN_PATH"},
Quoted 5 lines
.....
}
}I have done the initialization by hand, not sure if syntax is ok.
Pretty close, no '.] before the [] - we end up with (at the end of the series):
static struct mig_cmd_args {
ssize_t len; /* -1 = variable */
const char *name;
} mig_cmd_args[] = {
[MIG_CMD_INVALID] = { .len = -1, .name = "INVALID" },
[MIG_CMD_OPEN_RETURN_PATH] = { .len = 0, .name = "OPEN_RETURN_PATH" },
[MIG_CMD_PING] = { .len = sizeof(uint32_t), .name = "PING" },
[MIG_CMD_POSTCOPY_ADVISE] = { .len = 16, .name = "POSTCOPY_ADVISE" },
[MIG_CMD_POSTCOPY_LISTEN] = { .len = 0, .name = "POSTCOPY_LISTEN" },
[MIG_CMD_POSTCOPY_RUN] = { .len = 0, .name = "POSTCOPY_RUN" },
[MIG_CMD_POSTCOPY_RAM_DISCARD] =
{ .len = -1, .name = "POSTCOPY_RAM_DISCARD" },
[MIG_CMD_MAX] = { .len = -1, .name = "MAX" },
};Quoted 7 lines
Quoted 7 lines
include/sysemu/sysemu.h +2
-0static int loadvm_process_command (QEMUFile *f) { +MigrationIncomingState *mis =migration_incoming_get_current ();uint16_t cmd;uint16_t len; +uint32_t tmp32;
Quoted 6 lines
Quoted 4 lines
cmd = qemu_get_be16(f); len = qemu_get_be16(f); trace_loadvm_process_command(cmd, len);/* yes, this should go in previous patch */
Yes, done.
Quoted 24 lines
if (cmd > MIG_CMD_MAX ) { error_report("VM_COMMAND 0x%x unknown (len 0x%x)", cmd, len); ..... }if (loadvm_process_command_simple_lencheck(cmd_args[cmd].name, len,
cmd_args[cmd].len) { return -1; }switch (cmd) {
caseMIG_CMD_OPEN_RETURN_PATH:if (mis->return_path) { error_report("CMD_OPEN_RETURN_PATH called when RP already open"); /* Not really a problem, so don't give up */ return 0; } mis->return_path = qemu_file_get_return_path(f); if (!mis->return_path) { error_report("CMD_OPEN_RETURN_PATH failed"); return -1; } break;
Quoted 8 lines
.....
}You get the idea. I normally even put the code from the command in a function pointer, but I know that other people don't like it. This way you factor the common code at the beginning of the function.
What do you think?
Yep, done.
Dave
Quoted 44 lines
Quoted 44 lines
switch (cmd) {include/sysemu/sysemu.h +31
-0+case MIG_CMD_OPEN_RETURN_PATH : +if (loadvm_process_command_simple_lencheck ("CMD_OPEN_RETURN_PATH" , + len,0 )) { +return -1 ; + } +if (mis->return_path) { +error_report ("CMD_OPEN_RETURN_PATH called when RP already open" ); +/* Not really a problem, so don't give up */ +return 0 ; + } + mis->return_path =qemu_file_get_return_path (f); +if (!mis->return_path) { +error_report ("CMD_OPEN_RETURN_PATH failed" ); +return -1 ; + } +break ; + +case MIG_CMD_PING : +if (loadvm_process_command_simple_lencheck ("CMD_PING" , len, +sizeof (tmp32))) { +return -1 ; + } + tmp32 =qemu_get_be32 (f); +trace_loadvm_process_command_ping (tmp32); +if (!mis->return_path) { +error_report ("CMD_PING (0x%x) received with no return path" , + tmp32); +return -1 ; + } +/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +break ;default :error_report ("VM_COMMAND 0x%x unknown (len 0x%x)" , cmd, len);trace-events +0
-0@@ -1184,8 +1184,10 @@ qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u"
Quoted 8 lines
trace-events qemu_loadvm_state_section(unsigned int section_type) "%d" · line 1193
Quoted 8 lines
trace-events +2
-0loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" +loadvm_process_command_ping(uint32_t val) "%x" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" +savevm_send_ping(uint32_t val) "%x" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 13/42] Return path: Control commands
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add two
src->destcommands:
*OPEN_RETURN_PATH- To request that the destination open the return path
* PING - Request an acknowledge from the destinationSigned-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Amit
PATCH v7 14/42 Return path: Send responses from destination to source
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler
--- include/migration/migration.h | 17 ++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 64 insertions(+), 1 deletion(-)
include/migration/migration.h +17 -0
@@ -42,12 +42,20 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;
@@ -179,6 +187,15 @@ int migrate_compress_level(void);int migrate_compress_threads (void );int migrate_decompress_threads (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);
migration/migration.c +45 -0
57 lines not on this page · raw message
migration/savevm.c +1 -1
@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f) tmp32);return -1 ; } -/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +migrate_send_rp_pong (mis, tmp32);break ;default :
trace-events +1 -0
@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 # migration/rdma.c
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source
Quoted 76 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 17 ++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 64 insertions(+), 1 deletion(-)include/migration/migration.h +17
-0@@ -42,12 +42,20 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;@@ -179,6 +187,15 @@ int migrate_compress_level(void);int migrate_compress_threads (void );int migrate_decompress_threads (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);migration/migration.c +3
-0@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current =g_malloc0 (sizeof (MigrationIncomingState)); mis_current->file = f;QLIST_INIT (&mis_current->loadvm_handlers); +qemu_mutex_init (&mis_current->rp_mutex);return mis_current; }@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)qemu_coroutine_enter (co, f); } +/* +* Send a message on the return channel back to the source
migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197
Quoted 10 lines
migration/migration.c +10
-0+* of the migration. +*/ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data) +{ +trace_migrate_send_rp_message ((int )message_type, len); +qemu_mutex_lock (&mis->rp_mutex); +qemu_put_be16 (mis->return_path, (unsigned int )message_type); +qemu_put_be16 (mis->return_path, len);
if (len) {
migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198
Quoted 1 line
migration/migration.c +1
-0+qemu_put_buffer (mis->return_path, data, len);
}
?
We check for zero sized command on control commands but not on responses?
Quoted 59 lines
migration/migration.c +31
-0+qemu_fflush (mis->return_path); +qemu_mutex_unlock (&mis->rp_mutex); +} + +/* +* Send a 'SHUT' message on the return channel with the given value +* to indicate that we've finished with the RP. Non-0 value indicates +* error. +*/ +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value) +{ +uint32_t buf; + + buf =cpu_to_be32 (value); +migrate_send_rp_message (mis,MIG_RP_MSG_SHUT ,sizeof (buf), &buf); +} + +/* +* Send a 'PONG' message on the return channel with the given value +* (normally in response to a 'PING') +*/ +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value) +{ +uint32_t buf; + + buf =cpu_to_be32 (value); +migrate_send_rp_message (mis,MIG_RP_MSG_PONG ,sizeof (buf), &buf); +} +/* amount of nanoseconds we are willing to wait for migration to be down. * the choice of nanoseconds is because it is the maximum resolution that * get_clock() can achieve. It is an internal measure. All user-visible migration/savevm.c +1
-1@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f) tmp32);return -1 ; } -/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +migrate_send_rp_pong (mis, tmp32);break ;default :trace-events +1
-0@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 # migration/rdma.c
Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source
Quoted 89 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 17 ++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 64 insertions(+), 1 deletion(-)include/migration/migration.h +17
-0@@ -42,12 +42,20 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;@@ -179,6 +187,15 @@ int migrate_compress_level(void);int migrate_compress_threads (void );int migrate_decompress_threads (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);migration/migration.c +3
-0@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current =g_malloc0 (sizeof (MigrationIncomingState)); mis_current->file = f;QLIST_INIT (&mis_current->loadvm_handlers); +qemu_mutex_init (&mis_current->rp_mutex);return mis_current; }@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)qemu_coroutine_enter (co, f); } +/* +* Send a message on the return channel back to the source migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197
Quoted 10 lines
migration/migration.c +10
-0+* of the migration. +*/ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data) +{ +trace_migrate_send_rp_message ((int )message_type, len); +qemu_mutex_lock (&mis->rp_mutex); +qemu_put_be16 (mis->return_path, (unsigned int )message_type); +qemu_put_be16 (mis->return_path, len);if (len) {
Quoted 8 lines
migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198
Quoted 1 line
migration/migration.c +1
-0+qemu_put_buffer (mis->return_path, data, len);}
?
We check for zero sized command on control commands but not on responses?
Or should I remove the check in the control commands case? qemu_put_buffer looks like it's safe for size == 0
Dave
Quoted 59 lines
Quoted 59 lines
migration/migration.c +31
-0+qemu_fflush (mis->return_path); +qemu_mutex_unlock (&mis->rp_mutex); +} + +/* +* Send a 'SHUT' message on the return channel with the given value +* to indicate that we've finished with the RP. Non-0 value indicates +* error. +*/ +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value) +{ +uint32_t buf; + + buf =cpu_to_be32 (value); +migrate_send_rp_message (mis,MIG_RP_MSG_SHUT ,sizeof (buf), &buf); +} + +/* +* Send a 'PONG' message on the return channel with the given value +* (normally in response to a 'PING') +*/ +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value) +{ +uint32_t buf; + + buf =cpu_to_be32 (value); +migrate_send_rp_message (mis,MIG_RP_MSG_PONG ,sizeof (buf), &buf); +} +/* amount of nanoseconds we are willing to wait for migration to be down. * the choice of nanoseconds is because it is the maximum resolution that * get_clock() can achieve. It is an internal measure. All user-visible migration/savevm.c +1
-1@@ -1071,7 +1071,7 @@ static int loadvm_process_command(QEMUFile *f) tmp32);return -1 ; } -/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +migrate_send_rp_pong (mis, tmp32);break ;default :trace-events +1
-0@@ -1399,6 +1399,7 @@ migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
Quoted 1 line
trace-events migrate_fd_cleanup(void) "" · line 1405
Quoted 1 line
trace-events +0
-0# migration/rdma.c
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source
Quoted 94 lines · Dr. David Alan Gilbert
Quoted 93 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 17 ++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 64 insertions(+), 1 deletion(-)include/migration/migration.h +17
-0@@ -42,12 +42,20 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;@@ -179,6 +187,15 @@ int migrate_compress_level(void);int migrate_compress_threads (void );int migrate_decompress_threads (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);migration/migration.c +3
-0@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current =g_malloc0 (sizeof (MigrationIncomingState)); mis_current->file = f;QLIST_INIT (&mis_current->loadvm_handlers); +qemu_mutex_init (&mis_current->rp_mutex);return mis_current; }@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)qemu_coroutine_enter (co, f); } +/* +* Send a message on the return channel back to the source migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197
Quoted 10 lines
migration/migration.c +10
-0+* of the migration. +*/ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data) +{ +trace_migrate_send_rp_message ((int )message_type, len); +qemu_mutex_lock (&mis->rp_mutex); +qemu_put_be16 (mis->return_path, (unsigned int )message_type); +qemu_put_be16 (mis->return_path, len);if (len) {
migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198
Quoted 1 line
migration/migration.c +1
-0+qemu_put_buffer (mis->return_path, data, len);}
Quoted 7 lines
Quoted 4 lines
?
We check for zero sized command on control commands but not on responses?
Or should I remove the check in the control commands case?
qemu_put_bufferlooks like it's safe for size == 0
I would go for this just for consistence?
Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source
Quoted 97 lines · * Juan Quintela ([email protected]) wrote:
Quoted 96 lines · Dr. David Alan Gilbert
Quoted 95 lines · * Juan Quintela ([email protected]) wrote:
Quoted 76 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 17 ++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 64 insertions(+), 1 deletion(-)include/migration/migration.h +17
-0@@ -42,12 +42,20 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;@@ -179,6 +187,15 @@ int migrate_compress_level(void);int migrate_compress_threads (void );int migrate_decompress_threads (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);migration/migration.c +3
-0@@ -85,6 +85,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current =g_malloc0 (sizeof (MigrationIncomingState)); mis_current->file = f;QLIST_INIT (&mis_current->loadvm_handlers); +qemu_mutex_init (&mis_current->rp_mutex);return mis_current; }@@ -182,6 +183,50 @@ void process_incoming_migration(QEMUFile *f)qemu_coroutine_enter (co, f); } +/* +* Send a message on the return channel back to the source migration/migration.c void process_incoming_migration(QEMUFile *f) · line 197
Quoted 10 lines
migration/migration.c +10
-0+* of the migration. +*/ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data) +{ +trace_migrate_send_rp_message ((int )message_type, len); +qemu_mutex_lock (&mis->rp_mutex); +qemu_put_be16 (mis->return_path, (unsigned int )message_type); +qemu_put_be16 (mis->return_path, len);if (len) {
migration/migration.c void process_incoming_migration(QEMUFile *f) · line 198
Quoted 1 line
migration/migration.c +1
-0+qemu_put_buffer (mis->return_path, data, len);}
?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
We check for zero sized command on control commands but not on responses?
Or should I remove the check in the control commands case?
qemu_put_bufferlooks like it's safe for size == 0I would go for this just for consistence?
OK, check removed in both cases.
Dave
(Resend: I dropped the list off the first message) --
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 14/42] Return path: Send responses from destination to source
In reply to Dr. David Alan Gilbert (git)Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 15/42 Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
--- include/migration/migration.h | 8 ++ migration/migration.c | 177 +++++++++++++++++++++++++++++++++++++++++- trace-events | 12 +++ 3 files changed, 196 insertions(+), 1 deletion(-)
include/migration/migration.h +8 -0
@@ -77,6 +77,14 @@ struct MigrationStateint state;MigrationParams params; + +/* State related to return path */ +struct { +QEMUFile *file; +QemuThread rp_thread; +bool error; + } rp_state; +double mbps;int64_t total_time;int64_t downtime;
migration/migration.c +176 -1
208 lines not on this page · raw message
trace-events +12 -0
@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d" flic_reset_failed(int err) "flic: reset failed %d" # migration.c +await_return_path_close_on_source_close(void) "" +await_return_path_close_on_source_joining(void) "" migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" +migrate_fd_cleanup_src_rp(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +open_return_path_on_source(void) "" +open_return_path_on_source_continue(void) "" +source_return_path_thread_bad_end(void) "" +source_return_path_thread_end(void) "" +source_return_path_thread_entry(void) "" +source_return_path_thread_failed_read_cmd_data(void) "" +source_return_path_thread_loop_top(void) "" +source_return_path_thread_pong(uint32_t val) "%x" +source_return_path_thread_shut(uint32_t val) "%x" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 # migration/rdma.c
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/migration.c +6
-0+/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 868
Quoted 10 lines
migration/migration.c +10
-0+MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); + while (rp && !qemu_file_get_error(rp) &&
What can make rp == NULL?
THinking about that, could you mean *rp here?
Quoted 13 lines
migration/migration.c +13
-0+ migration_already_active(ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : + error_report("RP: Received invalid message 0x%04x length 0x%04x",
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 891
Quoted 10 lines
migration/migration.c +9
-0+ header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len);
I know this is a big request, but getting an array with messages length and message names to be able to print nice error messages looks ilke good?
Quoted 6 lines
migration/migration.c +6
-0+source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len);
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 907
Quoted 10 lines
migration/migration.c +10
-0+if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + } + +/* OK, we have the message and the data */ +switch (header_type) { +case MIG_RP_MSG_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf);
make local variable and call it sibling_error or whatever you like?
Quoted 4 lines
migration/migration.c +4
-0+trace_source_return_path_thread_shut (tmp32); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms);
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 921
Quoted 10 lines
migration/migration.c +10
-0+ } +/* +* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RP_MSG_PONG : + tmp32 =be32_to_cpup ((uint32_t *)buf);
unused?
Althought I guess it is used somewhere to make sure that the value is the same that whatever we did the ping. credentials?
I can't see with this and previous patch what value is sent here.
Quoted 8 lines
migration/migration.c +8
-0+trace_source_return_path_thread_pong (tmp32); +break ; + +default : +break ; + } + } +if (rp &&qemu_file_get_error (rp)) {
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 939
Quoted 10 lines
migration/migration.c +10
-0+trace_source_return_path_thread_bad_end (); +source_return_path_bad (ms); + } + +trace_source_return_path_thread_end (); +out: +return NULL ; +} + +__attribute__ (( unused )) /* Until later in patch series */
unused_by_know attribute required O:-)
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/migration.c +11
-0+/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len];
Quoted 8 lines
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 868
Quoted 5 lines
migration/migration.c +5
-0+uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); + while (rp && !qemu_file_get_error(rp) &&What can make rp == NULL?
THinking about that, could you mean *rp here?
I've reworked this; it was meant to catch the case of the rp being closed early, but was racy.
I've now got:
while (!ms->rp_state.error && !qemu_file_get_error(rp) &&
migration_already_active(ms)) {and now the rp qemu_file gets closed at the end of the thread, anyone else that wants the rp_thread to exit sets the error flag.
Quoted 18 lines
Quoted 18 lines
migration/migration.c +17
-0+ migration_already_active(ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + }
Quoted 8 lines
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 891
Quoted 5 lines
migration/migration.c +5
-0+if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len);I know this is a big request, but getting an array with messages length and message names to be able to print nice error messages looks ilke good?
Done; (same way as for the commands on the forward path).
Quoted 10 lines
Quoted 10 lines
migration/migration.c +10
-0+source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out;
Quoted 8 lines
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 907
Quoted 6 lines
migration/migration.c +6
-0+ } + +/* OK, we have the message and the data */ +switch (header_type) { +case MIG_RP_MSG_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf);make local variable and call it
sibling_erroror whatever you like?
Done.
Quoted 15 lines
Quoted 4 lines
migration/migration.c +4
-0+trace_source_return_path_thread_shut (tmp32); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms);migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 921
Quoted 10 lines
migration/migration.c +10
-0+ } +/* +* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RP_MSG_PONG : + tmp32 =be32_to_cpup ((uint32_t *)buf);
Quoted 5 lines
unused?
Althought I guess it is used somewhere to make sure that the value is the same that whatever we did the ping. credentials?I can't see with this and previous patch what value is sent here.
I'm not using the ping/pong messages for anything active, they exist primarily as a debug and tracing aid.
Quoted 12 lines
Quoted 12 lines
migration/migration.c +12
-0+trace_source_return_path_thread_pong (tmp32); +break ; + +default : +break ; + } + } +if (rp &&qemu_file_get_error (rp)) { +trace_source_return_path_thread_bad_end (); +source_return_path_bad (ms); + } +
Quoted 8 lines
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 939
Quoted 6 lines
migration/migration.c +6
-0+trace_source_return_path_thread_end (); +out: +return NULL ; +} + +__attribute__ (( unused )) /* Until later in patch series */
unused_by_knowattribute required O:-)
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 847
Quoted 13 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/migration.c +5
-1-/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s)
Can you rename this to something like
mark_source_rp_bad()
?
Intent is clearer that way.
Also, the comment says caller will print something, but the invocations below are a mix of printfs and traces. Not saying the caller has to print always, but maybe only comment needs update.
Quoted 45 lines
migration/migration.c +44
-0+{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} + +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len); +source_return_path_bad (ms);
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 902
Quoted 10 lines
migration/migration.c +10
-0+ goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + }
Amit
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
Quoted 17 lines · * Amit Shah ([email protected]) wrote:
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 847
Quoted 13 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/migration.c +5
-1-/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s)Can you rename this to something like
Quoted 5 lines
mark_source_rp_bad()?
Intent is clearer that way.
Done.
Quoted 3 lines
Also, the comment says caller will print something, but the invocations below are a mix of printfs and traces. Not saying the caller has to print always, but maybe only comment needs update.
Yes, I've changed the comment, and changed one of the traces into an error_report.
Thanks,
Dave
Quoted 50 lines
Quoted 50 lines
migration/migration.c +48
-0+{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} + +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len); +source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */
Quoted 8 lines
migration/migration.c int64_t migrate_xbzrle_cache_size(void) · line 902
Quoted 6 lines
migration/migration.c +6
-0+ res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + }Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)Hi Dave,
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 8 ++ migration/migration.c | 177 +++++++++++++++++++++++++++++++++++++++++- trace-events | 12 +++ 3 files changed, 196 insertions(+), 1 deletion(-)include/migration/migration.h +0
-0
include/migration/migration.h struct MigrationState · line 83
Quoted 7 lines
include/migration/migration.h +4
-0@@ -77,6 +77,14 @@ struct MigrationStateint state;MigrationParams params; + +/* State related to return path */ +struct { +QEMUFile *file;
There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.
But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.
Thanks,
zhanghailiang
Quoted 254 lines
include/migration/migration.h +4
-0+QemuThread rp_thread; +bool error; + } rp_state; +double mbps;int64_t total_time;int64_t downtime;migration/migration.c +176
-1@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)return params; } +/* +* Return true if we're already in the middle of a migration +* (i.e. any of the active or setup states) +*/ +static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIGRATION_STATUS_ACTIVE : +case MIGRATION_STATUS_SETUP : +return true; + +default : +return false; + + } +} +static void get_xbzrle_cache_stats (MigrationInfo *info) {if (migrate_use_xbzrle ()) {@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state) } } +static void migrate_fd_cleanup_src_rp (MigrationState *ms) +{ +QEMUFile *rp = ms->rp_state.file; + +/* +* When stuff goes wrong (e.g. failing destination) on the rp, it can get +* cleaned up from a few threads; make sure not to do it twice in parallel +*/ + rp =atomic_cmpxchg (&ms->rp_state.file, rp,NULL ); +if (rp) { +trace_migrate_fd_cleanup_src_rp (); +qemu_fclose (rp); + } +} +static void migrate_fd_cleanup (void *opaque) {MigrationState *s = opaque;@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)qemu_bh_delete (s->cleanup_bh); s->cleanup_bh =NULL ; +migrate_fd_cleanup_src_rp (s); +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)QEMUFile *f =migrate_get_current ()->file;trace_migrate_fd_cancel (); +if (s->rp_state.file) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->rp_state.file); + } + do { old_state = s->state; if (old_state != MIGRATION_STATUS_SETUP &&@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)return s->xbzrle_cache_size; } -/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s) +{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} + +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len); +source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + } + +/* OK, we have the message and the data */ +switch (header_type) { +case MIG_RP_MSG_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_shut (tmp32); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms); + } +/* +* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RP_MSG_PONG : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_pong (tmp32); +break ; + +default : +break ; + } + } +if (rp &&qemu_file_get_error (rp)) { +trace_source_return_path_thread_bad_end (); +source_return_path_bad (ms); + } + +trace_source_return_path_thread_end (); +out: +return NULL ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +static int open_return_path_on_source (MigrationState *ms) +{ + + ms->rp_state.file =qemu_file_get_return_path (ms->file); +if (!ms->rp_state.file) { +return -1 ; + } + +trace_open_return_path_on_source (); +qemu_thread_create (&ms->rp_state.rp_thread,"return path" , + source_return_path_thread, ms,QEMU_THREAD_JOINABLE ); + +trace_open_return_path_on_source_continue (); + +return 0 ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +/* Returns 0 if the RP was ok, otherwise there was an error on the RP */ +static int await_return_path_close_on_source (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place) +*/ +if (qemu_file_get_error (ms->file) && ms->rp_state.file) { +qemu_file_shutdown (ms->rp_state.file); + } +trace_await_return_path_close_on_source_joining (); +qemu_thread_join (&ms->rp_state.rp_thread); +trace_await_return_path_close_on_source_close (); +return ms->rp_state.error; +} + +/* +* Master migration thread on the source VM. +* It drives the migration and pumps the data down the outgoing channel. +*/ static void *migration_thread (void *opaque) {MigrationState *s = opaque;trace-events +12
-0@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d" flic_reset_failed(int err) "flic: reset failed %d" # migration.c +await_return_path_close_on_source_close(void) "" +await_return_path_close_on_source_joining(void) "" migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" +migrate_fd_cleanup_src_rp(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +open_return_path_on_source(void) "" +open_return_path_on_source_continue(void) "" +source_return_path_thread_bad_end(void) "" +source_return_path_thread_end(void) "" +source_return_path_thread_entry(void) "" +source_return_path_thread_failed_read_cmd_data(void) "" +source_return_path_thread_loop_top(void) "" +source_return_path_thread_pong(uint32_t val) "%x" +source_return_path_thread_shut(uint32_t val) "%x" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 # migration/rdma.c
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
Quoted 31 lines · * zhanghailiang ([email protected]) wrote:
Hi Dave,
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 8 ++ migration/migration.c | 177 +++++++++++++++++++++++++++++++++++++++++- trace-events | 12 +++ 3 files changed, 196 insertions(+), 1 deletion(-)include/migration/migration.h +0
-0include/migration/migration.h struct MigrationState · line 83
Quoted 7 lines
include/migration/migration.h +4
-0@@ -77,6 +77,14 @@ struct MigrationStateint state;MigrationParams params; + +/* State related to return path */ +struct { +QEMUFile *file;There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.
Quoted 5 lines
But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.
Would the following be clearer:
On the source make the existing migration file:
QEMUFile *to_dst_file;
and for the return path
QEMUFile *from_dst_dile;and then on the destination, the incoming migration stream:
QEMUFile *from_src_file;
and then the return path on the destionation:
QEMUFile *to_src_file;Dave
Quoted 256 lines
Thanks,
zhanghailiangQuoted 252 lines
include/migration/migration.h +4
-0+QemuThread rp_thread; +bool error; + } rp_state; +double mbps;int64_t total_time;int64_t downtime;migration/migration.c +176
-1@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)return params; } +/* +* Return true if we're already in the middle of a migration +* (i.e. any of the active or setup states) +*/ +static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIGRATION_STATUS_ACTIVE : +case MIGRATION_STATUS_SETUP : +return true; + +default : +return false; + + } +} +static void get_xbzrle_cache_stats (MigrationInfo *info) {if (migrate_use_xbzrle ()) {@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state) } } +static void migrate_fd_cleanup_src_rp (MigrationState *ms) +{ +QEMUFile *rp = ms->rp_state.file; + +/* +* When stuff goes wrong (e.g. failing destination) on the rp, it can get +* cleaned up from a few threads; make sure not to do it twice in parallel +*/ + rp =atomic_cmpxchg (&ms->rp_state.file, rp,NULL ); +if (rp) { +trace_migrate_fd_cleanup_src_rp (); +qemu_fclose (rp); + } +} +static void migrate_fd_cleanup (void *opaque) {MigrationState *s = opaque;@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)qemu_bh_delete (s->cleanup_bh); s->cleanup_bh =NULL ; +migrate_fd_cleanup_src_rp (s); +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)QEMUFile *f =migrate_get_current ()->file;trace_migrate_fd_cancel (); +if (s->rp_state.file) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->rp_state.file); + } + do { old_state = s->state; if (old_state != MIGRATION_STATUS_SETUP &&@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)return s->xbzrle_cache_size; } -/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s) +{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} + +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len); +source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + } + +/* OK, we have the message and the data */ +switch (header_type) { +case MIG_RP_MSG_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_shut (tmp32); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms); + } +/* +* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RP_MSG_PONG : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_pong (tmp32); +break ; + +default : +break ; + } + } +if (rp &&qemu_file_get_error (rp)) { +trace_source_return_path_thread_bad_end (); +source_return_path_bad (ms); + } + +trace_source_return_path_thread_end (); +out: +return NULL ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +static int open_return_path_on_source (MigrationState *ms) +{ + + ms->rp_state.file =qemu_file_get_return_path (ms->file); +if (!ms->rp_state.file) { +return -1 ; + } + +trace_open_return_path_on_source (); +qemu_thread_create (&ms->rp_state.rp_thread,"return path" , + source_return_path_thread, ms,QEMU_THREAD_JOINABLE ); + +trace_open_return_path_on_source_continue (); + +return 0 ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +/* Returns 0 if the RP was ok, otherwise there was an error on the RP */ +static int await_return_path_close_on_source (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place) +*/ +if (qemu_file_get_error (ms->file) && ms->rp_state.file) { +qemu_file_shutdown (ms->rp_state.file); + } +trace_await_return_path_close_on_source_joining (); +qemu_thread_join (&ms->rp_state.rp_thread); +trace_await_return_path_close_on_source_close (); +return ms->rp_state.error; +} + +/* +* Master migration thread on the source VM. +* It drives the migration and pumps the data down the outgoing channel. +*/ static void *migration_thread (void *opaque) {MigrationState *s = opaque;trace-events +12
-0@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d" flic_reset_failed(int err) "flic: reset failed %d" # migration.c +await_return_path_close_on_source_close(void) "" +await_return_path_close_on_source_joining(void) "" migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" +migrate_fd_cleanup_src_rp(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +open_return_path_on_source(void) "" +open_return_path_on_source_continue(void) "" +source_return_path_thread_bad_end(void) "" +source_return_path_thread_end(void) "" +source_return_path_thread_entry(void) "" +source_return_path_thread_failed_read_cmd_data(void) "" +source_return_path_thread_loop_top(void) "" +source_return_path_thread_pong(uint32_t val) "%x" +source_return_path_thread_shut(uint32_t val) "%x" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64
Quoted 4 lines
trace-events flic_no_device_api(int err) "flic: no Device Contral API support %d" · line 1417
Quoted 2 lines
trace-events +0
-0# migration/rdma.c
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 15/42] Return path: Source handling of return path
Quoted 32 lines · Dr. David Alan Gilbert
Quoted 31 lines · * zhanghailiang ([email protected]) wrote:
Hi Dave,
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 8 ++ migration/migration.c | 177 +++++++++++++++++++++++++++++++++++++++++- trace-events | 12 +++ 3 files changed, 196 insertions(+), 1 deletion(-)include/migration/migration.h +0
-0include/migration/migration.h struct MigrationState · line 83
Quoted 7 lines
include/migration/migration.h +4
-0@@ -77,6 +77,14 @@ struct MigrationStateint state;MigrationParams params; + +/* State related to return path */ +struct { +QEMUFile *file;There is already a 'file' member in MigrationState,
and since for migration, there is only one path direction, just from source side to destination side, so it is ok to use that name.
Quoted 8 lines
Quoted 5 lines
But for post-copy and COLO, we need two-way communication,
So we can rename the original 'file' member of MigrationState to 'ouput_file', and add a new 'input_file' member. For MigrationIncomingState struct, rename its original 'file' member to 'input_file',and add a new 'output_file'.
IMHO, this will make things more clear.Would the following be clearer:
Yes, it is clearer and more graceful :)
Quoted 5 lines
On the source make the existing migration file:
QEMUFile *to_dst_file;
and for the return path
QEMUFile *from_dst_dile;
^ from_dst_file
Quoted 272 lines
and then on the destination, the incoming migration stream:
QEMUFile *from_src_file; and then the return path on the destionation: QEMUFile *to_src_file;Dave
Quoted 256 lines
Thanks,
zhanghailiangQuoted 252 lines
include/migration/migration.h +4
-0+QemuThread rp_thread; +bool error; + } rp_state; +double mbps;int64_t total_time;int64_t downtime;migration/migration.c +176
-1@@ -278,6 +278,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)return params; } +/* +* Return true if we're already in the middle of a migration +* (i.e. any of the active or setup states) +*/ +static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIGRATION_STATUS_ACTIVE : +case MIGRATION_STATUS_SETUP : +return true; + +default : +return false; + + } +} +static void get_xbzrle_cache_stats (MigrationInfo *info) {if (migrate_use_xbzrle ()) {@@ -441,6 +458,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state) } } +static void migrate_fd_cleanup_src_rp (MigrationState *ms) +{ +QEMUFile *rp = ms->rp_state.file; + +/* +* When stuff goes wrong (e.g. failing destination) on the rp, it can get +* cleaned up from a few threads; make sure not to do it twice in parallel +*/ + rp =atomic_cmpxchg (&ms->rp_state.file, rp,NULL ); +if (rp) { +trace_migrate_fd_cleanup_src_rp (); +qemu_fclose (rp); + } +} +static void migrate_fd_cleanup (void *opaque) {MigrationState *s = opaque;@@ -448,6 +480,8 @@ static void migrate_fd_cleanup(void *opaque)qemu_bh_delete (s->cleanup_bh); s->cleanup_bh =NULL ; +migrate_fd_cleanup_src_rp (s); +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -487,6 +521,11 @@ static void migrate_fd_cancel(MigrationState *s)QEMUFile *f =migrate_get_current ()->file;trace_migrate_fd_cancel (); +if (s->rp_state.file) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->rp_state.file); + } + do { old_state = s->state; if (old_state != MIGRATION_STATUS_SETUP &&@@ -801,8 +840,144 @@ int64_t migrate_xbzrle_cache_size(void)return s->xbzrle_cache_size; } -/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s) +{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} + +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->rp_state.file; +uint16_t expected_len, header_len, header_type; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +trace_source_return_path_thread_entry (); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +trace_source_return_path_thread_loop_top (); + header_type =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_type) { +case MIG_RP_MSG_SHUT : +case MIG_RP_MSG_PONG : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid message 0x%04x length 0x%04x" , + header_type, header_len); +source_return_path_bad (ms); + goto out; + } +if (header_len > expected_len) { +error_report ("RP: Received message 0x%04x with" +"incorrect length %d expecting %d" , + header_type, header_len, + expected_len); +source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +trace_source_return_path_thread_failed_read_cmd_data (); +source_return_path_bad (ms); + goto out; + } + +/* OK, we have the message and the data */ +switch (header_type) { +case MIG_RP_MSG_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_shut (tmp32); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms); + } +/* +* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RP_MSG_PONG : + tmp32 =be32_to_cpup ((uint32_t *)buf); +trace_source_return_path_thread_pong (tmp32); +break ; + +default : +break ; + } + } +if (rp &&qemu_file_get_error (rp)) { +trace_source_return_path_thread_bad_end (); +source_return_path_bad (ms); + } + +trace_source_return_path_thread_end (); +out: +return NULL ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +static int open_return_path_on_source (MigrationState *ms) +{ + + ms->rp_state.file =qemu_file_get_return_path (ms->file); +if (!ms->rp_state.file) { +return -1 ; + } + +trace_open_return_path_on_source (); +qemu_thread_create (&ms->rp_state.rp_thread,"return path" , + source_return_path_thread, ms,QEMU_THREAD_JOINABLE ); + +trace_open_return_path_on_source_continue (); + +return 0 ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +/* Returns 0 if the RP was ok, otherwise there was an error on the RP */ +static int await_return_path_close_on_source (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place) +*/ +if (qemu_file_get_error (ms->file) && ms->rp_state.file) { +qemu_file_shutdown (ms->rp_state.file); + } +trace_await_return_path_close_on_source_joining (); +qemu_thread_join (&ms->rp_state.rp_thread); +trace_await_return_path_close_on_source_close (); +return ms->rp_state.error; +} + +/* +* Master migration thread on the source VM. +* It drives the migration and pumps the data down the outgoing channel. +*/ static void *migration_thread (void *opaque) {MigrationState *s = opaque;trace-events +12
-0@@ -1394,12 +1394,24 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d" flic_reset_failed(int err) "flic: reset failed %d" # migration.c +await_return_path_close_on_source_close(void) "" +await_return_path_close_on_source_joining(void) "" migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" +migrate_fd_cleanup_src_rp(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +open_return_path_on_source(void) "" +open_return_path_on_source_continue(void) "" +source_return_path_thread_bad_end(void) "" +source_return_path_thread_end(void) "" +source_return_path_thread_entry(void) "" +source_return_path_thread_failed_read_cmd_data(void) "" +source_return_path_thread_loop_top(void) "" +source_return_path_thread_pong(uint32_t val) "%x" +source_return_path_thread_shut(uint32_t val) "%x" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64Quoted 4 lines
trace-events flic_no_device_api(int err) "flic: no Device Contral API support %d" · line 1417
Quoted 2 lines
trace-events +0
-0# migration/rdma.c--
Dr. David Alan Gilbert / [email protected] / Manchester, UK.
PATCH v7 16/42 Rework loadvm path for subloops
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Split the core of qemu_loadvm_state out so we can use it for both.
Allow the inner loadvm loop to quit and cause the parent loops to exit as well.
--- include/migration/migration.h | 6 ++ migration/migration.c | 2 + migration/savevm.c | 131 +++++++++++++++++++++++------------------- trace-events | 4 ++ 4 files changed, 83 insertions(+), 60 deletions(-)
include/migration/migration.h +6 -0
@@ -54,6 +54,12 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *file; +/* +* Free at the start of the main state load, set as the main thread finishes +* loading state. +*/ +QemuEvent main_thread_load_event; +QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */
migration/migration.c +2 -0
@@ -86,12 +86,14 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current->file = f;QLIST_INIT (&mis_current->loadvm_handlers);qemu_mutex_init (&mis_current->rp_mutex); +qemu_event_init (&mis_current->main_thread_load_event, false);return mis_current; }void migration_incoming_state_destroy (void ) { +qemu_event_destroy (&mis_current->main_thread_load_event);loadvm_free_handlers (mis_current);g_free (mis_current); mis_current =NULL ;
migration/savevm.c +71 -60
205 lines not on this page · raw message
trace-events +4 -0
@@ -1181,7 +1181,11 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 # migration/savevm.c qemu_loadvm_state_section(unsigned int section_type) "%d" +qemu_loadvm_state_section_command(int ret) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" +qemu_loadvm_state_main(void) "" +qemu_loadvm_state_main_quit_parent(void) "" +qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x"
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 16/42] Rework loadvm path for subloops
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 6 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and cause the parent loops to exit as well.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Re: [Qemu-devel] [PATCH v7 16/42] Rework loadvm path for subloops
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 6 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and cause the parent loops to exit as well.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 17/42 Add migration-capability boolean for postcopy-ram.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).
--- include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 6 +++++- 3 files changed, 29 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
@@ -183,6 +183,7 @@ void migrate_add_blocker(Error *reason);*/ void migrate_del_blocker (Error *reason); +bool migrate_postcopy_ram (void );bool migrate_zero_blocks (void );bool migrate_auto_converge (void );
migration/migration.c +23 -0
@@ -408,6 +408,20 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,for (cap = params; cap; cap = cap->next) { s->enabled_capabilities[cap->value->capability] = cap->value->state; } + +if (migrate_postcopy_ram ()) { +if (migrate_use_compression ()) { +/* The decompression threads asynchronously write into RAM +* rather than use the atomic copies needed to avoid +* userfaulting. It should be possible to fix the decompression +* threads for compatibility in future. +*/ +error_report ("Postcopy is not currently compatible with " +"compression" ); + s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM ] = + false; + } + } } void qmp_migrate_set_parameters(bool has_compress_level,
@@ -770,6 +784,15 @@ void qmp_migrate_set_downtime(double value, Error **errp) max_downtime = (uint64_t )value; } +bool migrate_postcopy_ram (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM ]; +} +bool migrate_auto_converge (void ) {MigrationState *s;
qapi-schema.json +5 -1
@@ -526,11 +526,15 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has +# been migrated, pulling the remaining pages along as needed. NOTE: If +# the migration fails during postcopy the VM will fail. (since 2.4) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', - 'compress'] } + 'compress', 'x-postcopy-ram'] } ## # @MigrationCapabilityStatus
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.
Quoted 17 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the
migrate_start_postcopypatch later in the series).Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 6 +++++- 3 files changed, 29 insertions(+), 1 deletion(-)+++
b/qapi-schema.json
qapi-schema.json line 531
Quoted 6 lines
qapi-schema.json +3
-0@@ -526,11 +526,15 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has +# been migrated, pulling the remaining pages along as needed. NOTE: If +# the migration fails during postcopy the VM will fail. (since 2.4)
Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.
Quoted 16 lines · * Eric Blake ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the
migrate_start_postcopypatch later in the series).Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 6 +++++- 3 files changed, 29 insertions(+), 1 deletion(-)
Quoted 11 lines
qapi-schema.json line 531
Quoted 8 lines
+++
b/qapi-schema.jsonqapi-schema.json +3
-0@@ -526,11 +526,15 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has +# been migrated, pulling the remaining pages along as needed. NOTE: If +# the migration fails during postcopy the VM will fail. (since 2.4)Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?
It's easy enough to remove the x- once we're all happy; it seems pretty stable at the moment but when we're done I'll just submit a one liner to take the x- off.
Dave
Quoted 6 lines
Reviewed-by Eric Blake <[email protected]>Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Eric Blake ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the
migrate_start_postcopypatch later in the series).Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 6 +++++- 3 files changed, 29 insertions(+), 1 deletion(-)Quoted 11 lines
qapi-schema.json line 531
Quoted 8 lines
+++
b/qapi-schema.jsonqapi-schema.json +3
-0@@ -526,11 +526,15 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has +# been migrated, pulling the remaining pages along as needed. NOTE: If +# the migration fails during postcopy the VM will fail. (since 2.4)Marking it experimental because it might change? Or is the interface pretty stable, but you want more testing time to minimize bugs?
Quoted 3 lines
It's easy enough to remove the x- once we're all happy; it seems pretty stable at the moment but when we're done I'll just submit a one liner to take the x- off.
We shouldn't mark it stable till we have a released kernel (which freezes the kernel API for us).
We could pick this patchset without a released kernel, but this will have to remain x- if that happens.
Amit
Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the
migrate_start_postcopypatch later in the series).Signed-off-by Dr. David Alan Gilbert <[email protected]>
Re: [Qemu-devel] [PATCH v7 17/42] Add migration-capability boolean for postcopy-ram.
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the
migrate_start_postcopypatch later in the series).Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 18/42 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy
--- include/migration/migration.h | 16 +++ include/sysemu/sysemu.h | 20 ++++ migration/migration.c | 13 +++ migration/savevm.c | 247 ++++++++++++++++++++++++++++++++++++++++++ trace-events | 10 ++ 5 files changed, 306 insertions(+)
include/migration/migration.h +16 -0
@@ -50,6 +50,15 @@ enum mig_rp_message_type { }; typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head; + +typedef enum { +POSTCOPY_INCOMING_NONE =0 ,/* Initial state - no postcopy */ +POSTCOPY_INCOMING_ADVISE , +POSTCOPY_INCOMING_LISTENING , +POSTCOPY_INCOMING_RUNNING , +POSTCOPY_INCOMING_END +}PostcopyState ; +/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;
@@ -62,6 +71,7 @@ struct MigrationIncomingState {QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ +PostcopyState postcopy_state;/* See savevm.c */ LoadStateEntry_Head loadvm_handlers;
@@ -231,4 +241,10 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,void ram_mig_init (void );void savevm_skip_section_footers (void ); + +PostcopyState postcopy_state_get (MigrationIncomingState *mis); + +/* Set the state and return the old state */ +PostcopyState postcopy_state_set (MigrationIncomingState *mis, +PostcopyState new_state);#endif
include/sysemu/sysemu.h +20 -0
@@ -87,6 +87,17 @@ enum qemu_vm_cmd {MIG_CMD_INVALID =0 ,/* Must be 0 */ MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ MIG_CMD_PING ,/* Request a PONG on the RP */ + +MIG_CMD_POSTCOPY_ADVISE =20 ,/* Prior to any page transfers, just +warn we might want to do PC */ +MIG_CMD_POSTCOPY_LISTEN ,/* Start listening for incoming +pages as it's running. */ +MIG_CMD_POSTCOPY_RUN ,/* Start execution */ + +MIG_CMD_POSTCOPY_RAM_DISCARD ,/* A list of pages to discard that +were previously sent during +precopy but are dirty. */ + }; bool qemu_savevm_state_blocked(Error **errp);
@@ -101,6 +112,15 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);void qemu_savevm_send_open_return_path (QEMUFile *f); +void qemu_savevm_send_postcopy_advise (QEMUFile *f); +void qemu_savevm_send_postcopy_listen (QEMUFile *f); +void qemu_savevm_send_postcopy_run (QEMUFile *f); + +void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *end_list); +int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/migration.c +13 -0
@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); } + +PostcopyState postcopy_state_get (MigrationIncomingState *mis) +{ +return atomic_fetch_add (&mis->postcopy_state,0 ); +} + +/* Set the state and return the old state */ +PostcopyState postcopy_state_set (MigrationIncomingState *mis, +PostcopyState new_state) +{ +return atomic_xchg (&mis->postcopy_state, new_state); +} +
migration/savevm.c +247 -0
277 lines not on this page · raw message
trace-events +10 -0
@@ -1187,11 +1187,21 @@ qemu_loadvm_state_main(void) "" qemu_loadvm_state_main_quit_parent(void) "" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_postcopy_handle_advise(void) "" +loadvm_postcopy_handle_listen(void) "" +loadvm_postcopy_handle_run(void) "" +loadvm_postcopy_ram_handle_discard(void) "" +loadvm_postcopy_ram_handle_discard_end(void) "" +loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x" +qemu_savevm_send_postcopy_advise(void) "" +qemu_savevm_send_postcopy_ram_discard(const char *id, uint16_t len) "%s: %ud" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_send_ping(uint32_t val) "%x" +savevm_send_postcopy_listen(void) "" +savevm_send_postcopy_run(void) "" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
Quoted 20 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopySigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 16 +++ include/sysemu/sysemu.h | 20 ++++ migration/migration.c | 13 +++ migration/savevm.c | 247 ++++++++++++++++++++++++++++++++++++++++++ trace-events | 10 ++ 5 files changed, 306 insertions(+)migration/migration.c +0
-0
migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1134
Quoted 7 lines
migration/migration.c +4
-0@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); } + +PostcopyState postcopy_state_get (MigrationIncomingState *mis) +{ +return atomic_fetch_add (&mis->postcopy_state,0 );
What is wrong with atomic_read() here?
As the set of the state is atomic, even a normal read would do (I think)
migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1138
Quoted 4 lines
migration/migration.c +4
-0+void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *end_list)
I haven't looked at the following patches where this function is used, but it appears that getting an iovec could be a good idea?
migration/migration.c void migrate_fd_connect(MigrationState *s)
Quoted 8 lines
migration/migration.c +8
-0+{ +uint8_t *buf; +uint16_t tmplen; +uint16_t t; +size_t name_len =strlen (name); + +trace_qemu_savevm_send_postcopy_ram_discard (name, len); + buf =g_malloc0 (len*16 + name_len +3 );
I would suggest
gmalloc0(1 + 1 + name_len + 1 + (8 + 8) * len)
just to be clear where things came from.
I think that we don't need the \0 at all. If \0 is not there, strlen() return is going to be "funny". So, we can just change the assert to name_len < 255?
migration/migration.c
Quoted 2 lines
migration/migration.c +2
-0+ buf[0 ] =0 ;/* Version */ +assert (name_len <256 );
Can we move the assert before the malloc()?
My guess is that in a perfect world the assert would be a return -EINVAL, but I know that it is complicated.
migration/migration.c
Quoted 2 lines
migration/migration.c +2
-0+ buf[1 ] = name_len; +memcpy (buf+2 , name, name_len);
spaces around '+' (same around)
migration/migration.c
Quoted 8 lines
migration/migration.c +8
-0+ tmplen =2 +name_len; + buf[tmplen++] ='\0' ; + +for (t =0 ; t < len; t++) { +cpu_to_be64w ((uint64_t *)(buf + tmplen), start_list[t]); + tmplen +=8 ; +cpu_to_be64w ((uint64_t *)(buf + tmplen), end_list[t]); + tmplen +=8 ;
trace_qemu_savevm_send_postcopy_range(name, start_list[t], end_list[t]);
??
migration/migration.c
Quoted 6 lines
migration/migration.c +6
-0+/* We're expecting a +* Version (0) +* a RAM ID string (length byte, name, 0 term) +* then at least 1 16 byte chunk +*/ + if (len < 20) { 1 +
1+1+1+1+2*8
Humm, thinking about it, .... why are we not needing a length field of number of entries?
migration/migration.c
Quoted 6 lines
migration/migration.c +6
-0+error_report ("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)" , len); +return -1 ; + } + + tmp =qemu_get_byte (mis->file); +if (tmp !=0 ) {
I think that a constant telling POSTCOPY_VERSION0 or whatever?
Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
Quoted 11 lines · Juan Quintela
migration/savevm.c enum LoadVMExitCodes {
Quoted 6 lines
migration/savevm.c +6
-0+/* We're expecting a +* Version (0) +* a RAM ID string (length byte, name, 0 term) +* then at least 1 16 byte chunk +*/ + if (len < 20) { 1 +1+1+1+1+2*8Humm, thinking about it, .... why are we not needing a length field of number of entries?
hm, yea.
Quoted 8 lines
migration/savevm.c enum LoadVMExitCodes { · line 1169
Quoted 6 lines
migration/savevm.c +6
-0+error_report ("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)" , len); +return -1 ; + } + + tmp =qemu_get_byte (mis->file); +if (tmp !=0 ) {I think that a constant telling
POSTCOPY_VERSION0or whatever?
agreed.
Amit
Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
In reply to Juan QuintelaQuoted 16 lines · * Juan Quintela ([email protected]) wrote:
Quoted 15 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopySigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 16 +++ include/sysemu/sysemu.h | 20 ++++ migration/migration.c | 13 +++ migration/savevm.c | 247 ++++++++++++++++++++++++++++++++++++++++++ trace-events | 10 ++ 5 files changed, 306 insertions(+)
Quoted 15 lines
migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1134
Quoted 12 lines
migration/migration.c +4
-0@@ -1128,3 +1128,16 @@ void migrate_fd_connect(MigrationState *s)qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); } + +PostcopyState postcopy_state_get (MigrationIncomingState *mis) +{ +return atomic_fetch_add (&mis->postcopy_state,0 );What is wrong with
atomic_read()here?
As the set of the state is atomic, even a normal read would do (I think)
Actually, I made this an atomic_mb_read as per Paolo's comment on my v5 version (31st March).
I also added a comment documenting which threads read/write the state.
Quoted 7 lines
migration/migration.c void migrate_fd_connect(MigrationState *s) · line 1138
Quoted 4 lines
migration/migration.c +4
-0+void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *end_list)I haven't looked at the following patches where this function is used, but it appears that getting an iovec could be a good idea?
Yes, although I wouldn't want to make the wire format dependent on the host size_t or pointer size or anything.
Quoted 13 lines
migration/migration.c void migrate_fd_connect(MigrationState *s)
Quoted 9 lines
migration/migration.c +8
-0+{ +uint8_t *buf; +uint16_t tmplen; +uint16_t t; +size_t name_len =strlen (name); + +trace_qemu_savevm_send_postcopy_ram_discard (name, len); + buf =g_malloc0 (len*16 + name_len +3 );I would suggest
gmalloc0(1 + 1 + name_len + 1 + (8 + 8) * len)
Quoted 1 line
just to be clear where things came from.
Done.
Quoted 3 lines
I think that we don't need the \0 at all. If \0 is not there,
strlen()return is going to be "funny". So, we can just change the assert toname_len< 255?
Dave Gibson asked for the \0 in a previous review.
Quoted 5 lines
migration/migration.c
Quoted 3 lines
migration/migration.c +2
-0+ buf[0 ] =0 ;/* Version */ +assert (name_len <256 );Can we move the assert before the
malloc()?
Done.
Quoted 7 lines
My guess is that in a perfect world the assert would be a return
-EINVAL, but I know that it is complicated.migration/migration.c
Quoted 2 lines
migration/migration.c +2
-0+ buf[1 ] = name_len; +memcpy (buf+2 , name, name_len);spaces around '+' (same around)
Done.
Quoted 9 lines
migration/migration.c
Quoted 9 lines
migration/migration.c +8
-0+ tmplen =2 +name_len; + buf[tmplen++] ='\0' ; + +for (t =0 ; t < len; t++) { +cpu_to_be64w ((uint64_t *)(buf + tmplen), start_list[t]); + tmplen +=8 ; +cpu_to_be64w ((uint64_t *)(buf + tmplen), end_list[t]); + tmplen +=8 ;
Quoted 3 lines
trace_qemu_savevm_send_postcopy_range(name, start_list[t], end_list[t]);??
???
Quoted 8 lines
migration/migration.c
Quoted 6 lines
migration/migration.c +6
-0+/* We're expecting a +* Version (0) +* a RAM ID string (length byte, name, 0 term) +* then at least 1 16 byte chunk +*/ + if (len < 20) { 1 +1+1+1+1+2*8
Done.
Quoted 2 lines
Humm, thinking about it, .... why are we not needing a length field of number of entries?
Because we've got the size of the whole message from the command header.
Quoted 8 lines
migration/migration.c
Quoted 6 lines
migration/migration.c +6
-0+error_report ("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)" , len); +return -1 ; + } + + tmp =qemu_get_byte (mis->file); +if (tmp !=0 ) {I think that a constant telling
POSTCOPY_VERSION0or whatever?
Done; (as a const postcopy_ram_discard_version)
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopySigned-off-by Dr. David Alan Gilbert <[email protected]>
But:
Quoted 8 lines
migration/savevm.c +8
-0+void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *end_list) +{ +uint8_t *buf; +uint16_t tmplen; +uint16_t t;
migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 766
Quoted 10 lines
migration/savevm.c +10
-0+size_t name_len =strlen (name); + +trace_qemu_savevm_send_postcopy_ram_discard (name, len); + buf =g_malloc0 (len*16 + name_len +3 ); + buf[0 ] =0 ;/* Version */ +assert (name_len <256 ); + buf[1 ] = name_len; +memcpy (buf+2 , name, name_len); + tmplen =2 +name_len; + buf[tmplen++] ='\0' ;
whitespace around operators missing
migration/savevm.c enum LoadVMExitCodes { · line 1184
Quoted 4 lines
migration/savevm.c +3
-0+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis, + uint16_t len) + len -= 3+strlen(ramid);
ditto
Amit
Re: [Qemu-devel] [PATCH v7 18/42] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
Quoted 10 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopySigned-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>
Thanks,
Quoted 23 lines
But:
Quoted 8 lines
migration/savevm.c +8
-0+void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *end_list) +{ +uint8_t *buf; +uint16_t tmplen; +uint16_t t;migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 766
Quoted 10 lines
migration/savevm.c +10
-0+size_t name_len =strlen (name); + +trace_qemu_savevm_send_postcopy_ram_discard (name, len); + buf =g_malloc0 (len*16 + name_len +3 ); + buf[0 ] =0 ;/* Version */ +assert (name_len <256 ); + buf[1 ] = name_len; +memcpy (buf+2 , name, name_len); + tmplen =2 +name_len; + buf[tmplen++] ='\0' ;whitespace around operators missing
Quoted 6 lines
migration/savevm.c enum LoadVMExitCodes { · line 1184
Quoted 4 lines
migration/savevm.c +3
-0+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis, + uint16_t len) + len -= 3+strlen(ramid);ditto
Fixed.
Dave
Quoted 2 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 19/42 MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.
This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
--- include/sysemu/sysemu.h | 4 +++ migration/savevm.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ trace-events | 4 +++ 3 files changed, 102 insertions(+)
include/sysemu/sysemu.h +4 -0
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {MIG_CMD_INVALID =0 ,/* Must be 0 */ MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ MIG_CMD_PING ,/* Request a PONG on the RP */ +MIG_CMD_PACKAGED ,/* Send a wrapped stream within this stream */ MIG_CMD_POSTCOPY_ADVISE = 20, /* Prior to any page transfers, justwarn we might want to do PC */
@@ -100,6 +101,8 @@ enum qemu_vm_cmd { }; +#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24) +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -112,6 +115,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);void qemu_savevm_send_open_return_path (QEMUFile *f); +int qemu_savevm_send_packaged (QEMUFile *f,const QEMUSizedBuffer *qsb);void qemu_savevm_send_postcopy_advise (QEMUFile *f);void qemu_savevm_send_postcopy_listen (QEMUFile *f);void qemu_savevm_send_postcopy_run (QEMUFile *f);
migration/savevm.c +94 -0
112 lines not on this page · raw message
trace-events +4 -0
@@ -1187,6 +1187,10 @@ qemu_loadvm_state_main(void) "" qemu_loadvm_state_main_quit_parent(void) "" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +qemu_savevm_send_packaged(void) "" +loadvm_handle_cmd_packaged(unsigned int length) "%u" +loadvm_handle_cmd_packaged_main(int ret) "%d" +loadvm_handle_cmd_packaged_received(int ret) "%d" loadvm_postcopy_handle_advise(void) "" loadvm_postcopy_handle_listen(void) "" loadvm_postcopy_handle_run(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGEDis a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the wholeMIG_CMD_PACKAGEDis read off the stream prior to parsing the contents.This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
Quoted 7 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
Famous words of advise O:-)
I have to read the rest of series to make my mind on this one.
Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
In reply to Dr. David Alan Gilbert (git)Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGEDis a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the wholeMIG_CMD_PACKAGEDis read off the stream prior to parsing the contents.This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
Not sure why this is necessary. I suppose I'll have to go read the documentation in patch 1..
However:
Quoted 28 lines
migration/savevm.c +22
-0@@ -718,6 +718,50 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)qemu_savevm_command_send (f,MIG_CMD_OPEN_RETURN_PATH ,0 ,NULL ); } +/* We have a buffer of data to send; we don't want that all to be loaded +* by the command itself, so the command contains just the length of the +* extra buffer that we then send straight after it. +* TODO: Must be a better way to organise that +* +* Returns: +* 0 on success +* -ve on error +*/ +int qemu_savevm_send_packaged (QEMUFile *f,const QEMUSizedBuffer *qsb) +{ +size_t cur_iov; +size_t len =qsb_get_length (qsb); +uint32_t tmp; + +if (len >MAX_VM_CMD_PACKAGED_SIZE ) { +error_report ("%s: Unreasonably large packaged state: %zu" , + __func__, len); +return -1 ; + } + + tmp =cpu_to_be32 (len);
migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 752
Quoted 10 lines
migration/savevm.c +10
-0+ +trace_qemu_savevm_send_packaged (); +qemu_savevm_command_send (f,MIG_CMD_PACKAGED ,4 , (uint8_t *)&tmp); + +/* all the data follows (concatinating the iov's) */ +for (cur_iov =0 ; cur_iov < qsb->n_iov; cur_iov++) { +/* The iov entries are partially filled */ +size_t towrite = (qsb->iov[cur_iov].iov_len > len) ? + len : + qsb->iov[cur_iov].iov_len;
If iov_len was > len, we only wrote part of the current buffer, and we skip to the next?
Amit
Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
Quoted 8 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGEDis a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the wholeMIG_CMD_PACKAGEDis read off the stream prior to parsing the contents.
Quoted 5 lines
Quoted 2 lines
This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
Not sure why this is necessary. I suppose I'll have to go read the documentation in patch 1..
Yep - or one of the previous replies where I explained it.
Quoted 35 lines
However:
Quoted 33 lines
migration/savevm.c +27
-0@@ -718,6 +718,50 @@ void qemu_savevm_send_open_return_path(QEMUFile *f)qemu_savevm_command_send (f,MIG_CMD_OPEN_RETURN_PATH ,0 ,NULL ); } +/* We have a buffer of data to send; we don't want that all to be loaded +* by the command itself, so the command contains just the length of the +* extra buffer that we then send straight after it. +* TODO: Must be a better way to organise that +* +* Returns: +* 0 on success +* -ve on error +*/ +int qemu_savevm_send_packaged (QEMUFile *f,const QEMUSizedBuffer *qsb) +{ +size_t cur_iov; +size_t len =qsb_get_length (qsb); +uint32_t tmp; + +if (len >MAX_VM_CMD_PACKAGED_SIZE ) { +error_report ("%s: Unreasonably large packaged state: %zu" , + __func__, len); +return -1 ; + } + + tmp =cpu_to_be32 (len); + +trace_qemu_savevm_send_packaged (); +qemu_savevm_command_send (f,MIG_CMD_PACKAGED ,4 , (uint8_t *)&tmp); + +/* all the data follows (concatinating the iov's) */
Quoted 8 lines
migration/savevm.c void qemu_savevm_send_open_return_path(QEMUFile *f) · line 752
Quoted 5 lines
migration/savevm.c +5
-0+for (cur_iov =0 ; cur_iov < qsb->n_iov; cur_iov++) { +/* The iov entries are partially filled */ +size_t towrite = (qsb->iov[cur_iov].iov_len > len) ? + len : + qsb->iov[cur_iov].iov_len;If
iov_lenwas > len, we only wrote part of the current buffer, and we skip to the next?
Yes; this is just the end case; the qsb allocates iov entries in 'chunks' but then the data that gets added often doesn't use the whole chunk. 'len' - set above from qsb_get_length - gives the used contents of the qsb, and that's all we want to write. This is normally the case on the last entry in the qsb anyway; however since 'len' gets decremented by the amount written we might go once more around the loop and the 'if (!towrite) { break; }' might break us out of the loop instead.
Dave
Quoted 3 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 19/42] MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
In reply to Dr. David Alan Gilbert (git)Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGEDis a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the wholeMIG_CMD_PACKAGEDis read off the stream prior to parsing the contents.This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 20/42 Modify save_live_pending for postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Modify save_live_pending to return separate postcopiable and non-postcopiable counts.
--- include/migration/vmstate.h | 5 +++-- include/sysemu/sysemu.h | 4 +++- migration/block.c | 7 +++++-- migration/migration.c | 9 +++++++-- migration/ram.c | 8 ++++++-- migration/savevm.c | 21 +++++++++++++++++---- trace-events | 2 +- 7 files changed, 42 insertions(+), 14 deletions(-)
include/migration/vmstate.h +3 -2
@@ -54,8 +54,9 @@ typedef struct SaveVMHandlers {/* This runs outside the iothread lock! */ int (*save_live_setup)(QEMUFile *f,void *opaque); -uint64_t (*save_live_pending)(QEMUFile *f,void *opaque,uint64_t max_size); - +void (*save_live_pending)(QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending);LoadStateHandler *load_state; } SaveVMHandlers;
include/sysemu/sysemu.h +3 -1
@@ -110,7 +110,9 @@ void qemu_savevm_state_header(QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void ); -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);
migration/block.c +5 -2
@@ -754,7 +754,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)return 0 ; } -static uint64_t block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {/* Estimate pending number of bytes to send */ uint64_t pending;
@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread ();DPRINTF ("Enter save live pending %" PRIu64"\n" , pending); -return pending; + *non_postcopiable_pending = pending; + *postcopiable_pending =0 ; }static int block_load (QEMUFile *f,void *opaque,int version_id)
migration/migration.c +7 -2
@@ -1024,8 +1024,13 @@ static void *migration_thread(void *opaque)uint64_t pending_size;if (!qemu_file_rate_limit (s->file)) { - pending_size =qemu_savevm_state_pending (s->file, max_size); -trace_migrate_pending (pending_size, max_size); +uint64_t pend_post, pend_nonpost; + +qemu_savevm_state_pending (s->file, max_size, &pend_nonpost, + &pend_post); + pending_size = pend_nonpost + pend_post; +trace_migrate_pending (pending_size, max_size, + pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) {qemu_savevm_state_iterate (s->file); }else {
migration/ram.c +6 -2
@@ -1273,7 +1273,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)return 0 ; } -static uint64_t ram_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void ram_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {uint64_t remaining_size;
@@ -1287,7 +1289,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread (); remaining_size =ram_save_remaining () *TARGET_PAGE_SIZE ; } -return remaining_size; + + *non_postcopiable_pending =0 ; + *postcopiable_pending = remaining_size; }static int load_xbzrle (QEMUFile *f,ram_addr_t addr,void *host)
migration/savevm.c +17 -4
@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)qemu_fflush (f); } -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size) +/* Give an estimate of the amount left to be transferred, +* the result is split into the amount for units that can and +* for units that can't do postcopy. +*/ +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable) {SaveStateEntry *se; -uint64_t ret =0 ; +uint64_t tmp_non_postcopiable, tmp_postcopiable; + + *res_non_postcopiable =0 ; + *res_postcopiable =0 ; + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || !se->ops->save_live_pending) {
@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)continue ; } } - ret += se->ops->save_live_pending (f, se->opaque, max_size); + se->ops->save_live_pending (f, se->opaque, max_size, + &tmp_non_postcopiable, &tmp_postcopiable); + + *res_postcopiable += tmp_postcopiable; + *res_non_postcopiable += tmp_non_postcopiable; } -return ret; }void qemu_savevm_state_cancel (void )
trace-events +1 -1
@@ -1419,7 +1419,7 @@ migrate_fd_cleanup(void) "" migrate_fd_cleanup_src_rp(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" -migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 20/42] Modify save_live_pending for postcopy
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Modify
save_live_pendingto return separate postcopiable and non-postcopiable counts.Signed-off-by Dr. David Alan Gilbert <[email protected]>
I think that if you make a small change of meaning, everything gots easier:
Quoted 8 lines
migration/block.c +3
-1-static uint64_t block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {/* Estimate pending number of bytes to send */ uint64_t pending;
migration/block.c static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size) · line 779
Quoted 6 lines
migration/block.c +2
-1@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread ();DPRINTF ("Enter save live pending %" PRIu64"\n" , pending); -return pending; + *non_postcopiable_pending = pending; + *postcopiable_pending =0 ;
Change that two lines to:
*non_postcopiable_pending += pending;
*postcopiable_pending += 0; /* ok, equivalent of doing nothing */This way, chaining gots easier?
Quoted 29 lines
migration/savevm.c +12
-2@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)qemu_fflush (f); } -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size) +/* Give an estimate of the amount left to be transferred, +* the result is split into the amount for units that can and +* for units that can't do postcopy. +*/ +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable) {SaveStateEntry *se; -uint64_t ret =0 ; +uint64_t tmp_non_postcopiable, tmp_postcopiable; + + *res_non_postcopiable =0 ; + *res_postcopiable =0 ; + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || !se->ops->save_live_pending) {@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)continue ;
migration/savevm.c uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size) · line 1031
Quoted 10 lines
migration/savevm.c +5
-2} } - ret += se->ops->save_live_pending (f, se->opaque, max_size); + se->ops->save_live_pending (f, se->opaque, max_size, + &tmp_non_postcopiable, &tmp_postcopiable); + + *res_postcopiable += tmp_postcopiable; + *res_non_postcopiable += tmp_non_postcopiable; } -return ret;
With the change, we don't care in the other functions, and this one gets simpler IMHO.
Re: [Qemu-devel] [PATCH v7 20/42] Modify save_live_pending for postcopy
Quoted 9 lines · * Juan Quintela ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Modify
save_live_pendingto return separate postcopiable and non-postcopiable counts.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>
Thanks,
Quoted 19 lines
I think that if you make a small change of meaning, everything gots easier:
Quoted 8 lines
migration/block.c +3
-1-static uint64_t block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {/* Estimate pending number of bytes to send */ uint64_t pending;migration/block.c static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size) · line 779
Quoted 6 lines
migration/block.c +2
-1@@ -773,7 +775,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread ();DPRINTF ("Enter save live pending %" PRIu64"\n" , pending); -return pending; + *non_postcopiable_pending = pending; + *postcopiable_pending =0 ;Change that two lines to:
Quoted 4 lines
*non_postcopiable_pending += pending; *postcopiable_pending += 0; /* ok, equivalent of doing nothing */This way, chaining gots easier?
OK, done; I did it as:
+ /* We can do postcopy, and all the data is postcopiable */
+ *postcopiable_pending += remaining_size;
rather than having the odd += 0;
Quoted 34 lines
Quoted 34 lines
migration/savevm.c +14
-3@@ -1012,10 +1012,20 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)qemu_fflush (f); } -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size) +/* Give an estimate of the amount left to be transferred, +* the result is split into the amount for units that can and +* for units that can't do postcopy. +*/ +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable) {SaveStateEntry *se; -uint64_t ret =0 ; +uint64_t tmp_non_postcopiable, tmp_postcopiable; + + *res_non_postcopiable =0 ; + *res_postcopiable =0 ; + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || !se->ops->save_live_pending) {@@ -1026,9 +1036,12 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)continue ; } } - ret += se->ops->save_live_pending (f, se->opaque, max_size); + se->ops->save_live_pending (f, se->opaque, max_size, + &tmp_non_postcopiable, &tmp_postcopiable);
Quoted 8 lines
migration/savevm.c uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size) · line 1031
Quoted 5 lines
migration/savevm.c +3
-1+ + *res_postcopiable += tmp_postcopiable; + *res_non_postcopiable += tmp_non_postcopiable; } -return ret;With the change, we don't care in the other functions, and this one gets simpler IMHO.
Yep,
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 20/42] Modify save_live_pending for postcopy
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Modify
save_live_pendingto return separate postcopiable and non-postcopiable counts.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 21/42 postcopy: OS support test
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates postcopy-ram.c which will get most of the other helpers we need.
--- include/migration/postcopy-ram.h | 19 +++++ migration/Makefile.objs | 2 +- migration/postcopy-ram.c | 158 +++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 5 ++ 4 files changed, 183 insertions(+), 1 deletion(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
@@ -0,0 +1,19 @@ +/* +* Postcopy migration for RAM +* +* Copyright 2013 Red Hat, Inc. and/or its affiliates +* +* Authors: +* Dave Gilbert <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_POSTCOPY_RAM_H +#define QEMU_POSTCOPY_RAM_H + +/* Return true if the host supports everything we need to do postcopy-ram */ +bool postcopy_ram_supported_by_host (void ); + +#endif
migration/Makefile.objs +1 -1
@@ -1,7 +1,7 @@ common-obj-y += migration.o tcp.o common-obj-y += vmstate.o common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o -common-obj-y += xbzrle.o +common-obj-y += xbzrle.o postcopy-ram.o common-obj-$(CONFIG_RDMA) += rdma.o common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +158 -0
158 lines not on this page · raw message
migration/savevm.c +5 -0
@@ -37,6 +37,7 @@#include "qemu/timer.h" #include "audio/audio.h" #include "migration/migration.h" +#include "migration/postcopy-ram.h" #include "qemu/sockets.h" #include "qemu/queue.h" #include "sysemu/cpus.h"
@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (!postcopy_ram_supported_by_host ()) { +return -1 ; + } +if (remote_hps !=getpagesize ()) {/* * Some combinations of mismatch are probably possible but it gets
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>
I am guessing that test is ok, but we are doing the test each time that we change the function. We always end calling that kind of functions in several places. Shouldn't be good to rename the function to __postcopy_ram_supported_by_host()
and do a toplevel function that is:
bool postcopy_ram_supported_by_host(void)
{
static bool first_time = true;
static supported = false;
if (firt_time) {
first_time = false;
supported = __postcopy_ram_supported_by_host()
}
return supported;
}Notice that I don't know how slow the mmap + usefault thing is, but I guess that the values would not change while running, no?
It has a review-by because I don't see anything wrong with it.
Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test
Quoted 31 lines · * Juan Quintela ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>I am guessing that test is ok, but we are doing the test each time that we change the function. We always end calling that kind of functions in several places. Shouldn't be good to rename the function to
__postcopy_ram_supported_by_host()and do a toplevel function that is:
bool
postcopy_ram_supported_by_host(void){ static bool first_time = true; static supported = false; if (firt_time) { first_time = false; supported = __postcopy_ram_supported_by_host() } return supported; }
Quoted 2 lines
Notice that I don't know how slow the mmap + usefault thing is, but I guess that the values would not change while running, no?
Since we only call this once, at the start of an incoming migration, it seems overkill to do that.
Dave
Quoted 2 lines
It has a review-by because I don't see anything wrong with it.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test
In reply to Dr. David Alan Gilbert (git)Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>
migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1172
Quoted 7 lines
migration/savevm.c +4
-0@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (!postcopy_ram_supported_by_host ()) { +return -1 ; + } +
So this is just advise: if we receive this, and we can't handle postcopy, we're going to abort migration? Shouldn't we continue and let src know that we can't accept postcopy? ie the problem is currently punted to higher levels, should we try to handle it ourselves?
Amit
Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test
Quoted 14 lines · * Amit Shah ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/savevm.c +0
-0@@ -1165,6 +1166,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; }
Quoted 10 lines
migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1172
Quoted 4 lines
migration/savevm.c +4
-0+if (!postcopy_ram_supported_by_host ()) { +return -1 ; + } +So this is just advise: if we receive this, and we can't handle postcopy, we're going to abort migration? Shouldn't we continue and let src know that we can't accept postcopy? ie the problem is currently punted to higher levels, should we try to handle it ourselves?
We could, although it happens right at the start of migration, and it only happens if you explicitly enabled the postcopy capability, so I'm not sure there's any advantage in trying to fall back when you've been told to use it.
Dave
Quoted 4 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 21/42] postcopy: OS support test
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 22/42 migrate_start_postcopy: Command to trigger transition to postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopy
command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
--- hmp-commands.hx | 15 +++++++++++++++ hmp.c | 7 +++++++ hmp.h | 1 + include/migration/migration.h | 3 +++ migration/migration.c | 22 ++++++++++++++++++++++ qapi-schema.json | 8 ++++++++ qmp-commands.hx | 19 +++++++++++++++++++ 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
@@ -1006,6 +1006,21 @@ Set the parameter @var{parameter} for migration. ETEXI { + .name = "migrate_start_postcopy", + .args_type = "", + .params = "", + .help = "Switch migration to postcopy mode", + .mhandler.cmd = hmp_migrate_start_postcopy, + }, + +STEXI +@item migrate_start_postcopy +@findex migrate_start_postcopy +Switch in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). +ETEXI + + { .name = "client_migrate_info", .args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?", .params = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
@@ -1268,6 +1268,13 @@ void hmp_client_migrate_info(Monitor *mon, const QDict *qdict)hmp_handle_error (mon, &err); } +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict) +{ +Error *err =NULL ; +qmp_migrate_start_postcopy (&err); +hmp_handle_error (mon, &err); +} +void hmp_set_password (Monitor *mon,const QDict *qdict) {const char *protocol =qdict_get_str (qdict,"protocol" );
hmp.h +1 -0
@@ -68,6 +68,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);void hmp_migrate_set_parameter (Monitor *mon,const QDict *qdict);void hmp_migrate_set_cache_size (Monitor *mon,const QDict *qdict);void hmp_client_migrate_info (Monitor *mon,const QDict *qdict); +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict);void hmp_set_password (Monitor *mon,const QDict *qdict);void hmp_expire_password (Monitor *mon,const QDict *qdict);void hmp_eject (Monitor *mon,const QDict *qdict);
include/migration/migration.h +3 -0
@@ -111,6 +111,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);
migration/migration.c +22 -0
@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) { +error_setg (errp,"Postcopy must be started after migration has been" +" started" ); +return ; + } +/* +* we don't error if migration has finished since that would be racy +* with issuing this command. +*/ +atomic_set (&s->start_postcopy, true); +} +/* shared migration helpers */ static void migrate_set_state (MigrationState *s,int old_state,int new_state)
qapi-schema.json +8 -0
@@ -661,6 +661,14 @@ '*tls-port': 'int', '*cert-subject': 'str' } } ## +# @migrate-start-postcopy +# +# Switch migration to postcopy mode +# +# Since: 2.4 +{ 'command': 'migrate-start-postcopy' } + +## # @MouseInfo: # # Information about a mouse device.
qmp-commands.hx +19 -0
@@ -713,6 +713,25 @@ Example: EQMP { + .name = "migrate-start-postcopy", + .args_type = "", + .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy, + }, + +SQMP +migrate-start-postcopy +---------------------- + +Switch an in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). + +Example: +-> { "execute": "migrate-start-postcopy" } +<- { "return": {} } + +EQMP + + { .name = "query-migrate-cache-size", .args_type = "", .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 40 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with
migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopycommand must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>include/migration/migration.h +3
-0@@ -111,6 +111,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);migration/migration.c +1
-0@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp)
migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 10 lines
migration/migration.c +10
-0+{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {
I would claim that this check should be:
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
Thanks, Juan.
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with
migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopycommand must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>include/migration/migration.h +3
-0@@ -111,6 +111,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);migration/migration.c +1
-0@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp)migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 10 lines
migration/migration.c +10
-0+{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {I would claim that this check should be:
Quoted 4 lines
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereever
which wont work, because migrate_init reinitialises
the flag that start previously set.
However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy
if you're unlucky, and the migration finishes just at the same time you do the migrate_start_postcopy, do you want migrate_start_postcopy to fail? My guess was it was best for it not to fail, in this case.
Dave
Quoted 2 lines
Thanks, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 73 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with
migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopycommand must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>include/migration/migration.h +3
-0@@ -111,6 +111,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);migration/migration.c +1
-0@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp)migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 10 lines
migration/migration.c +10
-0+{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {I would claim that this check should be:
Quoted 4 lines
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereeverwhich wont work, because
migrate_initreinitialises
the flag that start previously set.However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy
Quoted 4 lines
if you're unlucky, and the migration finishes just at the same time you do the
migrate_start_postcopy, do you wantmigrate_start_postcopyto fail? My guess was it was best for it not to fail, in this case.
Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.
Later, Juan.
Quoted 7 lines
Dave
Quoted 2 lines
Thanks, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 35 lines · Juan Quintela
Quoted 34 lines · Dr. David Alan Gilbert
Quoted 14 lines · * Juan Quintela ([email protected]) wrote:
migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 11 lines
migration/migration.c +11
-0+void qmp_migrate_start_postcopy (Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {I would claim that this check should be:
Quoted 4 lines
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereeverwhich wont work, because
migrate_initreinitialises
the flag that start previously set.However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy
Quoted 8 lines
Quoted 4 lines
if you're unlucky, and the migration finishes just at the same time you do the
migrate_start_postcopy, do you wantmigrate_start_postcopyto fail? My guess was it was best for it not to fail, in this case.Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.
Yeah, I tend to agree, because in the cases where migration has failed or has been cancelled, we'll end up setting the postcopy bit. Then, upon the next migration, this bit could get reused - resulting in the previous condition of setting postcopy bit before starting migration.
Amit
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 41 lines · * Amit Shah ([email protected]) wrote:
Quoted 40 lines · Juan Quintela
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 14 lines · * Juan Quintela ([email protected]) wrote:
migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 11 lines
migration/migration.c +11
-0+void qmp_migrate_start_postcopy (Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {I would claim that this check should be:
Quoted 4 lines
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereeverwhich wont work, because
migrate_initreinitialises
the flag that start previously set.However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopyif you're unlucky, and the migration finishes just at the same time you do the
migrate_start_postcopy, do you wantmigrate_start_postcopyto fail? My guess was it was best for it not to fail, in this case.
Quoted 8 lines
Quoted 3 lines
Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.
Yeah, I tend to agree, because in the cases where migration has failed or has been cancelled, we'll end up setting the postcopy bit. Then, upon the next migration, this bit could get reused - resulting in the previous condition of setting postcopy bit before starting migration.
No, that doesn't happen; the bit is cleared at the start of migration so that race condition doesn't exist.
Dave
Quoted 3 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 22/42] migrate_start_postcopy: Command to trigger transition to postcopy
In reply to Juan QuintelaQuoted 74 lines · * Juan Quintela ([email protected]) wrote:
Quoted 73 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Juan Quintela ([email protected]) wrote:
Quoted 40 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with
migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopycommand must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>include/migration/migration.h +3
-0@@ -111,6 +111,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);migration/migration.c +1
-0@@ -465,6 +465,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp)migration/migration.c void qmp_migrate_set_parameters(bool has_compress_level, · line 478
Quoted 10 lines
migration/migration.c +10
-0+{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) {I would claim that this check should be:
Quoted 4 lines
if (s->state != MIGRATION_STATUS_ACTIVE) {??
FAILED, COMPLETED, CANCELL* don't make sense, right?
What I'm trying to catch here is people doing:
migrate_start_postcopy
migrate tcp:pppp:whereeverwhich wont work, because
migrate_initreinitialises
the flag that start previously set.However, I also don't want to create a race, since what you do is typically:
migrate tcp:pppp:whereever
<wait some time, get bored>
migrate_start_postcopy
Quoted 8 lines
Quoted 4 lines
if you're unlucky, and the migration finishes just at the same time you do the
migrate_start_postcopy, do you wantmigrate_start_postcopyto fail? My guess was it was best for it not to fail, in this case.Change the order, if it is ACTIVE: do the postcopy thing, otherwise, do the clause that is protected now? Moving to postcopy only make sense if we are in active.
The problem is that produces a race-condition for the command. If you wait too long and the migration finishes before you issue the command you get an error, when the migration has completed perfectly happily.
Dave
Quoted 11 lines
Later, Juan.
Quoted 7 lines
Dave
Quoted 2 lines
Thanks, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 23/42 MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy
'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.
--- include/migration/migration.h | 2 ++ migration/migration.c | 56 ++++++++++++++++++++++++++++++++++++------- qapi-schema.json | 4 +++- trace-events | 1 + 4 files changed, 54 insertions(+), 9 deletions(-)
include/migration/migration.h +2 -0
@@ -154,6 +154,8 @@ MigrationState *migrate_init(const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *); +/* True if outgoing migration has entered postcopy phase */ +bool migration_postcopy_phase (MigrationState *);MigrationState *migrate_get_current (void );void migrate_compress_threads_create (void );
migration/migration.c +48 -8
104 lines not on this page · raw message
qapi-schema.json +3 -1
@@ -424,6 +424,8 @@ # # @active: in the process of doing migration. # +# @postcopy-active: like active, but now in postcopy mode. (since 2.4) +# # @completed: migration is finished. # # @failed: some error occurred during migration process.
@@ -433,7 +435,7 @@ ## { 'enum': 'MigrationStatus', 'data': [ 'none', 'setup', 'cancelling', 'cancelled', - 'active', 'completed', 'failed' ] } + 'active', 'postcopy-active', 'completed', 'failed' ] } ## # @MigrationInfo
trace-events +1 -0
@@ -1421,6 +1421,7 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migration_thread_setup_complete(void) "" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" source_return_path_thread_bad_end(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Eric Blake <[email protected]>
But (there is always a but....)
Quoted 24 lines
migration/migration.c +20
-0@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)get_xbzrle_cache_stats (info);break ; +case MIGRATION_STATUS_POSTCOPY_ACTIVE : +/* Mostly the same as active; TODO add some postcopy stats */ + info->has_status = true; + info->has_total_time = true; + info->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) + - s->total_time; + info->has_expected_downtime = true; + info->expected_downtime = s->expected_downtime; + info->has_setup_time = true; + info->setup_time = s->setup_time; + + info->has_ram = true; + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->transferred =ram_bytes_transferred (); + info->ram->remaining =ram_bytes_remaining (); + info->ram->total =ram_bytes_total (); + info->ram->duplicate =dup_mig_pages_transferred (); + info->ram->skipped =skipped_mig_pages_transferred (); + info->ram->normal =norm_mig_pages_transferred (); + info->ram->normal_bytes =norm_mig_bytes_transferred ();
migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391
Quoted 10 lines
migration/migration.c +10
-0+ info->ram->dirty_pages_rate = s->dirty_pages_rate; + info->ram->mbps = s->mbps; + +if (blk_mig_active ()) { + info->has_disk = true; + info->disk =g_malloc0 (sizeof (*info->disk)); + info->disk->transferred =blk_mig_bytes_transferred (); + info->disk->remaining =blk_mig_bytes_remaining (); + info->disk->total =blk_mig_bytes_total (); + }
Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?
Once here, I guess we can get the migrate_already_active() bit without problem?
Later, Juan.
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Juan Quintela <[email protected]>But (there is always a but....)
Quoted 30 lines
migration/migration.c +26
-0@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)get_xbzrle_cache_stats (info);break ; +case MIGRATION_STATUS_POSTCOPY_ACTIVE : +/* Mostly the same as active; TODO add some postcopy stats */ + info->has_status = true; + info->has_total_time = true; + info->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) + - s->total_time; + info->has_expected_downtime = true; + info->expected_downtime = s->expected_downtime; + info->has_setup_time = true; + info->setup_time = s->setup_time; + + info->has_ram = true; + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->transferred =ram_bytes_transferred (); + info->ram->remaining =ram_bytes_remaining (); + info->ram->total =ram_bytes_total (); + info->ram->duplicate =dup_mig_pages_transferred (); + info->ram->skipped =skipped_mig_pages_transferred (); + info->ram->normal =norm_mig_pages_transferred (); + info->ram->normal_bytes =norm_mig_bytes_transferred (); + info->ram->dirty_pages_rate = s->dirty_pages_rate; + info->ram->mbps = s->mbps; + +if (blk_mig_active ()) { + info->has_disk = true; + info->disk =g_malloc0 (sizeof (*info->disk));
Quoted 8 lines
migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391
Quoted 4 lines
migration/migration.c +4
-0+ info->disk->transferred =blk_mig_bytes_transferred (); + info->disk->remaining =blk_mig_bytes_remaining (); + info->disk->total =blk_mig_bytes_total (); + }Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?
I think you can, I've not tried it; however I also think it should be safe.
migration/block.c's block_save_pending always puts a value in thenon_postcopiable_pending return value (and 0 in the postcopiable_pending); the migrate thread checks the non_postcopiable_pending size to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.
A possibly harder question is what happens if block.c did implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.
Quoted 2 lines
Once here, I guess we can get the
migrate_already_active()bit without problem?
I'm not sure of the question here; but the idea of migration_already_active() is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.
Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Juan Quintela <[email protected]>But (there is always a but....)
Quoted 30 lines
migration/migration.c +26
-0@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)get_xbzrle_cache_stats (info);break ; +case MIGRATION_STATUS_POSTCOPY_ACTIVE : +/* Mostly the same as active; TODO add some postcopy stats */ + info->has_status = true; + info->has_total_time = true; + info->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) + - s->total_time; + info->has_expected_downtime = true; + info->expected_downtime = s->expected_downtime; + info->has_setup_time = true; + info->setup_time = s->setup_time; + + info->has_ram = true; + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->transferred =ram_bytes_transferred (); + info->ram->remaining =ram_bytes_remaining (); + info->ram->total =ram_bytes_total (); + info->ram->duplicate =dup_mig_pages_transferred (); + info->ram->skipped =skipped_mig_pages_transferred (); + info->ram->normal =norm_mig_pages_transferred (); + info->ram->normal_bytes =norm_mig_bytes_transferred (); + info->ram->dirty_pages_rate = s->dirty_pages_rate; + info->ram->mbps = s->mbps; + +if (blk_mig_active ()) { + info->has_disk = true; + info->disk =g_malloc0 (sizeof (*info->disk));Quoted 8 lines
migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391
Quoted 4 lines
migration/migration.c +4
-0+ info->disk->transferred =blk_mig_bytes_transferred (); + info->disk->remaining =blk_mig_bytes_remaining (); + info->disk->total =blk_mig_bytes_total (); + }Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?
I think you can, I've not tried it; however I also think it should be safe.
Quoted 7 lines
migration/block.c's
block_save_pendingalways puts a value in thenon_postcopiable_pendingreturn value (and 0 in thepostcopiable_pending); the migrate thread checks thenon_postcopiable_pendingsize to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.
I mean that as it is right now, the info under blk_mig_active() check would be zero/the same than before entering postcopy.
Quoted 6 lines
A possibly harder question is what happens if
block.cdid implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.
Quoted 6 lines
Quoted 2 lines
Once here, I guess we can get the
migrate_already_active()bit without problem?I'm not sure of the question here; but the idea of
migration_already_active()is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.
Sorry. I mean that the migrate_already_active() bits can get in without further ado. Don't need to wait for postcopy to be integrated.
Quoted 7 lines
Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 70 lines · * Juan Quintela ([email protected]) wrote:
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Juan Quintela <[email protected]>But (there is always a but....)
Quoted 30 lines
migration/migration.c +26
-0@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)get_xbzrle_cache_stats (info);break ; +case MIGRATION_STATUS_POSTCOPY_ACTIVE : +/* Mostly the same as active; TODO add some postcopy stats */ + info->has_status = true; + info->has_total_time = true; + info->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) + - s->total_time; + info->has_expected_downtime = true; + info->expected_downtime = s->expected_downtime; + info->has_setup_time = true; + info->setup_time = s->setup_time; + + info->has_ram = true; + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->transferred =ram_bytes_transferred (); + info->ram->remaining =ram_bytes_remaining (); + info->ram->total =ram_bytes_total (); + info->ram->duplicate =dup_mig_pages_transferred (); + info->ram->skipped =skipped_mig_pages_transferred (); + info->ram->normal =norm_mig_pages_transferred (); + info->ram->normal_bytes =norm_mig_bytes_transferred (); + info->ram->dirty_pages_rate = s->dirty_pages_rate; + info->ram->mbps = s->mbps; + +if (blk_mig_active ()) { + info->has_disk = true; + info->disk =g_malloc0 (sizeof (*info->disk));Quoted 8 lines
migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391
Quoted 4 lines
migration/migration.c +4
-0+ info->disk->transferred =blk_mig_bytes_transferred (); + info->disk->remaining =blk_mig_bytes_remaining (); + info->disk->total =blk_mig_bytes_total (); + }Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?
I think you can, I've not tried it; however I also think it should be safe.
Quoted 7 lines
migration/block.c's
block_save_pendingalways puts a value in thenon_postcopiable_pendingreturn value (and 0 in thepostcopiable_pending); the migrate thread checks thenon_postcopiable_pendingsize to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.
Quoted 2 lines
I mean that as it is right now, the info under
blk_mig_active()check would be zero/the same than before entering postcopy.
Ah, yes; would blk_mig_bytes_total/transferred still have valid values you would want to display, even at the end of the block migration phase?
Quoted 9 lines
Quoted 9 lines
A possibly harder question is what happens if
block.cdid implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.Quoted 2 lines
Once here, I guess we can get the
migrate_already_active()bit without problem?
Quoted 6 lines
Quoted 3 lines
I'm not sure of the question here; but the idea of
migration_already_active()is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.Sorry. I mean that the
migrate_already_active()bits can get in without further ado. Don't need to wait for postcopy to be integrated.
Yes; do you want it split out?
Dave
Quoted 8 lines
Quoted 8 lines
Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 86 lines · Dr. David Alan Gilbert
Quoted 70 lines · * Juan Quintela ([email protected]) wrote:
Quoted 61 lines · Dr. David Alan Gilbert
Quoted 48 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Juan Quintela <[email protected]>But (there is always a but....)
Quoted 30 lines
migration/migration.c +26
-0@@ -358,6 +359,39 @@ MigrationInfo *qmp_query_migrate(Error **errp)get_xbzrle_cache_stats (info);break ; +case MIGRATION_STATUS_POSTCOPY_ACTIVE : +/* Mostly the same as active; TODO add some postcopy stats */ + info->has_status = true; + info->has_total_time = true; + info->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) + - s->total_time; + info->has_expected_downtime = true; + info->expected_downtime = s->expected_downtime; + info->has_setup_time = true; + info->setup_time = s->setup_time; + + info->has_ram = true; + info->ram =g_malloc0 (sizeof (*info->ram)); + info->ram->transferred =ram_bytes_transferred (); + info->ram->remaining =ram_bytes_remaining (); + info->ram->total =ram_bytes_total (); + info->ram->duplicate =dup_mig_pages_transferred (); + info->ram->skipped =skipped_mig_pages_transferred (); + info->ram->normal =norm_mig_pages_transferred (); + info->ram->normal_bytes =norm_mig_bytes_transferred (); + info->ram->dirty_pages_rate = s->dirty_pages_rate; + info->ram->mbps = s->mbps; + +if (blk_mig_active ()) { + info->has_disk = true; + info->disk =g_malloc0 (sizeof (*info->disk));Quoted 8 lines
migration/migration.c MigrationInfo *qmp_query_migrate(Error **errp) · line 391
Quoted 4 lines
migration/migration.c +4
-0+ info->disk->transferred =blk_mig_bytes_transferred (); + info->disk->remaining =blk_mig_bytes_remaining (); + info->disk->total =blk_mig_bytes_total (); + }Can we have block migration active with postcopy? I would assume that this would get disk corruption, no? Or if you preffer the other question, what protects us from disk corruption?
I think you can, I've not tried it; however I also think it should be safe.
Quoted 7 lines
migration/block.c's
block_save_pendingalways puts a value in thenon_postcopiable_pendingreturn value (and 0 in thepostcopiable_pending); the migrate thread checks thenon_postcopiable_pendingsize to decide when it can switch to postcopy, and performs a call to the complete method on each device before it does. Thus the block migration should be finished before we start doing the actual postcopy stage, and thus before the destination CPU starts running.Quoted 2 lines
I mean that as it is right now, the info under
blk_mig_active()check would be zero/the same than before entering postcopy.Ah, yes; would blk_mig_bytes_total/transferred still have valid values you would want to display, even at the end of the block migration phase?
Quoted 9 lines
Quoted 9 lines
A possibly harder question is what happens if
block.cdid implement postcopy and you had both block postcopy and ram postcopy active at the same time; again I think it should work but I'm not sure if one would starve the other.Quoted 2 lines
Once here, I guess we can get the
migrate_already_active()bit without problem?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I'm not sure of the question here; but the idea of
migration_already_active()is just to avoid all of the open-coded checks for each possible state; now we've added anothe state they were getting messy.Sorry. I mean that the
migrate_already_active()bits can get in without further ado. Don't need to wait for postcopy to be integrated.Yes; do you want it split out?
Whatever is easier for you. I mean that it can be integrated independently of postcopy.
so, it is up to you.
Juan.
Quoted 13 lines
Dave
Quoted 8 lines
Quoted 8 lines
Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
In reply to Dr. David Alan Gilbert (git)migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 436
Quoted 3 lines · Dr. David Alan Gilbert (git)
migration/migration.c +1
-2-if (s->state ==MIGRATION_STATUS_ACTIVE || - s->state ==MIGRATION_STATUS_SETUP ) { +if (migration_already_active (s)) {
(I know, not introduced here, but:)
A better name is migration_is_active()
migration/migration.c bool migration_has_failed(MigrationState *s) · line 650
Quoted 4 lines
migration/migration.c +4
-0+bool migration_postcopy_phase (MigrationState *s) +{ +return (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ); +}
And this is better named migration_in_postcopy()
otherwise,
Amit
Re: [Qemu-devel] [PATCH v7 23/42] MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
Quoted 9 lines · * Amit Shah ([email protected]) wrote:
migration/migration.c void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, · line 436
Quoted 3 lines · Dr. David Alan Gilbert (git)
migration/migration.c +1
-2-if (s->state ==MIGRATION_STATUS_ACTIVE || - s->state ==MIGRATION_STATUS_SETUP ) { +if (migration_already_active (s)) {(I know, not introduced here, but:)
A better name is
migration_is_active()
Done.
Quoted 7 lines
migration/migration.c bool migration_has_failed(MigrationState *s) · line 650
Quoted 5 lines
migration/migration.c +4
-0+bool migration_postcopy_phase (MigrationState *s) +{ +return (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ); +}And this is better named
migration_in_postcopy()
Done
Quoted 8 lines
otherwise,
Reviewed-by Amit Shah <[email protected]>Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 24/42 Add qemu_savevm_state_complete_postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.
The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.
The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.
--- include/migration/vmstate.h | 1 + include/sysemu/sysemu.h | 1 + migration/ram.c | 1 + migration/savevm.c | 51 ++++++++++++++++++++++++++++++++++++++++++--- 4 files changed, 51 insertions(+), 3 deletions(-)
include/migration/vmstate.h +1 -0
@@ -40,6 +40,7 @@ typedef struct SaveVMHandlers {SaveStateHandler *save_state;void (*cancel)(void *opaque); +int (*save_live_complete_postcopy)(QEMUFile *f,void *opaque);int (*save_live_complete_precopy)(QEMUFile *f,void *opaque);/* This runs both outside and inside the iothread lock. */
include/sysemu/sysemu.h +1 -0
@@ -108,6 +108,7 @@ void qemu_savevm_state_begin(QEMUFile *f, const MigrationParams *params);void qemu_savevm_state_header (QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f); +void qemu_savevm_state_complete_postcopy (QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);void qemu_savevm_state_cancel (void ); void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
migration/ram.c +1 -0
@@ -1608,6 +1608,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate, + .save_live_complete_postcopy = ram_save_complete, .save_live_complete_precopy = ram_save_complete, .save_live_pending = ram_save_pending, .load_state = ram_load,
migration/savevm.c +48 -3
75 lines not on this page · raw message
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
qemu_savevm_state_complete_postcopyto complementqemu_savevm_state_complete_precopytogether with a newsave_live_complete_postcopymethod on devices.
Quoted 8 lines
The
save_live_complete_precopymethod is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.The
save_live_complete_postcopymethod is called at the end of postcopy for all postcopiable devices.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Quoted 6 lines
migration/savevm.c +1
-0@@ -947,13 +987,15 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)int vmdesc_len;SaveStateEntry *se;int ret; +bool in_postcopy =migration_postcopy_phase (migrate_get_current ());
migration/savevm.c void qemu_savevm_state_complete_precopy(QEMUFile *f) · line 1000
Quoted 10 lines
migration/savevm.c +2
-1trace_savevm_state_complete_precopy ();cpu_synchronize_all_states (); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { -if (!se->ops || !se->ops->save_live_complete_precopy) { +if (!se->ops || !se->ops->save_live_complete_precopy || + (in_postcopy && se->ops->save_live_complete_postcopy)) {continue ; }
I would change the formatting to something like:
if (!se->ops ||
(in_postcopy && se->ops->save_live_complete_postcopy)
!se->ops->save_live_complete_precopy) {
continue
}Just to make easier to see when we exit?
Later, Juan.
Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy
Quoted 11 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
qemu_savevm_state_complete_postcopyto complementqemu_savevm_state_complete_precopytogether with a newsave_live_complete_postcopymethod on devices.The
save_live_complete_precopymethod is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.
Quoted 6 lines
Quoted 4 lines
The
save_live_complete_postcopymethod is called at the end of postcopy for all postcopiable devices.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>
Thanks.
Quoted 20 lines
Quoted 7 lines
migration/savevm.c +1
-0@@ -947,13 +987,15 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)int vmdesc_len;SaveStateEntry *se;int ret; +bool in_postcopy =migration_postcopy_phase (migrate_get_current ());migration/savevm.c void qemu_savevm_state_complete_precopy(QEMUFile *f) · line 1000
Quoted 10 lines
migration/savevm.c +2
-1trace_savevm_state_complete_precopy ();cpu_synchronize_all_states (); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { -if (!se->ops || !se->ops->save_live_complete_precopy) { +if (!se->ops || !se->ops->save_live_complete_precopy || + (in_postcopy && se->ops->save_live_complete_postcopy)) {continue ; }I would change the formatting to something like:
Quoted 7 lines
if (!se->ops || (in_postcopy && se->ops->save_live_complete_postcopy) !se->ops->save_live_complete_precopy) { continue }Just to make easier to see when we exit?
Done
Dave (Starting with the easy fix first :-)
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
qemu_savevm_state_complete_postcopyto complementqemu_savevm_state_complete_precopytogether with a newsave_live_complete_postcopymethod on devices.
Quoted 8 lines
The
save_live_complete_precopymethod is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.The
save_live_complete_postcopymethod is called at the end of postcopy for all postcopiable devices.Signed-off-by Dr. David Alan Gilbert <[email protected]>
But:
migration/savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 943
Quoted 10 lines
migration/savevm.c +3
-1@@ -938,7 +938,47 @@ int qemu_savevm_state_iterate(QEMUFile *f)static bool should_send_vmdesc (void ) {MachineState *machine =MACHINE (qdev_get_machine ()); -return !machine->suppress_vmdesc; +bool in_postcopy =migration_postcopy_phase (migrate_get_current ()); +return !machine->suppress_vmdesc && !in_postcopy; +}
This should be split in its own patch.
Amit
Re: [Qemu-devel] [PATCH v7 24/42] Add qemu_savevm_state_complete_postcopy
Quoted 20 lines · * Amit Shah ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
qemu_savevm_state_complete_postcopyto complementqemu_savevm_state_complete_precopytogether with a newsave_live_complete_postcopymethod on devices.Quoted 8 lines
The
save_live_complete_precopymethod is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.The
save_live_complete_postcopymethod is called at the end of postcopy for all postcopiable devices.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>But:
Quoted 12 lines
migration/savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 943
Quoted 10 lines
migration/savevm.c +3
-1@@ -938,7 +938,47 @@ int qemu_savevm_state_iterate(QEMUFile *f)static bool should_send_vmdesc (void ) {MachineState *machine =MACHINE (qdev_get_machine ()); -return !machine->suppress_vmdesc; +bool in_postcopy =migration_postcopy_phase (migrate_get_current ()); +return !machine->suppress_vmdesc && !in_postcopy; +}This should be split in its own patch.
Thanks, split.
Dave
Quoted 3 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 25/42 Postcopy: Maintain sentmap and calculate discard
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.
--- include/migration/migration.h | 12 +++ include/migration/postcopy-ram.h | 35 +++++++ include/qemu/typedefs.h | 1 + migration/migration.c | 1 + migration/postcopy-ram.c | 108 +++++++++++++++++++++ migration/ram.c | 203 ++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 2 - trace-events | 5 + 8 files changed, 363 insertions(+), 4 deletions(-)
include/migration/migration.h +12 -0
@@ -114,6 +114,13 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; + +/* bitmap of pages that have been sent at least once +* only maintained and used in postcopy at the moment +* where it's used to send the dirtymap at the start +* of the postcopy phase +*/ +unsigned long *sentmap; };void process_incoming_migration (QEMUFile *f);
@@ -183,6 +190,11 @@ double xbzrle_mig_cache_miss_rate(void);void ram_handle_compressed (void *host,uint8_t ch,uint64_t size);void ram_debug_dump_bitmap (unsigned long *todump,bool expected); +/* For outgoing discard bitmap */ +int ram_postcopy_send_discard_bitmap (MigrationState *ms); +/* For incoming postcopy discard */ +int ram_discard_range (MigrationIncomingState *mis,const char *block_name, +uint64_t start,uint64_t end);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +35 -0
@@ -16,4 +16,39 @@/* Return true if the host supports everything we need to do postcopy-ram */ bool postcopy_ram_supported_by_host (void ); +/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end); + + +/* +* Called at the start of each RAMBlock by the bitmap code +* 'offset' is the bitmap offset of the named RAMBlock in the migration +* bitmap. +* Returns a new PDS +*/ +PostcopyDiscardState *postcopy_discard_send_init (MigrationState *ms, +unsigned long offset, +const char *name); + +/* +* Called by the bitmap code for each chunk to discard +* May send a discard message, may just leave it queued to +* be sent later +* 'start' and 'end' describe an inclusive range of pages in the +* migration bitmap in the RAM block passed to postcopy_discard_send_init +*/ +void postcopy_discard_send_range (MigrationState *ms,PostcopyDiscardState *pds, +unsigned long start,unsigned long end); + +/* +* Called at the end of each RAMBlock by the bitmap code +* Sends any outstanding discard messages, frees the PDS +*/ +void postcopy_discard_send_finish (MigrationState *ms, +PostcopyDiscardState *pds); +#endif
include/qemu/typedefs.h +1 -0
@@ -61,6 +61,7 @@ typedef struct PCIExpressHost PCIExpressHost;typedef struct PCIHostState PCIHostState ;typedef struct PCMCIACardState PCMCIACardState ;typedef struct PixelFormat PixelFormat ; +typedef struct PostcopyDiscardState PostcopyDiscardState ;typedef struct PropertyInfo PropertyInfo ;typedef struct Property Property ;typedef struct QEMUBH QEMUBH ;
migration/migration.c +1 -0
@@ -22,6 +22,7 @@#include "block/block.h" #include "qemu/sockets.h" #include "migration/block.h" +#include "migration/postcopy-ram.h" #include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h"
migration/postcopy-ram.c +108 -0
125 lines not on this page · raw message
migration/ram.c +201 -2
271 lines not on this page · raw message
migration/savevm.c +0 -2
@@ -1295,7 +1295,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis, }trace_loadvm_postcopy_ram_handle_discard_header (ramid, len);while (len) { -/* TODO - ram_discard_range gets added in a later patch uint64_t start_addr, end_addr; start_addr =qemu_get_be64 (mis->file); end_addr =qemu_get_be64 (mis->file);
@@ -1305,7 +1304,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,if (ret) {return ret; } -*/ }trace_loadvm_postcopy_ram_handle_discard_end ();
trace-events +5 -0
@@ -1231,6 +1231,7 @@ qemu_file_fclose(void) "" migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64"" migration_throttle(void) "" +ram_postcopy_send_discard_bitmap(void) "" # hw/display/qxl.c disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
@@ -1495,6 +1496,10 @@ rdma_start_incoming_migration_after_rdma_listen(void) "" rdma_start_outgoing_migration_after_rdma_connect(void) "" rdma_start_outgoing_migration_after_rdma_source_init(void) "" +# migration/postcopy-ram.c +postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" +postcopy_ram_discard_range(void *start, void *end) "%p,%p" + # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p" kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Quoted 5 lines
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Not a patch without a suggestion O:-)
Quoted 17 lines
--- include/migration/migration.h | 12 +++ include/migration/postcopy-ram.h | 35 +++++++ include/qemu/typedefs.h | 1 + migration/migration.c | 1 + migration/postcopy-ram.c | 108 +++++++++++++++++++++ migration/ram.c | 203 ++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 2 - trace-events | 5 + 8 files changed, 363 insertions(+), 4 deletions(-)include/migration/migration.h +0
-0@@ -114,6 +114,13 @@ struct MigrationState
include/migration/migration.h struct MigrationState · line 124
Quoted 10 lines
include/migration/migration.h +7
-0/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; + +/* bitmap of pages that have been sent at least once +* only maintained and used in postcopy at the moment +* where it's used to send the dirtymap at the start +* of the postcopy phase +*/ +unsigned long *sentmap; };
We can use this sentmap for zero page optimization. If page is on sentmap, we need to sent a zero page, otherwise, just sent sentmap at the end of migration and clean everything not there?
include/migration/migration.h struct MigrationState
Quoted 9 lines
include/migration/migration.h +9
-0+/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end) +{ +trace_postcopy_ram_discard_range (start, end); +if (madvise (start, (end-start)+1 ,MADV_DONTNEED )) {
Can we s/end/lenght/ and adjust everywhere?
Not here, but putting a comment explaining where magic 12 cames from on definition of constant?
I think that the sentbitmap bits could we used without the rest.
Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard
Quoted 33 lines · * Juan Quintela ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Quoted 5 lines
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>Not a patch without a suggestion O:-)
Quoted 17 lines
--- include/migration/migration.h | 12 +++ include/migration/postcopy-ram.h | 35 +++++++ include/qemu/typedefs.h | 1 + migration/migration.c | 1 + migration/postcopy-ram.c | 108 +++++++++++++++++++++ migration/ram.c | 203 ++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 2 - trace-events | 5 + 8 files changed, 363 insertions(+), 4 deletions(-)include/migration/migration.h +0
-0@@ -114,6 +114,13 @@ struct MigrationState
Quoted 14 lines
include/migration/migration.h struct MigrationState · line 124
Quoted 10 lines
include/migration/migration.h +7
-0/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; + +/* bitmap of pages that have been sent at least once +* only maintained and used in postcopy at the moment +* where it's used to send the dirtymap at the start +* of the postcopy phase +*/ +unsigned long *sentmap; };We can use this sentmap for zero page optimization. If page is on sentmap, we need to sent a zero page, otherwise, just sent sentmap at the end of migration and clean everything not there?
Just as a compact way of sending zero pages? I'm not sure it would help.
Quoted 11 lines
include/migration/migration.h struct MigrationState
Quoted 9 lines
include/migration/migration.h +9
-0+/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end) +{ +trace_postcopy_ram_discard_range (start, end); +if (madvise (start, (end-start)+1 ,MADV_DONTNEED )) {Can we s/end/lenght/ and adjust everywhere?
Done - partially; everything that works in bytes is now start & length, everything that works in indexes into the RAM bitmap is still start/end, since generally that's what they're working with already.
Quoted 2 lines
Not here, but putting a comment explaining where magic 12 cames from on definition of constant?
Done.
Quoted 1 line
I think that the sentbitmap bits could we used without the rest.
Possibly, I can kind of see it's useful - but I'm not convinced what else for yet.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Quoted 5 lines
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Some whitespace issues, and some sentences in comments don't have a full-stop:
include/migration/postcopy-ram.h line 45
Quoted 9 lines
include/migration/postcopy-ram.h +9
-0+/* +* Called by the bitmap code for each chunk to discard +* May send a discard message, may just leave it queued to +* be sent later +* 'start' and 'end' describe an inclusive range of pages in the +* migration bitmap in the RAM block passed to postcopy_discard_send_init +*/ +void postcopy_discard_send_range (MigrationState *ms,PostcopyDiscardState *pds, +unsigned long start,unsigned long end);
unaligned line; no full-stop in comment above (similar elsewhere, not repeating that).
migration/postcopy-ram.c out: · line 172
Quoted 9 lines
migration/postcopy-ram.c +9
-0+/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end) +{ +trace_postcopy_ram_discard_range (start, end); +if (madvise (start, (end-start)+1 ,MADV_DONTNEED )) {
whitespace around operators
Quoted 6 lines
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed topostcopy_discard_send_init
missing punctuation
(also, you had started doing doxygen-style comments, want to keep on following that style?)
Quoted 1 line
+static RAMBlock *ram_find_block(const char *id)
just a suggestion, not very particular about this: rename to
ram_find_block_by_id()
instead, so that it's clear what method of finding we're using; also no name conflicts when there might be other ways of doing a find.
Amit
Re: [Qemu-devel] [PATCH v7 25/42] Postcopy: Maintain sentmap and calculate discard
Quoted 19 lines · * Amit Shah ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Quoted 5 lines
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>Some whitespace issues, and some sentences in comments don't have a full-stop:
Quoted 12 lines
include/migration/postcopy-ram.h line 45
Quoted 9 lines
include/migration/postcopy-ram.h +9
-0+/* +* Called by the bitmap code for each chunk to discard +* May send a discard message, may just leave it queued to +* be sent later +* 'start' and 'end' describe an inclusive range of pages in the +* migration bitmap in the RAM block passed to postcopy_discard_send_init +*/ +void postcopy_discard_send_range (MigrationState *ms,PostcopyDiscardState *pds, +unsigned long start,unsigned long end);unaligned line; no full-stop in comment above (similar elsewhere, not repeating that).
Fixed.
Quoted 11 lines
migration/postcopy-ram.c out: · line 172
Quoted 9 lines
migration/postcopy-ram.c +9
-0+/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end) +{ +trace_postcopy_ram_discard_range (start, end); +if (madvise (start, (end-start)+1 ,MADV_DONTNEED )) {whitespace around operators
Fixed.
Quoted 11 lines
Quoted 6 lines
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ * 'start' and 'end' describe an inclusive range of pages in the
+ * migration bitmap in the RAM block passed topostcopy_discard_send_initmissing punctuation
(also, you had started doing doxygen-style comments, want to keep on following that style?)
Fixed (I'm sure there are others, still getting into the hang of that).
Quoted 8 lines
Quoted 1 line
+static RAMBlock *ram_find_block(const char *id)
just a suggestion, not very particular about this: rename to
ram_find_block_by_id()instead, so that it's clear what method of finding we're using; also no name conflicts when there might be other ways of doing a find.
Done.
Dave
Quoted 2 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v7 26/42 postcopy: Incoming initialisation
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- include/migration/migration.h | 3 + include/migration/postcopy-ram.h | 12 ++++ migration/postcopy-ram.c | 116 +++++++++++++++++++++++++++++++++++++++ migration/ram.c | 11 ++++ migration/savevm.c | 4 ++ trace-events | 2 + 6 files changed, 148 insertions(+)
include/migration/migration.h +3 -0
@@ -69,6 +69,8 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ PostcopyState postcopy_state;
@@ -195,6 +197,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms);/* For incoming postcopy discard */ int ram_discard_range (MigrationIncomingState *mis,const char *block_name,uint64_t start,uint64_t end); +int ram_postcopy_incoming_init (MigrationIncomingState *mis);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +12 -0
@@ -17,6 +17,18 @@bool postcopy_ram_supported_by_host (void );/* +* Initialise postcopy-ram, setting the RAM to a state where we can go into +* postcopy later; must be called prior to any precopy. +* called from ram.c's similarly named ram_postcopy_incoming_init +*/ +int postcopy_ram_incoming_init (MigrationIncomingState *mis,size_t ram_pages); + +/* +* At the end of a migration where postcopy_ram_incoming_init was called. +*/ +int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis); + +/* * Discard the contents of memory start..end inclusive. * We can assume that if we've been called postcopy_ram_hosttest returned true */
migration/postcopy-ram.c +116 -0
128 lines not on this page · raw message
migration/ram.c +11 -0
@@ -1667,6 +1667,17 @@ static void decompress_data_with_multi_threads(uint8_t *compbuf, } } +/* +* Allocate data structures etc needed by incoming migration with postcopy-ram +* postcopy-ram's similarly names postcopy_ram_incoming_init does the work +*/ +int ram_postcopy_incoming_init (MigrationIncomingState *mis) +{ +size_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +return postcopy_ram_incoming_init (mis, ram_pages); +} +static int ram_load (QEMUFile *f,void *opaque,int version_id) {int flags =0 , ret =0 ;
migration/savevm.c +4 -0
@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } +postcopy_state_set (mis,POSTCOPY_INCOMING_ADVISE );return 0 ;
trace-events +2 -0
@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) "" # migration/postcopy-ram.c postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" +postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_ram_discard_range(void *start, void *end) "%p,%p" +postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 26/42] postcopy: Incoming initialisation
Quoted 4 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Quoted 9 lines
--- include/migration/migration.h | 3 + include/migration/postcopy-ram.h | 12 ++++ migration/postcopy-ram.c | 116 +++++++++++++++++++++++++++++++++++++++ migration/ram.c | 11 ++++ migration/savevm.c | 4 ++ trace-events | 2 + 6 files changed, 148 insertions(+)
qemu_hugepage_enable(host_addr, length)?
migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 214
Quoted 6 lines
migration/postcopy-ram.c +6
-0+#ifdef MADV_NOHUGEPAGE +if (madvise (host_addr, length,MADV_NOHUGEPAGE )) { +error_report ("%s: NOHUGEPAGE: %s" , __func__,strerror (errno)); +return -1 ; + } +#endif
qemu_hugepage_disable(host_addr, length)?
migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 245
Quoted 12 lines
migration/postcopy-ram.c +12
-0+#ifdef MADV_HUGEPAGE +if (madvise (host_addr, length,MADV_HUGEPAGE )) { +error_report ("%s HUGEPAGE: %s" , __func__,strerror (errno)); +return -1 ; + } +#endif + +/* +* We can also turn off userfault now since we should have all the +* pages. It can be useful to leave it on to debug postcopy +* if you're not sure it's always getting every page. +*/
qemu_userfault_unregister(host_addr, length)?
Quoted 11 lines
(quoted diff) +9
-0+ range_struct.start = (uintptr_t)host_addr; + range_struct.len = length; + + if (ioctl(mis->userfault_fd, UFFDIO_UNREGISTER, &range_struct)) { + error_report("%s: userfault unregister %s", __func__, strerror(errno)); + + return -1; + } +/*
Review
Quoted 10 lines
(quoted diff) +10
-0+ * Allocate data structures etc needed by incoming migration with postcopy-ram + * postcopy-ram's similarly names postcopy_ram_incoming_init does the work + */ +int ram_postcopy_incoming_init(MigrationIncomingState *mis) +{ + size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS; + + return postcopy_ram_incoming_init(mis, ram_pages); +} +
ram_postocpy_incoming_init()
andpostcopy_ram_incoming_init()
ouch Thinking about better names ....
Quoted 8 lines
static int
ram_load(QEMUFile *f, void *opaque, int version_id){ int flags = 0, ret = 0;migration/savevm.c +0
-0
migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1244
Quoted 7 lines
migration/savevm.c +4
-0@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } +
how/where we know that this is called soon enough?
Quoted 8 lines
migration/savevm.c +0
-0postcopy_state_set (mis,POSTCOPY_INCOMING_ADVISE );return 0 ;trace-events +0
-0
trace-events rdma_start_outgoing_migration_after_rdma_source_init(void) "" · line 1503
Quoted 6 lines
trace-events +2
-0@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) "" # migration/postcopy-ram.c postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" +postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_ram_discard_range(void *start, void *end) "%p,%p" +postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"
once here, if we have range names before, what about:
postcopy_ram_cleanup_range()postcopy_ram_init_range()
And let the ram* functions the same?
Re: [Qemu-devel] [PATCH v7 26/42] postcopy: Incoming initialisation
Quoted 17 lines · * Juan Quintela ([email protected]) wrote:
Quoted 4 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Quoted 9 lines
--- include/migration/migration.h | 3 + include/migration/postcopy-ram.h | 12 ++++ migration/postcopy-ram.c | 116 +++++++++++++++++++++++++++++++++++++++ migration/ram.c | 11 ++++ migration/savevm.c | 4 ++ trace-events | 2 + 6 files changed, 148 insertions(+)
qemu_hugepage_enable(host_addr, length)?
Quoted 8 lines
migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 214
Quoted 6 lines
migration/postcopy-ram.c +6
-0+#ifdef MADV_NOHUGEPAGE +if (madvise (host_addr, length,MADV_NOHUGEPAGE )) { +error_report ("%s: NOHUGEPAGE: %s" , __func__,strerror (errno)); +return -1 ; + } +#endif
qemu_hugepage_disable(host_addr, length)?
I've flipped those both to use the qemu_madvise which is how it's done in most other places in the codebase (I added
the QEMU_MADV_NOHUGEPAGE since HUGEPAGE was there but not the opposite).
Quoted 6 lines
Quoted 6 lines
migration/postcopy-ram.c +6
-0+#ifdef MADV_HUGEPAGE +if (madvise (host_addr, length,MADV_HUGEPAGE )) { +error_report ("%s HUGEPAGE: %s" , __func__,strerror (errno)); +return -1 ; + } +#endif
Quoted 8 lines
migration/postcopy-ram.c int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, · line 245
Quoted 6 lines
migration/postcopy-ram.c +6
-0+ +/* +* We can also turn off userfault now since we should have all the +* pages. It can be useful to leave it on to debug postcopy +* if you're not sure it's always getting every page. +*/
qemu_userfault_unregister(host_addr, length)?
Is it worth wrapping that ioctl, when it's already in a function that has to do other calls around it?
Quoted 26 lines
Quoted 11 lines
(quoted diff) +9
-0+ range_struct.start = (uintptr_t)host_addr; + range_struct.len = length; + + if (ioctl(mis->userfault_fd, UFFDIO_UNREGISTER, &range_struct)) { + error_report("%s: userfault unregister %s", __func__, strerror(errno)); + + return -1; + } +/*Review
Quoted 10 lines
(quoted diff) +10
-0+ * Allocate data structures etc needed by incoming migration with postcopy-ram + * postcopy-ram's similarly names postcopy_ram_incoming_init does the work + */ +int ram_postcopy_incoming_init(MigrationIncomingState *mis) +{ + size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS; + + return postcopy_ram_incoming_init(mis, ram_pages); +} +
ram_postocpy_incoming_init()
andpostcopy_ram_incoming_init()
Quoted 1 line
ouch Thinking about better names ....
Agreed; suggestions welcome. the 'ram' and 'postcopy_ram' are both the convention based on the filename.
Quoted 11 lines
Quoted 11 lines
static int
ram_load(QEMUFile *f, void *opaque, int version_id){ int flags = 0, ret = 0;migration/savevm.c +0
-0@@ -1238,6 +1238,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis,return -1 ; }
Quoted 6 lines
migration/savevm.c static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis, · line 1244
Quoted 4 lines
migration/savevm.c +4
-0+if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } +how/where we know that this is called soon enough?
It's driven by the sequence of commands byte that walk it through the state machine.
Quoted 17 lines
Quoted 8 lines
migration/savevm.c +0
-0postcopy_state_set (mis,POSTCOPY_INCOMING_ADVISE );return 0 ;trace-events +0
-0trace-events rdma_start_outgoing_migration_after_rdma_source_init(void) "" · line 1503
Quoted 6 lines
trace-events +2
-0@@ -1498,7 +1498,9 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) "" # migration/postcopy-ram.c postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" +postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_ram_discard_range(void *start, void *end) "%p,%p" +postcopy_init_area(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx"once here, if we have range names before, what about:
Quoted 2 lines
postcopy_ram_cleanup_range()postcopy_ram_init_range()
Done.
Quoted 1 line
And let the ram* functions the same?
Not sure which ones those refer to?
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 26/42] postcopy: Incoming initialisation
In reply to Dr. David Alan Gilbert (git)Quoted 4 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Amit
PATCH v7 27/42 postcopy: ram_enable_notify to switch on userfault
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)
--- include/migration/migration.h | 3 ++ include/migration/postcopy-ram.h | 6 ++++ migration/postcopy-ram.c | 69 +++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 9 ++++++ 4 files changed, 86 insertions(+), 1 deletion(-)
include/migration/migration.h +3 -0
@@ -69,6 +69,9 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +QemuThread fault_thread; +QemuSemaphore fault_thread_sem; +/* For the kernel to send us notifications */ int userfault_fd;QEMUFile *return_path;
include/migration/postcopy-ram.h +6 -0
@@ -17,6 +17,12 @@bool postcopy_ram_supported_by_host (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from ram.c's similarly named ram_postcopy_incoming_init
migration/postcopy-ram.c +68 -1
83 lines not on this page · raw message
migration/savevm.c +9 -0
@@ -1324,6 +1324,15 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)return -1 ; } +/* +* Sensitise RAM - can now generate requests for blocks that don't exist +* However, at this point the CPU shouldn't be running, and the IO +* shouldn't be doing anything yet so don't actually expect requests +*/ +if (postcopy_ram_enable_notify (mis)) { +return -1 ; + } +/* TODO start up the postcopy listening thread */ return 0 ; }
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault
Quoted 20 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>--- include/migration/migration.h | 3 ++ include/migration/postcopy-ram.h | 6 ++++ migration/postcopy-ram.c | 69 +++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 9 ++++++ 4 files changed, 86 insertions(+), 1 deletion(-)include/migration/migration.h +0
-0
include/migration/migration.h struct MigrationIncomingState { · line 72
Quoted 4 lines
include/migration/migration.h +1
-0@@ -69,6 +69,9 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +QemuThread fault_thread;
Best name ever? Well, it could be "wrong_thread" O:-)
include/migration/migration.h struct MigrationIncomingState { · line 75
Quoted 3 lines
include/migration/migration.h +3
-0+static void *postcopy_ram_fault_thread (void *opaque) +{ +MigrationIncomingState *mis = (MigrationIncomingState *)opaque;
Uneeded cast.
Quoted 21 lines
include/migration/migration.h +21
-0+ +fprintf (stderr,"postcopy_ram_fault_thread\n" ); +/* TODO: In later patch */ +qemu_sem_post (&mis->fault_thread_sem); +while (1 ) { +/* TODO: In later patch */ + } + +return NULL ; +} + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +/* Create the fault handler thread and wait for it to be ready */ +qemu_sem_init (&mis->fault_thread_sem,0 ); +qemu_thread_create (&mis->fault_thread,"postcopy/fault" , + postcopy_ram_fault_thread, mis,QEMU_THREAD_JOINABLE ); +qemu_sem_wait (&mis->fault_thread_sem); +qemu_sem_destroy (&mis->fault_thread_sem); + +/* Mark so that we get notified of accesses to unwritten areas */
include/migration/migration.h struct MigrationIncomingState {
Quoted 10 lines
include/migration/migration.h +7
-1+if (qemu_ram_foreach_block (ram_block_enable_notify, mis)) { +return -1 ; + } + + return 0; +} +#else /* No target OS support, stubs just fail */ -
This belongs in a different patch O:-)
If you have to resend, just change them.
Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault
Quoted 24 lines · * Juan Quintela ([email protected]) wrote:
Quoted 23 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>--- include/migration/migration.h | 3 ++ include/migration/postcopy-ram.h | 6 ++++ migration/postcopy-ram.c | 69 +++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 9 ++++++ 4 files changed, 86 insertions(+), 1 deletion(-)include/migration/migration.h +0
-0@@ -69,6 +69,9 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event;
Quoted 3 lines
include/migration/migration.h struct MigrationIncomingState { · line 72
Quoted 1 line
include/migration/migration.h +1
-0+QemuThread fault_thread;Best name ever? Well, it could be "
wrong_thread" O:-)
Well they are 'user faults'! Not that we blame the user for
them personally of course.
Quoted 5 lines
include/migration/migration.h struct MigrationIncomingState { · line 75
Quoted 3 lines
include/migration/migration.h +3
-0+static void *postcopy_ram_fault_thread (void *opaque) +{ +MigrationIncomingState *mis = (MigrationIncomingState *)opaque;Uneeded cast.
Thanks, gone.
Quoted 25 lines
Quoted 25 lines
include/migration/migration.h +25
-0+ +fprintf (stderr,"postcopy_ram_fault_thread\n" ); +/* TODO: In later patch */ +qemu_sem_post (&mis->fault_thread_sem); +while (1 ) { +/* TODO: In later patch */ + } + +return NULL ; +} + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +/* Create the fault handler thread and wait for it to be ready */ +qemu_sem_init (&mis->fault_thread_sem,0 ); +qemu_thread_create (&mis->fault_thread,"postcopy/fault" , + postcopy_ram_fault_thread, mis,QEMU_THREAD_JOINABLE ); +qemu_sem_wait (&mis->fault_thread_sem); +qemu_sem_destroy (&mis->fault_thread_sem); + +/* Mark so that we get notified of accesses to unwritten areas */ +if (qemu_ram_foreach_block (ram_block_enable_notify, mis)) { +return -1 ; + } +
Quoted 8 lines
include/migration/migration.h
Quoted 6 lines
include/migration/migration.h +3
-1+ return 0; +} +#else /* No target OS support, stubs just fail */ -This belongs in a different patch O:-)
Thanks, gone.
Quoted 3 lines
If you have to resend, just change them.
Reviewed-by Juan Quintela <[email protected]>
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 27/42] postcopy: ram_enable_notify to switch on userfault
In reply to Dr. David Alan Gilbert (git)Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>
Amit
PATCH v7 28/42 Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
--- include/migration/migration.h | 3 + migration/migration.c | 166 ++++++++++++++++++++++++++++++++++++++++-- trace-events | 4 + 3 files changed, 167 insertions(+), 6 deletions(-)
include/migration/migration.h +3 -0
@@ -120,6 +120,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; +/* Flag set once the migration thread is running (and needs joining) */ +bool started_migration_thread; +/* bitmap of pages that have been sent at least once * only maintained and used in postcopy at the moment * where it's used to send the dirtymap at the start
migration/migration.c +160 -6
239 lines not on this page · raw message
trace-events +4 -0
@@ -1422,9 +1422,13 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migration_thread_after_loop(void) "" +migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" +postcopy_start(void) "" +postcopy_start_set_run(void) "" source_return_path_thread_bad_end(void) "" source_return_path_thread_end(void) "" source_return_path_thread_entry(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration/migration.c | 166 ++++++++++++++++++++++++++++++++++++++++-- trace-events | 4 + 3 files changed, 167 insertions(+), 6 deletions(-)include/migration/migration.h +0
-0
include/migration/migration.h struct MigrationState · line 125
Quoted 6 lines
include/migration/migration.h +3
-0@@ -120,6 +120,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; +/* Flag set once the migration thread is running (and needs joining) */ +bool started_migration_thread; +
migration_thread_started?
include/migration/migration.h struct MigrationState
Quoted 7 lines
include/migration/migration.h +7
-0+ +/* +* send rest of state - note things that are doing postcopy +* will notice we're in POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX );
Do we undo this? or, are we sure that it is ok to maximize network output?
include/migration/migration.h
Quoted 2 lines
include/migration/migration.h +2
-0+/* Ping just for debugging, helps line traces up */ +qemu_savevm_send_ping (ms->file,2 );
Change the values 1, 2, 3 to constants?
Quoted 13 lines
include/migration/migration.h +13
-0+* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ +QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); + goto fail; + } + +qemu_savevm_state_complete_precopy (fb); +qemu_savevm_send_ping (fb,3 ); +
include/migration/migration.h
Quoted 10 lines
include/migration/migration.h +10
-0+qemu_savevm_send_postcopy_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ +if (qemu_savevm_send_packaged (ms->file, qsb)) { + goto fail_closefb; + } +qemu_fclose (fb);
Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?
Quoted 1 line
+
ms->downtime=qemu_clock_get_ms(QEMU_CLOCK_REALTIME)-time_at_stop;
Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.
Quoted 7 lines
include/migration/migration.h +1
-0+/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ static void *migration_thread (void *opaque) {MigrationState *s = opaque;
include/migration/migration.h
Quoted 10 lines
include/migration/migration.h +4
-0+/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time;bool old_vm_running = false; +bool entered_postcopy = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationStatus current_active_type =MIGRATION_STATUS_ACTIVE ;
current_active_state?
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
Quoted 20 lines · * Juan Quintela ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration/migration.c | 166 ++++++++++++++++++++++++++++++++++++++++-- trace-events | 4 + 3 files changed, 167 insertions(+), 6 deletions(-)include/migration/migration.h +0
-0@@ -120,6 +120,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy;
Quoted 5 lines
include/migration/migration.h struct MigrationState · line 125
Quoted 3 lines
include/migration/migration.h +3
-0+/* Flag set once the migration thread is running (and needs joining) */ +bool started_migration_thread; +
migration_thread_started?
Changed.
Quoted 10 lines
include/migration/migration.h struct MigrationState
Quoted 7 lines
include/migration/migration.h +7
-0+ +/* +* send rest of state - note things that are doing postcopy +* will notice we're in POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX );Do we undo this? or, are we sure that it is ok to maximize network output?
No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.
Quoted 4 lines
include/migration/migration.h
Quoted 2 lines
include/migration/migration.h +2
-0+/* Ping just for debugging, helps line traces up */ +qemu_savevm_send_ping (ms->file,2 );Change the values 1, 2, 3 to constants?
Suggestions to names? - they purely for debugging so you can match it up on the destination.
Quoted 19 lines
Quoted 19 lines
include/migration/migration.h +19
-0+* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ +QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); + goto fail; + } + +qemu_savevm_state_complete_precopy (fb); +qemu_savevm_send_ping (fb,3 ); + +qemu_savevm_send_postcopy_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */
Quoted 8 lines
include/migration/migration.h
Quoted 4 lines
include/migration/migration.h +4
-0+if (qemu_savevm_send_packaged (ms->file, qsb)) { + goto fail_closefb; + } +qemu_fclose (fb);Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?
The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.
Quoted 4 lines
Quoted 1 line
+
ms->downtime=qemu_clock_get_ms(QEMU_CLOCK_REALTIME)-time_at_stop;Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.
No, not separate.
Quoted 18 lines
Quoted 7 lines
include/migration/migration.h +1
-0+/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ static void *migration_thread (void *opaque) {MigrationState *s = opaque;include/migration/migration.h
Quoted 10 lines
include/migration/migration.h +4
-0+/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time;bool old_vm_running = false; +bool entered_postcopy = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationStatus current_active_type =MIGRATION_STATUS_ACTIVE ;
Quoted 1 line
current_active_state?
Changed.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 10 lines
migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1105
Quoted 7 lines
migration/migration.c +7
-0+ +/* +* send rest of state - note things that are doing postcopy +* will notice we're in POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX );Do we undo this? or, are we sure that it is ok to maximize network output?
Quoted 8 lines
No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.
I see.
Quoted 37 lines
Quoted 5 lines
migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1107
Quoted 3 lines
migration/migration.c +2
-0+/* Ping just for debugging, helps line traces up */ +qemu_savevm_send_ping (ms->file,2 );Change the values 1, 2, 3 to constants?
Suggestions to names? - they purely for debugging so you can match it up on the destination.
Quoted 19 lines
Quoted 19 lines
migration/migration.c +19
-0+* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ +QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); + goto fail; + } + +qemu_savevm_state_complete_precopy (fb); +qemu_savevm_send_ping (fb,3 ); + +qemu_savevm_send_postcopy_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ Quoted 8 lines
migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1132
Quoted 4 lines
migration/migration.c +4
-0+if (qemu_savevm_send_packaged (ms->file, qsb)) { + goto fail_closefb; + } +qemu_fclose (fb);Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?
Quoted 7 lines
The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.
Arghhh. ok. Comment?
Quoted 34 lines
Quoted 5 lines
Quoted 2 lines
+
ms->downtime=qemu_clock_get_ms(QEMU_CLOCK_REALTIME)-time_at_stop;Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.
No, not separate.
Quoted 18 lines
Quoted 7 lines
migration/migration.c +1
-0+/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ static void *migration_thread (void *opaque) {MigrationState *s = opaque;migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1177
Quoted 10 lines
migration/migration.c +4
-0+/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time;bool old_vm_running = false; +bool entered_postcopy = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationStatus current_active_type =MIGRATION_STATUS_ACTIVE ;Quoted 1 line
current_active_state?Changed.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
Quoted 38 lines · * Juan Quintela ([email protected]) wrote:
Quoted 28 lines · Dr. David Alan Gilbert
Quoted 19 lines
Quoted 19 lines
migration/migration.c +19
-0+* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ +QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); + goto fail; + } + +qemu_savevm_state_complete_precopy (fb); +qemu_savevm_send_ping (fb,3 ); + +qemu_savevm_send_postcopy_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ Quoted 8 lines
migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1132
Quoted 4 lines
migration/migration.c +4
-0+if (qemu_savevm_send_packaged (ms->file, qsb)) { + goto fail_closefb; + } +qemu_fclose (fb);Why can't we send this directly without the extra copy?
I guess that there are some missing/extra section starts/end whatever? Anything specific?Quoted 7 lines
The problem is that the destination has to be able to read the chunk of migration stream off the fd and leave the fd free for page requests that may be required during loading the device state.
Since the migration-stream is unstructured, there is no way to read a chunk of stream off without knowing the length of that chunk, and the only way to know that chunk is to write it to a buffer and then see how big it is.
Quoted 1 line
Arghhh. ok. Comment?
I've changed the comment at the start of that section to:
* While loading the device state we may trigger page transfer
* requests and the fd must be free to process those, and thus
* the destination must read the whole device state off the fd before
* it starts processing it. Unfortunately the ad-hoc migration format
* doesn't allow the destination to know the size to read without fully
* parsing it through each devices load-state code (especially the open
* coded devices that use get/put).
* So we wrap the device state up in a package with a length at the start;
* to do this we use a qemu_buf to hold the whole of the device state.
Dave
Quoted 27 lines
Quoted 27 lines
Quoted 6 lines
Quoted 3 lines
+
ms->downtime=qemu_clock_get_ms(QEMU_CLOCK_REALTIME)-time_at_stop;Now, that we are here, is there a counter of the time that takes the postcopy stage? Just curious.
No, not separate.
Quoted 18 lines
Quoted 17 lines
migration/migration.c +5
-0+/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ static void *migration_thread (void *opaque) {MigrationState *s = opaque; +/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time;bool old_vm_running = false; +bool entered_postcopy = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationStatus current_active_type =MIGRATION_STATUS_ACTIVE ;
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
current_active_state?Changed.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan GilbertQuoted 12 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Juan Quintela ([email protected]) wrote:
migration/migration.c static int await_return_path_close_on_source(MigrationState *ms) · line 1105
Quoted 6 lines
migration/migration.c +6
-0+/* +* send rest of state - note things that are doing postcopy +* will notice we're in POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX );Do we undo this? or, are we sure that it is ok to maximize network output?
Quoted 8 lines
No we don't undo it; it's a good question what we can do better. I'm trying to avoid delaying the postcopy-requested pages; ideally I'd like to separate those out so they get satisfied but still meet the bandwidth limit for the background transfer. The ideal is separate fd's, however something else I've considered is getting incoming postcopy requests to wake the outgoing side up when it's sleeping for the bandwidth limit, although I've not tried implementing that yet.
Might be a conflict in the knobs we expose (max_bandwidth) and us not adhering to that.
I agree we want this to go full-throttle, so maybe document that postcopy will override that knob? It's tricky to get everyone to understand that postcopy will do that. Plus there'll be other questions like what else does postcopy override? -- not that there's anythign more, but users will wonder.
Amit
Re: [Qemu-devel] [PATCH v7 28/42] Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan Gilbert (git)Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 29/42 Postcopy end in migration_thread
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
The end of migration code is getting a bit complciated now, so move out into its own function.
--- migration/migration.c | 91 +++++++++++++++++++++++++++++++++++++-------------- trace-events | 6 ++++ 2 files changed, 72 insertions(+), 25 deletions(-)
migration/migration.c +66 -25
@@ -1041,7 +1041,6 @@ static int open_return_path_on_source(MigrationState *ms)return 0 ; } -__attribute__ (( unused ))/* Until later in patch series */ /* Returns 0 if the RP was ok, otherwise there was an error on the RP */ static int await_return_path_close_on_source (MigrationState *ms) {
@@ -1159,6 +1158,68 @@ fail: }/* +* Used by migration_thread when there's not much left pending. +* The caller 'breaks' the loop when this returns. +*/ +static void migration_thread_end_of_iteration (MigrationState *s, +int current_active_state, +bool *old_vm_running, +int64_t *start_time) +{ +int ret; +if (s->state ==MIGRATION_STATUS_ACTIVE ) { +qemu_mutex_lock_iothread (); + *start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + *old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); +if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete_precopy (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { + goto fail; + } + }else if (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ) { +trace_migration_thread_end_of_iteration_postcopy_end (); + +qemu_savevm_state_complete_postcopy (s->file); +trace_migration_thread_end_of_iteration_postcopy_end_after_complete (); + } + +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up (since if there are no failures +* it will wait for the destination to send it's status in +* a SHUT command). +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +int rp_error; +trace_migration_thread_end_of_iteration_postcopy_end_before_rp (); + rp_error =await_return_path_close_on_source (s); +trace_migration_thread_end_of_iteration_postcopy_end_after_rp (rp_error); +if (rp_error) { + goto fail; + } + } + +if (qemu_file_get_error (s->file)) { +trace_migration_thread_end_of_iteration_file_err (); + goto fail; + } + +migrate_set_state (s, current_active_state,MIGRATION_STATUS_COMPLETED ); +return ; + +fail: +migrate_set_state (s, current_active_state,MIGRATION_STATUS_FAILED ); +} + +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */
@@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else { -int ret; - -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); - - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete_precopy (s->file); - } -qemu_mutex_unlock_iothread (); +trace_migration_thread_low_pending (pending_size); -if (ret <0 ) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_FAILED ); -break ; - } - -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_COMPLETED ); -break ; - } +migration_thread_end_of_iteration (s, current_active_type, + &old_vm_running, &start_time); +break ; } }
trace-events +6 -0
@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" +migration_thread_low_pending(uint64_t pending) "%" PRIu64 +migration_thread_end_of_iteration_file_err(void) "" +migration_thread_end_of_iteration_postcopy_end(void) "" +migration_thread_end_of_iteration_postcopy_end_after_complete(void) "" +migration_thread_end_of_iteration_postcopy_end_before_rp(void) "" +migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" postcopy_start(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
The end of migration code is getting a bit complciated now, so move out into its own function.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
I think that I would splint the function and then add the postcopy code.
BTW, it is a local function, we can use shorter names:
migration_completion()?
trace names specifically get hugggggggggge.
migration/migration.c fail: · line 1165
Quoted 2 lines
migration/migration.c +2
-0+static void migration_thread_end_of_iteration(MigrationState *s, + int current_active_state,
RunState?
And it is not needed as parameter.
migration/migration.c fail: · line 1170
Quoted 5 lines
migration/migration.c +5
-0+bool *old_vm_running, + int64_t *start_time) +{ + int ret; +if (s->state ==MIGRATION_STATUS_ACTIVE ) {
current_active_state = s->state;
Quoted 6 lines
migration/migration.c +6
-0+qemu_mutex_lock_iothread (); + *start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + *old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE );
migration/migration.c fail: · line 1186
Quoted 10 lines
migration/migration.c +10
-0+if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete_precopy (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { + goto fail; + } + }else if (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ) {
current_active_state = s->state;
Quoted 92 lines
migration/migration.c +40
-24+trace_migration_thread_end_of_iteration_postcopy_end (); + +qemu_savevm_state_complete_postcopy (s->file); +trace_migration_thread_end_of_iteration_postcopy_end_after_complete (); + } + +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up (since if there are no failures +* it will wait for the destination to send it's status in +* a SHUT command). +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +int rp_error; +trace_migration_thread_end_of_iteration_postcopy_end_before_rp (); + rp_error =await_return_path_close_on_source (s); +trace_migration_thread_end_of_iteration_postcopy_end_after_rp (rp_error); +if (rp_error) { + goto fail; + } + } + +if (qemu_file_get_error (s->file)) { +trace_migration_thread_end_of_iteration_file_err (); + goto fail; + } + +migrate_set_state (s, current_active_state,MIGRATION_STATUS_COMPLETED ); +return ; + +fail: +migrate_set_state (s, current_active_state,MIGRATION_STATUS_FAILED ); +} + +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ @@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else { -int ret; - -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); - - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete_precopy (s->file); - } -qemu_mutex_unlock_iothread (); +trace_migration_thread_low_pending (pending_size); -if (ret <0 ) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_FAILED ); -break ; - } - -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_COMPLETED ); -break ; - } +migration_thread_end_of_iteration (s, current_active_type, + &old_vm_running, &start_time); +break ; } }trace-events +6
-0@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" +migration_thread_low_pending(uint64_t pending) "%" PRIu64 +migration_thread_end_of_iteration_file_err(void) "" +migration_thread_end_of_iteration_postcopy_end(void) "" +migration_thread_end_of_iteration_postcopy_end_after_complete(void) "" +migration_thread_end_of_iteration_postcopy_end_before_rp(void) "" +migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" postcopy_start(void) ""
Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread
Quoted 7 lines · Juan Quintela
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
Quoted 6 lines
Quoted 4 lines
The end of migration code is getting a bit complciated now, so move out into its own function.
Signed-off-by Dr. David Alan Gilbert <[email protected]>I think that I would splint the function and then add the postcopy code.
Yeah, esp since this code was added / modified in the previous patch.
Amit
Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread
In reply to Juan QuintelaQuoted 7 lines · * Juan Quintela ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
Quoted 6 lines
Quoted 4 lines
The end of migration code is getting a bit complciated now, so move out into its own function.
Signed-off-by Dr. David Alan Gilbert <[email protected]>I think that I would splint the function and then add the postcopy code.
Done; I now have two patches:
Split out end of migration code from migration_thread
Postcopy: End of iteration
Quoted 5 lines
BTW, it is a local function, we can use shorter names:
migration_completion()?trace names specifically get hugggggggggge.
Done.
Quoted 5 lines
migration/migration.c fail: · line 1165
Quoted 2 lines
migration/migration.c +2
-0+static void migration_thread_end_of_iteration(MigrationState *s, + int current_active_state,RunState?
And it is not needed as parameter.
No, it's not RunState, it's derived from s->state which is still an int; it's also not the current state, but the current state we're expecting to be in, i.e. one of MIGRATION_STATUS_ACTIVE or MIGRATION_STATE_POSTCOPY_ACTIVE (which is why it's current_*active*_state); and it's only used as the parameter to migrate_set_state - in the same way the current code does:
migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COMPLETED);to ensure that any failure or cancel occuring at the same time isn't lost.
Dave
Quoted 109 lines
migration/migration.c fail: · line 1170
Quoted 7 lines
migration/migration.c +5
-0+bool *old_vm_running, + int64_t *start_time) +{ + int ret; +if (s->state ==MIGRATION_STATUS_ACTIVE ) {current_active_state = s->state;Quoted 6 lines
migration/migration.c +6
-0+qemu_mutex_lock_iothread (); + *start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + *old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE );migration/migration.c fail: · line 1186
Quoted 10 lines
migration/migration.c +10
-0+if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete_precopy (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { + goto fail; + } + }else if (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ) {current_active_state = s->state;Quoted 84 lines
migration/migration.c +40
-24+trace_migration_thread_end_of_iteration_postcopy_end (); + +qemu_savevm_state_complete_postcopy (s->file); +trace_migration_thread_end_of_iteration_postcopy_end_after_complete (); + } + +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up (since if there are no failures +* it will wait for the destination to send it's status in +* a SHUT command). +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +int rp_error; +trace_migration_thread_end_of_iteration_postcopy_end_before_rp (); + rp_error =await_return_path_close_on_source (s); +trace_migration_thread_end_of_iteration_postcopy_end_after_rp (rp_error); +if (rp_error) { + goto fail; + } + } + +if (qemu_file_get_error (s->file)) { +trace_migration_thread_end_of_iteration_file_err (); + goto fail; + } + +migrate_set_state (s, current_active_state,MIGRATION_STATUS_COMPLETED ); +return ; + +fail: +migrate_set_state (s, current_active_state,MIGRATION_STATUS_FAILED ); +} + +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. */ @@ -1233,31 +1294,11 @@ static void *migration_thread(void *opaque)/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else { -int ret; - -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); - - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete_precopy (s->file); - } -qemu_mutex_unlock_iothread (); +trace_migration_thread_low_pending (pending_size); -if (ret <0 ) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_FAILED ); -break ; - } - -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIGRATION_STATUS_ACTIVE , -MIGRATION_STATUS_COMPLETED ); -break ; - } +migration_thread_end_of_iteration (s, current_active_type, + &old_vm_running, &start_time); +break ; } }trace-events +1
-0@@ -1425,6 +1425,12 @@ migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" +migration_thread_low_pending(uint64_t pending) "%" PRIu64
Quoted 8 lines
trace-events migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" · line 1436
Quoted 8 lines
trace-events +5
-0+migration_thread_end_of_iteration_file_err(void) "" +migration_thread_end_of_iteration_postcopy_end(void) "" +migration_thread_end_of_iteration_postcopy_end_after_complete(void) "" +migration_thread_end_of_iteration_postcopy_end_before_rp(void) "" +migration_thread_end_of_iteration_postcopy_end_after_rp(int rp_error) "%d" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" postcopy_start(void) ""
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 29/42] Postcopy end in migration_thread
In reply to Dr. David Alan Gilbert (git)Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
The end of migration code is getting a bit complciated now, so move out into its own function.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Amit
PATCH v7 30/42 Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.
--- include/migration/migration.h | 4 +++ migration/migration.c | 70 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 3 files changed, 75 insertions(+)
include/migration/migration.h +4 -0
@@ -47,6 +47,8 @@ enum mig_rp_message_type {MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ + + MIG_RP_MSG_REQ_PAGES, /* data (start: be64, len: be64) */ }; typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
@@ -246,6 +248,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, uint32_t value);void migrate_send_rp_pong (MigrationIncomingState *mis,uint32_t value); +void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,ram_addr_t len);void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);
migration/migration.c +70 -0
100 lines not on this page · raw message
trace-events +1 -0
@@ -1422,6 +1422,7 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at %zx len %zx" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) ""
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
Quoted 17 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RP_MSG_REQ_PAGEScommand on Return path for the postcopy destination to request a page from the source.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 4 +++ migration/migration.c | 70 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 3 files changed, 75 insertions(+)include/migration/migration.h +0
-0
include/migration/migration.h enum mig_rp_message_type { · line 51
Quoted 5 lines
include/migration/migration.h +2
-0@@ -47,6 +47,8 @@ enum mig_rp_message_type {MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ + + MIG_RP_MSG_REQ_PAGES, /* data (start: be64, len: be64) */
Not that I really care, buht I think that leng could be 32bits. I am not seing networking getting good at multigigabytes transfers soon O:-)
include/migration/migration.h void migrate_send_rp_shut(MigrationIncomingState *mis, · line 252
Quoted 2 lines
include/migration/migration.h +2
-0+void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,ram_addr_t len);
Shouldn't len be a size_t?
(yes, I know that migration code is not really consistent about that)
Quoted 15 lines
include/migration/migration.h +0
-0void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);migration/migration.c +5
-0@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp) deferred_incoming = true; } +/* Request a range of pages from the source VM at the given +* start address. +* rbname: Name of the RAMBlock to request the page in, if NULL it's the same +* as the last request (a name must have been given previously) +* Start: Address offset within the RB
migration/migration.c static void deferred_incoming_migration(Error **errp) · line 130
Quoted 10 lines
migration/migration.c +10
-0+* Len: Length in bytes required - must be a multiple of pagesize +*/ +void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char *rbname, +ram_addr_t start,ram_addr_t len) +{ +uint8_t bufc[16 +1 +255 ];/* start (8 byte), len (8 byte), rbname upto 256 */ +uint64_t *buf64 = (uint64_t *)bufc; +size_t msglen =16 ;/* start + len */ + +assert (!(len &1 ));
ohhhh, why can't we get a real flags field?
Scratch that. Seeing the rest of the code, can't we have two commands:
MIG_RP_MSG_REQ_PAGESMIG_RP_MSG_REQ_PAGES_WITH_ID
I am not really sure that it makes sense getting a command that can be of two different lengths only for that?
I am not sure, but having a command with two different payloads look strange.
Later, Juan.
Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
Quoted 13 lines · * Juan Quintela ([email protected]) wrote:
Quoted 12 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RP_MSG_REQ_PAGEScommand on Return path for the postcopy destination to request a page from the source.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 4 +++ migration/migration.c | 70 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 3 files changed, 75 insertions(+)
Quoted 13 lines
include/migration/migration.h enum mig_rp_message_type { · line 51
Quoted 10 lines
include/migration/migration.h +2
-0@@ -47,6 +47,8 @@ enum mig_rp_message_type {MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ + + MIG_RP_MSG_REQ_PAGES, /* data (start: be64, len: be64) */Not that I really care, buht I think that leng could be 32bits. I am not seing networking getting good at multigigabytes transfers soon O:-)
Done.
Quoted 5 lines
include/migration/migration.h void migrate_send_rp_shut(MigrationIncomingState *mis, · line 252
Quoted 2 lines
include/migration/migration.h +2
-0+void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,ram_addr_t len);Shouldn't len be a
size_t?
(yes, I know that migration code is not really consistent about that)
Done (fun combination with the change above, but still)
Quoted 30 lines
Quoted 15 lines
include/migration/migration.h +0
-0void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);migration/migration.c +5
-0@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp) deferred_incoming = true; } +/* Request a range of pages from the source VM at the given +* start address. +* rbname: Name of the RAMBlock to request the page in, if NULL it's the same +* as the last request (a name must have been given previously) +* Start: Address offset within the RB migration/migration.c static void deferred_incoming_migration(Error **errp) · line 130
Quoted 10 lines
migration/migration.c +10
-0+* Len: Length in bytes required - must be a multiple of pagesize +*/ +void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char *rbname, +ram_addr_t start,ram_addr_t len) +{ +uint8_t bufc[16 +1 +255 ];/* start (8 byte), len (8 byte), rbname upto 256 */ +uint64_t *buf64 = (uint64_t *)bufc; +size_t msglen =16 ;/* start + len */ + +assert (!(len &1 ));ohhhh, why can't we get a real flags field?
Scratch that. Seeing the rest of the code, can't we have two commands:
Quoted 8 lines
MIG_RP_MSG_REQ_PAGESMIG_RP_MSG_REQ_PAGES_WITH_IDI am not really sure that it makes sense getting a command that can be of two different lengths only for that?
I am not sure, but having a command with two different payloads look strange.
Done (I made it _ID rather than WITH_ID - it was getting a bit long).
Dave
Quoted 2 lines
Later, Juan.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
In reply to Dr. David Alan Gilbert (git)Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RP_MSG_REQ_PAGEScommand on Return path for the postcopy destination to request a page from the source.Signed-off-by Dr. David Alan Gilbert <[email protected]>
migration/migration.c static void deferred_incoming_migration(Error **errp) · line 119
Quoted 7 lines
migration/migration.c +4
-0@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp) deferred_incoming = true; } +/* Request a range of pages from the source VM at the given +* start address. +* rbname: Name of the RAMBlock to request the page in, if NULL it's the same +* as the last request (a name must have been given previously)
Why not just send the name all the time?
Quoted 5 lines
migration/migration.c +1
-0@@ -1010,6 +1058,28 @@ static void *source_return_path_thread(void *opaque)trace_source_return_path_thread_pong (tmp32);break ; +case MIG_RP_MSG_REQ_PAGES :
migration/migration.c static void *source_return_path_thread(void *opaque) · line 1071
Quoted 10 lines
migration/migration.c +10
-0+ start =be64_to_cpup ((uint64_t *)buf); + len =be64_to_cpup (((uint64_t *)buf)+1 ); + tmpstr =NULL ; +if (len &1 ) { + len -=1 ;/* Remove the flag */ +/* Now we expect an idstr */ + tmp32 = buf[16 ];/* Length of the following idstr */ + tmpstr = (char *)&buf[17 ]; + buf[17 +tmp32] ='\0' ; + expected_len =16 +1 +tmp32;
Whitespace missing around operators
migration/migration.c static void *source_return_path_thread(void *opaque) · line 1074
Quoted 3 lines
migration/migration.c +3
-0+ }else { + expected_len =16 ; + }
This else can be removed if expected_len is set before the if
Amit
Re: [Qemu-devel] [PATCH v7 30/42] Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
Quoted 14 lines · * Amit Shah ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RP_MSG_REQ_PAGEScommand on Return path for the postcopy destination to request a page from the source.Signed-off-by Dr. David Alan Gilbert <[email protected]>migration/migration.c +0
-0@@ -113,6 +113,36 @@ static void deferred_incoming_migration(Error **errp) deferred_incoming = true; }
Quoted 6 lines
migration/migration.c static void deferred_incoming_migration(Error **errp) · line 119
Quoted 4 lines
migration/migration.c +4
-0+/* Request a range of pages from the source VM at the given +* start address. +* rbname: Name of the RAMBlock to request the page in, if NULL it's the same +* as the last request (a name must have been given previously) Why not just send the name all the time?
It does shrink the messages quite a bit, and you have to do a name lookup on them when you receive it, rather than just knowing it's the same one.
Quoted 4 lines
Quoted 4 lines
migration/migration.c +0
-0@@ -1010,6 +1058,28 @@ static void *source_return_path_thread(void *opaque)trace_source_return_path_thread_pong (tmp32);break ;
Quoted 13 lines
migration/migration.c static void *source_return_path_thread(void *opaque) · line 1071
Quoted 11 lines
migration/migration.c +11
-0+case MIG_RP_MSG_REQ_PAGES : + start =be64_to_cpup ((uint64_t *)buf); + len =be64_to_cpup (((uint64_t *)buf)+1 ); + tmpstr =NULL ; +if (len &1 ) { + len -=1 ;/* Remove the flag */ +/* Now we expect an idstr */ + tmp32 = buf[16 ];/* Length of the following idstr */ + tmpstr = (char *)&buf[17 ]; + buf[17 +tmp32] ='\0' ; + expected_len =16 +1 +tmp32;Whitespace missing around operators
Done.
Quoted 5 lines
migration/migration.c static void *source_return_path_thread(void *opaque) · line 1074
Quoted 3 lines
migration/migration.c +3
-0+ }else { + expected_len =16 ; + }This else can be removed if
expected_lenis set before the if
That's simplified out with the change Juan suggested.
Dave
Quoted 2 lines
Amit
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK