Postcopy implementation
v4 cover Postcopy implementation
From: "Dr. David Alan Gilbert" <[email protected]>
Hi,
This is the 4th cut of my version of postcopy; it is designed for use with
the Linux kernel additions just posted by Andrea Arcangeli here:
http://marc.info/?l=linux-kernel&m=141235633015100&w=2
(Note: This is a new version compared to my previous postcopy patchset; you'll need to update the kernel to the new version.)
Other than the new kernel ABI (which is only a small change to the userspace side); the major changes are;
a) Code for host page size != target page size
b) Support for migration over fd
From Cristian Klein; this is for libvirt support which Cristian recently posted to the libvirt list.
c) It's now build bisectable and builds on 32bit
Testing wise; I've now done many thousand of postcopy migrations without failure (both of idle and busy guests); so it seems pretty solid.
Must-TODO's:
1) A partially repeatable migration_cancel failure
2) virt_test's migrate.with_reboot test is failing
3) The ACPI fix in 2.1 that allowed migrating RAMBlocks to be larger than the source feels like it needs looking at for postcopy.
4) Paolo's comments with respect to the wakeup_request/is_running code in the migration thread
5) xbzrle needs disabling once in postcopy
Later-TODO's:
1) Control the rate of background page transfers during postcopy to reduce their impact on the latency of postcopy requests.
2) Work with RDMA
3) Could destination RP be made blocking (as per discussion with Paolo; I'm still worried that that changes too many assumptions)
V4:
Initial support for host page size != target page size
- tested heavily on hps==tps
- only partially tested on hps!=tps systems
- This involved quite a bit of rework around the discard code
Updated to new kernel userfault ABI
- It won't work with the previous version
Fix mis-optimisation of postcopy request for wrong RAMBlock
request for block A offset n
un-needed fault for block B/m (already received - no req sent) request for block B/l - wrongly sent as request for A/l
Fix thinko in discard bitmap processing (missed last word of bitmap)
Symptom: remap failures near the top of RAM if postcopy started late
Fix bug that caused kernel page acknowledgments to be misaligned
May have meant the guest was paused for longer than required
Fix potential for crashing cleaning up failed RP
Fixes in docs (from Yang)
Handle migration by fd as sockets if they are sockets
Build tested on 32bit
Fully build bisectable (x86-64)
Dave
Cristian Klein (1):
Handle bi-directional communication for fd migration
Dr. David Alan Gilbert (46):
QEMUSizedBuffer based QEMUFile
Tests: QEMUSizedBuffer/QEMUBuffer
Start documenting how postcopy works.
qemu_ram_foreach_block: pass up error value, and down the ramblock
name
improve DPRINTF macros, add to savevm
Add qemu_get_counted_string to read a string prefixed by a count byte Create MigrationIncomingState
socket shutdown
Provide runtime Target page information
Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source Return path: Source handling of return path
qemu_loadvm errors and debug
ram_debug_dump_bitmap: Dump a migration bitmap as text Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
migrate_init: Call from savevm
Allow savevm handlers to state whether they could go into postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
MIG_STATE_POSTCOPY_ACTIVE: Add new migration state qemu_savevm_state_complete: Postcopy changes
Postcopy page-map-incoming (PMI) structure
Postcopy: Maintain sentmap and calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: Create a fault handler thread before marking the ram as
userfault
Page request: Add MIG_RPCOMM_REQPAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue
Add assertion to check migration_dirty_pages postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration qemu_ram_block_from_host
Don't sync dirty bitmaps in postcopy
Host page!=target page: Cleanup bitmaps
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_ram_handle_{run,end} commands End of migration for postcopy
Makefile.objs | 2 +- arch_init.c | 739 +++++++++++++++++++++++++-- docs/migration.txt | 189 +++++++ exec.c | 76 ++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + include/exec/cpu-common.h | 8 +- include/migration/migration.h | 130 +++++ include/migration/postcopy-ram.h | 106 ++++ include/migration/qemu-file.h | 47 ++ include/migration/vmstate.h | 2 +- include/qemu/sockets.h | 1 + include/qemu/typedefs.h | 9 +- include/sysemu/sysemu.h | 43 +- migration-fd.c | 24 +- migration-rdma.c | 4 +- migration.c | 693 +++++++++++++++++++++++++- postcopy-ram.c | 1016 ++++++++++++++++++++++++++++++++++++++ qapi-schema.json | 14 +- qemu-file.c | 598 +++++++++++++++++++++- qmp-commands.hx | 19 + savevm.c | 881 +++++++++++++++++++++++++++++++-- tests/Makefile | 2 +- tests/test-vmstate.c | 74 +-- util/qemu-sockets.c | 28 ++ 26 files changed, 4550 insertions(+), 178 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 postcopy-ram.c
Signature · 1 line
1.9.3
PATCH v4 1/47 QEMUSizedBuffer based QEMUFile
From: "Dr. David Alan Gilbert" <[email protected]>
* Please comment on separate thread for this QEMUSizedBuffer patch *
This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:
http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html
Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.
For fixes/tweeks I've done:
--- include/migration/qemu-file.h | 28 +++ include/qemu/typedefs.h | 1 + qemu-file.c | 456 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 485 insertions(+)
include/migration/qemu-file.h +28 -0
@@ -25,6 +25,8 @@#define QEMU_FILE_H 1#include "exec/cpu-common.h" +#include <stdint.h> +/* This function writes a chunk of data to a file at the given position. * The pos argument can be ignored if the file is only being used for * streaming. The handler should try to write all of the data it can.
@@ -94,11 +96,21 @@ typedef struct QEMUFileOps {QEMURamSaveFunc *save_page; } QEMUFileOps; +struct QEMUSizedBuffer { +struct iovec *iov; +size_t n_iov; +size_t size;/* total allocated size in all iov's */ +size_t used;/* number of used bytes */ +}; + +typedef struct QEMUSizedBuffer QEMUSizedBuffer ; +QEMUFile *qemu_fopen_ops (void *opaque,const QEMUFileOps *ops);QEMUFile *qemu_fopen (const char *filename,const char *mode);QEMUFile *qemu_fdopen (int fd,const char *mode);QEMUFile *qemu_fopen_socket (int fd,const char *mode);QEMUFile *qemu_popen_cmd (const char *command,const char *mode); +QEMUFile *qemu_bufopen (const char *mode,QEMUSizedBuffer *input);int qemu_get_fd (QEMUFile *f);int qemu_fclose (QEMUFile *f);int64_t qemu_ftell (QEMUFile *f);
@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);void qemu_put_buffer_async (QEMUFile *f,const uint8_t *buf,int size);bool qemu_file_mode_is_not_valid (const char *mode); +QEMUSizedBuffer *qsb_create (const uint8_t *buffer,size_t len); +QEMUSizedBuffer *qsb_clone (const QEMUSizedBuffer *); +void qsb_free (QEMUSizedBuffer *); +size_t qsb_set_length (QEMUSizedBuffer *qsb,size_t length); +size_t qsb_get_length (const QEMUSizedBuffer *qsb); +ssize_t qsb_get_buffer (const QEMUSizedBuffer *,off_t start,size_t count, +uint8_t *buf); +ssize_t qsb_write_at (QEMUSizedBuffer *qsb,const uint8_t *buf, +off_t pos,size_t count); + + +/* +* For use on files opened with qemu_bufopen +*/ +const QEMUSizedBuffer *qemu_buf_get (QEMUFile *f); +static inline void qemu_put_ubyte (QEMUFile *f,unsigned int v) {qemu_put_byte (f, (int )v);
include/qemu/typedefs.h +1 -0
@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;typedef struct PCIEAERErr PCIEAERErr ;typedef struct PCIEPort PCIEPort ;typedef struct PCIESlot PCIESlot ; +typedef struct QEMUSizedBuffer QEMUSizedBuffer ;typedef struct MSIMessage MSIMessage ;typedef struct SerialState SerialState ;typedef struct PCMCIACardState PCMCIACardState ;
qemu-file.c +456 -0
459 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 01/47] QEMUSizedBuffer based QEMUFile
Quoted 41 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
* Please comment on separate thread for this QEMUSizedBuffer patch *
This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:
http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html
Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.
Signed-off-by Stefan Berger <[email protected]>Signed-off-by Joel Schopp <[email protected]>For fixes/tweeks I've done:
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>--- include/migration/qemu-file.h | 28 +++ include/qemu/typedefs.h | 1 + qemu-file.c | 456 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 485 insertions(+)include/migration/qemu-file.h +2
-0@@ -25,6 +25,8 @@#define QEMU_FILE_H 1#include "exec/cpu-common.h" +#include <stdint.h> +/* This function writes a chunk of data to a file at the given position. * The pos argument can be ignored if the file is only being used for * streaming. The handler should try to write all of the data it can. @@ -94,11 +96,21 @@ typedef struct QEMUFileOps {QEMURamSaveFunc *save_page; } QEMUFileOps;
include/migration/qemu-file.h typedef struct QEMUFileOps { · line 107
Quoted 10 lines
include/migration/qemu-file.h +9
-0+struct QEMUSizedBuffer { +struct iovec *iov; +size_t n_iov; +size_t size;/* total allocated size in all iov's */ +size_t used;/* number of used bytes */ +}; + +typedef struct QEMUSizedBuffer QEMUSizedBuffer ; +
There is a redefinition of typedef ‘QEMUSizedBuffer’ in 'include/qemu/typedefs.h:68', when i compile qemu, it complains;)
Quoted 37 lines
include/migration/qemu-file.h +17
-0QEMUFile *qemu_fopen_ops (void *opaque,const QEMUFileOps *ops);QEMUFile *qemu_fopen (const char *filename,const char *mode);QEMUFile *qemu_fdopen (int fd,const char *mode);QEMUFile *qemu_fopen_socket (int fd,const char *mode);QEMUFile *qemu_popen_cmd (const char *command,const char *mode); +QEMUFile *qemu_bufopen (const char *mode,QEMUSizedBuffer *input);int qemu_get_fd (QEMUFile *f);int qemu_fclose (QEMUFile *f);int64_t qemu_ftell (QEMUFile *f);@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);void qemu_put_buffer_async (QEMUFile *f,const uint8_t *buf,int size);bool qemu_file_mode_is_not_valid (const char *mode); +QEMUSizedBuffer *qsb_create (const uint8_t *buffer,size_t len); +QEMUSizedBuffer *qsb_clone (const QEMUSizedBuffer *); +void qsb_free (QEMUSizedBuffer *); +size_t qsb_set_length (QEMUSizedBuffer *qsb,size_t length); +size_t qsb_get_length (const QEMUSizedBuffer *qsb); +ssize_t qsb_get_buffer (const QEMUSizedBuffer *,off_t start,size_t count, +uint8_t *buf); +ssize_t qsb_write_at (QEMUSizedBuffer *qsb,const uint8_t *buf, +off_t pos,size_t count); + + +/* +* For use on files opened with qemu_bufopen +*/ +const QEMUSizedBuffer *qemu_buf_get (QEMUFile *f); +static inline void qemu_put_ubyte (QEMUFile *f,unsigned int v) {qemu_put_byte (f, (int )v);include/qemu/typedefs.h +0
-0
include/qemu/typedefs.h typedef struct PCIEAERLog PCIEAERLog; · line 63
Quoted 4 lines
include/qemu/typedefs.h +1
-0@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;typedef struct PCIEAERErr PCIEAERErr ;typedef struct PCIEPort PCIEPort ;typedef struct PCIESlot PCIESlot ; +typedef struct QEMUSizedBuffer QEMUSizedBuffer ;
Here!, see above comment. Thanks.
Quoted 468 lines
include/qemu/typedefs.h +0
-0typedef struct MSIMessage MSIMessage ;typedef struct SerialState SerialState ;typedef struct PCMCIACardState PCMCIACardState ;qemu-file.c +456
-0@@ -878,3 +878,459 @@ uint64_t qemu_get_be64(QEMUFile *f) v |=qemu_get_be32 (f);return v; } + +#define QSB_CHUNK_SIZE (1 << 10) +#define QSB_MAX_CHUNK_SIZE (16 * QSB_CHUNK_SIZE) + +/** +* Create a QEMUSizedBuffer +* This type of buffer uses scatter-gather lists internally and +* can grow to any size. Any data array in the scatter-gather list +* can hold different amount of bytes. +* +* @buffer: Optional buffer to copy into the QSB +* @len: size of initial buffer; if @buffer is given, buffer must +* hold at least len bytes +* +* Returns a pointer to a QEMUSizedBuffer or NULL on allocation failure +*/ +QEMUSizedBuffer *qsb_create (const uint8_t *buffer,size_t len) +{ +QEMUSizedBuffer *qsb; +size_t alloc_len, num_chunks, i, to_copy; +size_t chunk_size = (len >QSB_MAX_CHUNK_SIZE ) + ?QSB_MAX_CHUNK_SIZE + :QSB_CHUNK_SIZE ; + + num_chunks =DIV_ROUND_UP (len ? len :QSB_CHUNK_SIZE , chunk_size); + alloc_len = num_chunks * chunk_size; + + qsb =g_try_new0 (QEMUSizedBuffer,1 ); +if (!qsb) { +return NULL ; + } + + qsb->iov = g_try_new0(struct iovec, num_chunks); +if (!qsb->iov) { +g_free (qsb); +return NULL ; + } + + qsb->n_iov = num_chunks; + +for (i =0 ; i < num_chunks; i++) { + qsb->iov[i].iov_base =g_try_malloc0 (chunk_size); +if (!qsb->iov[i].iov_base) { +/* qsb_free is safe since g_free can cope with NULL */ +qsb_free (qsb); +return NULL ; + } + + qsb->iov[i].iov_len = chunk_size; +if (buffer) { + to_copy = (len - qsb->used) > chunk_size + ? chunk_size : (len - qsb->used); +memcpy (qsb->iov[i].iov_base, &buffer[qsb->used], to_copy); + qsb->used += to_copy; + } + } + + qsb->size = alloc_len; + +return qsb; +} + +/** +* Free the QEMUSizedBuffer +* +* @qsb: The QEMUSizedBuffer to free +*/ +void qsb_free (QEMUSizedBuffer *qsb) +{ +size_t i; + +if (!qsb) { +return ; + } + +for (i =0 ; i < qsb->n_iov; i++) { +g_free (qsb->iov[i].iov_base); + } +g_free (qsb->iov); +g_free (qsb); +} + +/** +* Get the number of used bytes in the QEMUSizedBuffer +* +* @qsb: A QEMUSizedBuffer +* +* Returns the number of bytes currently used in this buffer +*/ +size_t qsb_get_length (const QEMUSizedBuffer *qsb) +{ +return qsb->used; +} + +/** +* Set the length of the buffer; the primary usage of this +* function is to truncate the number of used bytes in the buffer. +* The size will not be extended beyond the current number of +* allocated bytes in the QEMUSizedBuffer. +* +* @qsb: A QEMUSizedBuffer +* @new_len: The new length of bytes in the buffer +* +* Returns the number of bytes the buffer was truncated or extended +* to. +*/ +size_t qsb_set_length (QEMUSizedBuffer *qsb,size_t new_len) +{ +if (new_len <= qsb->size) { + qsb->used = new_len; + }else { + qsb->used = qsb->size; + } +return qsb->used; +} + +/** +* Get the iovec that holds the data for a given position @pos. +* +* @qsb: A QEMUSizedBuffer +* @pos: The index of a byte in the buffer +* @d_off: Pointer to an offset that this function will indicate +* at what position within the returned iovec the byte +* is to be found +* +* Returns the index of the iovec that holds the byte at the given +* index @pos in the byte stream; a negative number if the iovec +* for the given position @pos does not exist. +*/ +static ssize_t qsb_get_iovec (const QEMUSizedBuffer *qsb, +off_t pos,off_t *d_off) +{ +ssize_t i; +off_t curr =0 ; + +if (pos > qsb->used) { +return -1 ; + } + +for (i =0 ; i < qsb->n_iov; i++) { +if (curr + qsb->iov[i].iov_len > pos) { + *d_off = pos - curr; +return i; + } + curr += qsb->iov[i].iov_len; + } +return -1 ; +} + +/* +* Convert the QEMUSizedBuffer into a flat buffer. +* +* Note: If at all possible, try to avoid this function since it +* may unnecessarily copy memory around. +* +* @qsb: pointer to QEMUSizedBuffer +* @start: offset to start at +* @count: number of bytes to copy +* @buf: a pointer to a buffer to write into (at least @count bytes) +* +* Returns the number of bytes copied into the output buffer +*/ +ssize_t qsb_get_buffer (const QEMUSizedBuffer *qsb,off_t start, +size_t count,uint8_t *buffer) +{ +const struct iovec *iov; +size_t to_copy, all_copy; +ssize_t index; +off_t s_off; +off_t d_off =0 ; +char *s; + +if (start > qsb->used) { +return 0 ; + } + + all_copy = qsb->used - start; +if (all_copy > count) { + all_copy = count; + }else { + count = all_copy; + } + + index =qsb_get_iovec (qsb, start, &s_off); +if (index <0 ) { +return 0 ; + } + +while (all_copy >0 ) { + iov = &qsb->iov[index]; + + s = iov->iov_base; + + to_copy = iov->iov_len - s_off; +if (to_copy > all_copy) { + to_copy = all_copy; + } +memcpy (&buffer[d_off], &s[s_off], to_copy); + + d_off += to_copy; + all_copy -= to_copy; + + s_off =0 ; + index++; + } + +return count; +} + +/** +* Grow the QEMUSizedBuffer to the given size and allocate +* memory for it. +* +* @qsb: A QEMUSizedBuffer +* @new_size: The new size of the buffer +* +* Return: +* a negative error code in case of memory allocation failure +* or +* the new size of the buffer. The returned size may be greater or equal +* to @new_size. +*/ +static ssize_t qsb_grow (QEMUSizedBuffer *qsb,size_t new_size) +{ +size_t needed_chunks, i; + +if (qsb->size < new_size) { +struct iovec *new_iov; +size_t size_diff = new_size - qsb->size; +size_t chunk_size = (size_diff >QSB_MAX_CHUNK_SIZE ) + ?QSB_MAX_CHUNK_SIZE :QSB_CHUNK_SIZE ; + + needed_chunks =DIV_ROUND_UP (size_diff, chunk_size); + + new_iov =g_try_malloc_n (qsb->n_iov + needed_chunks, +sizeof (struct iovec )); +if (new_iov ==NULL ) { +return -ENOMEM ; + } + +/* Allocate new chunks as needed into new_iov */ +for (i = qsb->n_iov; i < qsb->n_iov + needed_chunks; i++) { + new_iov[i].iov_base =g_try_malloc0 (chunk_size); + new_iov[i].iov_len = chunk_size; +if (!new_iov[i].iov_base) { +size_t j; + +/* Free previously allocated new chunks */ +for (j = qsb->n_iov; j < i; j++) { +g_free (new_iov[j].iov_base); + } +g_free (new_iov); + +return -ENOMEM ; + } + } + +/* +* Now we can't get any allocation errors, copy over to new iov +* and switch. +*/ +for (i =0 ; i < qsb->n_iov; i++) { + new_iov[i] = qsb->iov[i]; + } + + qsb->n_iov += needed_chunks; +g_free (qsb->iov); + qsb->iov = new_iov; + qsb->size += (needed_chunks * chunk_size); + } + +return qsb->size; +} + +/** +* Write into the QEMUSizedBuffer at a given position and a given +* number of bytes. This function will automatically grow the +* QEMUSizedBuffer. +* +* @qsb: A QEMUSizedBuffer +* @source: A byte array to copy data from +* @pos: The position within the @qsb to write data to +* @size: The number of bytes to copy into the @qsb +* +* Returns @size or a negative error code in case of memory allocation failure, +* or with an invalid 'pos' +*/ +ssize_t qsb_write_at (QEMUSizedBuffer *qsb,const uint8_t *source, +off_t pos,size_t count) +{ +ssize_t rc =qsb_grow (qsb, pos + count); +size_t to_copy; +size_t all_copy = count; +const struct iovec *iov; +ssize_t index; +char *dest; +off_t d_off, s_off =0 ; + +if (rc <0 ) { +return rc; + } + +if (pos + count > qsb->used) { + qsb->used = pos + count; + } + + index =qsb_get_iovec (qsb, pos, &d_off); +if (index <0 ) { +return -EINVAL ; + } + +while (all_copy >0 ) { + iov = &qsb->iov[index]; + + dest = iov->iov_base; + + to_copy = iov->iov_len - d_off; +if (to_copy > all_copy) { + to_copy = all_copy; + } + +memcpy (&dest[d_off], &source[s_off], to_copy); + + s_off += to_copy; + all_copy -= to_copy; + + d_off =0 ; + index++; + } + +return count; +} + +/** +* Create a deep copy of the given QEMUSizedBuffer. +* +* @qsb: A QEMUSizedBuffer +* +* Returns a clone of @qsb or NULL on allocation failure +*/ +QEMUSizedBuffer *qsb_clone (const QEMUSizedBuffer *qsb) +{ +QEMUSizedBuffer *out =qsb_create (NULL ,qsb_get_length (qsb)); +size_t i; +ssize_t res; +off_t pos =0 ; + +if (!out) { +return NULL ; + } + +for (i =0 ; i < qsb->n_iov; i++) { + res =qsb_write_at (out, qsb->iov[i].iov_base, + pos, qsb->iov[i].iov_len); +if (res <0 ) { +qsb_free (out); +return NULL ; + } + pos += res; + } + +return out; +} + +typedef struct QEMUBuffer { +QEMUSizedBuffer *qsb; +QEMUFile *file; +}QEMUBuffer ; + +static int buf_get_buffer (void *opaque,uint8_t *buf,int64_t pos,int size) +{ +QEMUBuffer *s = opaque; +ssize_t len =qsb_get_length (s->qsb) - pos; + +if (len <=0 ) { +return 0 ; + } + +if (len > size) { + len = size; + } +return qsb_get_buffer (s->qsb, pos, len, buf); +} + +static int buf_put_buffer (void *opaque,const uint8_t *buf, +int64_t pos,int size) +{ +QEMUBuffer *s = opaque; + +return qsb_write_at (s->qsb, buf, pos, size); +} + +static int buf_close (void *opaque) +{ +QEMUBuffer *s = opaque; + +qsb_free (s->qsb); + +g_free (s); + +return 0 ; +} + +const QEMUSizedBuffer *qemu_buf_get (QEMUFile *f) +{ +QEMUBuffer *p; + +qemu_fflush (f); + + p = f->opaque; + +return p->qsb; +} + +static const QEMUFileOps buf_read_ops = { + .get_buffer = buf_get_buffer, + .close = buf_close, +}; + +static const QEMUFileOps buf_write_ops = { + .put_buffer = buf_put_buffer, + .close = buf_close, +}; + +QEMUFile *qemu_bufopen (const char *mode,QEMUSizedBuffer *input) +{ +QEMUBuffer *s; + +if (mode ==NULL || (mode[0 ] !='r' && mode[0 ] !='w' ) || + mode[1 ] !='\0' ) { +error_report ("qemu_bufopen: Argument validity check failed" ); +return NULL ; + } + + s =g_malloc0 (sizeof (QEMUBuffer)); +if (mode[0 ] =='r' ) { + s->qsb = input; + } + +if (s->qsb ==NULL ) { + s->qsb =qsb_create (NULL ,0 ); + } +if (!s->qsb) { +g_free (s); +error_report ("qemu_bufopen: qsb_create failed" ); +return NULL ; + } + + +if (mode[0 ] =='r' ) { + s->file =qemu_fopen_ops (s, &buf_read_ops); + }else { + s->file =qemu_fopen_ops (s, &buf_write_ops); + } +return s->file; +}
Re: [Qemu-devel] [PATCH v4 01/47] QEMUSizedBuffer based QEMUFile
In reply to Dr. David Alan Gilbert (git)Quoted 12 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
* Please comment on separate thread for this QEMUSizedBuffer patch *
This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:
http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html
Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.
Quoted 7 lines
Signed-off-by Stefan Berger <[email protected]>Signed-off-by Joel Schopp <[email protected]>For fixes/tweeks I've done:
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 2/47 Tests: QEMUSizedBuffer/QEMUBuffer
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
* Please comment on separate thread for this QEMUSziedBuffer patch *
Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.
Only some of the tests are changed so that the fd backed QEMUFile is still tested.
--- tests/Makefile | 2 +- tests/test-vmstate.c | 74 +++++++++++++++++++++++++++------------------------- 2 files changed, 39 insertions(+), 37 deletions(-)
tests/Makefile +1 -1
@@ -260,7 +260,7 @@ tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \ libqemuutil.a libqemustub.a tests/test-vmstate$(EXESUF): tests/test-vmstate.o \ vmstate.o qemu-file.o \ - libqemuutil.a + libqemuutil.a libqemustub.a tests/test-qapi-types.c tests/test-qapi-types.h :\ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py
tests/test-vmstate.c +38 -36
143 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 02/47] Tests: QEMUSizedBuffer/QEMUBuffer
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
* Please comment on separate thread for this QEMUSziedBuffer patch *
Modify some of
tests/test-vmstate.cto use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.
Quoted 5 lines
Only some of the tests are changed so that the fd backed QEMUFile is still tested.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>
Looks ok, although I do wonder if the removal of some of the test paths for regular files is a concern.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 3/47 Start documenting how postcopy works.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- docs/migration.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+)
docs/migration.txt +189 -0
@@ -294,3 +294,192 @@ save/send this state when we are in the middle of a pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. + += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the Postcopy destination +needs to be able to request pages on demand from the source. + +For these scenarios there is a 'return path' from the destination to the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return +path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread AND postcopy + thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to converge; +its plus side is that there is an upper bound on the amount of migration traffic +and time it takes, the down side is that during the postcopy phase, a failure of +*either* side or the network connection causes the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU. + +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy (prior to the start of migration): + +migrate_set_capability x-postcopy-ram on + +The migration will still start in precopy mode, however issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy ram listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy ram run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_ram_state) +from ADVISE->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + (Triggered by reception of POSTCOPY_RAM_ADVISE command) + + Listen: The first command in the package, POSTCOPY_RAM_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RAM_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'sent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'sent map' is used for the transition to postcopy. It is a bitmap that +has a bit set whenever a page is sent to the destination, however during +the transition to postcopy mode it is masked against the migration bitmap +(sentmap &= migrationbitmap) to generate a bitmap recording pages that +have been previously been sent but are now dirty again. This masked +sentmap is sent to the destination which discards those now dirty pages +before starting the CPUs. + +Note that once in postcopy mode, the sent map is still updated; however, +its contents are not necessarily consistent with the pages already sent +due to the masking with the migration bitmap. + +=== Destination side page maps === + +(Needs to be changed so we can update both easily - at the moment updates are done + with a lock) +The destination keeps a 'requested map' and a 'received map'. +Both maps are initially 0, as pages are received the bits are set in 'received map'. +Incoming requests from the kernel cause the bit to be set in the 'requested map'. +When a page is received that is marked as 'requested' the kernel is notified. +If the kernel requests a page that has already been 'received' the kernel is notified +without re-requesting. + +This leads to three valid page states: +page states: + missing (!rc,!rq) - page not yet received or requested + received (rc,!rq) - Page received + requested (!rc,rq) - page requested but not yet received + +state transitions: + received -> missing (only during setup/discard) + + missing -> received (normal incoming page) + requested -> received (incoming page previously requested) + missing -> requested (userfault request) +
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 03/47] Start documenting how postcopy works.
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 4/47 qemu_ram_foreach_block: pass up error value, and down the ramblock name
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.
Pass the name of the ramblock to the function; helps in debugging.
--- exec.c | 10 ++++++++-- include/exec/cpu-common.h | 4 ++-- migration-rdma.c | 4 ++-- 3 files changed, 12 insertions(+), 6 deletions(-)
exec.c +8 -2
@@ -2867,12 +2867,18 @@ bool cpu_physical_memory_is_io(hwaddr phys_addr) memory_region_is_romd(mr)); } -void qemu_ram_foreach_block (RAMBlockIterFunc func,void *opaque) +int qemu_ram_foreach_block (RAMBlockIterFunc func,void *opaque) {RAMBlock *block; +int ret; QTAILQ_FOREACH(block, &ram_list.blocks, next) { -func (block->host, block->offset, block->length, opaque); + ret =func (block->idstr, block->host, block->offset, block->length, + opaque); +if (ret) { +return ret; + } } +return 0 ; }#endif
include/exec/cpu-common.h +2 -2
@@ -118,10 +118,10 @@ void cpu_flush_icache_range(hwaddr start, int len);extern struct MemoryRegion io_mem_rom;extern struct MemoryRegion io_mem_notdirty; -typedef void (RAMBlockIterFunc )(void *host_addr, +typedef int (RAMBlockIterFunc )(const char *block_name,void *host_addr,ram_addr_t offset,ram_addr_t length,void *opaque); -void qemu_ram_foreach_block (RAMBlockIterFunc func,void *opaque); +int qemu_ram_foreach_block (RAMBlockIterFunc func,void *opaque);#endif
migration-rdma.c +2 -2
@@ -595,10 +595,10 @@ static int __qemu_rdma_add_block(RDMAContext *rdma, void *host_addr,* in advanced before the migration starts. This tells us where the RAM blocks * are so that we can register them individually. */ -static void qemu_rdma_init_one_block (void *host_addr, +static int qemu_rdma_init_one_block (const char *block_name,void *host_addr,ram_addr_t block_offset,ram_addr_t length,void *opaque) { -__qemu_rdma_add_block (opaque, host_addr, block_offset, length); +return __qemu_rdma_add_block (opaque, host_addr, block_offset, length); }/*
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 04/47] qemu_ram_foreach_block: pass up error value, and down the ramblock name
Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
check the return value of the function it calls and error if it's non-0 Fixup
qemu_rdma_init_one_blockthat is the only current caller,
and__qemu_rdma_add_blockthe only function it calls using it.Pass the name of the ramblock to the function; helps in debugging.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 5/47 improve DPRINTF macros, add to savevm
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Improve the existing DPRINTF macros in migration.c and arch_init by:
1) Making them go to stderr rather than stdout (so you can run with
-nographic and redirect your debug to a file)
2) Making them print the ms time with each debug - useful for
debugging latency issues
Add the same macro to savevm
--- arch_init.c | 5 ++++- migration.c | 12 ++++++++++++ savevm.c | 10 ++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-)
arch_init.c +4 -1
@@ -53,9 +53,12 @@#include "hw/acpi/acpi.h" #include "qemu/host-utils.h" +// #define DEBUG_ARCH_INIT #ifdef DEBUG_ARCH_INIT #define DPRINTF (fmt, ...) \ - do { fprintf(stdout, "arch_init: " fmt, ## __VA_ARGS__); } while (0) + do { fprintf(stderr, "arch_init@%" PRId64 " " fmt "\n", \ + qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \ + ## __VA_ARGS__); } while (0)#else #define DPRINTF (fmt, ...) \ do { } while (0)
migration.c +12 -0
@@ -26,6 +26,18 @@#include "qmp-commands.h" #include "trace.h" +//#define DEBUG_MIGRATION + +#ifdef DEBUG_MIGRATION +#define DPRINTF (fmt, ...) \ + do { fprintf(stderr, "migration@%" PRId64 " " fmt "\n", \ + qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \ + ## __VA_ARGS__); } while (0) +#else +#define DPRINTF (fmt, ...) \ + do { } while (0) +#endif +enum {MIG_STATE_ERROR =-1 ,MIG_STATE_NONE ,
savevm.c +10 -0
@@ -43,6 +43,16 @@#include "block/snapshot.h" #include "block/qapi.h" +#ifdef DEBUG_SAVEVM +#define DPRINTF (fmt, ...) \ + do { fprintf(stderr, "savevm@%" PRId64 " " fmt "\n", \ + qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \ + ## __VA_ARGS__); } while (0) +#else +#define DPRINTF (fmt, ...) \ + do { } while (0) +#endif +#ifndef ETH_P_RARP #define ETH_P_RARP 0x8035
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 05/47] improve DPRINTF macros, add to savevm
Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Improve the existing DPRINTF macros in
migration.candarch_initby:
1) Making them go to stderr rather than stdout (so you can run with
-nographic and redirect your debug to a file)
2) Making them print the ms time with each debug - useful for
debugging latency issues
Quoted 3 lines
Add the same macro to savevm
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 6/47 Add qemu_get_counted_string to read a string prefixed by a count byte
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
and use it in loadvm_state.
--- include/migration/qemu-file.h | 2 ++ qemu-file.c | 15 +++++++++++++++ savevm.c | 18 ++++++++++-------- 3 files changed, 27 insertions(+), 8 deletions(-)
include/migration/qemu-file.h +2 -0
@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) {qemu_get_be64s (f, (uint64_t *)pv); } + +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf);#endif
qemu-file.c +15 -0
@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)return v; } +/* +* Get a string whose length is determined by a single preceding byte +* A preallocated 256 byte buffer must be passed in. +* Returns: 0 on success and a 0 terminated string in the buffer +*/ +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf) +{ +unsigned int len =qemu_get_byte (f); +int res =qemu_get_buffer (f, buf, len); + + buf[len] =0 ; + +return res != len; +} +#define QSB_CHUNK_SIZE (1 << 10)#define QSB_MAX_CHUNK_SIZE (16 * QSB_CHUNK_SIZE)
savevm.c +10 -8
@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f) v =qemu_get_be32 (f);if (v ==QEMU_VM_FILE_VERSION_COMPAT ) { -fprintf (stderr,"SaveVM v2 format is obsolete and don't work anymore\n" ); +error_report ("SaveVM v2 format is obsolete and don't work anymore" );return -ENOTSUP ; }if (v !=QEMU_VM_FILE_VERSION ) {
@@ -918,31 +918,33 @@ int qemu_loadvm_state(QEMUFile *f)while ((section_type =qemu_get_byte (f)) !=QEMU_VM_EOF ) {uint32_t instance_id, version_id, section_id;SaveStateEntry *se; -char idstr[257 ]; -int len; +char idstr[256 ];switch (section_type) {case QEMU_VM_SECTION_START :case QEMU_VM_SECTION_FULL :/* Read section start */ section_id =qemu_get_be32 (f); - len =qemu_get_byte (f); -qemu_get_buffer (f, (uint8_t *)idstr, len); - idstr[len] =0 ; +if (qemu_get_counted_string (f, (uint8_t *)idstr)) { +error_report ("Unable to read ID string for section %u" , + section_id); +return -EINVAL ; + } instance_id =qemu_get_be32 (f); version_id =qemu_get_be32 (f);/* Find savevm section */ se =find_se (idstr, instance_id);if (se ==NULL ) { -fprintf (stderr,"Unknown savevm section or instance '%s' %d\n" , idstr, instance_id); +error_report ("Unknown savevm section or instance '%s' %d" , + idstr, instance_id); ret = -EINVAL ; goto out; }/* Validate version */ if (version_id > se->version_id) { -fprintf (stderr,"savevm: unsupported version %d for '%s' v%d\n" , +error_report ("savevm: unsupported version %d for '%s' v%d" , version_id, idstr, se->version_id); ret = -EINVAL ; goto out;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 06/47] Add qemu_get_counted_string to read a string prefixed by a count byte
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
and use it in
loadvm_state.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/qemu-file.h | 2 ++ qemu-file.c | 15 +++++++++++++++ savevm.c | 18 ++++++++++-------- 3 files changed, 27 insertions(+), 8 deletions(-)include/migration/qemu-file.h +0
-0
include/migration/qemu-file.h static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) · line 304
Quoted 5 lines
include/migration/qemu-file.h +2
-0@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) {qemu_get_be64s (f, (uint64_t *)pv); } + +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf);
I'd suggest writing the prototype as
int qemu_get_counted_string(QEMUFile *f, uint8_t buf[256]);
The compiled code will be identical, of course, but it helps to document what the function expects.
Quoted 32 lines
include/migration/qemu-file.h +0
-0#endif qemu-file.c +15
-0@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)return v; } +/* +* Get a string whose length is determined by a single preceding byte +* A preallocated 256 byte buffer must be passed in. +* Returns: 0 on success and a 0 terminated string in the buffer +*/ +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf) +{ +unsigned int len =qemu_get_byte (f); +int res =qemu_get_buffer (f, buf, len); + + buf[len] =0 ; + +return res != len; +} +#define QSB_CHUNK_SIZE (1 << 10)#define QSB_MAX_CHUNK_SIZE (16 * QSB_CHUNK_SIZE)savevm.c +0
-0
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 911
Quoted 5 lines
savevm.c +1
-1@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f) v =qemu_get_be32 (f);if (v ==QEMU_VM_FILE_VERSION_COMPAT ) { -fprintf (stderr,"SaveVM v2 format is obsolete and don't work anymore\n" ); +error_report ("SaveVM v2 format is obsolete and don't work anymore" );
These changes of fprintf() to error_report() look like an unrelated cleanup.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 06/47] Add qemu_get_counted_string to read a string prefixed by a count byte
Quoted 25 lines · * David Gibson ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
and use it in
loadvm_state.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/qemu-file.h | 2 ++ qemu-file.c | 15 +++++++++++++++ savevm.c | 18 ++++++++++-------- 3 files changed, 27 insertions(+), 8 deletions(-)include/migration/qemu-file.h +0
-0include/migration/qemu-file.h static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) · line 304
Quoted 5 lines
include/migration/qemu-file.h +2
-0@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) {qemu_get_be64s (f, (uint64_t *)pv); } + +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf);I'd suggest writing the prototype as
Quoted 4 lines
int
qemu_get_counted_string(QEMUFile *f, uint8_t buf[256]);The compiled code will be identical, of course, but it helps to document what the function expects.
Good idea; done.
Quoted 33 lines
Quoted 33 lines
include/migration/qemu-file.h +0
-0#endif qemu-file.c +15
-0@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)return v; } +/* +* Get a string whose length is determined by a single preceding byte +* A preallocated 256 byte buffer must be passed in. +* Returns: 0 on success and a 0 terminated string in the buffer +*/ +int qemu_get_counted_string (QEMUFile *f,uint8_t *buf) +{ +unsigned int len =qemu_get_byte (f); +int res =qemu_get_buffer (f, buf, len); + + buf[len] =0 ; + +return res != len; +} +#define QSB_CHUNK_SIZE (1 << 10)#define QSB_MAX_CHUNK_SIZE (16 * QSB_CHUNK_SIZE)savevm.c +0
-0@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f)
Quoted 7 lines
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 911
Quoted 4 lines
savevm.c +1
-1v =qemu_get_be32 (f);if (v ==QEMU_VM_FILE_VERSION_COMPAT ) { -fprintf (stderr,"SaveVM v2 format is obsolete and don't work anymore\n" ); +error_report ("SaveVM v2 format is obsolete and don't work anymore" );These changes of
fprintf()toerror_report()look like an unrelated cleanup.
Not quite; with the use of qemu_get_counted_string it can return an error,
so I check it, and use error_report; while I was there I converted a couple of the surrounding fprintf's to error_report at the same time.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 7/47 Create MigrationIncomingState
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.
allocate MIS in process_incoming_migration_co
--- include/migration/migration.h | 9 +++++++++ include/qemu/typedefs.h | 2 ++ migration.c | 28 ++++++++++++++++++++++++++++ savevm.c | 2 ++ 4 files changed, 41 insertions(+)
include/migration/migration.h +9 -0
@@ -41,6 +41,15 @@ struct MigrationParams {typedef struct MigrationState MigrationState ; +/* State for the incoming migration */ +struct MigrationIncomingState { +QEMUFile *file; +}; + +MigrationIncomingState *migration_incoming_get_current (void ); +MigrationIncomingState *migration_incoming_state_init (QEMUFile *f); +void migration_incoming_state_destroy (void ); +struct MigrationState {int64_t bandwidth_limit;
include/qemu/typedefs.h +2 -0
@@ -14,6 +14,7 @@ typedef struct Visitor Visitor;struct Monitor ;typedef struct Monitor Monitor ; +typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct MigrationParams MigrationParams ;typedef struct Property Property ;
@@ -44,6 +45,7 @@ typedef struct PixelFormat PixelFormat;typedef struct QemuConsole QemuConsole ;typedef struct CharDriverState CharDriverState ;typedef struct MACAddr MACAddr ; +typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct NetClientState NetClientState ;typedef struct I2CBus I2CBus ;typedef struct ISABus ISABus ;
migration.c +28 -0
@@ -65,6 +65,7 @@ static NotifierList migration_state_notifiers =migrations at once. For now we don't need to add dynamic creation of migration */ +/* For outgoing */ MigrationState *migrate_get_current (void ) {static MigrationState current_migration = {
@@ -77,6 +78,28 @@ MigrationState *migrate_get_current(void)return ¤t_migration; } +/* For incoming */ +static MigrationIncomingState *mis_current; + +MigrationIncomingState *migration_incoming_get_current (void ) +{ +return mis_current; +} + +MigrationIncomingState *migration_incoming_state_init (QEMUFile * f) +{ + mis_current =g_malloc0 (sizeof (MigrationIncomingState)); + mis_current->file = f; + +return mis_current; +} + +void migration_incoming_state_destroy (void ) +{ +g_free (mis_current); + mis_current =NULL ; +} +void qemu_start_incoming_migration (const char *uri,Error **errp) {const char *p;
@@ -106,9 +129,14 @@ static void process_incoming_migration_co(void *opaque)Error *local_err =NULL ;int ret; +migration_incoming_state_init (f); + ret =qemu_loadvm_state (f); +qemu_fclose (f);free_xbzrle_decoded_buf (); +migration_incoming_state_destroy (); +if (ret <0 ) {error_report ("load of migration failed: %s" ,strerror (-ret));exit (EXIT_FAILURE );
savevm.c +2 -0
@@ -1244,9 +1244,11 @@ int load_vmstate(const char *name) }qemu_system_reset (VMRESET_SILENT ); +migration_incoming_state_init (f); ret =qemu_loadvm_state (f);qemu_fclose (f); +migration_incoming_state_destroy ();if (ret <0 ) {error_report ("Error %d while loading VM state" , ret);return ret;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 07/47] Create MigrationIncomingState
Quoted 21 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.
allocate MIS in
process_incoming_migration_coSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 9 +++++++++ include/qemu/typedefs.h | 2 ++ migration.c | 28 ++++++++++++++++++++++++++++ savevm.c | 2 ++ 4 files changed, 41 insertions(+)include/migration/migration.h +0
-0
include/migration/migration.h struct MigrationParams { · line 50
Quoted 10 lines
include/migration/migration.h +7
-0@@ -41,6 +41,15 @@ struct MigrationParams {typedef struct MigrationState MigrationState ; +/* State for the incoming migration */ +struct MigrationIncomingState { +QEMUFile *file; +}; + +MigrationIncomingState *migration_incoming_get_current (void ); +MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);
Hrm. I'd prefer to see this called migration_incoming_state_new(), since it allocates a new structure, rather than just initializing an already allocated one.
I guess you're trying to match migrate_init() in name, so i guess migrate_incoming_init() would work as well.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 07/47] Create MigrationIncomingState
Quoted 37 lines · * David Gibson ([email protected]) wrote:
Quoted 21 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.
allocate MIS in
process_incoming_migration_coSigned-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 9 +++++++++ include/qemu/typedefs.h | 2 ++ migration.c | 28 ++++++++++++++++++++++++++++ savevm.c | 2 ++ 4 files changed, 41 insertions(+)include/migration/migration.h +0
-0include/migration/migration.h struct MigrationParams { · line 50
Quoted 10 lines
include/migration/migration.h +7
-0@@ -41,6 +41,15 @@ struct MigrationParams {typedef struct MigrationState MigrationState ; +/* State for the incoming migration */ +struct MigrationIncomingState { +QEMUFile *file; +}; + +MigrationIncomingState *migration_incoming_get_current (void ); +MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);Hrm. I'd prefer to see this called
migration_incoming_state_new(), since it allocates a new structure, rather than just initializing an already allocated one.
Quoted 2 lines
I guess you're trying to match
migrate_init()in name, so i guessmigrate_incoming_init()would work as well.
No, you're right the 1st time, _new is more consistent; fixed.
Dave
Quoted 5 lines
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 8/47 socket shutdown
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add QEMUFile interface to allow a socket to be 'shut down' - i.e. any reads/writes will fail (and any blocking read/write will be woken).
Add qemu_socket wrapper to let OS dependencies be extracted out.
--- include/migration/qemu-file.h | 10 ++++++++++ include/qemu/sockets.h | 1 + qemu-file.c | 27 +++++++++++++++++++++++++-- util/qemu-sockets.c | 28 ++++++++++++++++++++++++++++ 4 files changed, 64 insertions(+), 2 deletions(-)
include/migration/qemu-file.h +10 -0
@@ -84,6 +84,14 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,size_t size, int *bytes_sent); +/* +* Stop any read or write (depending on flags) on the underlying +* transport on the QEMUFile. +* Existing blocking reads/writes must be woken +* Returns 0 on success, -err on error +*/ +typedef int (QEMUFileShutdownFunc )(void *opaque,bool rd,bool wr); + typedef struct QEMUFileOps { QEMUFilePutBufferFunc *put_buffer; QEMUFileGetBufferFunc *get_buffer;
@@ -94,6 +102,7 @@ typedef struct QEMUFileOps {QEMURamHookFunc *after_ram_iterate;QEMURamHookFunc *hook_ram_load;QEMURamSaveFunc *save_page; +QEMUFileShutdownFunc *shut_down; } QEMUFileOps;struct QEMUSizedBuffer {
@@ -178,6 +187,7 @@ void qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);int64_t qemu_file_get_rate_limit (QEMUFile *f);int qemu_file_get_error (QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret); +void qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv)
include/qemu/sockets.h +1 -0
@@ -41,6 +41,7 @@ int socket_set_nodelay(int fd);void qemu_set_block (int fd);void qemu_set_nonblock (int fd);int socket_set_fast_reuse (int fd); +int socket_shutdown (int fd,bool rd,bool wr);int send_all (int fd,const void *buf,int len1);int recv_all (int fd,void *buf,int len1,bool single_read);
qemu-file.c +25 -2
@@ -90,6 +90,14 @@ static int socket_close(void *opaque)return 0 ; } +/* qemufile_ to disambiguate from the qemu-sockets.c code which it uses */ +static int qemufile_socket_shutdown (void *opaque,bool rd,bool wr) +{ +QEMUFileSocket *s = opaque; + +return socket_shutdown (s->fd, rd, wr); +} +static int stdio_get_fd (void *opaque) {QEMUFileStdio *s = opaque;
@@ -337,15 +345,30 @@ QEMUFile *qemu_fdopen(int fd, const char *mode)static const QEMUFileOps socket_read_ops = { .get_fd = socket_get_fd, .get_buffer = socket_get_buffer, - .close = socket_close + .close = socket_close, + .shut_down = qemufile_socket_shutdown + };static const QEMUFileOps socket_write_ops = { .get_fd = socket_get_fd, .writev_buffer = socket_writev_buffer, - .close = socket_close + .close = socket_close, + .shut_down = qemufile_socket_shutdown + }; +/* +* Stop a file from being read/written - not all backing files can do this +* typically only sockets can. The caller should make sure they only +* call this for things that can. +*/ +void qemu_file_shutdown (QEMUFile *f) +{ +assert (f->ops->shut_down); + f->ops->shut_down (f, true, true); +} +bool qemu_file_mode_is_not_valid (const char *mode) { if (mode == NULL ||
util/qemu-sockets.c +28 -0
@@ -981,3 +981,31 @@ int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp)qemu_opts_del (opts);return fd; } + +int socket_shutdown (int fd,bool rd,bool wr) +{ +int how =0 ; + +#ifndef WIN32 +if (rd) { + how =SHUT_RD ; + } + +if (wr) { + how = rd ?SHUT_RDWR :SHUT_WR ; + } + +#else +/* Untested */ +if (rd) { + how =SD_RECEIVE ; + } + +if (wr) { + how = rd ?SD_BOTH :SD_SEND ; + } + +#endif + +return shutdown (fd, how); +}
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 11 lines
util/qemu-sockets.c +11
-0+#ifndef WIN32 +if (rd) { + how =SHUT_RD ; + } + +if (wr) { + how = rd ?SHUT_RDWR :SHUT_WR ; + } + +#else +/* Untested */
util/qemu-sockets.c int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp) · line 1009
Quoted 10 lines
util/qemu-sockets.c +10
-0+if (rd) { + how =SD_RECEIVE ; + } + +if (wr) { + how = rd ?SD_BOTH :SD_SEND ; + } + +#endif +
These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.
Paolo
Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown
Quoted 18 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 17 lines
(quoted diff) +17
-0+#ifndef WIN32 + if (rd) { + how = SHUT_RD; + } + + if (wr) { + how = rd ? SHUT_RDWR : SHUT_WR; + } + +#else + /* Untested */ + if (rd) { + how = SD_RECEIVE; + } + + if (wr) { + how = rd ? SD_BOTH : SD_SEND;
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ } + +#endif +These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.
OK, something like this? (the qemu-file.c abstraction is still needed to cover QEMUFile's that aren't simple sockets, but I've removed the second layer in util/qemu-sockets.c).
include/qemu/sockets.h +7 -0
@@ -44,6 +44,13 @@ int socket_set_fast_reuse(int fd);int send_all (int fd,const void *buf,int len1);int recv_all (int fd,void *buf,int len1,bool single_read); +#ifdef WIN32 +/* Windows has different names for the same constants with the same values */ +#define SHUT_RD 0 +#define SHUT_WR 1 +#define SHUT_RDWR 2 +#endif +/* callback function for nonblocking connect * valid fd on success, negative error code on failure */
qemu-file.c +7 -0
@@ -90,6 +90,13 @@ static int socket_close(void *opaque)return 0 ; } +static int socket_shutdown (void *opaque,bool rd,bool wr) +{ +QEMUFileSocket *s = opaque; + +return shutdown (s->fd, rd ? (wr ?SHUT_RDWR :SHUT_RD ) :SHUT_WR ); +} +static int stdio_get_fd (void *opaque) {QEMUFileStdio *s = opaque;
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown
Il 07/10/2014 12:00, Dr. David Alan Gilbert ha scritto:
Quoted 28 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 18 lines
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 17 lines
(quoted diff) +17
-0+#ifndef WIN32 + if (rd) { + how = SHUT_RD; + } + + if (wr) { + how = rd ? SHUT_RDWR : SHUT_WR; + } + +#else + /* Untested */ + if (rd) { + how = SD_RECEIVE; + } + + if (wr) { + how = rd ? SD_BOTH : SD_SEND;Quoted 8 lines
qemu-file.c static int socket_close(void *opaque)
Quoted 4 lines
qemu-file.c +4
-0+ } + +#endif +These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.
Quoted 3 lines
OK, something like this? (the
qemu-file.cabstraction is still needed to cover QEMUFile's that aren't simple sockets, but I've removed the second layer inutil/qemu-sockets.c).
Yes. Or just pass SHUT_* directly to socket_shutdown, of course.
Paolo
Quoted 37 lines
include/qemu/sockets.h +7
-0@@ -44,6 +44,13 @@ int socket_set_fast_reuse(int fd);int send_all (int fd,const void *buf,int len1);int recv_all (int fd,void *buf,int len1,bool single_read); +#ifdef WIN32 +/* Windows has different names for the same constants with the same values */ +#define SHUT_RD 0 +#define SHUT_WR 1 +#define SHUT_RDWR 2 +#endif +/* callback function for nonblocking connect * valid fd on success, negative error code on failure */ qemu-file.c +7
-1@@ -90,6 +90,13 @@ static int socket_close(void *opaque)return 0 ; } +static int socket_shutdown (void *opaque,bool rd,bool wr) +{ +QEMUFileSocket *s = opaque; + +return shutdown (s->fd, rd ? (wr ?SHUT_RDWR :SHUT_RD ) :SHUT_WR ); +} +static int stdio_get_fd (void *opaque) {QEMUFileStdio *s = opaque; --Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 9/47 Provide runtime Target page information
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init.
Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.
--- exec.c | 10 ++++++++++ include/sysemu/sysemu.h | 1 + 2 files changed, 11 insertions(+)
exec.c +10 -0
@@ -2881,4 +2881,14 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque) }return 0 ; } + +/* +* Allows code that needs to deal with migration bitmaps etc to still be built +* target independent. +*/ +size_t qemu_target_page_bits (void ) +{ +return TARGET_PAGE_BITS ; +} +#endif
include/sysemu/sysemu.h +1 -0
@@ -68,6 +68,7 @@ int qemu_reset_requested_get(void);void qemu_system_killed (int signal,pid_t pid);void qemu_devices_reset (void );void qemu_system_reset (bool report); +size_t qemu_target_page_bits (void );void qemu_add_exit_notifier (Notifier *notify);void qemu_remove_exit_notifier (Notifier *notify);
Signature · 1 line
1.9.3
PATCH v4 10/47 Return path: Open a return path on QEMUFile for sockets
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's using a dup'd fd.
--- include/migration/qemu-file.h | 7 +++++ qemu-file.c | 73 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 70 insertions(+), 10 deletions(-)
include/migration/qemu-file.h +7 -0
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, int *bytes_sent);/* +* Return a QEMUFile for comms in the opposite direction +*/ +typedef QEMUFile *(QEMURetPathFunc )(void *opaque); + +/* * Stop any read or write (depending on flags) on the underlying * transport on the QEMUFile. * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {QEMURamHookFunc *after_ram_iterate;QEMURamHookFunc *hook_ram_load;QEMURamSaveFunc *save_page; +QEMURetPathFunc *get_return_path;QEMUFileShutdownFunc *shut_down; } QEMUFileOps;
@@ -188,6 +194,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);int qemu_file_get_error (QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);void qemu_file_shutdown (QEMUFile *f); +QEMUFile *qemu_file_get_return_path (QEMUFile *f);void qemu_fflush (QEMUFile *f);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv)
qemu-file.c +63 -10
100 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's using a dup'd fd.
This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.
I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets
Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's using a dup'd fd.
Quoted 7 lines
This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.
I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.
Can you give an example of a transport where it would be a problem, so I can look at how that works?
It's a little tricky since, on the destination, at the time we create the connection we don't know that we're going to need the return path.
Dave
Quoted 5 lines
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's using a dup'd fd.
This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.
Quoted 6 lines
Quoted 3 lines
I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.
Can you give an example of a transport where it would be a problem, so I can look at how that works?
pipe
socket routed through some external transport / encoding layer that's one way only.
Quoted 2 lines
It's a little tricky since, on the destination, at the time we create the connection we don't know that we're going to need the return path.
Creating it and not using it shouldn't be a problem though, should it? As long as you just fall back to precopy rather than failing the migration if you're unable to open it.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 11/47 Return path: socket_writev_buffer: Block even on non-blocking fd's
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.
--- qemu-file.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)
qemu-file.c +35 -4
@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. +*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } } -return len; + +return offset; }static int socket_get_fd (void *opaque)
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's
Quoted 34 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- qemu-file.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)qemu-file.c +11
-3@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) {
qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109
Quoted 10 lines
qemu-file.c +10
-0+ err =socket_error (); + +if (err !=EAGAIN ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately.
Is that safe? This gives the caller no means to detect a partially completed send.
Quoted 19 lines
qemu-file.c +14
-1+*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } } -return len; + +return offset; }static int socket_get_fd (void *opaque)
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's
Quoted 40 lines · * David Gibson ([email protected]) wrote:
Quoted 39 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- qemu-file.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)qemu-file.c +16
-3@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len);
Quoted 8 lines
qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109
Quoted 5 lines
qemu-file.c +5
-0+/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. Is that safe? This gives the caller no means to detect a partially completed send.
Well I'm returning the -err, so the caller knows something has gone wrong - it just doesn't know whether it managed to send some part of the data before the failure.
Dave
Quoted 20 lines
Quoted 20 lines
qemu-file.c +14
-1+*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } } -return len; + +return offset; }
Quoted 7 lines
Quoted 1 line
static int
socket_get_fd(void *opaque)Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's
Quoted 50 lines · Dr. David Alan Gilbert
Quoted 40 lines · * David Gibson ([email protected]) wrote:
Quoted 39 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- qemu-file.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)qemu-file.c +16
-3@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len);Quoted 8 lines
qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109
Quoted 5 lines
qemu-file.c +5
-0+/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. Is that safe? This gives the caller no means to detect a partially completed send.
Quoted 3 lines
Well I'm returning the -err, so the caller knows something has gone wrong - it just doesn't know whether it managed to send some part of the data before the failure.
Right. Which seems like it could be pretty important.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 12/47 Handle bi-directional communication for fd migration
In reply to Dr. David Alan Gilbert (git)From: Cristian Klein <[email protected]>
--- migration-fd.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
migration-fd.c +22 -2
@@ -31,13 +31,29 @@ do { } while (0)#endif +static bool fd_is_socket (int fd) +{ +struct stat stat; +int ret =fstat (fd, &stat); +if (ret ==-1 ) { +/* When in doubt say no */ +return false; + } +return S_ISSOCK (stat.st_mode); +} +void fd_start_outgoing_migration (MigrationState *s,const char *fdname,Error **errp) {int fd =monitor_get_fd (cur_mon, fdname, errp);if (fd ==-1 ) {return ; } - s->file =qemu_fdopen (fd,"wb" ); + +if (fd_is_socket (fd)) { + s->file =qemu_fopen_socket (fd,"wb" ); + }else { + s->file =qemu_fdopen (fd,"wb" ); + }migrate_fd_connect (s); }
@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)DPRINTF ("Attempting to start an incoming migration via fd\n" ); fd =strtol (infd,NULL ,0 ); - f =qemu_fdopen (fd,"rb" ); +if (fd_is_socket (fd)) { + f =qemu_fopen_socket (fd,"rb" ); + }else { + f =qemu_fdopen (fd,"rb" ); + }if (f ==NULL ) {error_setg_errno (errp, errno,"failed to open the source descriptor" );return ;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration
Quoted 1 line · Dr. David Alan Gilbert (git)
From: Cristian Klein <[email protected]>
This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.
Quoted 54 lines
Signed-off-by Cristian Klein <[email protected]>--- migration-fd.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)migration-fd.c +22
-2@@ -31,13 +31,29 @@ do { } while (0)#endif +static bool fd_is_socket (int fd) +{ +struct stat stat; +int ret =fstat (fd, &stat); +if (ret ==-1 ) { +/* When in doubt say no */ +return false; + } +return S_ISSOCK (stat.st_mode); +} +void fd_start_outgoing_migration (MigrationState *s,const char *fdname,Error **errp) {int fd =monitor_get_fd (cur_mon, fdname, errp);if (fd ==-1 ) {return ; } - s->file =qemu_fdopen (fd,"wb" ); + +if (fd_is_socket (fd)) { + s->file =qemu_fopen_socket (fd,"wb" ); + }else { + s->file =qemu_fdopen (fd,"wb" ); + }migrate_fd_connect (s); }@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)DPRINTF ("Attempting to start an incoming migration via fd\n" ); fd =strtol (infd,NULL ,0 ); - f =qemu_fdopen (fd,"rb" ); +if (fd_is_socket (fd)) { + f =qemu_fopen_socket (fd,"rb" ); + }else { + f =qemu_fdopen (fd,"rb" ); + }if (f ==NULL ) {error_setg_errno (errp, errno,"failed to open the source descriptor" );return ;
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration
Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)
From: Cristian Klein <[email protected]>
This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.
“”"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu’s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. “”"
If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.
Cristian
Quoted 61 lines
Quoted 55 lines
Signed-off-by Cristian Klein <[email protected]>---
migration-fd.c| 24 ++++++++++++++++++++++--1 file changed, 22 insertions(+), 2 deletions(-)migration-fd.c +22
-2@@ -31,13 +31,29 @@ do { } while (0)#endif +static bool fd_is_socket (int fd) +{ +struct stat stat; +int ret =fstat (fd, &stat); +if (ret ==-1 ) { +/* When in doubt say no */ +return false; + } +return S_ISSOCK (stat.st_mode); +} +void fd_start_outgoing_migration (MigrationState *s,const char *fdname,Error **errp) {int fd =monitor_get_fd (cur_mon, fdname, errp);if (fd ==-1 ) {return ; } - s->file =qemu_fdopen (fd,"wb" ); + +if (fd_is_socket (fd)) { + s->file =qemu_fopen_socket (fd,"wb" ); + }else { + s->file =qemu_fdopen (fd,"wb" ); + }migrate_fd_connect (s); }@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)DPRINTF ("Attempting to start an incoming migration via fd\n" ); fd =strtol (infd,NULL ,0 ); - f =qemu_fdopen (fd,"rb" ); +if (fd_is_socket (fd)) { + f =qemu_fopen_socket (fd,"rb" ); + }else { + f =qemu_fdopen (fd,"rb" ); + }if (f ==NULL ) {error_setg_errno (errp, errno,"failed to open the source descriptor" );return ;Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration
Quoted 8 lines · Cristian Klein
Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)
From: Cristian Klein <[email protected]>
This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.
Quoted 6 lines
“”"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu’s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. “”"If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.
That would help, yes.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration
Quoted 9 lines · * David Gibson ([email protected]) wrote:
Quoted 8 lines · Cristian Klein
Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)
From: Cristian Klein <[email protected]>
This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.
Quoted 8 lines
Quoted 6 lines
??????"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu???s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. ??????"If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.
That would help, yes.
I've added that text into the commit message.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 13/47 Migration commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.
For use in postcopy.
--- include/migration/migration.h | 1 + include/sysemu/sysemu.h | 9 ++++++++ savevm.c | 48 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+)
include/migration/migration.h +1 -0
@@ -33,6 +33,7 @@#define QEMU_VM_SECTION_END 0x03#define QEMU_VM_SECTION_FULL 0x04#define QEMU_VM_SUBSECTION 0x05 +#define QEMU_VM_COMMAND 0x06struct MigrationParams {bool blk;
include/sysemu/sysemu.h +9 -0
@@ -82,6 +82,13 @@ void do_info_snapshots(Monitor *mon, const QDict *qdict);void qemu_announce_self (void ); +/* Subcommands for QEMU_VM_COMMAND */ +enum qemu_vm_cmd { +QEMU_VM_CMD_INVALID =0 ,/* Must be 0 */ + +QEMU_VM_CMD_AFTERLASTVALID +}; +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -89,6 +96,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);void qemu_savevm_state_complete (QEMUFile *f);void qemu_savevm_state_cancel (void );uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command, +uint16_t len,uint8_t *data);int qemu_loadvm_state (QEMUFile *f);/* SLIRP */
savevm.c +48 -0
66 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 13/47] Migration commands
Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Create
QEMU_VM_COMMANDsection type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.For use in postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 14/47 Return path: Control commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add two src->dest commands:
* OPENRP - To request that the destination open the return path
* REQACK - Request an acknowledge from the destination
--- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h | 4 +++ savevm.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+)
include/migration/migration.h +2 -0
@@ -45,6 +45,8 @@ typedef struct MigrationState MigrationState;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file; + +QEMUFile *return_path; };MigrationIncomingState *migration_incoming_get_current (void );
include/sysemu/sysemu.h +4 -0
@@ -85,6 +85,8 @@ void qemu_announce_self(void);/* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd {QEMU_VM_CMD_INVALID =0 ,/* Must be 0 */ +QEMU_VM_CMD_OPENRP ,/* Tell the dest to open the Return path */ +QEMU_VM_CMD_REQACK ,/* Request an ACK on the RP */ QEMU_VM_CMD_AFTERLASTVALID };
@@ -98,6 +100,8 @@ void qemu_savevm_state_cancel(void);uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data); +void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value); +void qemu_savevm_send_openrp (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f);/* SLIRP */
savevm.c +57 -0
83 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
QEMU_VM_CMD_INVALID = 0, /* Must be 0 */+
QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */
OPEN_RETURN_PATH?
Quoted 1 line
+
QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */
SEND_ACK or ACK_REQUESTED?
Quoted 1 line
QEMU_VM_CMD_AFTERLASTVALID
Pleaseseparatewords. Is this enum actually used at all?
Please avoid the difference between QEMU_VM_CMD and MIG_RPCOMM_.
Perhaps MIG_CMD and MIG_RPCMD_?
Paolo
Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands
Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
QEMU_VM_CMD_INVALID = 0, /* Must be 0 */+
QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */
OPEN_RETURN_PATH?Quoted 1 line
+
QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */
SEND_ACKorACK_REQUESTED?
Quoted 7 lines
Quoted 1 line
QEMU_VM_CMD_AFTERLASTVALIDPleaseseparatewords. Is this enum actually used at all?
Please avoid the difference between
QEMU_VM_CMDandMIG_RPCOMM_.Perhaps
MIG_CMDandMIG_RPCMD_?
Almost, I went with:
MIG_CMD_INVALID = 0, /* Must be 0 */
MIG_CMD_OPEN_RETURN_PATH, /* Tell the dest to open the Return path */
MIG_CMD_SEND_ACK, /* Request an ACK on the RP */
MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ MIG_CMD_POSTCOPY_ADVISE = 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */
MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */
and
MIG_RP_CMD_INVALID = 0, /* Must be 0 */
MIG_RP_CMD_SHUT, /* sibling will not send any more RP messages */
MIG_RP_CMD_ACK, /* data (seq: be32 ) */
MIG_RP_CMD_REQ_PAGES, /* data (start: be64, len: be64) */the only oddity I get from that is from the 'SEND_ACK' you suggested; since all my functions to send commands are send_ I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
QEMU_VM_CMD_INVALID = 0, /* Must be 0 */+
QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */
OPEN_RETURN_PATH?Quoted 1 line
+
QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */
SEND_ACKorACK_REQUESTED?Quoted 7 lines
Quoted 1 line
QEMU_VM_CMD_AFTERLASTVALIDPleaseseparatewords. Is this enum actually used at all?
Please avoid the difference between
QEMU_VM_CMDandMIG_RPCOMM_.Perhaps
MIG_CMDandMIG_RPCMD_?Almost, I went with:
MIG_CMD_INVALID = 0, /* Must be 0 */ MIG_CMD_OPEN_RETURN_PATH, /* Tell the dest to open the Return path */ MIG_CMD_SEND_ACK, /* Request an ACK on the RP */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */
MIG_CMD_POSTCOPY_ADVISE= 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */
MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */and
Quoted 8 lines
MIG_RP_CMD_INVALID = 0, /* Must be 0 */ MIG_RP_CMD_SHUT, /* sibling will not send any more RP messages */ MIG_RP_CMD_ACK, /* data (seq: be32 ) */ MIG_RP_CMD_REQ_PAGES, /* data (start: be64, len: be64) */the only oddity I get from that is from the '
SEND_ACK' you suggested; since all my functions to send commands aresend_I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.
Perhaps ping/pong?
Paolo
Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands
Quoted 46 lines · Paolo Bonzini
Quoted 43 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
QEMU_VM_CMD_INVALID = 0, /* Must be 0 */+
QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */
OPEN_RETURN_PATH?Quoted 1 line
+
QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */
SEND_ACKorACK_REQUESTED?Quoted 7 lines
Quoted 1 line
QEMU_VM_CMD_AFTERLASTVALIDPleaseseparatewords. Is this enum actually used at all?
Please avoid the difference between
QEMU_VM_CMDandMIG_RPCOMM_.Perhaps
MIG_CMDandMIG_RPCMD_?Almost, I went with:
MIG_CMD_INVALID = 0, /* Must be 0 */ MIG_CMD_OPEN_RETURN_PATH, /* Tell the dest to open the Return path */ MIG_CMD_SEND_ACK, /* Request an ACK on the RP */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */
MIG_CMD_POSTCOPY_ADVISE= 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */
MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */and
MIG_RP_CMD_INVALID = 0, /* Must be 0 */ MIG_RP_CMD_SHUT, /* sibling will not send any more RP messages */ MIG_RP_CMD_ACK, /* data (seq: be32 ) */ MIG_RP_CMD_REQ_PAGES, /* data (start: be64, len: be64) */
Quoted 5 lines
Quoted 3 lines
the only oddity I get from that is from the '
SEND_ACK' you suggested; since all my functions to send commands aresend_I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.Perhaps ping/pong?
I like that idea. Calling it "send_ack" looks like it's just asking for confusing names somewhere.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands
In reply to Paolo BonziniQuoted 5 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines
the only oddity I get from that is from the '
SEND_ACK' you suggested; since all my functions to send commands aresend_I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.Perhaps ping/pong?
Done (although I'm sure I'll find 'ack' in the comments somewhere).
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 15/47 Return path: Send responses from destination to source
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send an 'ack' message
Use it in the CMD_REQACK handler
--- include/migration/migration.h | 18 ++++++++++++++++++ migration.c | 41 +++++++++++++++++++++++++++++++++++++++++ savevm.c | 2 +- 3 files changed, 60 insertions(+), 1 deletion(-)
include/migration/migration.h +18 -0
@@ -40,6 +40,13 @@ struct MigrationParams {bool shared; }; +/* Commands sent on the return path from destination to source*/ +enum mig_rpcomm_cmd { +MIG_RPCOMM_INVALID =0 ,/* Must be 0 */ +MIG_RPCOMM_SHUT ,/* sibling will not send any more RP messages */ +MIG_RPCOMM_ACK ,/* data (seq: be32 ) */ +MIG_RPCOMM_AFTERLASTVALID +};typedef struct MigrationState MigrationState ;/* State for the incoming migration */
@@ -47,6 +54,7 @@ struct MigrationIncomingState {QEMUFile *file;QEMUFile *return_path; +QemuMutex rp_mutex;/* We send replies from multiple threads */ };MigrationIncomingState *migration_incoming_get_current (void );
@@ -168,6 +176,16 @@ int64_t migrate_xbzrle_cache_size(void);int64_t xbzrle_cache_resize (int64_t new_size); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rpcomm_cmd cmd, +uint16_t len,uint8_t *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_ack (MigrationIncomingState *mis, +uint32_t value); + +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags);
migration.c +41 -0
53 lines not on this page · raw message
savevm.c +1 -1
@@ -969,7 +969,7 @@ static int loadvm_process_command(QEMUFile *f) tmp32);return -1 ; } -/* migrate_send_rp_ack(mis, tmp32); TODO: gets added later */ +migrate_send_rp_ack (mis, tmp32);break ;default :
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 15/47] Return path: Send responses from destination to source
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send an 'ack' message
Use it in theCMD_REQACKhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 16/47 Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
--- include/migration/migration.h | 10 +++ migration.c | 181 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 190 insertions(+), 1 deletion(-)
include/migration/migration.h +10 -0
@@ -47,6 +47,14 @@ enum mig_rpcomm_cmd {MIG_RPCOMM_ACK ,/* data (seq: be32 ) */ MIG_RPCOMM_AFTERLASTVALID }; + +/* Source side RP state */ +struct MigrationRetPathState { +uint32_t latest_ack; +QemuThread rp_thread; +bool error; +}; +typedef struct MigrationState MigrationState ;/* State for the incoming migration */
@@ -69,9 +77,11 @@ struct MigrationStateQemuThread thread;QEMUBH *cleanup_bh;QEMUFile *file; +QEMUFile *return_path;int state;MigrationParams params; +struct MigrationRetPathState rp_state;double mbps;int64_t total_time;int64_t downtime;
migration.c +180 -1
212 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
include/migration/migration.h enum mig_rpcomm_cmd { · line 55
Quoted 5 lines
include/migration/migration.h +5
-0+/* Source side RP state */ +struct MigrationRetPathState { +uint32_t latest_ack; +QemuThread rp_thread; +bool error;
Should the QemuFile be in here?
include/migration/migration.h enum mig_rpcomm_cmd { · line 57
Quoted 2 lines
include/migration/migration.h +2
-0+}; +
Also please do not abbrev words, and add a typedef that matches the struct if it is useful. If it is not, just embed the struct without giving the type a name (struct { } rp_state).
migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264
Quoted 12 lines
migration.c +12
-0+static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIG_STATE_ACTIVE : +case MIG_STATE_SETUP : +return true; + +default : +return false; + + } +}
Should CANCELLING also be considered active? It is on the source->dest path.
migration.c int64_t migrate_xbzrle_cache_size(void) · line 829
Quoted 13 lines
migration.c +12
-0+static void await_outgoing_return_path_close (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place) +*/ +if (qemu_file_get_error (ms->file) && ms->return_path) { +qemu_file_shutdown (ms->return_path); + }
As mentioned early, I think it's simpler to let these function handle themselves the case where there is no return path, and call them unconditionally.
Paolo
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 8 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
include/migration/migration.h enum mig_rpcomm_cmd { · line 55
Quoted 5 lines
include/migration/migration.h +5
-0+/* Source side RP state */ +struct MigrationRetPathState { +uint32_t latest_ack; +QemuThread rp_thread; +bool error;Should the QemuFile be in here?
Yes, done.
Quoted 6 lines
include/migration/migration.h enum mig_rpcomm_cmd { · line 57
Quoted 2 lines
include/migration/migration.h +2
-0+}; +Also please do not abbrev words, and add a typedef that matches the struct if it is useful. If it is not, just embed the struct without giving the type a name (struct { }
rp_state).
Done.
Quoted 16 lines
migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264
Quoted 13 lines
migration.c +12
-0+static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIG_STATE_ACTIVE : +case MIG_STATE_SETUP : +return true; + +default : +return false; + + } +}Should CANCELLING also be considered active? It is on the
source->destpath.
Hmm, possibly - but my intention here was just to round up all of the places that already checked for ACTIVE+SETUP so that I could add POSTCOPY_ACTIVE; only one of those places also checked for CANCELLING, so I left it out.
Quoted 8 lines
Quoted 8 lines
migration.c +8
-0+static void await_outgoing_return_path_close (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place)
Quoted 8 lines
migration.c int64_t migrate_xbzrle_cache_size(void) · line 829
Quoted 4 lines
migration.c +4
-0+*/ +if (qemu_file_get_error (ms->file) && ms->return_path) { +qemu_file_shutdown (ms->return_path); + }As mentioned early, I think it's simpler to let these function handle themselves the case where there is no return path, and call them unconditionally.
I still need to think about that.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 15 lines · Dr. David Alan Gilbert
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
migration.c +7
-0+static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIG_STATE_ACTIVE : +case MIG_STATE_SETUP : +return true; +Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264
Quoted 5 lines
migration.c +5
-0+default : +return false; + + } +}Should CANCELLING also be considered active? It is on the
source->destpath.
Quoted 3 lines
Hmm, possibly - but my intention here was just to round up all of the places that already checked for ACTIVE+SETUP so that I could add
POSTCOPY_ACTIVE; only one of those places also checked for CANCELLING, so I left it out.
Ok, I would need to check the callers... There may be bugs waiting to be fixed. :) For now I guess it's ok as is.
Thanks for answering my comments!
Paolo
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)Quoted 189 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 10 +++ migration.c | 181 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 190 insertions(+), 1 deletion(-)include/migration/migration.h +10
-0@@ -47,6 +47,14 @@ enum mig_rpcomm_cmd {MIG_RPCOMM_ACK ,/* data (seq: be32 ) */ MIG_RPCOMM_AFTERLASTVALID }; + +/* Source side RP state */ +struct MigrationRetPathState { +uint32_t latest_ack; +QemuThread rp_thread; +bool error; +}; +typedef struct MigrationState MigrationState ;/* State for the incoming migration */ @@ -69,9 +77,11 @@ struct MigrationStateQemuThread thread;QEMUBH *cleanup_bh;QEMUFile *file; +QEMUFile *return_path;int state;MigrationParams params; +struct MigrationRetPathState rp_state;double mbps;int64_t total_time;int64_t downtime;migration.c +110
-1@@ -246,6 +246,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)return head; } +/* +* Return true if we're already in the middle of a migration +* (i.e. any of the active or setup states) +*/ +static bool migration_already_active (MigrationState *ms) +{ +switch (ms->state) { +case MIG_STATE_ACTIVE : +case MIG_STATE_SETUP : +return true; + +default : +return false; + + } +} +static void get_xbzrle_cache_stats (MigrationInfo *info) {if (migrate_use_xbzrle ()) {@@ -371,6 +388,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state) } } +static void migrate_fd_cleanup_src_rp (MigrationState *ms) +{ +QEMUFile *rp = ms->return_path; + +/* +* When stuff goes wrong (e.g. failing destination) on the rp, it can get +* cleaned up from a few threads; make sure not to do it twice in parallel +*/ + rp =atomic_cmpxchg (&ms->return_path, rp,NULL ); +if (rp) { +DPRINTF ("cleaning up return path\n" ); +qemu_fclose (rp); + } +} +static void migrate_fd_cleanup (void *opaque) {MigrationState *s = opaque;@@ -378,6 +410,8 @@ static void migrate_fd_cleanup(void *opaque)qemu_bh_delete (s->cleanup_bh); s->cleanup_bh =NULL ; +migrate_fd_cleanup_src_rp (s); +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel (); +if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path); + } + do { old_state = s->state; if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {@@ -655,8 +694,148 @@ int64_t migrate_xbzrle_cache_size(void)return s->xbzrle_cache_size; } -/* migration thread support */ +/* +* Something bad happened to the RP stream, mark an error +* The caller shall print something to indicate why +*/ +static void source_return_path_bad (MigrationState *s) +{ + s->rp_state.error = true; +migrate_fd_cleanup_src_rp (s); +} +/* +* Handles messages sent on the return path towards the source VM +* +*/ +static void *source_return_path_thread (void *opaque) +{ +MigrationState *ms = opaque; +QEMUFile *rp = ms->return_path; +uint16_t expected_len, header_len, header_com; +const int max_len =512 ; +uint8_t buf[max_len]; +uint32_t tmp32; +int res; + +DPRINTF ("RP: %s entry" , __func__); +while (rp && !qemu_file_get_error (rp) && +migration_already_active (ms)) { +DPRINTF ("RP: %s top of loop" , __func__); + header_com =qemu_get_be16 (rp); + header_len =qemu_get_be16 (rp); + +switch (header_com) { +case MIG_RPCOMM_SHUT : +case MIG_RPCOMM_ACK : + expected_len =4 ; +break ; + +default : +error_report ("RP: Received invalid cmd 0x%04x length 0x%04x" , + header_com, header_len); +source_return_path_bad (ms); + goto out; + } + +if (header_len > expected_len) { +error_report ("RP: Received command 0x%04x with" +"incorrect length %d expecting %d" , + header_com, header_len, + expected_len); +source_return_path_bad (ms); + goto out; + } + +/* We know we've got a valid header by this point */ + res =qemu_get_buffer (rp, buf, header_len); +if (res != header_len) { +DPRINTF ("RP: Failed to read command data" ); +source_return_path_bad (ms); + goto out; + } + +/* OK, we have the command and the data */ +switch (header_com) { +case MIG_RPCOMM_SHUT : + tmp32 =be32_to_cpup ((uint32_t *)buf); +if (tmp32) { +error_report ("RP: Sibling indicated error %d" , tmp32); +source_return_path_bad (ms); + }else { +DPRINTF ("RP: SHUT received" ); + } +/*
migration.c int64_t migrate_xbzrle_cache_size(void) · line 778
Quoted 10 lines
migration.c +10
-0+* We'll let the main thread deal with closing the RP +* we could do a shutdown(2) on it, but we're the only user +* anyway, so there's nothing gained. +*/ + goto out; + +case MIG_RPCOMM_ACK : + tmp32 =be32_to_cpup ((uint32_t *)buf); +DPRINTF ("RP: Received ACK 0x%x" , tmp32); +atomic_xchg (&ms->rp_state.latest_ack, tmp32);
I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)
Quoted 64 lines
migration.c +60
-0+break ; + +default : +/* This shouldn't happen because we should catch this above */ +DPRINTF ("RP: Bad header_com in dispatch" ); + } +/* Latest command processed, now leave a gap for the next one */ + header_com =MIG_RPCOMM_INVALID ; + } +if (rp &&qemu_file_get_error (rp)) { +DPRINTF ("%s: rp bad at end" , __func__); +source_return_path_bad (ms); + } + +DPRINTF ("%s: Bottom exit" , __func__); + +out: +return NULL ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file); +if (!ms->return_path) { +return -1 ; + } + +DPRINTF ("%s: starting thread" , __func__); +qemu_thread_create (&ms->rp_state.rp_thread,"return path" , + source_return_path_thread, ms,QEMU_THREAD_JOINABLE ); + +DPRINTF ("%s: continuing" , __func__); + +return 0 ; +} + +__attribute__ (( unused ))/* Until later in patch series */ +static void await_outgoing_return_path_close (MigrationState *ms) +{ +/* +* If this is a normal exit then the destination will send a SHUT and the +* rp_thread will exit, however if there's an error we need to cause +* it to exit, which we can do by a shutdown. +* (canceling must also shutdown to stop us getting stuck here if +* the destination died at just the wrong place) +*/ +if (qemu_file_get_error (ms->file) && ms->return_path) { +qemu_file_shutdown (ms->return_path); + } +DPRINTF ("%s: Joining" , __func__); +qemu_thread_join (&ms->rp_state.rp_thread); +DPRINTF ("%s: Exit" , __func__); +} + +/* +* Master migration thread on the source VM. +* It drives the migration and pumps the data down the outgoing channel. +*/ static void *migration_thread (void *opaque) {MigrationState *s = opaque;
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 6 lines · * zhanghailiang ([email protected]) wrote:
migration.c int64_t migrate_xbzrle_cache_size(void) · line 778
Quoted 4 lines
migration.c +4
-0+case MIG_RPCOMM_ACK : + tmp32 =be32_to_cpup ((uint32_t *)buf); +DPRINTF ("RP: Received ACK 0x%x" , tmp32); +atomic_xchg (&ms->rp_state.latest_ack, tmp32);I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)
Nothing currently; I've used the REQ/ACK as debug at the moment; I was thinking
that someone might want to wait on an ack being received before carrying on; but hadn't actually needed it in postcopy.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 5 lines · * zhanghailiang ([email protected]) wrote:
Quoted 4 lines
migration.c +4
-0+case MIG_RPCOMM_ACK : + tmp32 =be32_to_cpup ((uint32_t *)buf); +DPRINTF ("RP: Received ACK 0x%x" , tmp32); +atomic_xchg (&ms->rp_state.latest_ack, tmp32);
Quoted 6 lines
Quoted 1 line
I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)
Nothing currently; I've used the REQ/ACK as debug at the moment; I was thinking
that someone might want to wait on an ack being received before carrying on; but hadn't actually needed it in postcopy.
OK, i see, Thanks.
Quoted 7 lines
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK.
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
In reply to zhanghailiangQuoted 2 lines · zhanghailiang
Quoted 1 line · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[snip]
Quoted 6 lines
migration.c int64_t migrate_xbzrle_cache_size(void) · line 778
Quoted 4 lines
migration.c +4
-0+case MIG_RPCOMM_ACK : + tmp32 =be32_to_cpup ((uint32_t *)buf); +DPRINTF ("RP: Received ACK 0x%x" , tmp32); +atomic_xchg (&ms->rp_state.latest_ack, tmp32);I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)
Also, you don't appear to use tmp32 after that point, so what's the reason for the exchange, rather than just an assignment?
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 10 lines · * David Gibson ([email protected]) wrote:
Quoted 2 lines · zhanghailiang
Quoted 1 line · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
[snip]
Quoted 5 lines
Quoted 4 lines
migration.c +4
-0+case MIG_RPCOMM_ACK : + tmp32 =be32_to_cpup ((uint32_t *)buf); +DPRINTF ("RP: Received ACK 0x%x" , tmp32); +atomic_xchg (&ms->rp_state.latest_ack, tmp32);
Quoted 4 lines
Quoted 1 line
I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)
Also, you don't appear to use tmp32 after that point, so what's the reason for the exchange, rather than just an assignment?
I've killed the 'latest_ack' off; I've kept the DPRINTF (and might turn it into a trace).
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
[snip]
migration.c static void migrate_fd_cancel(MigrationState *s) · line 453
Quoted 7 lines
migration.c +3
-0@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel (); +if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path);
Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.
[snip]
migration.c int64_t migrate_xbzrle_cache_size(void) · line 803
Quoted 5 lines
migration.c +5
-0+__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file);
So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>[snip]
Quoted 4 lines
migration.c +0
-0@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel ();
Quoted 6 lines
migration.c static void migrate_fd_cancel(MigrationState *s) · line 453
Quoted 3 lines
migration.c +3
-0+if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path);Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.
The rp thread might be in the middle of a blocking read()/recv() so I'm doing a shutdown() to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.
Quoted 11 lines
[snip]
migration.c int64_t migrate_xbzrle_cache_size(void) · line 803
Quoted 5 lines
migration.c +5
-0+__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file);So, another reason this
get_return_pathabstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.
However, since the implementation of the get_return_path is a method on the particular implementation, and it can be different for a qemu_file opened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).
Dave
Quoted 5 lines
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>[snip]
Quoted 4 lines
migration.c +0
-0@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel ();Quoted 6 lines
migration.c static void migrate_fd_cancel(MigrationState *s) · line 453
Quoted 3 lines
migration.c +3
-0+if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path);Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.
Quoted 3 lines
The rp thread might be in the middle of a blocking read()/recv() so I'm doing a
shutdown()to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.
Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise pthread_cancel() should kill a thread, even if it's blocked at the moment.
Quoted 12 lines
Quoted 11 lines
[snip]
migration.c int64_t migrate_xbzrle_cache_size(void) · line 803
Quoted 5 lines
migration.c +5
-0+__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file);So, another reason this
get_return_pathabstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.
Quoted 4 lines
However, since the implementation of the
get_return_pathis a method on the particular implementation, and it can be different for aqemu_fileopened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).
So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.
The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.
The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.
The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 18 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>[snip]
migration.c static void migrate_fd_cancel(MigrationState *s) · line 453
Quoted 7 lines
migration.c +3
-0@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel (); +if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path);Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.
Quoted 7 lines
Quoted 3 lines
The rp thread might be in the middle of a blocking read()/recv() so I'm doing a
shutdown()to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise
pthread_cancel()should kill a thread, even if it's blocked at the moment.
It was Paolo's idea to use shutdown() and I agree - it works well; I'd originally thought about using pthread_cancel but it seemed to be generally disliked - you have to be very careful to either know exactly the points at which it might be killed (if you use the deferred version) or be prepared to deal with your thread disappearing at any time and ensure your data structures are always consistent. In addition there was some concern that there was no Windows equivalent to pthread_cancel.
Quoted 17 lines
Quoted 12 lines
Quoted 11 lines
[snip]
migration.c int64_t migrate_xbzrle_cache_size(void) · line 803
Quoted 5 lines
migration.c +5
-0+__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file);So, another reason this
get_return_pathabstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.Quoted 4 lines
However, since the implementation of the
get_return_pathis a method on the particular implementation, and it can be different for aqemu_fileopened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).
Quoted 4 lines
So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.
Yes, but hey we've got FILE* in C anyway, so it might be bad, but it's not inconsitent.
Quoted 2 lines
The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.
I think the bit that's specific, is as you say that I don't know whether I need it until later.
Quoted 5 lines
The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.
I'm OK with some transports not supporting this; I check for it and error out. At a higher level I do send an 'open_return_path' command from src->dest early on to say I'm going to want a return path, I guess a pipe might be able to open that fd then and pass it back over the original fd? But that might be hairy.
Quoted 5 lines
The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to
shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.
I'd be OK with setting some flag or similar at the same time if that would help; but I don't think there's a safe posix'y way of killing a thread that might be stuck in a recv()/read() other than shutdown().
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path
Quoted 30 lines · Dr. David Alan Gilbert
Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 18 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>[snip]
migration.c static void migrate_fd_cancel(MigrationState *s) · line 453
Quoted 7 lines
migration.c +3
-0@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)int old_state ;trace_migrate_fd_cancel (); +if (s->return_path) { +/* shutdown the rp socket, so causing the rp thread to shutdown */ +qemu_file_shutdown (s->return_path);Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.
Quoted 7 lines
Quoted 3 lines
The rp thread might be in the middle of a blocking read()/recv() so I'm doing a
shutdown()to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise
pthread_cancel()should kill a thread, even if it's blocked at the moment.
Quoted 7 lines
It was Paolo's idea to use
shutdown()and I agree - it works well; I'd originally thought about usingpthread_cancelbut it seemed to be generally disliked - you have to be very careful to either know exactly the points at which it might be killed (if you use the deferred version) or be prepared to deal with your thread disappearing at any time and ensure your data structures are always consistent. In addition there was some concern that there was no Windows equivalent topthread_cancel.
Hmm, yeah all right.
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
[snip]
migration.c int64_t migrate_xbzrle_cache_size(void) · line 803
Quoted 5 lines
migration.c +5
-0+__attribute__ (( unused ))/* Until later in patch series */ +static int open_outgoing_return_path (MigrationState *ms) +{ + + ms->return_path =qemu_file_get_return_path (ms->file);So, another reason this
get_return_pathabstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.However, since the implementation of the
get_return_pathis a method on the particular implementation, and it can be different for aqemu_fileopened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).
Quoted 7 lines
Quoted 4 lines
So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.
Yes, but hey we've got FILE* in C anyway, so it might be bad, but it's not inconsitent.
Uh.. not following the analogy here, sorry.
Quoted 12 lines
Quoted 2 lines
The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.
I think the bit that's specific, is as you say that I don't know whether I need it until later.
Quoted 5 lines
The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.
Quoted 5 lines
I'm OK with some transports not supporting this; I check for it and error out. At a higher level I do send an '
open_return_path' command fromsrc->destearly on to say I'm going to want a return path, I guess a pipe might be able to open that fd then and pass it back over the original fd? But that might be hairy.
You can pass fds over Unix sockets, but not over pipes AFAIK.
Quoted 12 lines
Quoted 5 lines
The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to
shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.I'd be OK with setting some flag or similar at the same time if that would help; but I don't think there's a safe posix'y way of killing a thread that might be stuck in a recv()/read() other than
shutdown().Dave
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 17/47 qemu_loadvm errors and debug
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Flip many fprintf's to error_report
Add lots of DPRINTF debug in qemu_loadvm*
--- savevm.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
savevm.c +17 -6
@@ -719,6 +719,8 @@ int qemu_savevm_state_iterate(QEMUFile *f)trace_savevm_section_end (se->idstr, se->section_id);if (ret <0 ) { +DPRINTF ("%s: setting error state after iterate on id=%d/%s" , + __func__, se->section_id, se->idstr);qemu_file_set_error (f, ret); }if (ret <=0 ) {
@@ -1019,6 +1021,7 @@ int qemu_loadvm_state(QEMUFile *f)SaveStateEntry *se;char idstr[256 ]; +DPRINTF ("qemu_loadvm_state loop: section_type=%d" , section_type);switch (section_type) {case QEMU_VM_SECTION_START :case QEMU_VM_SECTION_FULL :
@@ -1032,6 +1035,9 @@ int qemu_loadvm_state(QEMUFile *f) instance_id =qemu_get_be32 (f); version_id =qemu_get_be32 (f); +DPRINTF ("qemu_loadvm_state loop START/FULL: id=%d(%s)" , + section_id, idstr); +/* Find savevm section */ se =find_se (idstr, instance_id);if (se ==NULL ) {
@@ -1059,8 +1065,9 @@ int qemu_loadvm_state(QEMUFile *f) ret =vmstate_load (f, le->se, le->version_id);if (ret <0 ) { -fprintf (stderr,"qemu: warning: error while loading state for instance 0x%x of device '%s'\n" , - instance_id, idstr); +error_report ("qemu: error while loading state for" +"instance 0x%x of device '%s'" , + instance_id, idstr); goto out; }break ;
@@ -1068,23 +1075,25 @@ int qemu_loadvm_state(QEMUFile *f)case QEMU_VM_SECTION_END : section_id =qemu_get_be32 (f); +DPRINTF ("QEMU_VM_SECTION_PART/END entry for id=%d" , section_id); QLIST_FOREACH(le, &loadvm_handlers, entry) {if (le->section_id == section_id) {break ; } }if (le ==NULL ) { -fprintf (stderr,"Unknown savevm section %d\n" , section_id); +error_report ("Unknown savevm section %d" , section_id); ret = -EINVAL ; goto out; } ret =vmstate_load (f, le->se, le->version_id);if (ret <0 ) { -fprintf (stderr,"qemu: warning: error while loading state section id %d\n" , - section_id); +error_report ("qemu: error while loading state section" +" id %d (%s)" , section_id, le->se->idstr); goto out; } +DPRINTF ("QEMU_VM_SECTION_PART/END done for id=%d" , section_id);break ;case QEMU_VM_COMMAND : ret =loadvm_process_command (f);
@@ -1093,11 +1102,12 @@ int qemu_loadvm_state(QEMUFile *f) }break ;default : -fprintf (stderr,"Unknown savevm section type %d\n" , section_type); +error_report ("Unknown savevm section type %d" , section_type); ret = -EINVAL ; goto out; } } +DPRINTF ("qemu_loadvm_state loop: exited loop" );cpu_synchronize_all_post_init ();
@@ -1113,6 +1123,7 @@ out: ret =qemu_file_get_error (f); } +DPRINTF ("qemu_loadvm_state out: ret=%d" , ret);return ret; }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 17/47] qemu_loadvm errors and debug
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Flip many fprintf's to
error_report
Add lots of DPRINTF debug in qemu_loadvm*Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 18/47 ram_debug_dump_bitmap: Dump a migration bitmap as text
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.
--- arch_init.c | 39 +++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 1 + 2 files changed, 40 insertions(+)
arch_init.c +39 -0
@@ -769,6 +769,45 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of and it won't bother printing lines that are all this value +* if 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128l ; +char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + } + +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur+linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur+curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ; + found |= (thisbit ^ expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +} +static int ram_save_setup (QEMUFile *f,void *opaque) {RAMBlock *block;
include/migration/migration.h +1 -0
@@ -157,6 +157,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 18/47] ram_debug_dump_bitmap: Dump a migration bitmap as text
Quoted 20 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 39 +++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 1 + 2 files changed, 40 insertions(+)arch_init.c +1
-0@@ -769,6 +769,45 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */ +/*
arch_init.c static void reset_ram_globals(void) · line 782
Quoted 10 lines
arch_init.c +10
-0+* 'expected' is the value you expect the bitmap mostly to be full +* of and it won't bother printing lines that are all this value +* if 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128l ;
I don't think there's any point to the 'l' there. "long" isn't necessarily correct for an int64_t, and normal type promotion should get this right anyway.
Assuming the user has a >128 character wide terminal seems a little obnoxious, too.
Quoted 10 lines
arch_init.c +10
-0+char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + } + +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/*
arch_init.c static void reset_ram_globals(void) · line 802
Quoted 10 lines
arch_init.c +10
-0+* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur+linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur+curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ; + found |= (thisbit ^ expected);
I guess this will have the right result with the obvious encoding of a bool, but I don't think it's conceptually correct. It should be logical, not bitwise operations so:
found = found || (thisbit != expected);
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 18/47] ram_debug_dump_bitmap: Dump a migration bitmap as text
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 39 +++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 1 + 2 files changed, 40 insertions(+)arch_init.c +0
-0@@ -769,6 +769,45 @@ static void reset_ram_globals(void)#define MAX_WAIT 50/* ms, half buffered_file limit */
Quoted 15 lines
arch_init.c static void reset_ram_globals(void) · line 782
Quoted 11 lines
arch_init.c +11
-0+/* +* 'expected' is the value you expect the bitmap mostly to be full +* of and it won't bother printing lines that are all this value +* if 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128l ;I don't think there's any point to the 'l' there. "long" isn't necessarily correct for an
int64_t, and normal type promotion should get this right anyway.
Fixed.
Quoted 2 lines
Assuming the user has a >128 character wide terminal seems a little obnoxious, too.
This is a debug routine primarily to help me, but it seems right to leave it in for the next person who has to debug it; it's easy enough for them to tweak it to whatever their preference is.
Quoted 24 lines
Quoted 10 lines
arch_init.c +10
-0+char linebuf[129 ]; + +if (!todump) { + todump = migration_bitmap; + } + +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* arch_init.c static void reset_ram_globals(void) · line 802
Quoted 10 lines
arch_init.c +10
-0+* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur+linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur+curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ; + found |= (thisbit ^ expected);I guess this will have the right result with the obvious encoding of a bool, but I don't think it's conceptually correct. It should be logical, not bitwise operations so:
Quoted 1 line
found = found || (thisbit != expected);
Fixed.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 19/47 Rework loadvm path for subloops
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Split the core of qemu_loadvm_state out so we can use it for both.
Allow the inner loadvm loop to quit and signal whether the parent should.
loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.
--- savevm.c | 136 +++++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 84 insertions(+), 52 deletions(-)
savevm.c +84 -52
@@ -915,6 +915,26 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)return NULL ; } +/* These are ORable flags */ +const int LOADVM_EXITCODE_QUITLOOP =1 ; +const int LOADVM_EXITCODE_QUITPARENT =2 ; +const int LOADVM_EXITCODE_KEEPHANDLERS =4 ; + +typedef struct LoadStateEntry { +QLIST_ENTRY (LoadStateEntry) entry; +SaveStateEntry *se; +int section_id; +int version_id; +}LoadStateEntry ; + +typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head; + +static LoadStateEntry_Head loadvm_handlers = +QLIST_HEAD_INITIALIZER (loadvm_handlers); + +static int qemu_loadvm_state_main (QEMUFile *f, +LoadStateEntry_Head *loadvm_handlers); +static int loadvm_process_command_simple_lencheck (const char *name,unsigned int actual,unsigned int expected)
@@ -931,8 +951,11 @@ static int loadvm_process_command_simple_lencheck(const char *name,/* * Process an incoming 'QEMU_VM_COMMAND' * negative return on error (will issue error message) +* 0 just a normal return +* 1 All good, but exit the loop */ -static int loadvm_process_command (QEMUFile *f) +static int loadvm_process_command (QEMUFile *f, +LoadStateEntry_Head *loadvm_handlers) {MigrationIncomingState *mis =migration_incoming_get_current ();uint16_t com;
@@ -982,39 +1005,13 @@ static int loadvm_process_command(QEMUFile *f)return 0 ; } -typedef struct LoadStateEntry { -QLIST_ENTRY (LoadStateEntry) entry; -SaveStateEntry *se; -int section_id; -int version_id; -}LoadStateEntry ; - -int qemu_loadvm_state (QEMUFile *f) +static int qemu_loadvm_state_main (QEMUFile *f, +LoadStateEntry_Head *loadvm_handlers) { - QLIST_HEAD(, LoadStateEntry) loadvm_handlers = -QLIST_HEAD_INITIALIZER (loadvm_handlers); -LoadStateEntry *le, *new_le; +LoadStateEntry *le;uint8_t section_type; -unsigned int v;int ret; - -if (qemu_savevm_state_blocked (NULL )) { -return -EINVAL ; - } - - v =qemu_get_be32 (f); -if (v !=QEMU_VM_FILE_MAGIC ) { -return -EINVAL ; - } - - v =qemu_get_be32 (f); -if (v ==QEMU_VM_FILE_VERSION_COMPAT ) { -error_report ("SaveVM v2 format is obsolete and don't work anymore" ); -return -ENOTSUP ; - } -if (v !=QEMU_VM_FILE_VERSION ) { -return -ENOTSUP ; - } +int exitcode =0 ;while ((section_type =qemu_get_byte (f)) !=QEMU_VM_EOF ) {uint32_t instance_id, version_id, section_id;
@@ -1043,16 +1040,14 @@ int qemu_loadvm_state(QEMUFile *f)if (se ==NULL ) {error_report ("Unknown savevm section or instance '%s' %d" , idstr, instance_id); - ret = -EINVAL ; - goto out; +return -EINVAL ; }/* Validate version */ if (version_id > se->version_id) {error_report ("savevm: unsupported version %d for '%s' v%d" , version_id, idstr, se->version_id); - ret = -EINVAL ; - goto out; +return -EINVAL ; }/* Add entry */
@@ -1061,14 +1056,14 @@ int qemu_loadvm_state(QEMUFile *f) le->se = se; le->section_id = section_id; le->version_id = version_id; -QLIST_INSERT_HEAD (&loadvm_handlers, le, entry); +QLIST_INSERT_HEAD (loadvm_handlers, le, entry); ret =vmstate_load (f, le->se, le->version_id);if (ret <0 ) {error_report ("qemu: error while loading state for" "instance 0x%x of device '%s'" , instance_id, idstr); - goto out; +return ret; }break ;case QEMU_VM_SECTION_PART :
@@ -1076,47 +1071,84 @@ int qemu_loadvm_state(QEMUFile *f) section_id =qemu_get_be32 (f);DPRINTF ("QEMU_VM_SECTION_PART/END entry for id=%d" , section_id); - QLIST_FOREACH(le, &loadvm_handlers, entry) { + QLIST_FOREACH(le, loadvm_handlers, entry) {if (le->section_id == section_id) {break ; } }if (le ==NULL ) {error_report ("Unknown savevm section %d" , section_id); - ret = -EINVAL ; - goto out; +return -EINVAL ; } ret =vmstate_load (f, le->se, le->version_id);if (ret <0 ) {error_report ("qemu: error while loading state section" " id %d (%s)" , section_id, le->se->idstr); - goto out; +return ret; }DPRINTF ("QEMU_VM_SECTION_PART/END done for id=%d" , section_id);break ;case QEMU_VM_COMMAND : - ret =loadvm_process_command (f); -if (ret <0 ) { - goto out; + ret =loadvm_process_command (f, loadvm_handlers); +DPRINTF ("%s QEMU_VM_COMMAND ret: %d" , __func__, ret); +if ((ret <0 ) || (ret &LOADVM_EXITCODE_QUITLOOP )) { +return ret; } + exitcode |= ret;/* Lets us pass flags up to the parent */ break ;default :error_report ("Unknown savevm section type %d" , section_type); - ret = -EINVAL ; - goto out; +return -EINVAL ; } }DPRINTF ("qemu_loadvm_state loop: exited loop" ); -cpu_synchronize_all_post_init (); +if (exitcode &LOADVM_EXITCODE_QUITPARENT ) { +DPRINTF ("loadvm_handlers_state_main: End of loop with QUITPARENT" ); + exitcode &= ~LOADVM_EXITCODE_QUITPARENT ; + exitcode &=LOADVM_EXITCODE_QUITLOOP ; + } + +return exitcode; +} + +int qemu_loadvm_state (QEMUFile *f) +{ +LoadStateEntry *le, *new_le; +unsigned int v; +int ret; + +if (qemu_savevm_state_blocked (NULL )) { +return -EINVAL ; + } + + v =qemu_get_be32 (f); +if (v !=QEMU_VM_FILE_MAGIC ) { +return -EINVAL ; + } - ret =0 ; + v =qemu_get_be32 (f); +if (v ==QEMU_VM_FILE_VERSION_COMPAT ) { +error_report ("SaveVM v2 format is obsolete and don't work anymore" ); +return -ENOTSUP ; + } +if (v !=QEMU_VM_FILE_VERSION ) { +return -ENOTSUP ; + } + +QLIST_INIT (&loadvm_handlers); + ret =qemu_loadvm_state_main (f, &loadvm_handlers); -out: - QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) { -QLIST_REMOVE (le, entry); -g_free (le); +if (ret ==0 ) { +cpu_synchronize_all_post_init (); + } + +if ((ret <0 ) || !(ret &LOADVM_EXITCODE_KEEPHANDLERS )) { + QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) { +QLIST_REMOVE (le, entry); +g_free (le); + } }if (ret ==0 ) {
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
+/* These are ORable flags */
... make them an "enum".
savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 920
Quoted 2 lines
savevm.c +2
-0+const int LOADVM_EXITCODE_QUITLOOP =1 ; +const int LOADVM_EXITCODE_QUITPARENT =2 ;
LOADVM_QUIT_ALL, LOADVM_QUIT respectively?
Review
Quoted 2 lines
(quoted diff) +2
-0+const int LOADVM_EXITCODE_KEEPHANDLERS = 4; +
Is it more common to drop or keep handlers?
In either case, please add a comment to the three constants that details how to use them. In particular, please document why you should drop (resp. keep) handlers...
Is it by chance that they are only used in savevm.c? Should they be moved to a header file?
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110
Quoted 5 lines
savevm.c +4
-0+if (exitcode &LOADVM_EXITCODE_QUITPARENT ) { +DPRINTF ("loadvm_handlers_state_main: End of loop with QUITPARENT" ); + exitcode &= ~LOADVM_EXITCODE_QUITPARENT ; + exitcode &=LOADVM_EXITCODE_QUITLOOP ;
Either you want |=, or the first &= is useless.
Paolo
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Quoted 5 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 2 lines
+/* These are ORable flags */
... make them an "enum".
OK, will do - I'd generally tended to avoid using enum for things that were ORable where the combinations weren't themselves members of the enum; but I can do that.
Quoted 10 lines
savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 920
Quoted 2 lines
savevm.c +2
-0+const int LOADVM_EXITCODE_QUITLOOP =1 ; +const int LOADVM_EXITCODE_QUITPARENT =2 ;
LOADVM_QUIT_ALL,LOADVM_QUITrespectively?Review
Quoted 2 lines
(quoted diff) +2
-0+const int LOADVM_EXITCODE_KEEPHANDLERS = 4; +Is it more common to drop or keep handlers?
I'ts more common to drop them.
Quoted 3 lines
In either case, please add a comment to the three constants that details how to use them. In particular, please document why you should drop (resp. keep) handlers...
Does this make it clearer:
/* ORable flags that control the (potentially nested) loadvm_state loops */ enum LoadVMExitCodes {
/* Quit the loop level that received this command */
LOADVM_QUIT_LOOP = 1,
/* Quit this loop and our parent */
LOADVM_QUIT_PARENT = 2,
/*
* Keep the LoadStateEntry handler list after the loop exits,
* because they're being used in another thread.
*/
LOADVM_KEEP_HANDLERS = 4,
};
Quoted 2 lines
Is it by chance that they are only used in
savevm.c? Should they be moved to a header file?
They're local.
Quoted 6 lines
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110
Quoted 4 lines
savevm.c +4
-0+if (exitcode &LOADVM_EXITCODE_QUITPARENT ) { +DPRINTF ("loadvm_handlers_state_main: End of loop with QUITPARENT" ); + exitcode &= ~LOADVM_EXITCODE_QUITPARENT ; + exitcode &=LOADVM_EXITCODE_QUITLOOP ;Either you want |=, or the first &= is useless.
Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.
The idea is we have:
1 outer loadvm_state loop
2 receives packaged command
3 inner_loadvm_state loop
4 receives handle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits
so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.
Dave
Quoted 2 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Il 07/10/2014 10:58, Dr. David Alan Gilbert ha scritto:
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110
Quoted 5 lines
savevm.c +4
-0+if (exitcode &LOADVM_EXITCODE_QUITPARENT ) { +DPRINTF ("loadvm_handlers_state_main: End of loop with QUITPARENT" ); + exitcode &= ~LOADVM_EXITCODE_QUITPARENT ; + exitcode &=LOADVM_EXITCODE_QUITLOOP ;Either you want |=, or the first &= is useless.
Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.
The idea is we have:
1 outerloadvm_stateloop
2 receives packaged command
3inner_loadvm_stateloop
4 receiveshandle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits
Quoted 3 lines
so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.
Yes, that was my understanding as well.
We have only two nested loops, but if we had three, should it be QUIT_PARENT or QUIT_ALL?
Paolo
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Quoted 21 lines · * Paolo Bonzini ([email protected]) wrote:
Il 07/10/2014 10:58, Dr. David Alan Gilbert ha scritto:
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110
Quoted 5 lines
savevm.c +4
-0+if (exitcode &LOADVM_EXITCODE_QUITPARENT ) { +DPRINTF ("loadvm_handlers_state_main: End of loop with QUITPARENT" ); + exitcode &= ~LOADVM_EXITCODE_QUITPARENT ; + exitcode &=LOADVM_EXITCODE_QUITLOOP ;Either you want |=, or the first &= is useless.
Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.
The idea is we have:
1 outerloadvm_stateloop
2 receives packaged command
3inner_loadvm_stateloop
4 receiveshandle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits
Quoted 8 lines
Quoted 3 lines
so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.
Yes, that was my understanding as well.
We have only two nested loops, but if we had three, should it be
QUIT_PARENTorQUIT_ALL?
The answer probably depends on why you've got 3 nested loops; either way is a bit of guesswork about what some potential future user wants to do.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
In reply to Dr. David Alan Gilbert (git)Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 7 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and signal whether the parent should.
loadvm_handlersis made static since it's lifetime is greater than the outerqemu_loadvm_state.
Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.
It's also not clear from this patch alone why the lifetime of loadvm_handlers now needs to exceed that of qemu_loadvm_state().
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Quoted 15 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 7 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and signal whether the parent should.
loadvm_handlersis made static since it's lifetime is greater than the outerqemu_loadvm_state.
Quoted 7 lines
Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.
It's also not clear from this patch alone why the lifetime of
loadvm_handlersnow needs to exceed that ofqemu_loadvm_state().
OK, how about if I reworked that last sentence to be:
loadvm_handlers is made module-global to survive beyond the lifetime of the outer qemu_loadvm_state since it may still be in use by a subloop in the postcopy listen thread.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 7 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and signal whether the parent should.
loadvm_handlersis made static since it's lifetime is greater than the outerqemu_loadvm_state.Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.
Quoted 8 lines
Quoted 2 lines
It's also not clear from this patch alone why the lifetime of
loadvm_handlersnow needs to exceed that ofqemu_loadvm_state().OK, how about if I reworked that last sentence to be:
loadvm_handlersis made module-global to survive beyond the lifetime of the outerqemu_loadvm_statesince it may still be in use by a subloop in the postcopy listen thread.
Yeah, that's better. A global seems ugly though. Would it be better to dynamically allocate the list head and pass a pointer into the listen thread, or even to pass the list head by value into the listen thread.
The individual list elements need to be cleaned up at some point anyway, so I don't think that introduces any lifetime questions that weren't already there.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops
Quoted 31 lines · * David Gibson ([email protected]) wrote:
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Quoted 7 lines
Split the core of
qemu_loadvm_stateout so we can use it for both.Allow the inner loadvm loop to quit and signal whether the parent should.
loadvm_handlersis made static since it's lifetime is greater than the outerqemu_loadvm_state.Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.
Quoted 8 lines
Quoted 2 lines
It's also not clear from this patch alone why the lifetime of
loadvm_handlersnow needs to exceed that ofqemu_loadvm_state().OK, how about if I reworked that last sentence to be:
loadvm_handlersis made module-global to survive beyond the lifetime of the outerqemu_loadvm_statesince it may still be in use by a subloop in the postcopy listen thread.
Quoted 8 lines
Yeah, that's better. A global seems ugly though. Would it be better to dynamically allocate the list head and pass a pointer into the listen thread, or even to pass the list head by value into the listen thread.
The individual list elements need to be cleaned up at some point anyway, so I don't think that introduces any lifetime questions that weren't already there.
I've moved the loadvm_handlers out into the MigrationIncomingState structure, and free them when that is deallocated at the end of migration.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 20/47 Add migration-capability boolean for postcopy-ram.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- include/migration/migration.h | 1 + migration.c | 9 +++++++++ qapi-schema.json | 6 +++++- 3 files changed, 15 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
@@ -173,6 +173,7 @@ void migrate_add_blocker(Error *reason);*/ void migrate_del_blocker (Error *reason); +bool migrate_postcopy_ram (void );bool migrate_rdma_pin_all (void );bool migrate_zero_blocks (void );
migration.c +9 -0
@@ -658,6 +658,15 @@ bool migrate_rdma_pin_all(void)return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL ]; } +bool migrate_postcopy_ram (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM ]; +} +bool migrate_auto_converge (void ) {MigrationState *s;
qapi-schema.json +5 -1
@@ -494,10 +494,14 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has been +# migrated, pulling the remaining pages along as needed. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.2) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] } + 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] } ## # @MigrationCapabilityStatus
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 20/47] Add migration-capability boolean for postcopy-ram.
Quoted 12 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>--- include/migration/migration.h | 1 + migration.c | 9 +++++++++ qapi-schema.json | 6 +++++- 3 files changed, 15 insertions(+), 1 deletion(-)#
qapi-schema.json line 504
Quoted 9 lines
qapi-schema.json +5
-1+# @x-postcopy-ram: Start executing on the migration target before all of RAM has been +# migrated, pulling the remaining pages along as needed. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.2) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] } + 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] }
Can we wrap this to keep things in 80 columns? Also, the question was raised on the libvirt list on whether the interface is stable enough to name this 'postcopy-ram' from the get-go (rather than marking the interface experimental), so that libvirt can start using it sooner.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v4 20/47] Add migration-capability boolean for postcopy-ram.
Quoted 12 lines · * Eric Blake ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>--- include/migration/migration.h | 1 + migration.c | 9 +++++++++ qapi-schema.json | 6 +++++- 3 files changed, 15 insertions(+), 1 deletion(-)
Quoted 12 lines
qapi-schema.json line 504
Quoted 10 lines
#
qapi-schema.json +5
-1+# @x-postcopy-ram: Start executing on the migration target before all of RAM has been +# migrated, pulling the remaining pages along as needed. NOTE: If the +# migration fails during postcopy the VM will fail. (since 2.2) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] } + 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] }Can we wrap this to keep things in 80 columns?
Done.
Quoted 4 lines
Also, the question was
raised on the libvirt list on whether the interface is stable enough to name this 'postcopy-ram' from the get-go (rather than marking the interface experimental), so that libvirt can start using it sooner.
I'm still nervous about that, what I intend to do is add one patch at the end of the series that removes the x- so that can get discussed separately.
While I'm confident that the interface to libvirt is stable, removing the x- declares that the whole thing is stable and I then have to maintain migration compatibility; and it seemed sensible to let people try it for a release; however if libvirt have no way to support QEMUs ability to have experimental features, I guess no one is actually going to try it, which is very disappointing.
Dave
Quoted 4 lines
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 21/47 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add state variable showing current incoming postcopy state.
--- include/migration/migration.h | 8 + include/sysemu/sysemu.h | 20 +++ savevm.c | 335 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 363 insertions(+)
include/migration/migration.h +8 -0
@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;struct MigrationIncomingState {QEMUFile *file; +volatile enum { +POSTCOPY_RAM_INCOMING_NONE =0 ,/* Initial state - no postcopy */ +POSTCOPY_RAM_INCOMING_ADVISE , +POSTCOPY_RAM_INCOMING_LISTENING , +POSTCOPY_RAM_INCOMING_RUNNING , +POSTCOPY_RAM_INCOMING_END + } postcopy_ram_state; +QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ };
include/sysemu/sysemu.h +20 -0
@@ -88,6 +88,16 @@ enum qemu_vm_cmd {QEMU_VM_CMD_OPENRP ,/* Tell the dest to open the Return path */ QEMU_VM_CMD_REQACK ,/* Request an ACK on the RP */ +QEMU_VM_CMD_POSTCOPY_RAM_ADVISE =20 ,/* Prior to any page transfers, just +warn we might want to do PC */ +QEMU_VM_CMD_POSTCOPY_RAM_DISCARD ,/* A list of pages to discard that +were previously sent during +precopy but are dirty. */ +QEMU_VM_CMD_POSTCOPY_RAM_LISTEN ,/* Start listening for incoming +pages as it's running. */ +QEMU_VM_CMD_POSTCOPY_RAM_RUN ,/* Start execution */ +QEMU_VM_CMD_POSTCOPY_RAM_END ,/* Postcopy is finished. */ + QEMU_VM_CMD_AFTERLASTVALID };
@@ -102,6 +112,16 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);void qemu_savevm_send_openrp (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_advise (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len,uint8_t offset, +uint64_t *addrlist, +uint32_t *masklist); + +void qemu_savevm_send_postcopy_ram_listen (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_run (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_end (QEMUFile *f,uint8_t status); +int qemu_loadvm_state (QEMUFile *f);/* SLIRP */
savevm.c +335 -0
365 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add state variable showing current incoming postcopy state.
This appears to implement a lot more than just adding a state variable...
Quoted 12 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 8 + include/sysemu/sysemu.h | 20 +++ savevm.c | 335 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 363 insertions(+)include/migration/migration.h +0
-0
include/migration/migration.h typedef struct MigrationState MigrationState; · line 64
Quoted 4 lines
include/migration/migration.h +1
-0@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;struct MigrationIncomingState {QEMUFile *file; +volatile enum {
What's the reason for the volatile? I think that really needs a comment.
Quoted 73 lines
include/migration/migration.h +7
-0+POSTCOPY_RAM_INCOMING_NONE =0 ,/* Initial state - no postcopy */ +POSTCOPY_RAM_INCOMING_ADVISE , +POSTCOPY_RAM_INCOMING_LISTENING , +POSTCOPY_RAM_INCOMING_RUNNING , + POSTCOPY_RAM_INCOMING_END + } postcopy_ram_state; +QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ };include/sysemu/sysemu.h +20
-0@@ -88,6 +88,16 @@ enum qemu_vm_cmd {QEMU_VM_CMD_OPENRP ,/* Tell the dest to open the Return path */ QEMU_VM_CMD_REQACK ,/* Request an ACK on the RP */ +QEMU_VM_CMD_POSTCOPY_RAM_ADVISE =20 ,/* Prior to any page transfers, just +warn we might want to do PC */ +QEMU_VM_CMD_POSTCOPY_RAM_DISCARD ,/* A list of pages to discard that +were previously sent during +precopy but are dirty. */ +QEMU_VM_CMD_POSTCOPY_RAM_LISTEN ,/* Start listening for incoming +pages as it's running. */ +QEMU_VM_CMD_POSTCOPY_RAM_RUN ,/* Start execution */ +QEMU_VM_CMD_POSTCOPY_RAM_END ,/* Postcopy is finished. */ + QEMU_VM_CMD_AFTERLASTVALID };@@ -102,6 +112,16 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);void qemu_savevm_send_openrp (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_advise (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len,uint8_t offset, +uint64_t *addrlist, +uint32_t *masklist); + +void qemu_savevm_send_postcopy_ram_listen (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_run (QEMUFile *f); +void qemu_savevm_send_postcopy_ram_end (QEMUFile *f,uint8_t status); +int qemu_loadvm_state (QEMUFile *f);/* SLIRP */ savevm.c +10
-0@@ -39,6 +39,7 @@#include "exec/memory.h" #include "qmp-commands.h" #include "trace.h" +#include "qemu/bitops.h" #include "qemu/iov.h" #include "block/snapshot.h" #include "block/qapi.h" @@ -624,6 +625,92 @@ void qemu_savevm_send_openrp(QEMUFile *f) {qemu_savevm_command_send (f,QEMU_VM_CMD_OPENRP ,0 ,NULL ); } + +/* Send prior to any RAM transfer */ +void qemu_savevm_send_postcopy_ram_advise (QEMUFile *f) +{ +DPRINTF ("send postcopy-ram-advise" ); +uint64_t tmp[2 ]; + tmp[0 ] =cpu_to_be64 (sysconf (_SC_PAGESIZE)); + tmp[1 ] =cpu_to_be64 (1ul <<qemu_target_page_bits ()); +
savevm.c void qemu_savevm_send_openrp(QEMUFile *f) · line 646
Quoted 10 lines
savevm.c +10
-0+qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_ADVISE ,16 , + (uint8_t *)tmp); +} + +/* Prior to running, to cause pages that have been dirtied after precopy +* started to be discarded on the destination. +* CMD_POSTCOPY_RAM_DISCARD consist of: +* 3 byte header (filled in by qemu_savevm_send_postcopy_ram_discard) +* byte version (0) +* byte offset into the 1st data word containing 1st page of RAMBlock
I'm not able to follow what that description means.
Quoted 110 lines
savevm.c +103
-0+* byte Length of name field +* n x byte RAM block name (NOT 0 terminated) +* n x +* be64 Page addresses for start of an invalidation range +* be32 mask of 32 pages, '1' to discard' +* +* Hopefully this is pretty sparse so we don't get too many entries, +* and using the mask should deal with most pagesize differences +* just ending up as a single full mask +* +* The mask is always 32bits irrespective of the long size +* +* name: RAMBlock name that these entries are part of +* len: Number of page entries +* addrlist: 'len' addresses +* masklist: 'len' masks (corresponding to the addresses) +*/ +void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len,uint8_t offset, +uint64_t *addrlist, +uint32_t *masklist) +{ +uint8_t *buf; +uint16_t tmplen; +uint16_t t; + +DPRINTF ("send postcopy-ram-discard" ); + buf =g_malloc0 (len*12 +strlen (name) +3 ); + buf[0 ] =0 ;/* Version */ + buf[1 ] = offset; +assert (strlen (name) <256 ); + buf[2 ] =strlen (name); +memcpy (buf+3 , name,strlen (name)); + tmplen =3 +strlen (name); + +for (t =0 ; t < len; t++) { +cpu_to_be64w ((uint64_t *)(buf + tmplen), addrlist[t]); + tmplen +=8 ; +cpu_to_be32w ((uint32_t *)(buf + tmplen), masklist[t]); + tmplen +=4 ; + } +qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_DISCARD , + tmplen, buf); +g_free (buf); +} + +/* Get the destination into a state where it can receive page data. */ +void qemu_savevm_send_postcopy_ram_listen (QEMUFile *f) +{ +DPRINTF ("send postcopy-ram-listen" ); +qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_LISTEN ,0 ,NULL ); +} + +/* Kick the destination into running */ +void qemu_savevm_send_postcopy_ram_run (QEMUFile *f) +{ +DPRINTF ("send postcopy-ram-run" ); +qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_RUN ,0 ,NULL ); +} + +/* End of postcopy - with a status byte; 0 is good, anything else is a fail */ +void qemu_savevm_send_postcopy_ram_end (QEMUFile *f,uint8_t status) +{ +DPRINTF ("send postcopy-ram-end" ); +qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_END ,1 , &status); +} +bool qemu_savevm_state_blocked (Error **errp) {SaveStateEntry *se;@@ -935,6 +1022,220 @@ static LoadStateEntry_Head loadvm_handlers =static int qemu_loadvm_state_main (QEMUFile *f,LoadStateEntry_Head *loadvm_handlers); +/* ------ incoming postcopy-ram messages ------ */ +/* 'advise' arrives before any RAM transfers just to tell us that a postcopy +* *might* happen - it might be skipped if precopy transferred everything +* quickly. +*/ +static int loadvm_postcopy_ram_handle_advise (MigrationIncomingState *mis, +uint64_t remote_hps, +uint64_t remote_tps) +{ +DPRINTF ("%s" , __func__); +if (mis->postcopy_ram_state !=POSTCOPY_RAM_INCOMING_NONE ) { +error_report ("CMD_POSTCOPY_RAM_ADVISE in wrong postcopy state (%d)" , + mis->postcopy_ram_state); +return -1 ; + } + +if (remote_hps !=sysconf (_SC_PAGESIZE)) { +/* +* Some combinations of mismatch are probably possible but it gets +* a bit more complicated. In particular we need to place whole +* host pages on the dest at once, and we need to ensure that we +* handle dirtying to make sure we never end up sending part of +* a hostpage on it's own. +*/ +error_report ("Postcopy needs matching host page sizes (s=%d d=%d)" , + (int )remote_hps, (int )sysconf (_SC_PAGESIZE)); +return -1 ; + } + +if (remote_tps != (1ul <<qemu_target_page_bits ())) { +/* +* Again, some differences could be dealt with, but for now keep it +* simple. +*/ +error_report ("Postcopy needs matching target page sizes (s=%d d=%d)" , + (int )remote_tps,1 <<qemu_target_page_bits ());
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1070
Quoted 10 lines
savevm.c +10
-0+return -1 ; + } + + mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_ADVISE ; + +/* +* Postcopy will be sending lots of small messages along the return path +* that it needs quick answers to. +*/ +socket_set_nodelay (qemu_get_fd (mis->return_path));
So, here you break the QEMUFile abstraction and assume you have a socket.
Quoted 52 lines
savevm.c +52
-0+return 0 ; +} + +/* After postcopy we will be told to throw some pages away since they're +* dirty and will have to be demand fetched. Must happen before CPU is +* started. +* There can be 0..many of these messages, each encoding multiple pages. +* Bits set in the message represent a page in the source VMs bitmap, but +* since the guest/target page sizes can be different on s/d then we have +* to convert. +*/ +static int loadvm_postcopy_ram_handle_discard (MigrationIncomingState *mis, +uint16_t len) +{ +int tmp; +unsigned int first_bit_offset; +char ramid[256 ]; + +DPRINTF ("%s" , __func__); + +if (mis->postcopy_ram_state !=POSTCOPY_RAM_INCOMING_ADVISE ) { +error_report ("CMD_POSTCOPY_RAM_DISCARD in wrong postcopy state (%d)" , + mis->postcopy_ram_state); +return -1 ; + } +/* We're expecting a +* 3 byte header, +* a RAM ID string +* then at least 1 12 byte chunks +*/ +if (len <16 ) { +error_report ("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)" , len); +return -1 ; + } + + tmp =qemu_get_byte (mis->file); +if (tmp !=0 ) { +error_report ("CMD_POSTCOPY_RAM_DISCARD invalid version (%d)" , tmp); +return -1 ; + } + first_bit_offset =qemu_get_byte (mis->file); + +if (qemu_get_counted_string (mis->file, (uint8_t *)ramid)) { +error_report ("CMD_POSTCOPY_RAM_DISCARD Failed to read RAMBlock ID" ); +return -1 ; + } + + len -=3 +strlen (ramid); +if (len %12 ) { +error_report ("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)" , len); +return -1 ; + }
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1133
Quoted 10 lines
savevm.c +10
-0+while (len) { +uint64_t startaddr; +uint32_t mask; +/* +* We now have pairs of address, mask +* The mask is 32 bits of bitmask starting at 'startaddr'-offset +* RAMBlock; e.g. if the RAMBlock started at 8k where TPS=4k +* then first_bit_offset=2 and the 1st 2 bits of the mask +* aren't relevant to this RAMBlock, and bit 2 corresponds +* to the 1st page of this RAMBlock
Um.. yeah.. can't make much snse of this comment either.
Quoted 46 lines
savevm.c +46
-0+*/ + startaddr =qemu_get_be64 (mis->file); + mask =qemu_get_be32 (mis->file); + + len -=12 ; + +while (mask) { +/* mask= .....?10...0 */ +/* ^fs */ +int firstset =ctz32 (mask); + +/* tmp32=.....?11...1 */ +/* ^fs */ +uint32_t tmp32 = mask | ((((uint32_t )1 )<<firstset)-1 ); + +/* mask= .?01..10...0 */ +/* ^fz ^fs */ +int firstzero =cto32 (tmp32); + +if ((startaddr ==0 ) && (firstset < first_bit_offset)) { +error_report ("CMD_POSTCOPY_RAM_DISCARD bad data; bit set" +" prior to block; block=%s offset=%d" +" firstset=%d\n" , ramid, first_bit_offset, + firstzero); +return -1 ; + } + +/* +* we know there must be at least 1 bit set due to the loop entry +* If there is no 0 firstzero will be 32 +*/ +/* TODO - ram_discard_range gets added in a later patch +int ret = ram_discard_range(mis, ramid, +startaddr + firstset - first_bit_offset, +startaddr + (firstzero - 1) - first_bit_offset); +ret = -1; +if (ret) { +return ret; +} +*/ + +/* mask= .?0000000000 */ +/* ^fz ^fs */ +if (firstzero !=32 ) { + mask &= (((uint32_t )-1 ) << firstzero); + }else {
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1189
Quoted 10 lines
savevm.c +10
-0+ mask =0 ; + } + } + } +DPRINTF ("%s finished" , __func__); + +return 0 ; +} + +/* After this message we must be able to immediately receive page data */
The purpose of the listen message from a protocol point of view isn't really clear to me. I understand why the destination needs to set up the postcopy handling before processing the device data, but why does it need an assertion from the source to start this, rather than just an internal detail on the load path.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
Quoted 6 lines · * David Gibson ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add state variable showing current incoming postcopy state.
This appears to implement a lot more than just adding a state variable...
It's clearer with the title line; I've reworded it to:
Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy
Quoted 15 lines
Quoted 15 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 8 + include/sysemu/sysemu.h | 20 +++ savevm.c | 335 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 363 insertions(+)include/migration/migration.h +0
-0@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;struct MigrationIncomingState {QEMUFile *file;
Quoted 3 lines
include/migration/migration.h typedef struct MigrationState MigrationState; · line 64
Quoted 1 line
include/migration/migration.h +1
-0+ volatile enum {What's the reason for the volatile? I think that really needs a comment.
I've removed it and replaced it by atomic_ functions to access it; the state is accessed from multiple threads.
Quoted 10 lines
Quoted 10 lines
include/migration/migration.h +7
-0+POSTCOPY_RAM_INCOMING_NONE =0 ,/* Initial state - no postcopy */ +POSTCOPY_RAM_INCOMING_ADVISE , +POSTCOPY_RAM_INCOMING_LISTENING , +POSTCOPY_RAM_INCOMING_RUNNING , + POSTCOPY_RAM_INCOMING_END + } postcopy_ram_state; +QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ };
<snip>
Quoted 29 lines
Quoted 29 lines
savevm.c +14
-0@@ -39,6 +39,7 @@#include "exec/memory.h" #include "qmp-commands.h" #include "trace.h" +#include "qemu/bitops.h" #include "qemu/iov.h" #include "block/snapshot.h" #include "block/qapi.h" @@ -624,6 +625,92 @@ void qemu_savevm_send_openrp(QEMUFile *f) {qemu_savevm_command_send (f,QEMU_VM_CMD_OPENRP ,0 ,NULL ); } + +/* Send prior to any RAM transfer */ +void qemu_savevm_send_postcopy_ram_advise (QEMUFile *f) +{ +DPRINTF ("send postcopy-ram-advise" ); +uint64_t tmp[2 ]; + tmp[0 ] =cpu_to_be64 (sysconf (_SC_PAGESIZE)); + tmp[1 ] =cpu_to_be64 (1ul <<qemu_target_page_bits ()); + +qemu_savevm_command_send (f,QEMU_VM_CMD_POSTCOPY_RAM_ADVISE ,16 , + (uint8_t *)tmp); +} +
Quoted 8 lines
savevm.c void qemu_savevm_send_openrp(QEMUFile *f) · line 646
Quoted 6 lines
savevm.c +6
-0+/* Prior to running, to cause pages that have been dirtied after precopy +* started to be discarded on the destination. +* CMD_POSTCOPY_RAM_DISCARD consist of: +* 3 byte header (filled in by qemu_savevm_send_postcopy_ram_discard) +* byte version (0) +* byte offset into the 1st data word containing 1st page of RAMBlock I'm not able to follow what that description means.
I've reworded it as:
* byte offset to be subtracted from each page address to deal with
* RAMBlocks that don't start on a mask word boundary.
(I've tried reworking this protocol a few times, the offset still seems to work out easiest, otherwise you end up having to treat the address entries as potentially signed).
<snip>
Quoted 8 lines
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1070
Quoted 5 lines
savevm.c +5
-0+/* +* Postcopy will be sending lots of small messages along the return path +* that it needs quick answers to. +*/ +socket_set_nodelay (qemu_get_fd (mis->return_path));So, here you break the QEMUFile abstraction and assume you have a socket.
Ah yes; I put that in to see if it would help. Hmm; I've taken it out for now, I need to see if there's a sensible way to add the abstration.
Quoted 11 lines
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1133
Quoted 10 lines
savevm.c +10
-0+while (len) { +uint64_t startaddr; +uint32_t mask; +/* +* We now have pairs of address, mask +* The mask is 32 bits of bitmask starting at 'startaddr'-offset +* RAMBlock; e.g. if the RAMBlock started at 8k where TPS=4k +* then first_bit_offset=2 and the 1st 2 bits of the mask +* aren't relevant to this RAMBlock, and bit 2 corresponds +* to the 1st page of this RAMBlock
Quoted 1 line
Um.. yeah.. can't make much snse of this comment either.
Well, at least it's the one that corresponds to the comment above you couldn't understand either.
I've reworded it to:
* We now have pairs of address, mask
* Each word of mask is 32 bits, where each bit corresponds to one
* target page.
* RAMBlocks don't necessarily start on word boundaries,
* and the offset in the header indicates the offset into the 1st
* mask word that corresponds to the 1st page of the RAMBlock.
<snip>
Quoted 7 lines
savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1189
Quoted 1 line
savevm.c +1
-0+/* After this message we must be able to immediately receive page data */ The purpose of the listen message from a protocol point of view isn't really clear to me. I understand why the destination needs to set up the postcopy handling before processing the device data, but why does it need an assertion from the source to start this, rather than just an internal detail on the load path.
The device migration format isn't structured well enough to allow the receiver to know the size of the device data without parsing it all. The parsing isn't structured, so there's no way to do that without calling all the device code that may read from guest memory. We solve that by sending all the device data in the CMD_PACKAGED.
While we could assume that a CMD_PACKAGED is always for postcopy data, instead I don't special case the processing at all of the data inside the package, keeping that general. Then the listen message is used inside that package to start the listener thread off and set up the other associated state.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 22/47 QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.
This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.
--- include/sysemu/sysemu.h | 4 +++ savevm.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
include/sysemu/sysemu.h +4 -0
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {QEMU_VM_CMD_INVALID =0 ,/* Must be 0 */ QEMU_VM_CMD_OPENRP ,/* Tell the dest to open the Return path */ QEMU_VM_CMD_REQACK ,/* Request an ACK on the RP */ +QEMU_VM_CMD_PACKAGED ,/* Send a wrapped stream within this stream */ QEMU_VM_CMD_POSTCOPY_RAM_ADVISE = 20, /* Prior to any page transfers, justwarn we might want to do PC */
@@ -101,6 +102,8 @@ enum qemu_vm_cmd { QEMU_VM_CMD_AFTERLASTVALID }; +#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24) +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -112,6 +115,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);void qemu_savevm_send_openrp (QEMUFile *f); +void qemu_savevm_send_packaged (QEMUFile *f,const QEMUSizedBuffer *qsb);void qemu_savevm_send_postcopy_ram_advise (QEMUFile *f); void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint8_t offset,
savevm.c +82 -0
100 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
QEMU_VM_CMD_PACKAGEDis a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Though one minor comment:
[snip]
Quoted 4 lines
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that
I'm not quite understanding what that comment's getting at.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
Quoted 17 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
QEMU_VM_CMD_PACKAGEDis a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Though one minor comment:
Quoted 7 lines
[snip]
Quoted 4 lines
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise thatI'm not quite understanding what that comment's getting at.
We have these VM Commands; and they are a command type, and a length:
CMD_whatever
length: whatever
data for whatever
This comment is describing that, to make things easier for this code it's ended up as:
CMD_PACKAGED
CMD length: 4 <--- i.e. just enough to hold the next 'length' field
package length
---------------
The package
Which is a little different, hence i thought it needed the comment.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
Quoted 34 lines · Dr. David Alan Gilbert
Quoted 17 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
QEMU_VM_CMD_PACKAGEDis a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Though one minor comment:
Quoted 7 lines
[snip]
Quoted 4 lines
+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise thatI'm not quite understanding what that comment's getting at.
We have these VM Commands; and they are a command type, and a length:
CMD_whatever
length: whatever
data for whateverThis comment is describing that, to make things easier for this code it's ended up as:
Quoted 7 lines
CMD_PACKAGED
CMD length: 4 <--- i.e. just enough to hold the next 'length' field
package length
---------------
The packageWhich is a little different, hence i thought it needed the comment.
Ah.. right. That seems.. gratuitously easy to get wrong further down the track. Why not just use the cmd length as the package length.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 23/47 migrate_init: Call from savevm
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.
--- include/migration/migration.h | 1 + include/qemu/typedefs.h | 1 + migration.c | 2 +- savevm.c | 2 ++ 4 files changed, 5 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
@@ -140,6 +140,7 @@ int migrate_fd_close(MigrationState *s);void add_migration_state_change_notifier (Notifier *notify);void remove_migration_state_change_notifier (Notifier *notify); +MigrationState *migrate_init (const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *);
include/qemu/typedefs.h +1 -0
@@ -16,6 +16,7 @@ struct Monitor;typedef struct Monitor Monitor ;typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct MigrationParams MigrationParams ; +typedef struct MigrationState MigrationState ;typedef struct Property Property ;typedef struct PropertyInfo PropertyInfo ;
migration.c +1 -1
@@ -488,7 +488,7 @@ bool migration_has_failed(MigrationState *s) s->state == MIG_STATE_ERROR); } -static MigrationState *migrate_init (const MigrationParams *params) +MigrationState *migrate_init (const MigrationParams *params) {MigrationState *s =migrate_get_current ();int64_t bandwidth_limit = s->bandwidth_limit;
savevm.c +2 -0
@@ -949,6 +949,8 @@ static int qemu_savevm_state(QEMUFile *f) .blk = 0, .shared = 0 }; +MigrationState *ms =migrate_init (¶ms); + ms->file = f;if (qemu_savevm_state_blocked (NULL )) {return -EINVAL ;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 23/47] migrate_init: Call from savevm
Quoted 31 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the
savevm.ccode for suspending.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 1 + include/qemu/typedefs.h | 1 + migration.c | 2 +- savevm.c | 2 ++ 4 files changed, 5 insertions(+), 1 deletion(-)include/migration/migration.h +1
-0@@ -140,6 +140,7 @@ int migrate_fd_close(MigrationState *s);void add_migration_state_change_notifier (Notifier *notify);void remove_migration_state_change_notifier (Notifier *notify); +MigrationState *migrate_init (const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *);include/qemu/typedefs.h +0
-0
include/qemu/typedefs.h struct Monitor; · line 20
Quoted 5 lines
include/qemu/typedefs.h +1
-0@@ -16,6 +16,7 @@ struct Monitor;typedef struct Monitor Monitor ;typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct MigrationParams MigrationParams ; +typedef struct MigrationState MigrationState ;
Er, another redefinition, when compile, it complains there is a redefinition of typedef ‘MigrationState’ in 'include/migration/migration.h:59', is this a problem?
Quoted 29 lines
include/qemu/typedefs.h +0
-0typedef struct Property Property ;typedef struct PropertyInfo PropertyInfo ;migration.c +1
-1@@ -488,7 +488,7 @@ bool migration_has_failed(MigrationState *s) s->state == MIG_STATE_ERROR); } -static MigrationState *migrate_init (const MigrationParams *params) +MigrationState *migrate_init (const MigrationParams *params) {MigrationState *s =migrate_get_current ();int64_t bandwidth_limit = s->bandwidth_limit;savevm.c +2
-0@@ -949,6 +949,8 @@ static int qemu_savevm_state(QEMUFile *f) .blk = 0, .shared = 0 }; +MigrationState *ms =migrate_init (¶ms); + ms->file = f;if (qemu_savevm_state_blocked (NULL )) {return -EINVAL ;
Re: [Qemu-devel] [PATCH v4 23/47] migrate_init: Call from savevm
In reply to Dr. David Alan Gilbert (git)Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the
savevm.ccode for suspending.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 24/47 Allow savevm handlers to state whether they could go into postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts
--- arch_init.c | 7 +++++++ include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 4 +++- migration.c | 9 ++++++++- savevm.c | 23 +++++++++++++++++++---- 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +7 -0
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)return ret; } +/* RAM's always up for postcopying */ +static bool ram_can_postcopy (void *opaque) +{ +return true; +} +static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel, + .can_postcopy = ram_can_postcopy, };void ram_mig_init (void )
include/migration/vmstate.h +1 -1
@@ -54,7 +54,7 @@ typedef struct SaveVMHandlers {/* This runs outside the iothread lock! */ int (*save_live_setup)(QEMUFile *f,void *opaque);uint64_t (*save_live_pending)(QEMUFile *f,void *opaque,uint64_t max_size); - +bool (*can_postcopy)(void *opaque);LoadStateHandler *load_state; } SaveVMHandlers;
include/sysemu/sysemu.h +3 -1
@@ -110,7 +110,9 @@ void qemu_savevm_state_begin(QEMUFile *f,int qemu_savevm_state_iterate (QEMUFile *f);void qemu_savevm_state_complete (QEMUFile *f);void qemu_savevm_state_cancel (void ); -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);
migration.c +8 -1
@@ -865,8 +865,15 @@ static void *migration_thread(void *opaque)uint64_t pending_size;if (!qemu_file_rate_limit (s->file)) { - pending_size =qemu_savevm_state_pending (s->file, max_size); +uint64_t pend_post, pend_nonpost; +DPRINTF ("iterate\n" ); +qemu_savevm_state_pending (s->file, max_size, &pend_nonpost, + &pend_post); + pending_size = pend_nonpost + pend_post;trace_migrate_pending (pending_size, max_size); +DPRINTF ("pending size %" PRIu64" max %" PRIu64" (post=%" PRIu64 +" nonpost=%" PRIu64")\n" , + pending_size, max_size, pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) {qemu_savevm_state_iterate (s->file); }else {
savevm.c +19 -4
@@ -911,10 +911,18 @@ void qemu_savevm_state_complete(QEMUFile *f)qemu_fflush (f); } -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size) +/* Give an estimate of the amount left to be transferred, +* the result is split into the amount for units that can and +* for units that can't do postcopy. +*/ +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable) {SaveStateEntry *se; -uint64_t ret =0 ; +uint64_t res_nonpc =0 ; +uint64_t res_pc =0 ; +uint64_t tmp; QTAILQ_FOREACH(se, &savevm_handlers, entry) {if (!se->ops || !se->ops->save_live_pending) {
@@ -925,9 +933,16 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)continue ; } } - ret += se->ops->save_live_pending (f, se->opaque, max_size); + tmp = se->ops->save_live_pending (f, se->opaque, max_size); + +if (se->ops->can_postcopy (se->opaque)) { + res_pc += tmp; + }else { + res_nonpc += tmp; + } } -return ret; + *res_non_postcopiable = res_nonpc; + *res_postcopiable = res_pc; }void qemu_savevm_state_cancel (void )
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy
Quoted 32 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Use that to split the
qemu_savevm_state_pendingcounts into postcopiable and non-postcopiable amountsSigned-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 7 +++++++ include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 4 +++- migration.c | 9 ++++++++- savevm.c | 23 +++++++++++++++++++---- 5 files changed, 38 insertions(+), 7 deletions(-)arch_init.c +6
-0@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)return ret; } +/* RAM's always up for postcopying */ +static bool ram_can_postcopy (void *opaque) +{ +return true; +} +static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,
arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208
Quoted 4 lines
arch_init.c +1
-0@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel, + .can_postcopy = ram_can_postcopy,
Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?
On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy
Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Use that to split the
qemu_savevm_state_pendingcounts into postcopiable and non-postcopiable amountsSigned-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 7 +++++++ include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 4 +++- migration.c | 9 ++++++++- savevm.c | 23 +++++++++++++++++++---- 5 files changed, 38 insertions(+), 7 deletions(-)arch_init.c +6
-0@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)return ret; } +/* RAM's always up for postcopying */ +static bool ram_can_postcopy (void *opaque) +{ +return true; +} +static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208
Quoted 4 lines
arch_init.c +1
-0@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel, + .can_postcopy = ram_can_postcopy,Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?
Quoted 4 lines
On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.
It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.
Dave
Quoted 5 lines
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy
Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Use that to split the
qemu_savevm_state_pendingcounts into postcopiable and non-postcopiable amountsSigned-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 7 +++++++ include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 4 +++- migration.c | 9 ++++++++- savevm.c | 23 +++++++++++++++++++---- 5 files changed, 38 insertions(+), 7 deletions(-)arch_init.c +6
-0@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)return ret; } +/* RAM's always up for postcopying */ +static bool ram_can_postcopy (void *opaque) +{ +return true; +} +static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208
Quoted 4 lines
arch_init.c +1
-0@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel, + .can_postcopy = ram_can_postcopy,Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?
Quoted 7 lines
Quoted 4 lines
On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.
It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.
Ok. I still wonder if it might be better to instead extend the save_live_pending callback in order to return both non-postcopyable and postcopyable quantites. It allows for the case of a postcopyable device which has some non-postcopyable data - and with any postcopyable device other than RAM, it seems likely that there will need to be some precopied metadata at least. Plus it avoids adding another callback.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy
Quoted 51 lines · * David Gibson ([email protected]) wrote:
Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Use that to split the
qemu_savevm_state_pendingcounts into postcopiable and non-postcopiable amountsSigned-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 7 +++++++ include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 4 +++- migration.c | 9 ++++++++- savevm.c | 23 +++++++++++++++++++---- 5 files changed, 38 insertions(+), 7 deletions(-)arch_init.c +6
-0@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)return ret; } +/* RAM's always up for postcopying */ +static bool ram_can_postcopy (void *opaque) +{ +return true; +} +static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate,arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208
Quoted 4 lines
arch_init.c +1
-0@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = { .save_live_pending = ram_save_pending, .load_state = ram_load, .cancel = ram_migration_cancel, + .can_postcopy = ram_can_postcopy,Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?
Quoted 7 lines
Quoted 4 lines
On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.
It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.
Quoted 7 lines
Ok. I still wonder if it might be better to instead extend the
save_live_pendingcallback in order to return both non-postcopyable and postcopyable quantites. It allows for the case of a postcopyable device which has some non-postcopyable data - and with any postcopyable device other than RAM, it seems likely that there will need to be some precopied metadata at least. Plus it avoids adding another callback.
There are two separate suggestions there - which I'll address in opposite order:
1) Extending save_live_pending callback to avoid a new callback can_postcopy is used for a few diferent decisions; not just where to add the pending value to; so I don't think extending s_l_p saves the need for the other callback
2) Allowing a device to do both pre and postcopy
Yeh I can see you could theoretically have a device where that would be useful; but for reasonably small chunks of metadata it already gets the chance to send those during the _begin call. I'll make the change to save_live_pending's parameters to let this work.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 25/47 postcopy: OS support test
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates postcopy-ram.c which will get most of the other helpers we need.
--- Makefile.objs | 2 +- include/migration/postcopy-ram.h | 19 +++++ postcopy-ram.c | 160 +++++++++++++++++++++++++++++++++++++++ savevm.c | 6 ++ 4 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 postcopy-ram.c
Makefile.objs +1 -1
@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o common-obj-$(CONFIG_RDMA) += migration-rdma.o common-obj-y += qemu-char.o #aio.o common-obj-y += block-migration.o -common-obj-y += page_cache.o xbzrle.o +common-obj-y += page_cache.o xbzrle.o postcopy-ram.o common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
include/migration/postcopy-ram.h new file +19 -0
@@ -0,0 +1,19 @@ +/* +* Postcopy migration for RAM +* +* Copyright 2013 Red Hat, Inc. and/or its affiliates +* +* Authors: +* Dave Gilbert <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_POSTCOPY_RAM_H +#define QEMU_POSTCOPY_RAM_H + +/* Return 0 if the host supports everything we need to do postcopy-ram */ +int postcopy_ram_hosttest (void ); + +#endif
postcopy-ram.c new file +160 -0
160 lines not on this page · raw message
savevm.c +6 -0
@@ -33,6 +33,7 @@#include "qemu/timer.h" #include "audio/audio.h" #include "migration/migration.h" +#include "migration/postcopy-ram.h" #include "qemu/sockets.h" #include "qemu/queue.h" #include "sysemu/cpus.h"
@@ -1087,6 +1088,11 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,return -1 ; } +/* Check this host can do it */ +if (postcopy_ram_hosttest ()) { +return -1 ; + } +if (remote_hps !=sysconf (_SC_PAGESIZE)) {/* * Some combinations of mismatch are probably possible but it gets
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 25/47] postcopy: OS support test
Quoted 43 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- Makefile.objs | 2 +- include/migration/postcopy-ram.h | 19 +++++ postcopy-ram.c | 160 +++++++++++++++++++++++++++++++++++++++ savevm.c | 6 ++ 4 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 postcopy-ram.cMakefile.objs +1
-1@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o common-obj-$(CONFIG_RDMA) += migration-rdma.o common-obj-y += qemu-char.o #aio.o common-obj-y += block-migration.o -common-obj-y += page_cache.o xbzrle.o +common-obj-y += page_cache.o xbzrle.o postcopy-ram.o common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.oinclude/migration/postcopy-ram.h new file +7
-0@@ -0,0 +1,19 @@ +/* +* Postcopy migration for RAM +* +* Copyright 2013 Red Hat, Inc. and/or its affiliates +* +* Authors: +* Dave Gilbert <[email protected]>
include/migration/postcopy-ram.h line 17
Quoted 10 lines
include/migration/postcopy-ram.h new file +10
-0+* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_POSTCOPY_RAM_H +#define QEMU_POSTCOPY_RAM_H + +/* Return 0 if the host supports everything we need to do postcopy-ram */ +int postcopy_ram_hosttest (void );
Maybe postcopy_supported_by_host() would be a bit clearer?
[snip]
include/migration/postcopy-ram.h line 19
Quoted 2 lines
include/migration/postcopy-ram.h new file +2
-0+#ifdef HOST_X86_64 +/* NOTE: These are Andrea's 3.15.0 world */
I thought the usual approach in qemu was to import the updated headers first in a separate patch, rather than embeddeding new defines.
Quoted 20 lines
include/migration/postcopy-ram.h new file +20
-0+#ifndef MADV_USERFAULT +#define MADV_USERFAULT 18 +#define MADV_NOUSERFAULT 19 +#endif + +#ifndef __NR_remap_anon_pages +#define __NR_remap_anon_pages 321 +#endif + +#ifndef __NR_userfaultfd +#define __NR_userfaultfd 322 +#endif + +#endif + +#ifndef USERFAULTFD_PROTOCOL +#define USERFAULTFD_PROTOCOL (uint64_t)0xaa +#endif + +#endif
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 25/47] postcopy: OS support test
Quoted 48 lines · * David Gibson ([email protected]) wrote:
Quoted 47 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates
postcopy-ram.cwhich will get most of the other helpers we need.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- Makefile.objs | 2 +- include/migration/postcopy-ram.h | 19 +++++ postcopy-ram.c | 160 +++++++++++++++++++++++++++++++++++++++ savevm.c | 6 ++ 4 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 postcopy-ram.cMakefile.objs +1
-1@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o common-obj-$(CONFIG_RDMA) += migration-rdma.o common-obj-y += qemu-char.o #aio.o common-obj-y += block-migration.o -common-obj-y += page_cache.o xbzrle.o +common-obj-y += page_cache.o xbzrle.o postcopy-ram.o common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.oinclude/migration/postcopy-ram.h new file +11
-0@@ -0,0 +1,19 @@ +/* +* Postcopy migration for RAM +* +* Copyright 2013 Red Hat, Inc. and/or its affiliates +* +* Authors: +* Dave Gilbert <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*
Quoted 8 lines
include/migration/postcopy-ram.h line 17
Quoted 6 lines
include/migration/postcopy-ram.h new file +6
-0+*/ +#ifndef QEMU_POSTCOPY_RAM_H +#define QEMU_POSTCOPY_RAM_H + +/* Return 0 if the host supports everything we need to do postcopy-ram */ +int postcopy_ram_hosttest (void );Maybe
postcopy_supported_by_host()would be a bit clearer?
I went with postcopy_ram_supported_by_host
(and flipped the sense so it returns true if it's supported)
Quoted 6 lines
[snip]
Quoted 2 lines
+#ifdef
HOST_X86_64
+ /* NOTE: These are Andrea's 3.15.0 world */I thought the usual approach in qemu was to import the updated headers first in a separate patch, rather than embeddeding new defines.
Yes, those will sort themselves out when the syscalls land in the kernel and then we can import the headers.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 26/47 migrate_start_postcopy: Command to trigger transition to postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopy
command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
--- hmp-commands.hx | 15 +++++++++++++++ hmp.c | 7 +++++++ hmp.h | 1 + include/migration/migration.h | 3 +++ migration.c | 22 ++++++++++++++++++++++ qapi-schema.json | 8 ++++++++ qmp-commands.hx | 19 +++++++++++++++++++ 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration. ETEXI { + .name = "migrate_start_postcopy", + .args_type = "", + .params = "", + .help = "Switch migration to postcopy mode", + .mhandler.cmd = hmp_migrate_start_postcopy, + }, + +STEXI +@item migrate_start_postcopy +@findex migrate_start_postcopy +Switch in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). +ETEXI + + { .name = "client_migrate_info", .args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?", .params = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
@@ -1079,6 +1079,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict) } } +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict) +{ +Error *err =NULL ; +qmp_migrate_start_postcopy (&err); +hmp_handle_error (mon, &err); +} +void hmp_set_password (Monitor *mon,const QDict *qdict) {const char *protocol =qdict_get_str (qdict,"protocol" );
hmp.h +1 -0
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);void hmp_migrate_set_speed (Monitor *mon,const QDict *qdict);void hmp_migrate_set_capability (Monitor *mon,const QDict *qdict);void hmp_migrate_set_cache_size (Monitor *mon,const QDict *qdict); +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict);void hmp_set_password (Monitor *mon,const QDict *qdict);void hmp_expire_password (Monitor *mon,const QDict *qdict);void hmp_eject (Monitor *mon,const QDict *qdict);
include/migration/migration.h +3 -0
@@ -100,6 +100,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +volatile bool start_postcopy; };void process_incoming_migration (QEMUFile *f);
migration.c +22 -0
@@ -379,6 +379,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params, } } +void qmp_migrate_start_postcopy (Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIG_STATE_NONE ) { +error_setg (errp,"Postcopy must be started after migration has been" +" started" ); +return ; + } +/* +* we don't error if migration has finished since that would be racy +* with issuing this command. +*/ + s->start_postcopy = true; +} +/* shared migration helpers */ static void migrate_set_state (MigrationState *s,int old_state,int new_state)
qapi-schema.json +8 -0
@@ -541,6 +541,14 @@ { 'command': 'query-migrate-capabilities', 'returns': ['MigrationCapabilityStatus']} ## +# @migrate-start-postcopy +# +# Switch migration to postcopy mode +# +# Since: 2.2 +{ 'command': 'migrate-start-postcopy' } + +## # @MouseInfo: # # Information about a mouse device.
qmp-commands.hx +19 -0
@@ -685,6 +685,25 @@ Example: EQMP { + .name = "migrate-start-postcopy", + .args_type = "", + .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy, + }, + +SQMP +migrate-start-postcopy +---------------------- + +Switch an in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). + +Example: +-> { "execute": "migrate-start-postcopy" } +<- { "return": {} } + +EQMP + + { .name = "query-migrate-cache-size", .args_type = "", .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 26/47] migrate_start_postcopy: Command to trigger transition to postcopy
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with
migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopy
Quoted 8 lines
command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Eric Blake <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 27/47 MIG_STATE_POSTCOPY_ACTIVE: Add new migration state
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
'MIG_STATE_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy
'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.
--- include/migration/migration.h | 2 ++ migration.c | 58 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 54 insertions(+), 6 deletions(-)
include/migration/migration.h +2 -0
@@ -147,6 +147,8 @@ MigrationState *migrate_init(const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *); +/* True if outgoing migration has entered postcopy phase */ +bool migration_postcopy_phase (MigrationState *);MigrationState *migrate_get_current (void );uint64_t ram_bytes_remaining (void );
migration.c +52 -6
118 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 27/47] MIG_STATE_POSTCOPY_ACTIVE: Add new migration state
Quoted 8 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
'
MIG_STATE_POSTCOPY_ACTIVE' is entered aftermigrate_start_postcopy'
migration_postcopy_phase' is provided for other sections to know if they're in postcopy.Signed-off-by Dr. David Alan Gilbert <[email protected]>
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 28/47 qemu_savevm_state_complete: Postcopy changes
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.
We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the _complete call.
--- include/sysemu/sysemu.h | 1 + savevm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-)
include/sysemu/sysemu.h +1 -0
@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size,uint64_t *res_non_postcopiable,uint64_t *res_postcopiable); +void qemu_savevm_state_postcopy_complete (QEMUFile *f);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);
savevm.c +51 -1
74 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 28/47] qemu_savevm_state_complete: Postcopy changes
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When postcopy calls
qemu_savevm_state_completeit's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the_completecall.
So, we should probably rename qemu_savevm_state_complete() to reflect the fact that it's no longer actually a completion, but just the transition from pre-copy to post-copy phases. A good, brief name doesn't immediately occur to me, unfortunately.
Quoted 31 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/sysemu/sysemu.h | 1 + savevm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-)include/sysemu/sysemu.h +1
-0@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size,uint64_t *res_non_postcopiable,uint64_t *res_postcopiable); +void qemu_savevm_state_postcopy_complete (QEMUFile *f);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);savevm.c +5
-0@@ -854,10 +854,51 @@ int qemu_savevm_state_iterate(QEMUFile *f)return ret; } +/* +* Calls the complete routines just for those devices that are postcopiable; +* causing the last few pages to be sent immediately and doing any associated +* cleanup. +* Note postcopy also calls the plain qemu_savevm_state_complete to complete
savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 871
Quoted 10 lines
savevm.c +10
-0+* all the other devices, but that happens at the point we switch to postcopy. +*/ +void qemu_savevm_state_postcopy_complete (QEMUFile *f) +{ +SaveStateEntry *se; +int ret; + + QTAILQ_FOREACH(se, &savevm_handlers, entry) { +if (!se->ops || !se->ops->save_live_complete || + !se->ops->can_postcopy) {
So, you check for the presence of a can_postcopy callback, but you don't ever actually invoke it.
savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 884
Quoted 13 lines
savevm.c +13
-0+continue ; + } +if (se->ops && se->ops->is_active) { +if (!se->ops->is_active (se->opaque)) { +continue ; + } + } +trace_savevm_section_start (se->idstr, se->section_id); +/* Section type */ +qemu_put_byte (f,QEMU_VM_SECTION_END ); +qemu_put_be32 (f, se->section_id); + + ret = se->ops->save_live_complete (f, se->opaque);
I'm wondering if it might be clearer not to overload the save_live_complete hook, but instead allow both "execution transition" (old complete) and "final complete" (postcopy complete) hooks (expecting only one to be non-NULL in most cases).
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 28/47] qemu_savevm_state_complete: Postcopy changes
Quoted 13 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When postcopy calls
qemu_savevm_state_completeit's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the_completecall.
Quoted 4 lines
So, we should probably rename
qemu_savevm_state_complete()to reflect the fact that it's no longer actually a completion, but just the transition from pre-copy to post-copy phases. A good, brief name doesn't immediately occur to me, unfortunately.
Well it's still completion in the non-postcopy case; if you do think of a good obvious name then I'd be happy to change it. (Another way would be to add aparameter to qemu_savevm_state_complete to make it do one or the other, but some of the conditions in it were already a bit hairy).
Quoted 36 lines
Quoted 36 lines
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/sysemu/sysemu.h | 1 + savevm.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-)include/sysemu/sysemu.h +1
-0@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size,uint64_t *res_non_postcopiable,uint64_t *res_postcopiable); +void qemu_savevm_state_postcopy_complete (QEMUFile *f);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_reqack (QEMUFile *f,uint32_t value);savevm.c +10
-0@@ -854,10 +854,51 @@ int qemu_savevm_state_iterate(QEMUFile *f)return ret; } +/* +* Calls the complete routines just for those devices that are postcopiable; +* causing the last few pages to be sent immediately and doing any associated +* cleanup. +* Note postcopy also calls the plain qemu_savevm_state_complete to complete +* all the other devices, but that happens at the point we switch to postcopy. +*/ +void qemu_savevm_state_postcopy_complete (QEMUFile *f) +{ +SaveStateEntry *se;
Quoted 8 lines
savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 871
Quoted 5 lines
savevm.c +5
-0+int ret; + + QTAILQ_FOREACH(se, &savevm_handlers, entry) { +if (!se->ops || !se->ops->save_live_complete || + !se->ops->can_postcopy) {So, you check for the presence of a
can_postcopycallback, but you don't ever actually invoke it.
Thanks, fixed.
Quoted 14 lines
savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 884
Quoted 13 lines
savevm.c +13
-0+continue ; + } +if (se->ops && se->ops->is_active) { +if (!se->ops->is_active (se->opaque)) { +continue ; + } + } +trace_savevm_section_start (se->idstr, se->section_id); +/* Section type */ +qemu_put_byte (f,QEMU_VM_SECTION_END ); +qemu_put_be32 (f, se->section_id); + + ret = se->ops->save_live_complete (f, se->opaque);
Quoted 4 lines
I'm wondering if it might be clearer not to overload the
save_live_completehook, but instead allow both "execution transition" (old complete) and "final complete" (postcopy complete) hooks (expecting only one to be non-NULL in most cases).
Note that I only call save_live_complete once for any one device. Non-postcopied devices get done in the original loop, postcopied devices in the new ones, so from the point of view of a device it's still a complete.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 29/47 Postcopy page-map-incoming (PMI) structure
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.
--- include/migration/migration.h | 19 ++++ include/migration/postcopy-ram.h | 12 +++ include/qemu/typedefs.h | 1 + postcopy-ram.c | 220 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+)
include/migration/migration.h +19 -0
@@ -57,6 +57,24 @@ struct MigrationRetPathState {typedef struct MigrationState MigrationState ; +/* Postcopy page-map-incoming - data about each page on the inbound side */ + +typedef enum { +POSTCOPY_PMI_MISSING ,/* page hasn't yet been received */ +POSTCOPY_PMI_REQUESTED ,/* Kernel asked for a page, but we've not got it */ +POSTCOPY_PMI_RECEIVED /* We've got the page */ +}PostcopyPMIState ; + +struct PostcopyPMI { +QemuMutex mutex; +unsigned long *received_map;/* Pages that we have received */ +unsigned long *requested_map;/* Pages that we're sending a request for */ +unsigned long host_mask;/* A mask with enough bits set to cover one +host page in the PMI */ +unsigned long host_bits;/* The number of bits in the map representing +one host page */ +}; +/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;
@@ -71,6 +89,7 @@ struct MigrationIncomingState {QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ +PostcopyPMI postcopy_pmi; };MigrationIncomingState *migration_incoming_get_current (void );
include/migration/postcopy-ram.h +12 -0
@@ -13,7 +13,19 @@#ifndef QEMU_POSTCOPY_RAM_H #define QEMU_POSTCOPY_RAM_H +#include "migration/migration.h" +/* Return 0 if the host supports everything we need to do postcopy-ram */ int postcopy_ram_hosttest (void ); +/* +* In 'advise' mode record that a page has been received. +*/ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index); + +void postcopy_pmi_destroy (MigrationIncomingState *mis); +void postcopy_pmi_discard_range (MigrationIncomingState *mis, +size_t start,size_t npages); +void postcopy_pmi_dump (MigrationIncomingState *mis);#endif
include/qemu/typedefs.h +1 -0
@@ -77,6 +77,7 @@ typedef struct QEMUSGList QEMUSGList;typedef struct SHPCDevice SHPCDevice ;typedef struct FWCfgState FWCfgState ;typedef struct PcGuestInfo PcGuestInfo ; +typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ;typedef struct AdapterInfo AdapterInfo ;
postcopy-ram.c +220 -0
237 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 29/47] Postcopy page-map-incoming (PMI) structure
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
Though there are a couple of minor comments below:
Quoted 109 lines
--- include/migration/migration.h | 19 ++++ include/migration/postcopy-ram.h | 12 +++ include/qemu/typedefs.h | 1 + postcopy-ram.c | 220 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+)include/migration/migration.h +19
-0@@ -57,6 +57,24 @@ struct MigrationRetPathState {typedef struct MigrationState MigrationState ; +/* Postcopy page-map-incoming - data about each page on the inbound side */ + +typedef enum { +POSTCOPY_PMI_MISSING ,/* page hasn't yet been received */ +POSTCOPY_PMI_REQUESTED ,/* Kernel asked for a page, but we've not got it */ +POSTCOPY_PMI_RECEIVED /* We've got the page */ +}PostcopyPMIState ; + +struct PostcopyPMI { +QemuMutex mutex; +unsigned long *received_map;/* Pages that we have received */ +unsigned long *requested_map;/* Pages that we're sending a request for */ +unsigned long host_mask;/* A mask with enough bits set to cover one +host page in the PMI */ +unsigned long host_bits;/* The number of bits in the map representing +one host page */ +}; +/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *file;@@ -71,6 +89,7 @@ struct MigrationIncomingState {QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ +PostcopyPMI postcopy_pmi; };MigrationIncomingState *migration_incoming_get_current (void );include/migration/postcopy-ram.h +12
-0@@ -13,7 +13,19 @@#ifndef QEMU_POSTCOPY_RAM_H #define QEMU_POSTCOPY_RAM_H +#include "migration/migration.h" +/* Return 0 if the host supports everything we need to do postcopy-ram */ int postcopy_ram_hosttest (void ); +/* +* In 'advise' mode record that a page has been received. +*/ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index); + +void postcopy_pmi_destroy (MigrationIncomingState *mis); +void postcopy_pmi_discard_range (MigrationIncomingState *mis, +size_t start,size_t npages); +void postcopy_pmi_dump (MigrationIncomingState *mis);#endif include/qemu/typedefs.h +1
-0@@ -77,6 +77,7 @@ typedef struct QEMUSGList QEMUSGList;typedef struct SHPCDevice SHPCDevice ;typedef struct FWCfgState FWCfgState ;typedef struct PcGuestInfo PcGuestInfo ; +typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ;typedef struct AdapterInfo AdapterInfo ;postcopy-ram.c +14
-0@@ -23,6 +23,9 @@#include "qemu-common.h" #include "migration/migration.h" #include "migration/postcopy-ram.h" +#include "sysemu/sysemu.h" +#include "qemu/bitmap.h" +#include "qemu/error-report.h" //#define DEBUG_POSTCOPY @@ -82,6 +85,216 @@#if defined(__linux__) && defined(MADV_USERFAULT ) && \ defined(__NR_remap_anon_pages) +/* ---------------------------------------------------------------------- */ +/* Postcopy pagemap-inbound (pmi) - data structures that record the */ +/* state of each page used by the inbound postcopy */ +/* It's a pair of bitmaps (of the same structure as the migration bitmaps)*/ +/* holding one bit per target-page, although all operations work on host */ +/* pages. */ +__attribute__ (( unused ))/* Until later in patch series */ +static void postcopy_pmi_init (MigrationIncomingState *mis,size_t ram_pages) +{ +unsigned int tpb =qemu_target_page_bits (); +unsigned long host_bits;
postcopy-ram.c line 108
Quoted 10 lines
postcopy-ram.c +10
-0+ +qemu_mutex_init (&mis->postcopy_pmi.mutex); + mis->postcopy_pmi.received_map =bitmap_new (ram_pages); + mis->postcopy_pmi.requested_map =bitmap_new (ram_pages); +bitmap_clear (mis->postcopy_pmi.received_map,0 , ram_pages); +bitmap_clear (mis->postcopy_pmi.requested_map,0 , ram_pages); +/* +* Each bit in the map represents one 'target page' which is no bigger +* than a host page but can be smaller. It's useful to have some +* convenience masks for later
So, there's no inherent reason a target page couldn't be bigger than a host page. It's fair enough not to handle that case for now, but something somewhere should probably verify that it's no the case.
Quoted 17 lines
postcopy-ram.c +17
-0+*/ + +/* +* The number of bits one host page takes up in the bitmap +* e.g. on a 64k host page, 4k Target page, host_bits=64/4=16 +*/ + host_bits =sysconf (_SC_PAGESIZE) / (1ul << tpb); +/* Should be a power of 2 */ +assert (host_bits && !(host_bits & (host_bits -1 ))); +/* +* If the host_bits isn't a division of the number of bits in long +* then the code gets a lot more complex; disallow for now +* (I'm not aware of a system where it's true anyway) +*/ +assert (((sizeof (long ) *8 ) % host_bits) ==0 ); + + mis->postcopy_pmi.host_bits = host_bits;
postcopy-ram.c line 135
Quoted 10 lines
postcopy-ram.c +10
-0+/* A mask, starting at bit 0, containing host_bits continuous set bits */ + mis->postcopy_pmi.host_mask = (1ul << host_bits) -1 ; + +assert ((ram_pages % host_bits) ==0 ); +} + +void postcopy_pmi_destroy (MigrationIncomingState *mis) +{ +if (mis->postcopy_pmi.received_map) { +g_free (mis->postcopy_pmi.received_map);
g_free() is safe to call on NULL anyway, isn't it?
Quoted 67 lines
postcopy-ram.c +67
-0+ mis->postcopy_pmi.received_map =NULL ; + } +if (mis->postcopy_pmi.requested_map) { +g_free (mis->postcopy_pmi.requested_map); + mis->postcopy_pmi.requested_map =NULL ; + } +qemu_mutex_destroy (&mis->postcopy_pmi.mutex); +} + +/* +* Mark a set of pages in the PMI as being clear; this is used by the discard +* at the start of postcopy, and before the postcopy stream starts. +*/ +void postcopy_pmi_discard_range (MigrationIncomingState *mis, +size_t start,size_t npages) +{ +bitmap_clear (mis->postcopy_pmi.received_map, start, npages); +} + +/* +* Test a host-page worth of bits in the map starting at bitmap_index +* The bits should all be consistent +*/ +static bool test_hpbits (MigrationIncomingState *mis, +size_t bitmap_index,unsigned long *map) +{ +long masked; + +assert ((bitmap_index & (mis->postcopy_pmi.host_bits-1 )) ==0 ); + + masked = (map[BIT_WORD (bitmap_index)] >> + (bitmap_index %BITS_PER_LONG )) & + mis->postcopy_pmi.host_mask; + +assert ((masked ==0 ) || (masked == mis->postcopy_pmi.host_mask)); +return !!masked; +} + +/* +* Set host-page worth of bits in the map starting at bitmap_index +*/ +static void set_hpbits (MigrationIncomingState *mis, +size_t bitmap_index,unsigned long *map) +{ +assert ((bitmap_index & (mis->postcopy_pmi.host_bits-1 )) ==0 ); + + map[BIT_WORD (bitmap_index)] |= mis->postcopy_pmi.host_mask << + (bitmap_index %BITS_PER_LONG ); +} + +/* +* Clear host-page worth of bits in the map starting at bitmap_index +*/ +static void clear_hpbits (MigrationIncomingState *mis, +size_t bitmap_index,unsigned long *map) +{ +assert ((bitmap_index & (mis->postcopy_pmi.host_bits-1 )) ==0 ); + + map[BIT_WORD (bitmap_index)] &= ~(mis->postcopy_pmi.host_mask << + (bitmap_index %BITS_PER_LONG )); +} + +/* +* Retrieve the state of the given page +* Note: This version for use by callers already holding the lock +*/ +static PostcopyPMIState postcopy_pmi_get_state_nolock(
postcopy-ram.c line 212
Quoted 10 lines
postcopy-ram.c +10
-0+MigrationIncomingState *mis, + size_t bitmap_index) +{ + bool received, requested; + + received =test_hpbits (mis, bitmap_index, mis->postcopy_pmi.received_map); + requested =test_hpbits (mis, bitmap_index, mis->postcopy_pmi.requested_map); + +if (received) { +assert (!requested);
Clearing the requested bit when you set the received bit seems a bit pointless. (requested && received) isn't meaningfully different from (!requested && received) but there seems no reason to go to extra trouble to avoid that state, and having the record might be interesting for gathering statistics.
Quoted 66 lines
postcopy-ram.c +66
-0+return POSTCOPY_PMI_RECEIVED ; + }else { +return requested ?POSTCOPY_PMI_REQUESTED :POSTCOPY_PMI_MISSING ; + } +} + +/* Retrieve the state of the given page */ +__attribute__ (( unused ))/* Until later in patch series */ +static PostcopyPMIState postcopy_pmi_get_state (MigrationIncomingState *mis, +size_t bitmap_index) +{ +PostcopyPMIState ret; +qemu_mutex_lock (&mis->postcopy_pmi.mutex); + ret =postcopy_pmi_get_state_nolock (mis, bitmap_index); +qemu_mutex_unlock (&mis->postcopy_pmi.mutex); + +return ret; +} + +/* +* Set the page state to the given state if the previous state was as expected +* Return the actual previous state. +*/ +__attribute__ (( unused ))/* Until later in patch series */ +static PostcopyPMIState postcopy_pmi_change_state (MigrationIncomingState *mis, +size_t bitmap_index, +PostcopyPMIState expected_state, +PostcopyPMIState new_state) +{ +PostcopyPMIState old_state; + +qemu_mutex_lock (&mis->postcopy_pmi.mutex); + old_state =postcopy_pmi_get_state_nolock (mis, bitmap_index); + +if (old_state == expected_state) { +switch (new_state) { +case POSTCOPY_PMI_MISSING : +assert (0 );/* This shouldn't actually happen - use discard_range */ +break ; + +case POSTCOPY_PMI_REQUESTED : +assert (old_state ==POSTCOPY_PMI_MISSING ); +set_hpbits (mis, bitmap_index, mis->postcopy_pmi.requested_map); +break ; + +case POSTCOPY_PMI_RECEIVED : +assert (old_state ==POSTCOPY_PMI_MISSING || + old_state ==POSTCOPY_PMI_REQUESTED ); +set_hpbits (mis, bitmap_index, mis->postcopy_pmi.received_map); +clear_hpbits (mis, bitmap_index, mis->postcopy_pmi.requested_map); +break ; + } + } + +qemu_mutex_unlock (&mis->postcopy_pmi.mutex); +return old_state; +} + +/* +* Useful when debugging postcopy, although if it failed early the +* received map can be quite sparse and thus big when dumped. +*/ +void postcopy_pmi_dump (MigrationIncomingState *mis) +{ +fprintf (stderr,"postcopy_pmi_dump: requested\n" ); +ram_debug_dump_bitmap (mis->postcopy_pmi.requested_map, false);
postcopy-ram.c line 288
Quoted 10 lines
postcopy-ram.c +10
-0+fprintf (stderr,"postcopy_pmi_dump: received\n" ); +ram_debug_dump_bitmap (mis->postcopy_pmi.received_map, true); +fprintf (stderr,"postcopy_pmi_dump: end\n" ); +} + +/* Called by ram_load prior to mapping the page */ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index) +{ +if (mis->postcopy_ram_state ==POSTCOPY_RAM_INCOMING_ADVISE ) {
A silent no-op if you're not in the expected migration phase doesn't seem right. Should this be an assert() instead?
Quoted 25 lines
postcopy-ram.c +16
-0+/* +* If we're in precopy-advise mode we need to track received pages even +* though we don't need to place pages atomically yet. +* In advise mode there's only a single thread, so don't need locks +*/ +set_bit (bitmap_index, mis->postcopy_pmi.received_map); + } +} +int postcopy_ram_hosttest (void ) {/* TODO: Needs guarding with CONFIG_ once we have libc's that have the defs @@ -156,5 +369,12 @@ int postcopy_ram_hosttest(void) return -1; } +/* Called by ram_load prior to mapping the page */ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index) +{ +/* We don't support postcopy so don't care */ +} +#endif
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 29/47] Postcopy page-map-incoming (PMI) structure
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Though there are a couple of minor comments below:
<snip>
Quoted 17 lines
Quoted 17 lines
postcopy-ram.c +17
-0+/* ---------------------------------------------------------------------- */ +/* Postcopy pagemap-inbound (pmi) - data structures that record the */ +/* state of each page used by the inbound postcopy */ +/* It's a pair of bitmaps (of the same structure as the migration bitmaps)*/ +/* holding one bit per target-page, although all operations work on host */ +/* pages. */ +__attribute__ (( unused ))/* Until later in patch series */ +static void postcopy_pmi_init (MigrationIncomingState *mis,size_t ram_pages) +{ +unsigned int tpb =qemu_target_page_bits (); +unsigned long host_bits; + +qemu_mutex_init (&mis->postcopy_pmi.mutex); + mis->postcopy_pmi.received_map =bitmap_new (ram_pages); + mis->postcopy_pmi.requested_map =bitmap_new (ram_pages); +bitmap_clear (mis->postcopy_pmi.received_map,0 , ram_pages); +bitmap_clear (mis->postcopy_pmi.requested_map,0 , ram_pages);
Quoted 8 lines
postcopy-ram.c line 108
Quoted 4 lines
postcopy-ram.c +4
-0+/* +* Each bit in the map represents one 'target page' which is no bigger +* than a host page but can be smaller. It's useful to have some +* convenience masks for later So, there's no inherent reason a target page couldn't be bigger than a host page. It's fair enough not to handle that case for now, but something somewhere should probably verify that it's no the case.
I've added a guard for this in the host test.
Quoted 21 lines
Quoted 21 lines
postcopy-ram.c +21
-0+*/ + +/* +* The number of bits one host page takes up in the bitmap +* e.g. on a 64k host page, 4k Target page, host_bits=64/4=16 +*/ + host_bits =sysconf (_SC_PAGESIZE) / (1ul << tpb); +/* Should be a power of 2 */ +assert (host_bits && !(host_bits & (host_bits -1 ))); +/* +* If the host_bits isn't a division of the number of bits in long +* then the code gets a lot more complex; disallow for now +* (I'm not aware of a system where it's true anyway) +*/ +assert (((sizeof (long ) *8 ) % host_bits) ==0 ); + + mis->postcopy_pmi.host_bits = host_bits; +/* A mask, starting at bit 0, containing host_bits continuous set bits */ + mis->postcopy_pmi.host_mask = (1ul << host_bits) -1 ; + +assert ((ram_pages % host_bits) ==0 );
Quoted 8 lines
postcopy-ram.c line 135
Quoted 6 lines
postcopy-ram.c +6
-0+} + +void postcopy_pmi_destroy (MigrationIncomingState *mis) +{ +if (mis->postcopy_pmi.received_map) { +g_free (mis->postcopy_pmi.received_map);
g_free()is safe to call on NULL anyway, isn't it?
It is; fixed.
Quoted 13 lines
Quoted 13 lines
postcopy-ram.c +13
-0+/* +* Retrieve the state of the given page +* Note: This version for use by callers already holding the lock +*/ +static PostcopyPMIState postcopy_pmi_get_state_nolock ( +MigrationIncomingState *mis, +size_t bitmap_index) +{ +bool received, requested; + + received =test_hpbits (mis, bitmap_index, mis->postcopy_pmi.received_map); + requested =test_hpbits (mis, bitmap_index, mis->postcopy_pmi.requested_map); +
Quoted 8 lines
postcopy-ram.c line 212
Quoted 2 lines
postcopy-ram.c +2
-0+if (received) { +assert (!requested);Clearing the requested bit when you set the received bit seems a bit pointless. (requested && received) isn't meaningfully different from (!requested && received) but there seems no reason to go to extra trouble to avoid that state, and having the record might be interesting for gathering statistics.
Hmm yes I think you're right; but I want to think about it to convince me a bit more; this code originally started off as two really seprate bitmaps and has slowly morphed into really representing 3 states. I've added it to a TODO.
Quoted 8 lines
postcopy-ram.c line 288
Quoted 5 lines
postcopy-ram.c +5
-0+/* Called by ram_load prior to mapping the page */ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index) +{ +if (mis->postcopy_ram_state ==POSTCOPY_RAM_INCOMING_ADVISE ) {A silent no-op if you're not in the expected migration phase doesn't seem right. Should this be an
assert()instead?
No. This routine is called by the RAM code prior to doing anything with the page, but it does it in all postcopy states; it's just that we only care about it in the ADVISE state (it makes things a little cleaner - the RAM code no longer has to know about the postcopy stages).
Dave
Quoted 16 lines
Quoted 16 lines
postcopy-ram.c +9
-0+/* +* If we're in precopy-advise mode we need to track received pages even +* though we don't need to place pages atomically yet. +* In advise mode there's only a single thread, so don't need locks +*/ +set_bit (bitmap_index, mis->postcopy_pmi.received_map); + } +} +int postcopy_ram_hosttest (void ) {/* TODO: Needs guarding with CONFIG_ once we have libc's that have the defs @@ -156,5 +369,12 @@ int postcopy_ram_hosttest(void)return -1 ; }
Quoted 15 lines
Quoted 9 lines
postcopy-ram.c +7
-0+/* Called by ram_load prior to mapping the page */ +void postcopy_hook_early_receive (MigrationIncomingState *mis, +size_t bitmap_index) +{ +/* We don't support postcopy so don't care */ +} +#endif Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 30/47 Postcopy: Maintain sentmap and calculate discard
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.
--- arch_init.c | 271 ++++++++++++++++++++++++++++++++++++++- include/migration/migration.h | 12 ++ include/migration/postcopy-ram.h | 34 +++++ include/qemu/typedefs.h | 1 + migration.c | 2 + postcopy-ram.c | 111 ++++++++++++++++ savevm.c | 3 - 7 files changed, 428 insertions(+), 6 deletions(-)
arch_init.c +268 -3
@@ -40,6 +40,7 @@#include "hw/audio/audio.h" #include "sysemu/kvm.h" #include "migration/migration.h" +#include "migration/postcopy-ram.h" #include "hw/i386/smbios.h" #include "exec/address-spaces.h" #include "hw/audio/pcspk.h"
@@ -415,9 +416,15 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t **current_data,return bytes_sent; } +/* mr: The region to search for dirty pages in +* start: Start address (typically so we can continue from previous page) +* bitoffset: Pointer into which to store the offset into the dirty map +* at which the bit was found. +*/ static inline ram_addr_t migration_bitmap_find_and_reset_dirty (MemoryRegion *mr, -ram_addr_t start) +ram_addr_t start, +unsigned long *bitoffset) {unsigned long base = mr->ram_addr >>TARGET_PAGE_BITS ;unsigned long nr = base + (start >>TARGET_PAGE_BITS );
@@ -436,6 +443,7 @@ ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,clear_bit (next, migration_bitmap); migration_dirty_pages--; } + *bitoffset = next;return (next - base) <<TARGET_PAGE_BITS ; }
@@ -564,6 +572,19 @@ static void migration_bitmap_sync(void) } } +static RAMBlock *ram_find_block (const char *id) +{ +RAMBlock *block; + + QTAILQ_FOREACH(block, &ram_list.blocks, next) { +if (!strcmp (id, block->idstr)) { +return block; + } + } + +return NULL ; +} +/* * ram_save_page: Send the given page to the stream *
@@ -652,13 +673,14 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage)bool complete_round = false;int bytes_sent =0 ;MemoryRegion *mr; +unsigned long bitoffset;if (!block) block =QTAILQ_FIRST (&ram_list.blocks);while (true) { mr = block->mr; - offset =migration_bitmap_find_and_reset_dirty (mr, offset); + offset =migration_bitmap_find_and_reset_dirty (mr, offset, &bitoffset);if (complete_round && block == last_seen_block && offset >= last_offset) {break ;
@@ -676,6 +698,11 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage)/* if page is unmodified, continue to the next */ if (bytes_sent >0 ) { +MigrationState *s =migrate_get_current (); +if (s->sentmap) { +set_bit (bitoffset, s->sentmap); + } + last_sent_block = block;break ; }
@@ -735,12 +762,19 @@ void free_xbzrle_decoded_buf(void)static void migration_end (void ) { +MigrationState *s =migrate_get_current (); +if (migration_bitmap) {memory_global_dirty_log_stop ();g_free (migration_bitmap); migration_bitmap =NULL ; } +if (s->sentmap) { +g_free (s->sentmap); + s->sentmap =NULL ; + } +XBZRLE_cache_lock ();if (XBZRLE .cache) {cache_fini (XBZRLE .cache);
@@ -808,6 +842,232 @@ void ram_debug_dump_bitmap(unsigned long *todump, bool expected) } } +/* **** functions for postcopy ***** */ + +/* +* A helper to get 32 bits from a bit map; trivial for HOST_LONG_BITS=32 +* messier for 64; the bitmaps are actually long's that are 32 or 64bit +*/ +static uint32_t get_32bits_map (unsigned long *map,int64_t start) +{ +#if HOST_LONG_BITS ==64 +uint64_t tmp64; + + tmp64 = map[start /64 ]; +return (start &32 ) ? (tmp64 >>32 ) : (tmp64 &0xffffffffu ); +#elif HOST_LONG_BITS ==32 +/* +* Irrespective of host endianness, sentmap[n] is for pages earlier +* than sentmap[n+1] so we can't just cast up +*/ +return map[start /32 ]; +#else +#error "Host long other than 64/32 not supported" +#endif +} + +/* +* A helper to put 32 bits into a bit map; trivial for HOST_LONG_BITS=32 +* messier for 64; the bitmaps are actually long's that are 32 or 64bit +*/ +__attribute__ (( unused ))/* Until later in patch series */ +static void put_32bits_map (unsigned long *map,int64_t start, +uint32_t v) +{ +#if HOST_LONG_BITS ==64 +uint64_t tmp64 = v; +uint64_t mask =0xffffffffu ; + +if (start &32 ) { + tmp64 = tmp64 <<32 ; + mask = mask <<32 ; + } + + map[start /64 ] = (map[start /64 ] & ~mask) | tmp64; +#elif HOST_LONG_BITS ==32 +/* +* Irrespective of host endianness, sentmap[n] is for pages earlier +* than sentmap[n+1] so we can't just cast up +*/ + map[start /32 ] = v; +#else +#error "Host long other than 64/32 not supported" +#endif +} + +/* +* When working on 32bit chunks of a bitmap where the only valid section +* is between start..end (inclusive), generate a mask with only those +* valid bits set for the current 32bit word within that bitmask. +*/ +static int make_32bit_mask (unsigned long start,unsigned long end, +unsigned long cur32) +{ +unsigned long first32, last32; +uint32_t mask = ~(uint32_t )0 ; + first32 = start /32 ; + last32 = end /32 ; + +if ((cur32 == first32) && (start &31 )) { +/* e.g. (start & 31) = 3 +* 1 << . -> 2^3 +* . - 1 -> 2^3 - 1 i.e. mask 2..0 +* ~. -> mask 31..3 +*/ + mask &= ~((((uint32_t )1 ) << (start &31 )) -1 ); + } + +if ((cur32 == last32) && ((end &31 ) !=31 )) { +/* e.g. (end & 31) = 3 +* . +1 -> 4 +* 1 << . -> 2^4 +* . -1 -> 2^4 - 1 +* = mask set 3..0 +*/ + mask &= (((uint32_t )1 ) << ((end &31 ) +1 )) -1 ; + } + +return mask; +} + +/* +* Callback from ram_postcopy_each_ram_discard for each RAMBlock +* start,end: Indexes into the bitmap for the first and last bit +* representing the named block +*/ +static int pc_send_discard_bm_ram (MigrationState *ms, +PostcopyDiscardState *pds, +unsigned long start,unsigned long end) +{ +/* +* There is no guarantee that start, end are on convenient 32bit multiples +* (We always send 32bit chunks over the wire, irrespective of long size) +*/ +unsigned long first32, last32, cur32; + first32 = start /32 ; + last32 = end /32 ; + +for (cur32 = first32; cur32 <= last32; cur32++) { +/* Deal with start/end not on alignment */ +uint32_t mask =make_32bit_mask (start, end, cur32); + +uint32_t data =get_32bits_map (ms->sentmap, cur32 *32 ); + data &= mask; + +if (data) { +postcopy_discard_send_chunk (ms, pds, (cur32-first32) *32 , data); + } + } + +return 0 ; +} + +/* +* Utility for the outgoing postcopy code. +* Calls postcopy_send_discard_bm_ram for each RAMBlock +* passing it bitmap indexes and name. +* Returns: 0 on success +* (qemu_ram_foreach_block ends up passing unscaled lengths +* which would mean postcopy code would have to deal with target page) +*/ +static int pc_each_ram_discard (MigrationState *ms) +{ +struct RAMBlock *block; +int ret; + + QTAILQ_FOREACH(block, &ram_list.blocks, next) { +unsigned long first = block->offset >>TARGET_PAGE_BITS ; +unsigned long last = (block->offset + (block->length-1 )) + >>TARGET_PAGE_BITS ; +PostcopyDiscardState *pds =postcopy_discard_send_init (ms, + first &31 , + block->idstr); + +/* +* Postcopy sends chunks of bitmap over the wire, but it +* just needs indexes at this point, avoids it having +* target page specific code. +*/ + ret =pc_send_discard_bm_ram (ms, pds, first, last); +postcopy_discard_send_finish (ms, pds); +if (ret) { +return ret; + } + } + +return 0 ; +} + +/* +* Transmit the set of pages to be discarded after precopy to the target +* these are pages that have been sent previously but have been dirtied +* Hopefully this is pretty sparse +*/ +int ram_postcopy_send_discard_bitmap (MigrationState *ms) +{ +/* This should be our last sync, the src is now paused */ +migration_bitmap_sync (); + +/* +* Update the sentmap to be sentmap&=dirty +*/ +bitmap_and (ms->sentmap, ms->sentmap, migration_bitmap, +last_ram_offset () >>TARGET_PAGE_BITS ); + + +DPRINTF ("Dumping merged sentmap" ); +#ifdef DEBUG_POSTCOPY +ram_debug_dump_bitmap (ms->sentmap, false); +#endif + +return pc_each_ram_discard (ms); +} + +/* +* At the start of the postcopy phase of migration, any now-dirty +* precopied pages are discarded. +* +* start..end is an inclusive range of bits indexed in the source +* VMs bitmap for this RAMBlock, source_target_page_bits tells +* us what one of those bits represents. +* +* start/end are offsets from the start of the bitmap for RAMBlock 'block_name' +* +* Returns 0 on success. +*/ +int ram_discard_range (MigrationIncomingState *mis, +const char *block_name, +uint64_t start,uint64_t end) +{ +assert (end >= start); + +RAMBlock *rb =ram_find_block (block_name); + +if (!rb) { +error_report ("ram_discard_range: Failed to find block '%s'" , + block_name); +return -1 ; + } + +uint64_t index_offset = rb->offset >>TARGET_PAGE_BITS ; +postcopy_pmi_discard_range (mis, start + index_offset, (end - start) +1 ); + +/* +1 gives the byte after the end of the last page to be discarded */ +ram_addr_t end_offset = (end+1 ) <<TARGET_PAGE_BITS ; +uint8_t *host_startaddr = rb->host + (start <<TARGET_PAGE_BITS ); +uint8_t *host_endaddr; + +if (end_offset <= rb->length) { + host_endaddr = rb->host + (end_offset-1 ); +return postcopy_ram_discard_range (mis, host_startaddr, host_endaddr); + }else { +error_report ("ram_discard_range: Overrun block '%s' (%" PRIu64 +"/%" PRIu64"/%zu)" , + block_name, start, end, rb->length); +return -1 ; + } +} +static int ram_save_setup (QEMUFile *f,void *opaque) {RAMBlock *block;
@@ -846,7 +1106,6 @@ static int ram_save_setup(QEMUFile *f, void *opaque)acct_clear (); } -qemu_mutex_lock_iothread ();qemu_mutex_lock_ramlist (); bytes_transferred =0 ;
@@ -856,6 +1115,12 @@ static int ram_save_setup(QEMUFile *f, void *opaque) migration_bitmap =bitmap_new (ram_bitmap_pages);bitmap_set (migration_bitmap,0 , ram_bitmap_pages); +if (migrate_postcopy_ram ()) { +MigrationState *s =migrate_get_current (); + s->sentmap =bitmap_new (ram_bitmap_pages); +bitmap_clear (s->sentmap,0 , ram_bitmap_pages); + } +/* * Count the total number of pages used by ram blocks not including any * gaps due to alignment or unplugs.
include/migration/migration.h +12 -0
@@ -122,6 +122,13 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ volatile bool start_postcopy; + +/* bitmap of pages that have been sent at least once +* only maintained and used in postcopy at the moment +* where it's used to send the dirtymap at the start +* of the postcopy phase +*/ +unsigned long *sentmap; };void process_incoming_migration (QEMUFile *f);
@@ -191,6 +198,11 @@ double xbzrle_mig_cache_miss_rate(void);void ram_handle_compressed (void *host,uint8_t ch,uint64_t size);void ram_debug_dump_bitmap (unsigned long *todump,bool expected); +/* For outgoing discard bitmap */ +int ram_postcopy_send_discard_bitmap (MigrationState *ms); +/* For incoming postcopy discard */ +int ram_discard_range (MigrationIncomingState *mis,const char *block_name, +uint64_t start,uint64_t end);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +34 -0
@@ -28,4 +28,38 @@ void postcopy_pmi_destroy(MigrationIncomingState *mis);void postcopy_pmi_discard_range (MigrationIncomingState *mis,size_t start,size_t npages);void postcopy_pmi_dump (MigrationIncomingState *mis); + +/* +* Discard the contents of memory start..end inclusive. +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +uint8_t *end); + + +/* +* Called at the start of each RAMBlock by the bitmap code +* offset is the bit within the first 32bit chunk of mask +* that represents the first page of the RAM Block +* Returns a new PDS +*/ +PostcopyDiscardState *postcopy_discard_send_init (MigrationState *ms, +uint8_t offset, +const char *name); + +/* +* Called by the bitmap code for each chunk to discard +* May send a discard message, may just leave it queued to +* be sent later +*/ +void postcopy_discard_send_chunk (MigrationState *ms,PostcopyDiscardState *pds, +unsigned long pos,uint32_t bitmap); + +/* +* Called at the end of each RAMBlock by the bitmap code +* Sends any outstanding discard messages, frees the PDS +*/ +void postcopy_discard_send_finish (MigrationState *ms, +PostcopyDiscardState *pds); +#endif
include/qemu/typedefs.h +1 -0
@@ -63,6 +63,7 @@ typedef struct PCIEAERLog PCIEAERLog;typedef struct PCIEAERErr PCIEAERErr ;typedef struct PCIEPort PCIEPort ;typedef struct PCIESlot PCIESlot ; +typedef struct PostcopyDiscardState PostcopyDiscardState ;typedef struct QEMUSizedBuffer QEMUSizedBuffer ;typedef struct MSIMessage MSIMessage ;typedef struct SerialState SerialState ;
migration.c +2 -0
@@ -22,6 +22,7 @@#include "block/block.h" #include "qemu/sockets.h" #include "migration/block.h" +#include "migration/postcopy-ram.h" #include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h"
@@ -947,6 +948,7 @@ static void *migration_thread(void *opaque) }else {int ret; +DPRINTF ("done iterating\n" );qemu_mutex_lock_iothread (); start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER );
postcopy-ram.c +111 -0
128 lines not on this page · raw message
savevm.c +0 -3
@@ -1267,15 +1267,12 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,* we know there must be at least 1 bit set due to the loop entry * If there is no 0 firstzero will be 32 */ -/* TODO - ram_discard_range gets added in a later patch int ret =ram_discard_range (mis, ramid, startaddr + firstset - first_bit_offset, startaddr + (firstzero -1 ) - first_bit_offset); -ret = -1; if (ret) {return ret; } -*/ /* mask= .?0000000000 */ /* ^fz ^fs */
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 30/47] Postcopy: Maintain sentmap and calculate discard
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.
I find this one really hard to wrap my head around, and I'm having trouble putting my finger on why.
I do wonder if the "base + tiny bitmap" encodinng for the discard list over the wire is the best choice. It seems to involve a bunch of rather tedious code rejigging the bitmap into 32-bit chunks, and a bunch of rather hard to follow code moving back and forth between that encoding and simple address or page ranges for handling the actual discards. It also involves sending the bit offsets for the start of each ram block over the wire, which feels like it should be an internal detail.
Would just a simple list of start..end or start/len pairs end up simpler overall? Converting the bitmap used to track it on the source into ranges would be a little fiddly, but I suspect less so than the code to split into 32-bit pieces.
It might also be a bit more robust against possible future options for source host vs. dest host vs. target page size, since the source can construct it in terms if its granularity constraints, and destination can round each chunk out to its own granularity.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 30/47] Postcopy: Maintain sentmap and calculate discard
Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Quoted 6 lines
Quoted 3 lines
Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.I find this one really hard to wrap my head around, and I'm having trouble putting my finger on why.
It seems to turn out fairly compact, and a lot of the time the bitmap we're trying to transfer is very sparse, but seems to be a little clumpy.
Quoted 8 lines
I do wonder if the "base + tiny bitmap" encodinng for the discard list over the wire is the best choice. It seems to involve a bunch of rather tedious code rejigging the bitmap into 32-bit chunks, and a bunch of rather hard to follow code moving back and forth between that encoding and simple address or page ranges for handling the actual discards. It also involves sending the bit offsets for the start of each ram block over the wire, which feels like it should be an internal detail.
Yes the fiddling into 32bit chunks is a bit messier than it was originally; the problem here is that the migration bitmap (for no apparent reason) uses 'long' so the complexity all comes from having the internal structure be flexible and wanting not to pass that flexibility onto the wire.
Quoted 4 lines
Would just a simple list of start..end or start/len pairs end up simpler overall? Converting the bitmap used to track it on the source into ranges would be a little fiddly, but I suspect less so than the code to split into 32-bit pieces.
In the end you have to traverse that bitmap somewhere, since I already had the data in the form of a bitmap it seemed reasonable to take advantage of it as a compact representation.
Quoted 4 lines
It might also be a bit more robust against possible future options for source host vs. dest host vs. target page size, since the source can construct it in terms if its granularity constraints, and destination can round each chunk out to its own granularity.
That already happens here as the destination reconsistutes the addresses from the bitmap.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 31/47 postcopy: Incoming initialisation
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- arch_init.c | 11 ++++ include/migration/migration.h | 1 + include/migration/postcopy-ram.h | 12 +++++ migration.c | 1 + postcopy-ram.c | 110 ++++++++++++++++++++++++++++++++++++++- savevm.c | 4 ++ 6 files changed, 138 insertions(+), 1 deletion(-)
arch_init.c +11 -0
@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) } } +/* +* Allocate data structures etc needed by incoming migration with postcopy-ram +* postcopy-ram's similarly names postcopy_ram_incoming_init does the work +*/ +int ram_postcopy_incoming_init (MigrationIncomingState *mis) +{ +size_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +return postcopy_ram_incoming_init (mis, ram_pages); +} +static int ram_load (QEMUFile *f,void *opaque,int version_id) {ram_addr_t addr;
include/migration/migration.h +1 -0
@@ -203,6 +203,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms);/* For incoming postcopy discard */ int ram_discard_range (MigrationIncomingState *mis,const char *block_name,uint64_t start,uint64_t end); +int ram_postcopy_incoming_init (MigrationIncomingState *mis);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +12 -0
@@ -19,6 +19,18 @@int postcopy_ram_hosttest (void );/* +* Initialise postcopy-ram, setting the RAM to a state where we can go into +* postcopy later; must be called prior to any precopy. +* called from arch_init's similarly named ram_postcopy_incoming_init +*/ +int postcopy_ram_incoming_init (MigrationIncomingState *mis,size_t ram_pages); + +/* +* At the end of a migration where postcopy_ram_incoming_init was called. +*/ +int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis); + +/* * In 'advise' mode record that a page has been received. */ void postcopy_hook_early_receive(MigrationIncomingState *mis,
migration.c +1 -0
@@ -99,6 +99,7 @@ MigrationIncomingState *migration_incoming_state_init(QEMUFile* f)void migration_incoming_state_destroy (void ) { +postcopy_pmi_destroy (mis_current);g_free (mis_current); mis_current =NULL ; }
postcopy-ram.c +109 -1
128 lines not on this page · raw message
savevm.c +4 -0
@@ -1166,6 +1166,10 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } + mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_ADVISE ;/*
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 31/47] postcopy: Incoming initialisation
Quoted 20 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 11 ++++ include/migration/migration.h | 1 + include/migration/postcopy-ram.h | 12 +++++ migration.c | 1 + postcopy-ram.c | 110 ++++++++++++++++++++++++++++++++++++++- savevm.c | 4 ++ 6 files changed, 138 insertions(+), 1 deletion(-)arch_init.c +0
-0@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) } }
arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) · line 1357
Quoted 10 lines
arch_init.c +10
-0+/* +* Allocate data structures etc needed by incoming migration with postcopy-ram +* postcopy-ram's similarly names postcopy_ram_incoming_init does the work +*/ +int ram_postcopy_incoming_init (MigrationIncomingState *mis) +{ +size_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +return postcopy_ram_incoming_init (mis, ram_pages); +}
Um.. yeah. I'm sure ram_postcopy_incoming_init versus postcopy_ram_incoming_init won't get confusing o_O.
[snip]
Quoted 22 lines
arch_init.c +22
-0+/* +* Setup an area of RAM so that it *can* be used for postcopy later; this +* must be done right at the start prior to pre-copy. +* opaque should be the MIS. +*/ +static int init_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length,void *opaque) +{ +MigrationIncomingState *mis = opaque; + +DPRINTF ("init_area: %s: %p offset=%zx length=%zd(%zx)" , + block_name, host_addr, offset, length, length); +/* +* We need the whole of RAM to be truly empty for postcopy, so things +* like ROMs and any data tables built during init must be zero'd +* - we're going to get the copy from the source anyway. +*/ +if (postcopy_ram_discard_range (mis, host_addr, (host_addr + length -1 ))) { +return -1 ; + } + +/*
arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
Quoted 10 lines
arch_init.c +10
-0+* We also need the area to be normal 4k pages, not huge pages +* (otherwise we can't be sure we can use remap_anon_pages to put +* a 4k page in later). THP might come along and map a 2MB page +* and when it's partially accessed in precopy it might not break +* it down, but leave a 2MB zero'd page. +*/ +if (madvise (host_addr, length,MADV_NOHUGEPAGE )) { +perror ("init_area: NOHUGEPAGE" ); +return -1 ; + }
I'm assuming this is because remap_anon_pages() can't automatically split a THP itself. It's not immediately obvious to me why it can't though.
Also.. what effect will this have on an actual hugetlbfs memory region? If there's code to handle that case I haven't spotted it yet.
arch_init.c
Quoted 12 lines
arch_init.c +12
-0+ +return 0 ; +} + +/* +* At the end of migration, undo the effects of init_area +* opaque should be the MIS. +*/ +static int cleanup_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length,void *opaque) +{ +/* Turn off userfault here as well? */
This comment appears to be obsoleted by the code below.
Quoted 90 lines
arch_init.c +65
-0+ +DPRINTF ("cleanup_area: %s: %p offset=%zx length=%zd(%zx)" , + block_name, host_addr, offset, length, length); +/* +* We turned off hugepage for the precopy stage with postcopy enabled +* we can turn it back on now. +*/ +if (madvise (host_addr, length,MADV_HUGEPAGE )) { +perror ("init_area: HUGEPAGE" ); +return -1 ; + } + +/* +* We can also turn off userfault now since we should have all the +* pages. It can be useful to leave it on to debug postcopy +* if you're not sure it's always getting every page. +*/ +if (madvise (host_addr, length,MADV_NOUSERFAULT )) { +perror ("init_area: NOUSERFAULT" ); +return -1 ; + } + + return 0; +} + +/* +* Initialise postcopy-ram, setting the RAM to a state where we can go into +* postcopy later; must be called prior to any precopy. +* called from arch_init's similarly named ram_postcopy_incoming_init +*/ +int postcopy_ram_incoming_init (MigrationIncomingState *mis,size_t ram_pages) +{ +postcopy_pmi_init (mis, ram_pages); + +if (qemu_ram_foreach_block (init_area, mis)) { +return -1 ; + } + +return 0 ; +} + +/* +* At the end of a migration where postcopy_ram_incoming_init was called. +*/ +int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis) +{ +/* TODO: Join the fault thread once we're sure it will exit */ +if (qemu_ram_foreach_block (cleanup_area, mis)) { +return -1 ; + } + +return 0 ; +} +#else /* No target OS support, stubs just fail */ @@ -404,6 +501,17 @@ void postcopy_hook_early_receive(MigrationIncomingState *mis,/* We don't support postcopy so don't care */ } +int postcopy_ram_incoming_init (MigrationIncomingState *mis,size_t ram_pages) +{ +error_report ("postcopy_ram_incoming_init: No OS support" ); +return -1 ; +} + +int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis) +{ +assert (0 ); +} +void postcopy_pmi_destroy (MigrationIncomingState *mis) {/* Called in normal cleanup path - so it's OK */ savevm.c +4
-0@@ -1166,6 +1166,10 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,return -1 ; } +if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } + mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_ADVISE ;/*
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 31/47] postcopy: Incoming initialisation
Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 11 ++++ include/migration/migration.h | 1 + include/migration/postcopy-ram.h | 12 +++++ migration.c | 1 + postcopy-ram.c | 110 ++++++++++++++++++++++++++++++++++++++- savevm.c | 4 ++ 6 files changed, 138 insertions(+), 1 deletion(-)arch_init.c +0
-0@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) } }
Quoted 13 lines
arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) · line 1357
Quoted 10 lines
arch_init.c +10
-0+/* +* Allocate data structures etc needed by incoming migration with postcopy-ram +* postcopy-ram's similarly names postcopy_ram_incoming_init does the work +*/ +int ram_postcopy_incoming_init (MigrationIncomingState *mis) +{ +size_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +return postcopy_ram_incoming_init (mis, ram_pages); +}Um.. yeah. I'm sure
ram_postcopy_incoming_initversuspostcopy_ram_incoming_initwon't get confusingo_O.
agreed; that's why I put the comments on. My problem here is that:
1) last_ram_offset() comes from code that's poisoned so it can't be built in a target independent file
2) I'd managed so far (with a couple of hacks) to keep postcopy_ram.c target independent.
3) ram_ is the prefix for external names in arch_init.c
4) postcopy_ram_ is the prefix for external names in postcopy_ram.c
If I threw in the towel and made postcopy_ram target dependent it would remove the need for that wrapper; it might be the best bet. (Other naming suggestions also welcome)
Quoted 29 lines
[snip]
Quoted 28 lines
arch_init.c +28
-0+/* +* Setup an area of RAM so that it *can* be used for postcopy later; this +* must be done right at the start prior to pre-copy. +* opaque should be the MIS. +*/ +static int init_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length,void *opaque) +{ +MigrationIncomingState *mis = opaque; + +DPRINTF ("init_area: %s: %p offset=%zx length=%zd(%zx)" , + block_name, host_addr, offset, length, length); +/* +* We need the whole of RAM to be truly empty for postcopy, so things +* like ROMs and any data tables built during init must be zero'd +* - we're going to get the copy from the source anyway. +*/ +if (postcopy_ram_discard_range (mis, host_addr, (host_addr + length -1 ))) { +return -1 ; + } + +/* +* We also need the area to be normal 4k pages, not huge pages +* (otherwise we can't be sure we can use remap_anon_pages to put +* a 4k page in later). THP might come along and map a 2MB page +* and when it's partially accessed in precopy it might not break +* it down, but leave a 2MB zero'd page. +*/
Quoted 8 lines
arch_init.c
Quoted 4 lines
arch_init.c +4
-0+if (madvise (host_addr, length,MADV_NOHUGEPAGE )) { +perror ("init_area: NOHUGEPAGE" ); +return -1 ; + }I'm assuming this is because
remap_anon_pages()can't automatically split a THP itself. It's not immediately obvious to me why it can't though.
No, I think this restriction stems from two things:
1) remap_anon_pages not allowing us to map into an area that's already got a page present - it's a good protection mechanism against us doing something stupid and receiving a page that we already received and the destination is busy accessing.
2) We wouldn't want THP to decide to convert a page that we'd only partially received into a HP because we wouldn't then receive userfault messages for it.
(Although it might be best to check with Andrea).
(1) might disappear with the modifications to replace remap_anon_pages that Andrea is working on.
Quoted 2 lines
Also.. what effect will this have on an actual hugetlbfs memory region? If there's code to handle that case I haven't spotted it yet.
I wouldn't expect this code to work with hugetlbfs mappings.
Quoted 6 lines
Quoted 6 lines
arch_init.c +6
-0+ +return 0 ; +} + +/* +* At the end of migration, undo the effects of init_area
Quoted 8 lines
arch_init.c
Quoted 6 lines
+ * opaque should be the MIS.
+ */arch_init.c +4
-0+static int cleanup_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length,void *opaque) +{ +/* Turn off userfault here as well? */ This comment appears to be obsoleted by the code below.
Thanks; I've squashed it.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 32/47 postcopy: ram_enable_notify to switch on userfault
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- include/migration/migration.h | 2 ++ include/migration/postcopy-ram.h | 6 +++++ postcopy-ram.c | 49 +++++++++++++++++++++++++++++++++++++++- savevm.c | 9 ++++++++ 4 files changed, 65 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
@@ -87,6 +87,8 @@ struct MigrationIncomingState { POSTCOPY_RAM_INCOMING_END } postcopy_ram_state; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ PostcopyPMI postcopy_pmi;
include/migration/postcopy-ram.h +6 -0
@@ -19,6 +19,12 @@int postcopy_ram_hosttest (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from arch_init's similarly named ram_postcopy_incoming_init
postcopy-ram.c +48 -1
63 lines not on this page · raw message
savevm.c +9 -0
@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis) mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_LISTENING ; +/* +* Sensitise RAM - can now generate requests for blocks that don't exist +* However, at this point the CPU shouldn't be running, and the IO +* shouldn't be doing anything yet so don't actually expect requests +*/ +if (postcopy_ram_enable_notify (mis)) { +return -1 ; + } +/* TODO start up the postcopy listening thread */ return 0 ; }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 499
Quoted 3 lines
postcopy-ram.c +3
-0+static int postcopy_ram_sensitise_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length, +void *opaque)
Weird name, and I'm not referring to the British -ise. :)
Perhaps ram_block_enable_userfault or ram_block_enable_notify? It helps clarity to limit the use of the "postcopy_ram_" prefix for static function.
Paolo
Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault
Quoted 9 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 499
Quoted 3 lines
postcopy-ram.c +3
-0+static int postcopy_ram_sensitise_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length, +void *opaque)Weird name, and I'm not referring to the British -ise. :)
Perhaps
ram_block_enable_userfaultorram_block_enable_notify? It helps clarity to limit the use of the "postcopy_ram_" prefix for static function.
Yep, that's fair enough; I'll make it ram_block_enable_notify.
Dave
Quoted 1 line
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault
In reply to Dr. David Alan Gilbert (git)Quoted 45 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 2 ++ include/migration/postcopy-ram.h | 6 +++++ postcopy-ram.c | 49 +++++++++++++++++++++++++++++++++++++++- savevm.c | 9 ++++++++ 4 files changed, 65 insertions(+), 1 deletion(-)include/migration/migration.h +2
-0@@ -87,6 +87,8 @@ struct MigrationIncomingState { POSTCOPY_RAM_INCOMING_END } postcopy_ram_state; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ PostcopyPMI postcopy_pmi;include/migration/postcopy-ram.h +6
-0@@ -19,6 +19,12 @@int postcopy_ram_hosttest (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from arch_init's similarly named ram_postcopy_incoming_init postcopy-ram.c +0
-0
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494
Quoted 10 lines
postcopy-ram.c +7
-0@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)return 0 ; } +/* +* Mark the given area of RAM as requiring notification to unwritten areas +* Used as a callback on qemu_ram_foreach_block. +* host_addr: Base of area to mark +* offset: Offset in the whole ram arena +* length: Length of the section +* opaque: Unused
^^^^^^
This appears to be wrong - opaque is used to find the MIS.
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502
Quoted 8 lines
postcopy-ram.c +8
-0+* Returns 0 on success +*/ +static int postcopy_ram_sensitise_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length, +void *opaque) +{ +MigrationIncomingState *mis = opaque; +uint64_t tokern[2 ];
"tokern"?
Quoted 66 lines
postcopy-ram.c +33
-1+ +if (madvise (host_addr, length,MADV_USERFAULT )) { +perror ("postcopy_ram_sensitise_area madvise" ); +return -1 ; + } + +/* Now tell our userfault_fd that it's responsible for this area */ + tokern[0 ] = (uint64_t )(uintptr_t )host_addr |1 ;/* 1 means register area */ + tokern[1 ] = (uint64_t )(uintptr_t )host_addr + length; +if (write (mis->userfault_fd, tokern,16 ) !=16 ) { +perror ("postcopy_ram_sensitise_area write" ); +madvise (host_addr, length,MADV_NOUSERFAULT ); +return -1 ; + } + + return 0; +} + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +/* Mark so that we get notified of accesses to unwritten areas */ +if (qemu_ram_foreach_block (postcopy_ram_sensitise_area, mis)) { +return -1 ; + } + +return 0 ; +} +#else /* No target OS support, stubs just fail */ -int postcopy_ram_hosttest (void ) {error_report ("postcopy_ram_hosttest: No OS support" );@@ -528,6 +570,11 @@ int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, {assert (0 ); } + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +assert (0 ); +}#endif /* ------------------------------------------------------------------------- */ savevm.c +9
-0@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis) mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_LISTENING ; +/* +* Sensitise RAM - can now generate requests for blocks that don't exist +* However, at this point the CPU shouldn't be running, and the IO +* shouldn't be doing anything yet so don't actually expect requests +*/ +if (postcopy_ram_enable_notify (mis)) { +return -1 ; + } +/* TODO start up the postcopy listening thread */ return 0 ; }
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault
Quoted 58 lines · * David Gibson ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 2 ++ include/migration/postcopy-ram.h | 6 +++++ postcopy-ram.c | 49 +++++++++++++++++++++++++++++++++++++++- savevm.c | 9 ++++++++ 4 files changed, 65 insertions(+), 1 deletion(-)include/migration/migration.h +2
-0@@ -87,6 +87,8 @@ struct MigrationIncomingState { POSTCOPY_RAM_INCOMING_END } postcopy_ram_state; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ PostcopyPMI postcopy_pmi;include/migration/postcopy-ram.h +6
-0@@ -19,6 +19,12 @@int postcopy_ram_hosttest (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from arch_init's similarly named ram_postcopy_incoming_init postcopy-ram.c +0
-0postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494
Quoted 10 lines
postcopy-ram.c +7
-0@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)return 0 ; } +/* +* Mark the given area of RAM as requiring notification to unwritten areas +* Used as a callback on qemu_ram_foreach_block. +* host_addr: Base of area to mark +* offset: Offset in the whole ram arena +* length: Length of the section +* opaque: Unused ^^^^^^
Quoted 1 line
This appears to be wrong - opaque is used to find the MIS.
Fixed.
Quoted 11 lines
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502
Quoted 9 lines
postcopy-ram.c +8
-0+* Returns 0 on success +*/ +static int postcopy_ram_sensitise_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length, +void *opaque) +{ +MigrationIncomingState *mis = opaque; +uint64_t tokern[2 ];"tokern"?
Now "to_kernel"
Dave
Quoted 55 lines
Quoted 55 lines
postcopy-ram.c +33
-1+ +if (madvise (host_addr, length,MADV_USERFAULT )) { +perror ("postcopy_ram_sensitise_area madvise" ); +return -1 ; + } + +/* Now tell our userfault_fd that it's responsible for this area */ + tokern[0 ] = (uint64_t )(uintptr_t )host_addr |1 ;/* 1 means register area */ + tokern[1 ] = (uint64_t )(uintptr_t )host_addr + length; +if (write (mis->userfault_fd, tokern,16 ) !=16 ) { +perror ("postcopy_ram_sensitise_area write" ); +madvise (host_addr, length,MADV_NOUSERFAULT ); +return -1 ; + } + + return 0; +} + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +/* Mark so that we get notified of accesses to unwritten areas */ +if (qemu_ram_foreach_block (postcopy_ram_sensitise_area, mis)) { +return -1 ; + } + +return 0 ; +} +#else /* No target OS support, stubs just fail */ -int postcopy_ram_hosttest (void ) {error_report ("postcopy_ram_hosttest: No OS support" );@@ -528,6 +570,11 @@ int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start, {assert (0 ); } + +int postcopy_ram_enable_notify (MigrationIncomingState *mis) +{ +assert (0 ); +}#endif /* ------------------------------------------------------------------------- */ savevm.c +0
-0@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis) mis->postcopy_ram_state =POSTCOPY_RAM_INCOMING_LISTENING ;
Quoted 18 lines
Quoted 12 lines
savevm.c +9
-0+/* +* Sensitise RAM - can now generate requests for blocks that don't exist +* However, at this point the CPU shouldn't be running, and the IO +* shouldn't be doing anything yet so don't actually expect requests +*/ +if (postcopy_ram_enable_notify (mis)) { +return -1 ; + } +/* TODO start up the postcopy listening thread */ return 0 ; }Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault
Quoted 75 lines · Dr. David Alan Gilbert
Quoted 58 lines · * David Gibson ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 2 ++ include/migration/postcopy-ram.h | 6 +++++ postcopy-ram.c | 49 +++++++++++++++++++++++++++++++++++++++- savevm.c | 9 ++++++++ 4 files changed, 65 insertions(+), 1 deletion(-)include/migration/migration.h +2
-0@@ -87,6 +87,8 @@ struct MigrationIncomingState { POSTCOPY_RAM_INCOMING_END } postcopy_ram_state; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *return_path;QemuMutex rp_mutex;/* We send replies from multiple threads */ PostcopyPMI postcopy_pmi;include/migration/postcopy-ram.h +6
-0@@ -19,6 +19,12 @@int postcopy_ram_hosttest (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from arch_init's similarly named ram_postcopy_incoming_init postcopy-ram.c +0
-0postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494
Quoted 10 lines
postcopy-ram.c +7
-0@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)return 0 ; } +/* +* Mark the given area of RAM as requiring notification to unwritten areas +* Used as a callback on qemu_ram_foreach_block. +* host_addr: Base of area to mark +* offset: Offset in the whole ram arena +* length: Length of the section +* opaque: Unused ^^^^^^Quoted 1 line
This appears to be wrong - opaque is used to find the MIS.
Fixed.
Quoted 11 lines
postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502
Quoted 9 lines
postcopy-ram.c +8
-0+* Returns 0 on success +*/ +static int postcopy_ram_sensitise_area (const char *block_name,void *host_addr, +ram_addr_t offset,ram_addr_t length, +void *opaque) +{ +MigrationIncomingState *mis = opaque; +uint64_t tokern[2 ];"tokern"?
Quoted 1 line
Now "
to_kernel"
Ah! I thought it was just mispelled "token".
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 33/47 Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +3 -0
@@ -125,6 +125,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ volatile bool start_postcopy; +/* Flag set once the migration thread is running (and needs joining) */ +volatile bool started_migration_thread; +/* bitmap of pages that have been sent at least once * only maintained and used in postcopy at the moment * where it's used to send the dirtymap at the start
migration.c +182 -19
274 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 15 lines
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)include/migration/migration.h +0
-0
include/migration/migration.h struct MigrationState · line 129
Quoted 5 lines
include/migration/migration.h +2
-0@@ -125,6 +125,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ volatile bool start_postcopy; +/* Flag set once the migration thread is running (and needs joining) */ +volatile bool started_migration_thread;
volatile almost never does what you think it does. :)
In this case, I think only one thread reads/writes the variable so "volatile" is unnecessary.
Otherwise, you would need to add actual memory barriers, atomic operations, or synchronization primitives.
For start_postcopy, it is okay because it is just a hint to the compiler and the processor will eventually see the assignment. For this case QEMU has atomic_read/atomic_set (corresponding to __ATOMIC_RELAXED in C/C++1x), so you could use those as well.
Quoted 47 lines
include/migration/migration.h +0
-0/* bitmap of pages that have been sent at least once * only maintained and used in postcopy at the moment * where it's used to send the dirtymap at the start migration.c +12
-3@@ -475,7 +475,10 @@ static void migrate_fd_cleanup(void *opaque)if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread (); -qemu_thread_join (&s->thread); +if (s->started_migration_thread) { +qemu_thread_join (&s->thread); + s->started_migration_thread = false; + }qemu_mutex_lock_iothread ();qemu_fclose (s->file);@@ -872,7 +875,6 @@ out:return NULL ; } -__attribute__ (( unused ))/* Until later in patch series */ static int open_outgoing_return_path (MigrationState *ms) {@@ -890,7 +892,6 @@ static int open_outgoing_return_path(MigrationState *ms)return 0 ; } -__attribute__ (( unused ))/* Until later in patch series */ static void await_outgoing_return_path_close (MigrationState *ms) {/* @@ -908,6 +909,97 @@ static void await_outgoing_return_path_close(MigrationState *ms)DPRINTF ("%s: Exit" , __func__); } +/* Switch from normal iteration to postcopy +* Returns non-0 on error +*/ +static int postcopy_start (MigrationState *ms) +{ +int ret; +const QEMUSizedBuffer *qsb; +migrate_set_state (ms,MIG_STATE_ACTIVE ,MIG_STATE_POSTCOPY_ACTIVE );
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 929
Quoted 10 lines
migration.c +10
-0+ +DPRINTF ("postcopy_start\n" ); +qemu_mutex_lock_iothread (); +DPRINTF ("postcopy_start: setting run state\n" ); + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); + +if (ret <0 ) { +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ;
Please use "goto" for error returns, like
fail_locked:
qemu_mutex_unlock_iothread();
fail:
migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
return -1;Quoted 7 lines
migration.c +7
-0+ } + +/* +* in Finish migrate and with the io-lock held everything should +* be quiet, but we've potentially still got dirty pages and we +* need to tell the destination to throw any pages it's already received +* that are dirty
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 946
Quoted 10 lines
migration.c +10
-0+*/ +if (ram_postcopy_send_discard_bitmap (ms)) { +DPRINTF ("postcopy send discard bitmap failed\n" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +DPRINTF ("postcopy_start: sending req 2\n" ); +qemu_savevm_send_reqack (ms->file,2 );
Perhaps move it below qemu_file_set_rate_limit, and add trace_qemu_savevm_send_reqack?
Also what is 2/3/4? Is this just for debugging or is it part of the protocol?
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 959
Quoted 13 lines
migration.c +13
-0+/* +* send rest of state - note things that are doing postcopy +* will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX ); +DPRINTF ("postcopy_start: do state_complete\n" ); + +/* +* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/
The comments in the code are very nice. Thanks. This is a huge improvement from the last version I received.
Quoted 14 lines
migration.c +14
-0+QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +qemu_savevm_state_complete (fb); +DPRINTF ("postcopy_start: sending req 3\n" ); +qemu_savevm_send_reqack (fb,3 ); + +qemu_savevm_send_postcopy_ram_run (fb); +
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 983
Quoted 10 lines
migration.c +10
-0+/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ +if (qsb_get_length (qsb) >MAX_VM_CMD_PACKAGED_SIZE ) { +DPRINTF ("postcopy_start: Unreasonably large packaged state: %lu\n" , + (unsigned long )(qsb_get_length (qsb))); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +qemu_fclose (fb);
Close fb above migrate_set_state, and use goto as above. Or just have three labels.
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 996
Quoted 13 lines
migration.c +13
-0+return -1 ; + } +qemu_savevm_send_packaged (ms->file, qsb); +qemu_fclose (fb); + +qemu_mutex_unlock_iothread (); + +DPRINTF ("postcopy_start not finished sending ack\n" ); +qemu_savevm_send_reqack (ms->file,4 ); + + ret =qemu_file_get_error (ms->file); +if (ret) { +error_report ("postcopy_start: Migration stream errored" );
This should have been reported already.
Quoted 30 lines
migration.c +14
-0+migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); + } + +return ret; +} +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. @@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)static void *migration_thread (void *opaque) {MigrationState *s = opaque; +/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time; +bool old_vm_running = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationPhase current_active_type =MIG_STATE_ACTIVE ; +qemu_savevm_state_begin (s->file, &s->params); +if (migrate_postcopy_ram ()) { +/* Now tell the dest that it should open it's end so it can reply */ +qemu_savevm_send_openrp (s->file);
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1036
Quoted 10 lines
migration.c +10
-0+ +/* And ask it to send an ack that will make stuff easier to debug */ +qemu_savevm_send_reqack (s->file,1 ); + +/* Tell the destination that we *might* want to do postcopy later; +* if the other end can't do postcopy it should fail now, nice and +* early. +*/ +qemu_savevm_send_postcopy_ram_advise (s->file); + }
Should this be done here or in the save_state_begin function for RAM? In general, I'm curious if there are parts of postcopy_start that could/should be changed into new save state functions (with postcopy_start just iterating on all devices).
Quoted 67 lines
migration.c +41
-11s->setup_time =qemu_clock_get_ms (QEMU_CLOCK_HOST ) - setup_start; + current_active_type =MIG_STATE_ACTIVE ;migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ACTIVE );DPRINTF ("setup complete\n" );@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)" nonpost=%" PRIu64")\n" , pending_size, max_size, pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) { +/* Still a significant amount to transfer */ + + current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +if (migrate_postcopy_ram () && + s->state !=MIG_STATE_POSTCOPY_ACTIVE && + pend_nonpost ==0 && s->start_postcopy) { + +if (!postcopy_start (s)) { + current_active_type =MIG_STATE_POSTCOPY_ACTIVE ; + } + +continue ; + } +/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else {int ret; -DPRINTF ("done iterating\n" ); -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); +DPRINTF ("done iterating pending size %" PRIu64"\n" , + pending_size); + +if (s->state ==MIG_STATE_ACTIVE ) { +qemu_mutex_lock_iothread (); + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); +if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { +migrate_set_state (s, current_active_type, +MIG_STATE_ERROR ); +break ; + } + }else if (s->state ==MIG_STATE_POSTCOPY_ACTIVE ) { +DPRINTF ("postcopy end\n" ); + +qemu_savevm_state_postcopy_complete (s->file); +DPRINTF ("postcopy end after complete\n" ); - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete (s->file); } -qemu_mutex_unlock_iothread (); -if (ret <0 ) {
migration.c static void *migration_thread(void *opaque) · line 1114
Quoted 10 lines
migration.c +8
-2-migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); -break ; +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up. +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +DPRINTF ("before rp close" ); +await_outgoing_return_path_close (s);
Should this be done even if there is an error? Perhaps move it altogether out of the big migration thread while() loop?
migration.c static void *migration_thread(void *opaque) · line 1121
Quoted 9 lines
migration.c +3
-2+DPRINTF ("after rp close" ); } -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_COMPLETED ); +migrate_set_state (s, current_active_type, +MIG_STATE_COMPLETED );break ; }
This "else" is huge, can you extract it into its own function?
Quoted 39 lines
migration.c +16
-1} }if (qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); +migrate_set_state (s, current_active_type,MIG_STATE_ERROR ); +DPRINTF ("migration_thread: file is in error state\n" );break ; } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque) } } +DPRINTF ("migration_thread: After loop" );qemu_mutex_lock_iothread ();if (s->state ==MIG_STATE_COMPLETED ) {int64_t end_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)/* Notify before starting migration thread */ notifier_list_notify (&migration_state_notifiers, s); +/* Open the return path; currently for postcopy but other things might +* also want it. +*/ +if (migrate_postcopy_ram ()) { +if (open_outgoing_return_path (s)) { +error_report ("Unable to open return-path for postcopy" ); +migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ERROR ); +migrate_fd_cleanup (s); +return ; + } + } +qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); + s->started_migration_thread = true; }
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Quoted 19 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 18 lines
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)include/migration/migration.h +0
-0@@ -125,6 +125,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ volatile bool start_postcopy;
Quoted 4 lines
include/migration/migration.h struct MigrationState · line 129
Quoted 2 lines
include/migration/migration.h +2
-0+/* Flag set once the migration thread is running (and needs joining) */ +volatile bool started_migration_thread;volatile almost never does what you think it does. :)
True.
Quoted 2 lines
In this case, I think only one thread reads/writes the variable so "volatile" is unnecessary.
Lets just check that; so it's set by 'migrate_fd_connect' (from the main thread) when it spawns the thread, and it's cleared by migrate_fd_cleanup that's always run as a bh, so should always be in the main thread; so yes - always the same thread, that's nice and simple; volatile evaporated.
Quoted 5 lines
Otherwise, you would need to add actual memory barriers, atomic operations, or synchronization primitives.
For
start_postcopy, it is okay because it is just a hint to the compiler and the processor will eventually see the assignment.
Yes, in this case my understanding is that it's necessary to stop the compiler potentially moving the check outside the loop.
Quoted 3 lines
For this case
QEMU has atomic_read/atomic_set (corresponding to__ATOMIC_RELAXEDin C/C++1x), so you could use those as well.
Ah, so those look like they just volatile cast anyway.
(I've probably got some other flags I need to think about reading/writing atomically/safely).
Dave
(I'll take the other issues in this mail separately since there are quite a few). --
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Quoted 4 lines · Dr. David Alan Gilbert
Quoted 2 lines
For this case QEMU has atomic_read/atomic_set (corresponding to
__ATOMIC_RELAXEDin C/C++1x), so you could use those as well.Ah, so those look like they just volatile cast anyway.
Yeah, but it explicitly shows that the assignment is a) for a multi-threaded operation b) using relaxed semantics. It attaches the information to the use instead of the variable; it just happens that volatile is the pre-C11 way to express those.
Paolo
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Quoted 6 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Quoted 2 lines
For this case QEMU has atomic_read/atomic_set (corresponding to
__ATOMIC_RELAXEDin C/C++1x), so you could use those as well.
Quoted 6 lines
Quoted 1 line
Ah, so those look like they just volatile cast anyway.
Yeah, but it explicitly shows that the assignment is a) for a multi-threaded operation b) using relaxed semantics. It attaches the information to the use instead of the variable; it just happens that volatile is the pre-C11 way to express those.
OK, I'll use those anyway; Ideally what I'd have is a way to mark something so that it'd compile-time-fail if I didn't use an atomic_ on it, because it's the type of thing that I'm bound to forget somewhere.
Dave
Quoted 2 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
In reply to Paolo BonziniQuoted 15 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 14 lines
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)include/migration/migration.h +0
-0
<snip>
Quoted 21 lines
Quoted 8 lines
include/migration/migration.h +8
-0+/* Switch from normal iteration to postcopy +* Returns non-0 on error +*/ +static int postcopy_start (MigrationState *ms) +{ +int ret; +const QEMUSizedBuffer *qsb; +migrate_set_state (ms,MIG_STATE_ACTIVE ,MIG_STATE_POSTCOPY_ACTIVE );include/migration/migration.h
Quoted 10 lines
include/migration/migration.h +10
-0+ +DPRINTF ("postcopy_start\n" ); +qemu_mutex_lock_iothread (); +DPRINTF ("postcopy_start: setting run state\n" ); + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); + +if (ret <0 ) { +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ;Please use "goto" for error returns, like
Quoted 5 lines
fail_locked:qemu_mutex_unlock_iothread(); fail: migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR); return -1;
Done; they all end up unlocking, but I've got another label for a case that has to close the fb later.
Quoted 12 lines
Quoted 12 lines
include/migration/migration.h +12
-0+ } + +/* +* in Finish migrate and with the io-lock held everything should +* be quiet, but we've potentially still got dirty pages and we +* need to tell the destination to throw any pages it's already received +* that are dirty +*/ +if (ram_postcopy_send_discard_bitmap (ms)) { +DPRINTF ("postcopy send discard bitmap failed\n" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread ();
Quoted 8 lines
include/migration/migration.h
Quoted 5 lines
include/migration/migration.h +5
-0+return -1 ; + } + +DPRINTF ("postcopy_start: sending req 2\n" ); +qemu_savevm_send_reqack (ms->file,2 );Perhaps move it below
qemu_file_set_rate_limit, and addtrace_qemu_savevm_send_reqack?
Trace added, and also moved as requested - was the request to move it just to elimintate the other DPRINTF?
Quoted 2 lines
Also what is 2/3/4? Is this just for debugging or is it part of the protocol?
Debug; they're very useful for matching the debug streams up, especially when the timers on the two hosts are very different.
(I'm up for suggestions on how to mark the 2/3/4 for debug more clearly, especially if it meant that it didn't make the ping (ne reqack) dedicated to debug).
Quoted 36 lines
include/migration/migration.h
Quoted 13 lines
include/migration/migration.h +13
-0+/* +* send rest of state - note things that are doing postcopy +* will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX ); +DPRINTF ("postcopy_start: do state_complete\n" ); + +/* +* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ The comments in the code are very nice. Thanks. This is a huge improvement from the last version I received.
Quoted 19 lines
include/migration/migration.h +19
-0+QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +qemu_savevm_state_complete (fb); +DPRINTF ("postcopy_start: sending req 3\n" ); +qemu_savevm_send_reqack (fb,3 ); + +qemu_savevm_send_postcopy_ram_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ +if (qsb_get_length (qsb) >MAX_VM_CMD_PACKAGED_SIZE ) {
Quoted 8 lines
include/migration/migration.h
Quoted 5 lines
include/migration/migration.h +5
-0+DPRINTF ("postcopy_start: Unreasonably large packaged state: %lu\n" , + (unsigned long )(qsb_get_length (qsb))); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +qemu_fclose (fb);Close fb above
migrate_set_state, and use goto as above. Or just have three labels.
Done, it's a separate label.
Quoted 16 lines
Quoted 4 lines
include/migration/migration.h +3
-0+return -1 ; + } +qemu_savevm_send_packaged (ms->file, qsb);include/migration/migration.h
Quoted 10 lines
include/migration/migration.h +10
-0+qemu_fclose (fb); + +qemu_mutex_unlock_iothread (); + +DPRINTF ("postcopy_start not finished sending ack\n" ); +qemu_savevm_send_reqack (ms->file,4 ); + + ret =qemu_file_get_error (ms->file); +if (ret) { +error_report ("postcopy_start: Migration stream errored" );This should have been reported already.
No, sorry - I don't trust qemu_file reporting errors by itself.
Dave
(Again, the rest of the comments on this patch can wait for another mail)
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Quoted 2 lines · Dr. David Alan Gilbert
Trace added, and also moved as requested - was the request to move it just to elimintate the other DPRINTF?
Yes.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Also what is 2/3/4? Is this just for debugging or is it part of the protocol?
Debug; they're very useful for matching the debug streams up, especially when the timers on the two hosts are very different.
(I'm up for suggestions on how to mark the 2/3/4 for debug more clearly, especially if it meant that it didn't make the ping (ne reqack) dedicated to debug).
No problem, as long as it's clear to the guy matching the code against the debug output.
Paolo
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
In reply to Paolo BonziniQuoted 40 lines · * Paolo Bonzini ([email protected]) wrote:
Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:
Quoted 39 lines
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)migration.c +15
-0@@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)static void *migration_thread (void *opaque) {MigrationState *s = opaque; +/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time; +bool old_vm_running = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationPhase current_active_type =MIG_STATE_ACTIVE ; +qemu_savevm_state_begin (s->file, &s->params); +if (migrate_postcopy_ram ()) { +/* Now tell the dest that it should open it's end so it can reply */ +qemu_savevm_send_openrp (s->file); + +/* And ask it to send an ack that will make stuff easier to debug */ +qemu_savevm_send_reqack (s->file,1 ); + +/* Tell the destination that we *might* want to do postcopy later; +* if the other end can't do postcopy it should fail now, nice and +* early.
Quoted 8 lines
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1036
Quoted 3 lines
migration.c +3
-0+*/ +qemu_savevm_send_postcopy_ram_advise (s->file); + }Should this be done here or in the
save_state_beginfunction for RAM? In general, I'm curious if there are parts ofpostcopy_startthat could/should be changed into new save state functions (withpostcopy_startjust iterating on all devices).
The contents of this 'if' are generic to whatever is being postcopied, (and as per one of your other comments the _ram_ has been removed from the send_postcopy_ram_advise); so I think this is the right place for it.
Quoted 66 lines
Quoted 66 lines
migration.c +41
-10s->setup_time =qemu_clock_get_ms (QEMU_CLOCK_HOST ) - setup_start; + current_active_type =MIG_STATE_ACTIVE ;migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ACTIVE );DPRINTF ("setup complete\n" );@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)" nonpost=%" PRIu64")\n" , pending_size, max_size, pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) { +/* Still a significant amount to transfer */ + + current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +if (migrate_postcopy_ram () && + s->state !=MIG_STATE_POSTCOPY_ACTIVE && + pend_nonpost ==0 && s->start_postcopy) { + +if (!postcopy_start (s)) { + current_active_type =MIG_STATE_POSTCOPY_ACTIVE ; + } + +continue ; + } +/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else {int ret; -DPRINTF ("done iterating\n" ); -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); +DPRINTF ("done iterating pending size %" PRIu64"\n" , + pending_size); + +if (s->state ==MIG_STATE_ACTIVE ) { +qemu_mutex_lock_iothread (); + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); +if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { +migrate_set_state (s, current_active_type, +MIG_STATE_ERROR ); +break ; + } + }else if (s->state ==MIG_STATE_POSTCOPY_ACTIVE ) { +DPRINTF ("postcopy end\n" ); + +qemu_savevm_state_postcopy_complete (s->file); +DPRINTF ("postcopy end after complete\n" ); - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete (s->file); } -qemu_mutex_unlock_iothread ();
Quoted 14 lines
migration.c static void *migration_thread(void *opaque) · line 1114
Quoted 11 lines
migration.c +8
-3-if (ret <0 ) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); -break ; +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up. +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +DPRINTF ("before rp close" ); +await_outgoing_return_path_close (s);Should this be done even if there is an error? Perhaps move it altogether out of the big migration thread
while()loop?
Yes, I've made a note of that; I need to go and look at more error cases to see where it makes sense (e.g. the one above), however in the non-error case I do want it to wait here for the 'SHUT' from the destination to indicate that the destination believes migration completed correctly (or not), and that should happen before the state gets set to COMPLETED because we're waiting.
Quoted 11 lines
migration.c static void *migration_thread(void *opaque) · line 1121
Quoted 9 lines
migration.c +3
-2+DPRINTF ("after rp close" ); } -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_COMPLETED ); +migrate_set_state (s, current_active_type, +MIG_STATE_COMPLETED );break ; }This "else" is huge, can you extract it into its own function?
Done, and all the changes inside this "else" I've moved into a separate commit that deals with the end rather than the start of postcopy.
Dave
Quoted 32 lines
Quoted 32 lines
migration.c +12
-1} }if (qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); +migrate_set_state (s, current_active_type,MIG_STATE_ERROR ); +DPRINTF ("migration_thread: file is in error state\n" );break ; } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque) } } +DPRINTF ("migration_thread: After loop" );qemu_mutex_lock_iothread ();if (s->state ==MIG_STATE_COMPLETED ) {int64_t end_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)/* Notify before starting migration thread */ notifier_list_notify (&migration_state_notifiers, s); +/* Open the return path; currently for postcopy but other things might +* also want it. +*/ +if (migrate_postcopy_ram ()) { +if (open_outgoing_return_path (s)) { +error_report ("Unable to open return-path for postcopy" ); +migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ERROR ); +migrate_fd_cleanup (s); +return ;
Quoted 9 lines
migration.c void migrate_fd_connect(MigrationState *s) · line 1211
Quoted 8 lines
migration.c +4
-0+ } + } +qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); + s->started_migration_thread = true; }
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan Gilbert (git)Quoted 85 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)include/migration/migration.h +3
-0@@ -125,6 +125,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ volatile bool start_postcopy; +/* Flag set once the migration thread is running (and needs joining) */ +volatile bool started_migration_thread; +/* bitmap of pages that have been sent at least once * only maintained and used in postcopy at the moment * where it's used to send the dirtymap at the start migration.c +29
-3@@ -475,7 +475,10 @@ static void migrate_fd_cleanup(void *opaque)if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread (); -qemu_thread_join (&s->thread); +if (s->started_migration_thread) { +qemu_thread_join (&s->thread); + s->started_migration_thread = false; + }qemu_mutex_lock_iothread ();qemu_fclose (s->file);@@ -872,7 +875,6 @@ out:return NULL ; } -__attribute__ (( unused ))/* Until later in patch series */ static int open_outgoing_return_path (MigrationState *ms) {@@ -890,7 +892,6 @@ static int open_outgoing_return_path(MigrationState *ms)return 0 ; } -__attribute__ (( unused ))/* Until later in patch series */ static void await_outgoing_return_path_close (MigrationState *ms) {/* @@ -908,6 +909,97 @@ static void await_outgoing_return_path_close(MigrationState *ms)DPRINTF ("%s: Exit" , __func__); } +/* Switch from normal iteration to postcopy +* Returns non-0 on error +*/ +static int postcopy_start (MigrationState *ms) +{ +int ret; +const QEMUSizedBuffer *qsb; +migrate_set_state (ms,MIG_STATE_ACTIVE ,MIG_STATE_POSTCOPY_ACTIVE ); + +DPRINTF ("postcopy_start\n" ); +qemu_mutex_lock_iothread (); +DPRINTF ("postcopy_start: setting run state\n" ); + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); + +if (ret <0 ) { +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +/* +* in Finish migrate and with the io-lock held everything should +* be quiet, but we've potentially still got dirty pages and we +* need to tell the destination to throw any pages it's already received +* that are dirty
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 946
Quoted 10 lines
migration.c +10
-0+*/ +if (ram_postcopy_send_discard_bitmap (ms)) { +DPRINTF ("postcopy send discard bitmap failed\n" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +DPRINTF ("postcopy_start: sending req 2\n" ); +qemu_savevm_send_reqack (ms->file,2 );
Are these reqacks just for debugging, or do they affect the protocol?
Quoted 69 lines
migration.c +57
-0+/* +* send rest of state - note things that are doing postcopy +* will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually +* wrap their state up here +*/ +qemu_file_set_rate_limit (ms->file,INT64_MAX ); +DPRINTF ("postcopy_start: do state_complete\n" ); + +/* +* We need to leave the fd free for page transfers during the +* loading of the device state, so wrap all the remaining +* commands and state into a package that gets sent in one go +*/ +QEMUFile *fb =qemu_bufopen ("w" ,NULL ); +if (!fb) { +error_report ("Failed to create buffered file" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +return -1 ; + } + +qemu_savevm_state_complete (fb); +DPRINTF ("postcopy_start: sending req 3\n" ); +qemu_savevm_send_reqack (fb,3 ); + +qemu_savevm_send_postcopy_ram_run (fb); + +/* <><> end of stuff going into the package */ + qsb =qemu_buf_get (fb); + +/* Now send that blob */ +if (qsb_get_length (qsb) >MAX_VM_CMD_PACKAGED_SIZE ) { +DPRINTF ("postcopy_start: Unreasonably large packaged state: %lu\n" , + (unsigned long )(qsb_get_length (qsb))); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); +qemu_mutex_unlock_iothread (); +qemu_fclose (fb); +return -1 ; + } +qemu_savevm_send_packaged (ms->file, qsb); +qemu_fclose (fb); + +qemu_mutex_unlock_iothread (); + +DPRINTF ("postcopy_start not finished sending ack\n" ); +qemu_savevm_send_reqack (ms->file,4 ); + + ret =qemu_file_get_error (ms->file); +if (ret) { +error_report ("postcopy_start: Migration stream errored" ); +migrate_set_state (ms,MIG_STATE_POSTCOPY_ACTIVE ,MIG_STATE_ERROR ); + } + +return ret; +} +/* * Master migration thread on the source VM. * It drives the migration and pumps the data down the outgoing channel. @@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)static void *migration_thread (void *opaque) {MigrationState *s = opaque; +/* Used by the bandwidth calcs, updated later */ int64_t initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );int64_t setup_start =qemu_clock_get_ms (QEMU_CLOCK_HOST );int64_t initial_bytes =0 ;int64_t max_size =0 ;int64_t start_time = initial_time;
migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1025
Quoted 10 lines
migration.c +6
-0+bool old_vm_running = false; +/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */ +enum MigrationPhase current_active_type =MIG_STATE_ACTIVE ; +qemu_savevm_state_begin (s->file, &s->params); +if (migrate_postcopy_ram ()) { +/* Now tell the dest that it should open it's end so it can reply */
s/it's/its/
Quoted 18 lines
migration.c +13
-0+qemu_savevm_send_openrp (s->file); + +/* And ask it to send an ack that will make stuff easier to debug */ +qemu_savevm_send_reqack (s->file,1 ); + +/* Tell the destination that we *might* want to do postcopy later; +* if the other end can't do postcopy it should fail now, nice and +* early. +*/ +qemu_savevm_send_postcopy_ram_advise (s->file); + } + s->setup_time =qemu_clock_get_ms (QEMU_CLOCK_HOST ) - setup_start; + current_active_type =MIG_STATE_ACTIVE ;migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ACTIVE );DPRINTF ("setup complete\n" );
migration.c static void *migration_thread(void *opaque) · line 1065
Quoted 9 lines
migration.c +6
-0@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)" nonpost=%" PRIu64")\n" , pending_size, max_size, pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) { +/* Still a significant amount to transfer */ + + current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +if (migrate_postcopy_ram () && + s->state !=MIG_STATE_POSTCOPY_ACTIVE && + pend_nonpost ==0 && s->start_postcopy) {
Hrm. This is checking for pend_nonpost == 0, rather than just close to zero. IIUC this will only work if all "live sendable" state is also postcopyable. But if we have live sendable data that's not postcopyable - like the power hash page table - we'll need some threshold here, like we currently have for entering the stopped vm phase of a precopy migration.
Or am I missing something?
Quoted 109 lines
migration.c +61
-16+ +if (!postcopy_start (s)) { + current_active_type =MIG_STATE_POSTCOPY_ACTIVE ; + } + +continue ; + } +/* Just another iteration step */ qemu_savevm_state_iterate (s->file); }else {int ret; -DPRINTF ("done iterating\n" ); -qemu_mutex_lock_iothread (); - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - old_vm_running =runstate_is_running (); +DPRINTF ("done iterating pending size %" PRIu64"\n" , + pending_size); + +if (s->state ==MIG_STATE_ACTIVE ) { +qemu_mutex_lock_iothread (); + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + old_vm_running =runstate_is_running (); + + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); +if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete (s->file); + } +qemu_mutex_unlock_iothread (); + +if (ret <0 ) { +migrate_set_state (s, current_active_type, +MIG_STATE_ERROR ); +break ; + } + }else if (s->state ==MIG_STATE_POSTCOPY_ACTIVE ) { +DPRINTF ("postcopy end\n" ); + +qemu_savevm_state_postcopy_complete (s->file); +DPRINTF ("postcopy end after complete\n" ); - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete (s->file); } -qemu_mutex_unlock_iothread (); -if (ret <0 ) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); -break ; +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up. +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +DPRINTF ("before rp close" ); +await_outgoing_return_path_close (s); +DPRINTF ("after rp close" ); } -if (!qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_COMPLETED ); +migrate_set_state (s, current_active_type, +MIG_STATE_COMPLETED );break ; } } }if (qemu_file_get_error (s->file)) { -migrate_set_state (s,MIG_STATE_ACTIVE ,MIG_STATE_ERROR ); +migrate_set_state (s, current_active_type,MIG_STATE_ERROR ); +DPRINTF ("migration_thread: file is in error state\n" );break ; } current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque) } } +DPRINTF ("migration_thread: After loop" );qemu_mutex_lock_iothread ();if (s->state ==MIG_STATE_COMPLETED ) {int64_t end_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)/* Notify before starting migration thread */ notifier_list_notify (&migration_state_notifiers, s); +/* Open the return path; currently for postcopy but other things might +* also want it. +*/ +if (migrate_postcopy_ram ()) { +if (open_outgoing_return_path (s)) { +error_report ("Unable to open return-path for postcopy" ); +migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ERROR ); +migrate_fd_cleanup (s); +return ; + } + } +qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); + s->started_migration_thread = true; }
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread
Quoted 16 lines · * David Gibson ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 + migration.c | 201 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 185 insertions(+), 19 deletions(-)include/migration/migration.h +0
-0
Quoted 4 lines
include/migration/migration.h
Quoted 2 lines
include/migration/migration.h +2
-0+DPRINTF ("postcopy_start: sending req 2\n" ); +qemu_savevm_send_reqack (ms->file,2 );Are these reqacks just for debugging, or do they affect the protocol?
Just for debugging; comment added. They just make it easy to line traces up between the source and destination, and also make it easy to figure out how far stuff has got if it jams up.
Quoted 4 lines
include/migration/migration.h
Quoted 2 lines
include/migration/migration.h +2
-0+if (migrate_postcopy_ram ()) { +/* Now tell the dest that it should open it's end so it can reply */ s/it's/its/
Fixed.
Quoted 35 lines
Quoted 18 lines
include/migration/migration.h +13
-0+qemu_savevm_send_openrp (s->file); + +/* And ask it to send an ack that will make stuff easier to debug */ +qemu_savevm_send_reqack (s->file,1 ); + +/* Tell the destination that we *might* want to do postcopy later; +* if the other end can't do postcopy it should fail now, nice and +* early. +*/ +qemu_savevm_send_postcopy_ram_advise (s->file); + } + s->setup_time =qemu_clock_get_ms (QEMU_CLOCK_HOST ) - setup_start; + current_active_type =MIG_STATE_ACTIVE ;migrate_set_state (s,MIG_STATE_SETUP ,MIG_STATE_ACTIVE );DPRINTF ("setup complete\n" );include/migration/migration.h static void *migration_thread(void *opaque) · line 1065
Quoted 9 lines
include/migration/migration.h +6
-0@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)" nonpost=%" PRIu64")\n" , pending_size, max_size, pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) { +/* Still a significant amount to transfer */ + + current_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +if (migrate_postcopy_ram () && + s->state !=MIG_STATE_POSTCOPY_ACTIVE && + pend_nonpost ==0 && s->start_postcopy) {Hrm. This is checking for
pend_nonpost== 0, rather than just close to zero. IIUC this will only work if all "live sendable" state is also postcopyable. But if we have live sendable data that's not postcopyable - like the power hash page table - we'll need some threshold here, like we currently have for entering the stopped vm phase of a precopy migration.
Quoted 1 line
Or am I missing something?
Hmm, I think you're right; I've changed this to:
pend_nonpost <= max_size
so that it's the same cut-off logic as the normal end-of-migrate; I think that will work; i.e. it gets small enough to be expected to complete quickly in the _complete phase that's at the start of postcopy.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 34/47 Postcopy: Create a fault handler thread before marking the ram as userfault
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- include/migration/migration.h | 3 +++ postcopy-ram.c | 23 +++++++++++++++++++++++ 2 files changed, 26 insertions(+)
include/migration/migration.h +3 -0
@@ -87,6 +87,9 @@ struct MigrationIncomingState { POSTCOPY_RAM_INCOMING_END } postcopy_ram_state; +QemuThread fault_thread; +QemuSemaphore fault_thread_sem; +/* For the kernel to send us notifications */ int userfault_fd;QEMUFile *return_path;
postcopy-ram.c +23 -0
@@ -518,8 +518,31 @@ static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,return 0 ; } +/* +* Handle faults detected by the USERFAULT markings +*/ +static void *postcopy_ram_fault_thread (void *opaque) +{ +MigrationIncomingState *mis = (MigrationIncomingState *)opaque; + +fprintf (stderr,"postcopy_ram_fault_thread\n" ); +/* TODO: In later patch */ +qemu_sem_post (&mis->fault_thread_sem); +while (1 ) { +/* TODO: In later patch */ + } + +return NULL ; +} +int postcopy_ram_enable_notify (MigrationIncomingState *mis) { +/* Create the fault handler thread and wait for it to be ready */ +qemu_sem_init (&mis->fault_thread_sem,0 ); +qemu_thread_create (&mis->fault_thread,"postcopy/fault" , + postcopy_ram_fault_thread, mis,QEMU_THREAD_JOINABLE ); +qemu_sem_wait (&mis->fault_thread_sem); +/* Mark so that we get notified of accesses to unwritten areas */ if (qemu_ram_foreach_block (postcopy_ram_sensitise_area, mis)) {return -1 ;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 34/47] Postcopy: Create a fault handler thread before marking the ram as userfault
Quoted 1 line · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This could do with a bit more rationale in the commit message.
Also is there a reason not to fold this with the patch originally marking the RAM as userfault? IIRC that one wasn't partocularly long either.
Otherwise
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Dr. David Alan Gilbert
AppliedRe: [Qemu-devel] [PATCH v4 34/47] Postcopy: Create a fault handler thread before marking the ram as userfault
Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This could do with a bit more rationale in the commit message.
Also is there a reason not to fold this with the patch originally marking the RAM as userfault? IIRC that one wasn't partocularly long either.
I've merged it with 'ram_enable_notify to switch on userfault' and it's commit message now reads.
'Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)'
Dave
Quoted 10 lines
Otherwise
Reviewed-by David Gibson <[email protected]>Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 35/47 Page request: Add MIG_RPCOMM_REQPAGES reverse command
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source.
--- include/migration/migration.h | 3 ++ migration.c | 74 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+)
include/migration/migration.h +3 -0
@@ -45,6 +45,7 @@ enum mig_rpcomm_cmd {MIG_RPCOMM_INVALID =0 ,/* Must be 0 */ MIG_RPCOMM_SHUT ,/* sibling will not send any more RP messages */ MIG_RPCOMM_ACK ,/* data (seq: be32 ) */ +MIG_RPCOMM_REQPAGES ,/* data (start: be64, len: be64) */ MIG_RPCOMM_AFTERLASTVALID };
@@ -250,6 +251,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, uint32_t value);void migrate_send_rp_ack (MigrationIncomingState *mis,uint32_t value); +void migrate_send_rp_reqpages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,ram_addr_t len);void ram_control_before_iterate (QEMUFile *f,uint64_t flags);
migration.c +74 -0
104 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command
Quoted 57 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RPCOMM_REQPAGEScommand on Return path for the postcopy destination to request a page from the source.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- include/migration/migration.h | 3 ++ migration.c | 74 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+)include/migration/migration.h +3
-0@@ -45,6 +45,7 @@ enum mig_rpcomm_cmd {MIG_RPCOMM_INVALID =0 ,/* Must be 0 */ MIG_RPCOMM_SHUT ,/* sibling will not send any more RP messages */ MIG_RPCOMM_ACK ,/* data (seq: be32 ) */ +MIG_RPCOMM_REQPAGES ,/* data (start: be64, len: be64) */ MIG_RPCOMM_AFTERLASTVALID };@@ -250,6 +251,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, uint32_t value);void migrate_send_rp_ack (MigrationIncomingState *mis,uint32_t value); +void migrate_send_rp_reqpages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,ram_addr_t len);void ram_control_before_iterate (QEMUFile *f,uint64_t flags);migration.c +17
-0@@ -144,6 +144,38 @@ void migrate_send_rp_ack(MigrationIncomingState *mis,migrate_send_rp_message (mis,MIG_RPCOMM_ACK ,4 , (uint8_t *)&buf); } +/* Request a range of pages from the source VM at the given +* start address. +* rbname: Name of the RAMBlock to request the page in, if NULL it's the same +* as the last request (a name must have been given previously) +* Start: Address offset within the RB +* Len: Length in bytes required - must be a multiple of pagesize +*/ +void migrate_send_rp_reqpages (MigrationIncomingState *mis,const char *rbname, +ram_addr_t start,ram_addr_t len) +{ +uint8_t bufc[16 +1 +255 ];/* start (8 byte), len (8 byte), rbname upto 256 */ +uint64_t *buf64 = (uint64_t *)bufc; +size_t msglen =16 ;/* start + len */ + +assert (!(len &1 )); +if (rbname) { +int rbname_len =strlen (rbname);
migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173
Quoted 10 lines
migration.c +10
-0+assert (rbname_len <256 ); + + len |=1 ;/* Flag to say we've got a name */ + bufc[msglen++] = rbname_len; +memcpy (bufc + msglen, rbname, rbname_len); + msglen += rbname_len; + } + + buf64[0 ] = (uint64_t )start; + buf64[0 ] =cpu_to_be64 (buf64[0 ]);
I think this would be clearer as well as less verbose, as just:
buf64[0] = cpu_to_be64(start);
Quoted 27 lines
migration.c +16
-0+ buf64[1 ] = (uint64_t )len; + buf64[1 ] =cpu_to_be64 (buf64[1 ]); +migrate_send_rp_message (mis,MIG_RPCOMM_REQPAGES , msglen, bufc); +} +void qemu_start_incoming_migration (const char *uri,Error **errp) {const char *p;@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s) }/* +* Process a request for pages received on the return path, +* We're allowed to send more than requested (e.g. to round to our page size) +* and we don't need to send pages that have already been sent. +*/ +static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname, +ram_addr_t start,ram_addr_t len) +{ +DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +} + +/* * Handles messages sent on the return path towards the source VM * */
migration.c static void *source_return_path_thread(void *opaque) · line 841
Quoted 4 lines
migration.c +1
-0@@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)const int max_len =512 ;uint8_t buf[max_len];uint32_t tmp32; +uint64_t tmp64a, tmp64b;
Hrm.. calling everything "tmp*" doesn't help readability.
Quoted 47 lines
migration.c +30
-0+char *tmpstr;int res;DPRINTF ("RP: %s entry" , __func__);@@ -810,6 +855,11 @@ static void *source_return_path_thread(void *opaque) expected_len =4 ;break ; +case MIG_RPCOMM_REQPAGES : +/* 16 byte start/len _possibly_ plus an id str */ + expected_len =16 +256 ; +break ; +default :error_report ("RP: Received invalid cmd 0x%04x length 0x%04x" , header_com, header_len);@@ -857,6 +907,30 @@ static void *source_return_path_thread(void *opaque)atomic_xchg (&ms->rp_state.latest_ack, tmp32);break ; +case MIG_RPCOMM_REQPAGES : + tmp64a =be64_to_cpup ((uint64_t *)buf);/* Start */ + tmp64b =be64_to_cpup (((uint64_t *)buf)+1 );/* Len */ + tmpstr =NULL ; +if (tmp64b &1 ) { + tmp64b -=1 ;/* Remove the flag */ +/* Now we expect an idstr */ + tmp32 = buf[16 ];/* Length of the following idstr */ + tmpstr = (char *)&buf[17 ]; + buf[17 +tmp32] ='\0' ; + expected_len =16 +1 +tmp32; + }else { + expected_len =16 ; + } +if (header_len != expected_len) { +error_report ("RP: Received ReqPage with length %d expecting %d" , + header_len, expected_len); +source_return_path_bad (ms); + } +migrate_handle_rp_reqpages (ms, tmpstr, + (ram_addr_t )tmp64a, + (ram_addr_t )tmp64b); +break ; +default :/* This shouldn't happen because we should catch this above */ DPRINTF ("RP: Bad header_com in dispatch" );
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command
Quoted 11 lines · * David Gibson ([email protected]) wrote:
migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RPCOMM_REQPAGEScommand on Return path for the postcopy destination to request a page from the source.migration.c +2
-0+ buf64[0 ] = (uint64_t )start; + buf64[0 ] =cpu_to_be64 (buf64[0 ]);I think this would be clearer as well as less verbose, as just:
buf64[0] = cpu_to_be64(start);
I've gone with the halfway mark of:
buf64[0] = cpu_to_be64((uint64_t)start);
it jsut doesn't feel right passing something into a byteswap unless you know the size.
Quoted 25 lines
Quoted 25 lines
migration.c +16
-0+ buf64[1 ] = (uint64_t )len; + buf64[1 ] =cpu_to_be64 (buf64[1 ]); +migrate_send_rp_message (mis,MIG_RPCOMM_REQPAGES , msglen, bufc); +} +void qemu_start_incoming_migration (const char *uri,Error **errp) {const char *p;@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s) }/* +* Process a request for pages received on the return path, +* We're allowed to send more than requested (e.g. to round to our page size) +* and we don't need to send pages that have already been sent. +*/ +static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname, +ram_addr_t start,ram_addr_t len) +{ +DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +} + +/* * Handles messages sent on the return path towards the source VM *
Quoted 8 lines
migration.c static void *source_return_path_thread(void *opaque) · line 841
Quoted 6 lines
migration.c +1
-0*/ @@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)const int max_len =512 ;uint8_t buf[max_len];uint32_t tmp32; +uint64_t tmp64a, tmp64b;Hrm.. calling everything "tmp*" doesn't help readability.
True; most of the rest of those tmps are used by multiple commands and just read off the wire and immediately used.
They're now start/len for tmp64a/b.
Dave
Quoted 40 lines
Quoted 40 lines
migration.c +25
-0+char *tmpstr;int res;DPRINTF ("RP: %s entry" , __func__);@@ -810,6 +855,11 @@ static void *source_return_path_thread(void *opaque) expected_len =4 ;break ; +case MIG_RPCOMM_REQPAGES : +/* 16 byte start/len _possibly_ plus an id str */ + expected_len =16 +256 ; +break ; +default :error_report ("RP: Received invalid cmd 0x%04x length 0x%04x" , header_com, header_len);@@ -857,6 +907,30 @@ static void *source_return_path_thread(void *opaque)atomic_xchg (&ms->rp_state.latest_ack, tmp32);break ; +case MIG_RPCOMM_REQPAGES : + tmp64a =be64_to_cpup ((uint64_t *)buf);/* Start */ + tmp64b =be64_to_cpup (((uint64_t *)buf)+1 );/* Len */ + tmpstr =NULL ; +if (tmp64b &1 ) { + tmp64b -=1 ;/* Remove the flag */ +/* Now we expect an idstr */ + tmp32 = buf[16 ];/* Length of the following idstr */ + tmpstr = (char *)&buf[17 ]; + buf[17 +tmp32] ='\0' ; + expected_len =16 +1 +tmp32; + }else { + expected_len =16 ; + } +if (header_len != expected_len) { +error_report ("RP: Received ReqPage with length %d expecting %d" , + header_len, expected_len); +source_return_path_bad (ms); + }
Quoted 14 lines
Quoted 8 lines
migration.c +5
-0+migrate_handle_rp_reqpages (ms, tmpstr, + (ram_addr_t )tmp64a, + (ram_addr_t )tmp64b); +break ; +default :/* This shouldn't happen because we should catch this above */ DPRINTF ("RP: Bad header_com in dispatch" );Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command
Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:
migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
MIG_RPCOMM_REQPAGEScommand on Return path for the postcopy destination to request a page from the source.migration.c +2
-0+ buf64[0 ] = (uint64_t )start; + buf64[0 ] =cpu_to_be64 (buf64[0 ]);I think this would be clearer as well as less verbose, as just:
buf64[0] = cpu_to_be64(start);
Quoted 6 lines
I've gone with the halfway mark of:
buf64[0] = cpu_to_be64((uint64_t)start);
it jsut doesn't feel right passing something into a byteswap unless you know the size.
I've always thought of (this group of) byteswap functions as specifying the output size. It's a value parameter so integer promotion to the required type is pretty safe.
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
migration.c +16
-0+ buf64[1 ] = (uint64_t )len; + buf64[1 ] =cpu_to_be64 (buf64[1 ]); +migrate_send_rp_message (mis,MIG_RPCOMM_REQPAGES , msglen, bufc); +} +void qemu_start_incoming_migration (const char *uri,Error **errp) {const char *p;@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s) }/* +* Process a request for pages received on the return path, +* We're allowed to send more than requested (e.g. to round to our page size) +* and we don't need to send pages that have already been sent. +*/ +static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname, +ram_addr_t start,ram_addr_t len) +{ +DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +} + +/* * Handles messages sent on the return path towards the source VM * Quoted 8 lines
migration.c static void *source_return_path_thread(void *opaque) · line 841
Quoted 6 lines
migration.c +1
-0*/ @@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)const int max_len =512 ;uint8_t buf[max_len];uint32_t tmp32; +uint64_t tmp64a, tmp64b;Hrm.. calling everything "tmp*" doesn't help readability.
Quoted 3 lines
True; most of the rest of those tmps are used by multiple commands and just read off the wire and immediately used.
They're now start/len for tmp64a/b.
Ok, great. I find it's usually best to declare appropriate variables for each case (or even use local blocks), rather than share. The compiler's smart enough to coalesce them, so there's no real cost.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
PATCH v4 36/47 Page request: Process incoming page request
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.
--- arch_init.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 21 +++++++++++++++++ include/qemu/typedefs.h | 3 ++- migration.c | 34 +++++++++++++++++++++++++++- 4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +52 -0
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Queue the pages for transmission, e.g. a request from postcopy destination +* ms: MigrationStatus in which the queue is held +* rbname: The RAMBlock the request is for - may be NULL (to mean reuse last) +* start: Offset from the start of the RAMBlock +* len: Length (in bytes) to send +* Return: 0 on success +*/ +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len) +{ +RAMBlock *ramblock; + +if (!rbname) { +/* Reuse last RAMBlock */ + ramblock = ms->last_req_rb; + +if (!ramblock) { +error_report ("ram_save_queue_pages no previous block" ); +return -1 ; + } + }else { + ramblock =ram_find_block (rbname); + +if (!ramblock) { +error_report ("ram_save_queue_pages no block '%s'" , rbname); +return -1 ; + } + } +DPRINTF ("ram_save_queue_pages: Block %s start %zx len %zx" , + ramblock->idstr, start, len); + +if (start+len > ramblock->length) { +error_report ("%s request overrun start=%zx len=%zx blocklen=%zx" , + __func__, start, len, ramblock->length); +return -1 ; + } + +struct MigrationSrcPageRequest *new_entry = +g_malloc0 (sizeof (struct MigrationSrcPageRequest )); + new_entry->rb = ramblock; + new_entry->offset = start; + new_entry->len = len; + ms->last_req_rb = ramblock; + +qemu_mutex_lock (&ms->src_page_req_mutex); +QSIMPLEQ_INSERT_TAIL (&ms->src_page_requests, new_entry, next_req); +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return 0 ; +} + +/* * ram_find_and_save_block: Finds a page to send and sends it to f * * Returns: The number of bytes written.
include/migration/migration.h +21 -0
@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);void migration_incoming_state_destroy (void ); +/* +* An outstanding page request, on the source, having been received +* and queued +*/ +struct MigrationSrcPageRequest { +RAMBlock *rb; +hwaddr offset; +hwaddr len; + +QSIMPLEQ_ENTRY (MigrationSrcPageRequest) next_req; +}; +struct MigrationState {int64_t bandwidth_limit;
@@ -138,6 +150,12 @@ struct MigrationState* of the postcopy phase */ unsigned long *sentmap; + +/* Queue of outstanding page requests from the destination */ +QemuMutex src_page_req_mutex; + QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests; +/* The RAMBlock used in the last src_page_request */ +RAMBlock *last_req_rb; };void process_incoming_migration (QEMUFile *f);
@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, size_t size, int *bytes_sent); +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len); +#endif
include/qemu/typedefs.h +2 -1
@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;typedef struct QEMUFile QEMUFile ;typedef struct QEMUBH QEMUBH ; +typedef struct AdapterInfo AdapterInfo ;typedef struct AioContext AioContext ;typedef struct Visitor Visitor ;
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;#endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }/*
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 133 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On receiving
MIG_RPCOMM_REQPAGESlook up the address and queue the page.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 21 +++++++++++++++++ include/qemu/typedefs.h | 3 ++- migration.c | 34 +++++++++++++++++++++++++++- 4 files changed, 108 insertions(+), 2 deletions(-)arch_init.c +52
-0@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Queue the pages for transmission, e.g. a request from postcopy destination +* ms: MigrationStatus in which the queue is held +* rbname: The RAMBlock the request is for - may be NULL (to mean reuse last) +* start: Offset from the start of the RAMBlock +* len: Length (in bytes) to send +* Return: 0 on success +*/ +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len) +{ +RAMBlock *ramblock; + +if (!rbname) { +/* Reuse last RAMBlock */ + ramblock = ms->last_req_rb; + +if (!ramblock) { +error_report ("ram_save_queue_pages no previous block" ); +return -1 ; + } + }else { + ramblock =ram_find_block (rbname); + +if (!ramblock) { +error_report ("ram_save_queue_pages no block '%s'" , rbname); +return -1 ; + } + } +DPRINTF ("ram_save_queue_pages: Block %s start %zx len %zx" , + ramblock->idstr, start, len); + +if (start+len > ramblock->length) { +error_report ("%s request overrun start=%zx len=%zx blocklen=%zx" , + __func__, start, len, ramblock->length); +return -1 ; + } + +struct MigrationSrcPageRequest *new_entry = +g_malloc0 (sizeof (struct MigrationSrcPageRequest )); + new_entry->rb = ramblock; + new_entry->offset = start; + new_entry->len = len; + ms->last_req_rb = ramblock; + +qemu_mutex_lock (&ms->src_page_req_mutex); +QSIMPLEQ_INSERT_TAIL (&ms->src_page_requests, new_entry, next_req); +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return 0 ; +} + +/* * ram_find_and_save_block: Finds a page to send and sends it to f * * Returns: The number of bytes written. include/migration/migration.h +21
-0@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);void migration_incoming_state_destroy (void ); +/* +* An outstanding page request, on the source, having been received +* and queued +*/ +struct MigrationSrcPageRequest { +RAMBlock *rb; +hwaddr offset; +hwaddr len; + +QSIMPLEQ_ENTRY (MigrationSrcPageRequest) next_req; +}; +struct MigrationState {int64_t bandwidth_limit;@@ -138,6 +150,12 @@ struct MigrationState* of the postcopy phase */ unsigned long *sentmap; + +/* Queue of outstanding page requests from the destination */ +QemuMutex src_page_req_mutex; + QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests; +/* The RAMBlock used in the last src_page_request */ +RAMBlock *last_req_rb; };void process_incoming_migration (QEMUFile *f);@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, size_t size, int *bytes_sent); +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len); +#endif include/qemu/typedefs.h +1
-0@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;typedef struct QEMUFile QEMUFile ;typedef struct QEMUBH QEMUBH ; +typedef struct AdapterInfo AdapterInfo ;typedef struct AioContext AioContext ;typedef struct Visitor Visitor ;
include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85
Quoted 6 lines
include/qemu/typedefs.h +1
-1@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;
:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?
Quoted 68 lines
include/qemu/typedefs.h +0
-0#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }/*
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 11 lines · * zhanghailiang ([email protected]) wrote:
include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85
Quoted 8 lines
typedef struct Visitor Visitor;
include/qemu/typedefs.h +1
-1@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;:(, another redefinition, 'RAMBlock' also defined in '
include/exec/cpu-all.h:314', Am i miss something when compile qemu?
Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)
What compiler and flags are you using?
Dave
Quoted 67 lines
Quoted 67 lines
include/qemu/typedefs.h +0
-0#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }
Quoted 4 lines
migration.c static void source_return_path_bad(MigrationState *s) · line 861
Quoted 2 lines
migration.c +0
-0/*
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Il 08/10/2014 09:49, Dr. David Alan Gilbert ha scritto:
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
:(, another redefinition, 'RAMBlock' also defined in '
include/exec/cpu-all.h:314', Am i miss something when compile qemu?Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)
What compiler and flags are you using?
I think it is visible with CentOS 6.
Paolo
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
In reply to Dr. David Alan GilbertQuoted 14 lines · Dr. David Alan Gilbert
Quoted 11 lines · * zhanghailiang ([email protected]) wrote:
include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85
Quoted 8 lines
typedef struct Visitor Visitor;
include/qemu/typedefs.h +1
-1@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;:(, another redefinition, 'RAMBlock' also defined in '
include/exec/cpu-all.h:314', Am i miss something when compile qemu?
Quoted 6 lines
Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)
What compiler and flags are you using?
Dave
Hi Dave,
My compiler info:
gcc (SUSE Linux) 4.3.4
The configure info is:
#./configure --target-list=x86_64-softmmu --enable-debug --disable-gtk
...
CFLAGS -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -g QEMU_CFLAGS -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -I/usr/include/libpng12 -I/usr/include/pixman-1 LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g
...
Maybe its gcc's limitation, but why this redefinition need? After i remove one, it compiles successfully;)
Thanks,
zhanghailiang
Quoted 77 lines
Quoted 68 lines
Quoted 68 lines
include/qemu/typedefs.h +0
-0#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }Quoted 4 lines
migration.c static void source_return_path_bad(MigrationState *s) · line 861
Quoted 2 lines
migration.c +0
-0/* --
Dr. David Alan Gilbert / [email protected] / Manchester, UK.
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 30 lines · * zhanghailiang ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert
Quoted 11 lines · * zhanghailiang ([email protected]) wrote:
include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85
Quoted 8 lines
typedef struct Visitor Visitor;
include/qemu/typedefs.h +1
-1@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;:(, another redefinition, 'RAMBlock' also defined in '
include/exec/cpu-all.h:314', Am i miss something when compile qemu?Quoted 6 lines
Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)
What compiler and flags are you using?
Dave
Hi Dave,
My compiler info:
gcc (SUSE Linux) 4.3.4The configure info is:
#./configure --target-list=x86_64-softmmu --enable-debug --disable-gtk
...
Quoted 7 lines
CFLAGS -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -g QEMU_CFLAGS -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -I/usr/include/libpng12 -I/usr/include/pixman-1 LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g...
Maybe its gcc's limitation, but why this redefinition need? After i remove one, it compiles successfully;)
OK, thanks. I'll clean them up.
Dave
Quoted 83 lines
Thanks,
zhanghailiangQuoted 77 lines
Quoted 68 lines
Quoted 68 lines
include/qemu/typedefs.h +0
-0#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }Quoted 4 lines
migration.c static void source_return_path_bad(MigrationState *s) · line 861
Quoted 2 lines
migration.c +0
-0/* --
Dr. David Alan Gilbert / [email protected] / Manchester, UK.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
In reply to Dr. David Alan Gilbert (git)Quoted 30 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On receiving
MIG_RPCOMM_REQPAGESlook up the address and queue the page.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 21 +++++++++++++++++ include/qemu/typedefs.h | 3 ++- migration.c | 34 +++++++++++++++++++++++++++- 4 files changed, 108 insertions(+), 2 deletions(-)arch_init.c +9
-0@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Queue the pages for transmission, e.g. a request from postcopy destination +* ms: MigrationStatus in which the queue is held +* rbname: The RAMBlock the request is for - may be NULL (to mean reuse last) +* start: Offset from the start of the RAMBlock +* len: Length (in bytes) to send +* Return: 0 on success +*/ +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len)
arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681
Quoted 10 lines
arch_init.c +10
-0+{ +RAMBlock *ramblock; + +if (!rbname) { +/* Reuse last RAMBlock */ + ramblock = ms->last_req_rb; + +if (!ramblock) { +error_report ("ram_save_queue_pages no previous block" ); +return -1 ;
This should be an assert() shouldn't it?
arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 689
Quoted 8 lines
arch_init.c +8
-0+ } + }else { + ramblock =ram_find_block (rbname); + +if (!ramblock) { +error_report ("ram_save_queue_pages no block '%s'" , rbname); +return -1 ; + }
And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.
Quoted 158 lines
arch_init.c +25
-0+ } +DPRINTF ("ram_save_queue_pages: Block %s start %zx len %zx" , + ramblock->idstr, start, len); + +if (start+len > ramblock->length) { +error_report ("%s request overrun start=%zx len=%zx blocklen=%zx" , + __func__, start, len, ramblock->length); +return -1 ; + } + +struct MigrationSrcPageRequest *new_entry = +g_malloc0 (sizeof (struct MigrationSrcPageRequest )); + new_entry->rb = ramblock; + new_entry->offset = start; + new_entry->len = len; + ms->last_req_rb = ramblock; + +qemu_mutex_lock (&ms->src_page_req_mutex); +QSIMPLEQ_INSERT_TAIL (&ms->src_page_requests, new_entry, next_req); +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return 0 ; +} + +/* * ram_find_and_save_block: Finds a page to send and sends it to f * * Returns: The number of bytes written. include/migration/migration.h +21
-0@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);void migration_incoming_state_destroy (void ); +/* +* An outstanding page request, on the source, having been received +* and queued +*/ +struct MigrationSrcPageRequest { +RAMBlock *rb; +hwaddr offset; +hwaddr len; + +QSIMPLEQ_ENTRY (MigrationSrcPageRequest) next_req; +}; +struct MigrationState {int64_t bandwidth_limit;@@ -138,6 +150,12 @@ struct MigrationState* of the postcopy phase */ unsigned long *sentmap; + +/* Queue of outstanding page requests from the destination */ +QemuMutex src_page_req_mutex; + QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests; +/* The RAMBlock used in the last src_page_request */ +RAMBlock *last_req_rb; };void process_incoming_migration (QEMUFile *f);@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, size_t size, int *bytes_sent); +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len); +#endif include/qemu/typedefs.h +2
-1@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;typedef struct QEMUFile QEMUFile ;typedef struct QEMUBH QEMUBH ; +typedef struct AdapterInfo AdapterInfo ;typedef struct AioContext AioContext ;typedef struct Visitor Visitor ;@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }/*
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 53 lines · * David Gibson ([email protected]) wrote:
Quoted 30 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On receiving
MIG_RPCOMM_REQPAGESlook up the address and queue the page.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 21 +++++++++++++++++ include/qemu/typedefs.h | 3 ++- migration.c | 34 +++++++++++++++++++++++++++- 4 files changed, 108 insertions(+), 2 deletions(-)arch_init.c +9
-0@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Queue the pages for transmission, e.g. a request from postcopy destination +* ms: MigrationStatus in which the queue is held +* rbname: The RAMBlock the request is for - may be NULL (to mean reuse last) +* start: Offset from the start of the RAMBlock +* len: Length (in bytes) to send +* Return: 0 on success +*/ +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len)arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681
Quoted 10 lines
arch_init.c +10
-0+{ +RAMBlock *ramblock; + +if (!rbname) { +/* Reuse last RAMBlock */ + ramblock = ms->last_req_rb; + +if (!ramblock) { +error_report ("ram_save_queue_pages no previous block" ); +return -1 ;This should be an
assert()shouldn't it?arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 689
Quoted 8 lines
arch_init.c +8
-0+ } + }else { + ramblock =ram_find_block (rbname); + +if (!ramblock) { +error_report ("ram_save_queue_pages no block '%s'" , rbname); +return -1 ; + }
Quoted 2 lines
And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.
No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.
(On the incoming side it's a bit more OK since if you've not got a full running VM anyway yet then there's not much to lose).
Dave
Quoted 158 lines
Quoted 158 lines
arch_init.c +25
-0+ } +DPRINTF ("ram_save_queue_pages: Block %s start %zx len %zx" , + ramblock->idstr, start, len); + +if (start+len > ramblock->length) { +error_report ("%s request overrun start=%zx len=%zx blocklen=%zx" , + __func__, start, len, ramblock->length); +return -1 ; + } + +struct MigrationSrcPageRequest *new_entry = +g_malloc0 (sizeof (struct MigrationSrcPageRequest )); + new_entry->rb = ramblock; + new_entry->offset = start; + new_entry->len = len; + ms->last_req_rb = ramblock; + +qemu_mutex_lock (&ms->src_page_req_mutex); +QSIMPLEQ_INSERT_TAIL (&ms->src_page_requests, new_entry, next_req); +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return 0 ; +} + +/* * ram_find_and_save_block: Finds a page to send and sends it to f * * Returns: The number of bytes written. include/migration/migration.h +21
-0@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);MigrationIncomingState *migration_incoming_state_init (QEMUFile *f);void migration_incoming_state_destroy (void ); +/* +* An outstanding page request, on the source, having been received +* and queued +*/ +struct MigrationSrcPageRequest { +RAMBlock *rb; +hwaddr offset; +hwaddr len; + +QSIMPLEQ_ENTRY (MigrationSrcPageRequest) next_req; +}; +struct MigrationState {int64_t bandwidth_limit;@@ -138,6 +150,12 @@ struct MigrationState* of the postcopy phase */ unsigned long *sentmap; + +/* Queue of outstanding page requests from the destination */ +QemuMutex src_page_req_mutex; + QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests; +/* The RAMBlock used in the last src_page_request */ +RAMBlock *last_req_rb; };void process_incoming_migration (QEMUFile *f);@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, size_t size, int *bytes_sent); +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len); +#endif include/qemu/typedefs.h +2
-1@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;typedef struct QEMUFile QEMUFile ;typedef struct QEMUBH QEMUBH ; +typedef struct AdapterInfo AdapterInfo ;typedef struct AioContext AioContext ;typedef struct Visitor Visitor ;@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;typedef struct PcGuestInfo PcGuestInfo ;typedef struct PostcopyPMI PostcopyPMI ;typedef struct Range Range ; -typedef struct AdapterInfo AdapterInfo ; +typedef struct RAMBlock RAMBlock ;#endif /* QEMU_TYPEDEFS_H */ migration.c +33
-1@@ -26,6 +26,8 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" //#define DEBUG_MIGRATION @@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)migrate_fd_cleanup_src_rp (s); +/* This queue generally should be empty - but in the case of a failed +* migration might have some droppings in. +*/ +struct MigrationSrcPageRequest *mspr, *next_mspr; + QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) { +QSIMPLEQ_REMOVE_HEAD (&s->src_page_requests, next_req); +g_free (mspr); + } +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params) s->state =MIG_STATE_SETUP ;trace_migrate_set_state (MIG_STATE_SETUP ); +qemu_mutex_init (&s->src_page_req_mutex); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)static void migrate_handle_rp_reqpages (MigrationState *ms,const char * rbname,ram_addr_t start,ram_addr_t len) { -DPRINTF ("migrate_handle_rp_reqpages: at %zx for len %zx" , start, len); +DPRINTF ("migrate_handle_rp_reqpages: in %s start %zx len %zx" , + rbname, start, len); + +/* Round everything up to our host page size */ +long our_host_ps =sysconf (_SC_PAGESIZE); +if (start & (our_host_ps-1 )) { +long roundings = start & (our_host_ps-1 ); + start -= roundings; + len += roundings; + } +if (len & (our_host_ps-1 )) { +long roundings = len & (our_host_ps-1 ); + len -= roundings; + len += our_host_ps; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +source_return_path_bad (ms); + } }
Quoted 7 lines
Quoted 1 line
migration.c +0
-0/* Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 54 lines · Dr. David Alan Gilbert
Quoted 53 lines · * David Gibson ([email protected]) wrote:
Quoted 30 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
On receiving
MIG_RPCOMM_REQPAGESlook up the address and queue the page.Signed-off-by Dr. David Alan Gilbert <[email protected]>--- arch_init.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/migration/migration.h | 21 +++++++++++++++++ include/qemu/typedefs.h | 3 ++- migration.c | 34 +++++++++++++++++++++++++++- 4 files changed, 108 insertions(+), 2 deletions(-)arch_init.c +9
-0@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Queue the pages for transmission, e.g. a request from postcopy destination +* ms: MigrationStatus in which the queue is held +* rbname: The RAMBlock the request is for - may be NULL (to mean reuse last) +* start: Offset from the start of the RAMBlock +* len: Length (in bytes) to send +* Return: 0 on success +*/ +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len)arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681
Quoted 10 lines
arch_init.c +10
-0+{ +RAMBlock *ramblock; + +if (!rbname) { +/* Reuse last RAMBlock */ + ramblock = ms->last_req_rb; + +if (!ramblock) { +error_report ("ram_save_queue_pages no previous block" ); +return -1 ;This should be an
assert()shouldn't it?Quoted 8 lines
arch_init.c +8
-0+ } + }else { + ramblock =ram_find_block (rbname); + +if (!ramblock) { +error_report ("ram_save_queue_pages no block '%s'" , rbname); +return -1 ; + }
Quoted 8 lines
Quoted 2 lines
And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.
No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.
Ah, ok, that makes sense. Maybe adding something to the error message or a nearby comment indicating that if these happen it's certainly a bug, not the result of some external problem?
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request
Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 2 lines
And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.
No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.
Quoted 3 lines
Ah, ok, that makes sense. Maybe adding something to the error message or a nearby comment indicating that if these happen it's certainly a bug, not the result of some external problem?
Done.
Dave
Quoted 6 lines
Signature · 5 lines
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v4 37/47 Page request: Consume pages off the post-copy queue
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning.
Note:
a) After a queued page the linear walk carries on from after the
unqueued page; there is a reasonable chance that the destination was about to ask for other closeby pages anyway.
b) We have to be careful of any assumptions that the page walking
code makes, in particular it does some short cuts on its first linear walk that break as soon as we do a queued page.
--- arch_init.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 125 insertions(+), 24 deletions(-)
arch_init.c +125 -24
@@ -331,6 +331,7 @@ static RAMBlock *last_seen_block;/* This is the last block from where we have sent data */ static RAMBlock *last_sent_block;static ram_addr_t last_offset; +static bool last_was_from_queue;static unsigned long *migration_bitmap;static uint64_t migration_dirty_pages;static uint32_t last_version;
@@ -460,6 +461,19 @@ static inline bool migration_bitmap_set_dirty(ram_addr_t addr)return ret; } +static inline bool migration_bitmap_clear_dirty (ram_addr_t addr) +{ +bool ret; +int nr = addr >>TARGET_PAGE_BITS ; + + ret =test_and_clear_bit (nr, migration_bitmap); + +if (ret) { + migration_dirty_pages--; + } +return ret; +} +static void migration_bitmap_sync_range (ram_addr_t start,ram_addr_t length) {ram_addr_t addr;
@@ -660,6 +674,39 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, }/* +* Unqueue a page from the queue fed by postcopy page requests +* +* Returns: The RAMBlock* to transmit from (or NULL if the queue is empty) +* ms: MigrationState in +* offset: the byte offset within the RAMBlock for the start of the page +* bitoffset: global offset in the dirty/sent bitmaps +*/ +static RAMBlock *ram_save_unqueue_page (MigrationState *ms,ram_addr_t *offset, +unsigned long *bitoffset) +{ +RAMBlock *result =NULL ; +qemu_mutex_lock (&ms->src_page_req_mutex); +if (!QSIMPLEQ_EMPTY (&ms->src_page_requests)) { +struct MigrationSrcPageRequest *entry = +QSIMPLEQ_FIRST (&ms->src_page_requests); + result = entry->rb; + *offset = entry->offset; + *bitoffset = (entry->offset + entry->rb->offset) >>TARGET_PAGE_BITS ; + +if (entry->len >TARGET_PAGE_SIZE ) { + entry->len -=TARGET_PAGE_SIZE ; + entry->offset +=TARGET_PAGE_SIZE ; + }else { +QSIMPLEQ_REMOVE_HEAD (&ms->src_page_requests, next_req); +g_free (entry); + } + } +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return result; +} + +/* * Queue the pages for transmission, e.g. a request from postcopy destination * ms: MigrationStatus in which the queue is held * rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
@@ -720,44 +767,97 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,static int ram_find_and_save_block (QEMUFile *f,bool last_stage) { +MigrationState *ms =migrate_get_current ();RAMBlock *block = last_seen_block; +RAMBlock *tmpblock;ram_addr_t offset = last_offset; +ram_addr_t tmpoffset;bool complete_round = false;int bytes_sent =0 ; -MemoryRegion *mr;unsigned long bitoffset; +unsigned long hps =sysconf (_SC_PAGESIZE); -if (!block) +if (!block) { block =QTAILQ_FIRST (&ram_list.blocks); + last_was_from_queue = false; + } -while (true) { - mr = block->mr; - offset =migration_bitmap_find_and_reset_dirty (mr, offset, &bitoffset); -if (complete_round && block == last_seen_block && - offset >= last_offset) { -break ; +while (true) {/* Until we send a block or run out of stuff to send */ + tmpblock =NULL ; + +/* +* Don't break host-page chunks up with queue items +* so only unqueue if, +* a) The last item came from the queue anyway +* b) The last sent item was the last target-page in a host page +*/ +if (last_was_from_queue || (!last_sent_block) || + ((last_offset & (hps -1 )) == (hps -TARGET_PAGE_SIZE ))) { + tmpblock =ram_save_unqueue_page (ms, &tmpoffset, &bitoffset); } -if (offset >= block->length) { - offset =0 ; - block =QTAILQ_NEXT (block, next); -if (!block) { - block =QTAILQ_FIRST (&ram_list.blocks); - complete_round = true; - ram_bulk_stage = false; + +if (tmpblock) { +/* We've got a block from the postcopy queue */ +DPRINTF ("%s: Got postcopy item '%s' offset=%zx bitoffset=%zx" , + __func__, tmpblock->idstr, tmpoffset, bitoffset); +/* We're sending this page, and since it's postcopy nothing else +* will dirty it, and we must make sure it doesn't get sent again. +*/ +if (!migration_bitmap_clear_dirty (bitoffset <<TARGET_PAGE_BITS )) { +DPRINTF ("%s: Not dirty for postcopy %s/%zx bito=%zx (sent=%d)" , + __func__, tmpblock->idstr, tmpoffset, bitoffset, +test_bit (bitoffset, ms->sentmap)); +continue ; } +/* +* As soon as we start servicing pages out of order, then we have +* to kill the bulk stage, since the bulk stage assumes +* in (migration_bitmap_find_and_reset_dirty) that every page is +* dirty, that's no longer true. +*/ + ram_bulk_stage = false; +/* +* We mustn't change block/offset unless it's to a valid one +* otherwise we can go down some of the exit cases in the normal +* path. +*/ + block = tmpblock; + offset = tmpoffset; + last_was_from_queue = true; }else { - bytes_sent =ram_save_page (f, block, offset, last_stage); - -/* if page is unmodified, continue to the next */ -if (bytes_sent >0 ) { -MigrationState *s =migrate_get_current (); -if (s->sentmap) { -set_bit (bitoffset, s->sentmap); +MemoryRegion *mr; +/* priority queue empty, so just search for something dirty */ + mr = block->mr; + offset =migration_bitmap_find_and_reset_dirty (mr, offset, + &bitoffset); +if (complete_round && block == last_seen_block && + offset >= last_offset) { +break ; + } +if (offset >= block->length) { + offset =0 ; + block =QTAILQ_NEXT (block, next); +if (!block) { + block =QTAILQ_FIRST (&ram_list.blocks); + complete_round = true; + ram_bulk_stage = false; } +continue ;/* pick an offset in the new block */ + } + last_was_from_queue = false; + } - last_sent_block = block; -break ; +/* We have a page to send, so send it */ + bytes_sent =ram_save_page (f, block, offset, last_stage); + +/* if page is unmodified, continue to the next */ +if (bytes_sent >0 ) { +if (ms->sentmap) { +set_bit (bitoffset, ms->sentmap); } + + last_sent_block = block; +break ; } } last_seen_block = block;
@@ -851,6 +951,7 @@ static void reset_ram_globals(void) last_offset =0 ; last_version = ram_list.version; ram_bulk_stage = true; + last_was_from_queue = false; }#define MAX_WAIT 50/* ms, half buffered_file limit */
Signature · 1 line
1.9.3