Skip to contentSkip to conversation

Postcopy implementation

PATCH Superseded Dr. David Alan Gilbert (git) 8 participants 204 messages Oct 3, 2014 – Jan 28, 2015 Kbuild Page 1 of 2

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover Postcopy implementation

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This is the 4th cut of my version of postcopy; it is designed for use with
the Linux kernel additions just posted by Andrea Arcangeli here:

http://marc.info/?l=linux-kernel&m=141235633015100&w=2

(Note: This is a new version compared to my previous postcopy patchset; you'll need to update the kernel to the new version.)

Other than the new kernel ABI (which is only a small change to the userspace side); the major changes are;

a) Code for host page size != target page size
b) Support for migration over fd
From Cristian Klein; this is for libvirt support which Cristian recently posted to the libvirt list.
c) It's now build bisectable and builds on 32bit

Testing wise; I've now done many thousand of postcopy migrations without failure (both of idle and busy guests); so it seems pretty solid.

Must-TODO's:
1) A partially repeatable migration_cancel failure
2) virt_test's migrate.with_reboot test is failing
3) The ACPI fix in 2.1 that allowed migrating RAMBlocks to be larger than the source feels like it needs looking at for postcopy.
4) Paolo's comments with respect to the wakeup_request/is_running code in the migration thread
5) xbzrle needs disabling once in postcopy

Later-TODO's:
1) Control the rate of background page transfers during postcopy to reduce their impact on the latency of postcopy requests.
2) Work with RDMA
3) Could destination RP be made blocking (as per discussion with Paolo; I'm still worried that that changes too many assumptions)

V4:
Initial support for host page size != target page size
- tested heavily on hps==tps
- only partially tested on hps!=tps systems
- This involved quite a bit of rework around the discard code
Updated to new kernel userfault ABI
- It won't work with the previous version
Fix mis-optimisation of postcopy request for wrong RAMBlock
request for block A offset n
un-needed fault for block B/m (already received - no req sent) request for block B/l - wrongly sent as request for A/l
Fix thinko in discard bitmap processing (missed last word of bitmap)
Symptom: remap failures near the top of RAM if postcopy started late
Fix bug that caused kernel page acknowledgments to be misaligned
May have meant the guest was paused for longer than required
Fix potential for crashing cleaning up failed RP
Fixes in docs (from Yang)
Handle migration by fd as sockets if they are sockets
Build tested on 32bit
Fully build bisectable (x86-64)

Dave

Cristian Klein (1):
Handle bi-directional communication for fd migration

Dr. David Alan Gilbert (46):
QEMUSizedBuffer based QEMUFile
Tests: QEMUSizedBuffer/QEMUBuffer
Start documenting how postcopy works.
qemu_ram_foreach_block: pass up error value, and down the ramblock
name
improve DPRINTF macros, add to savevm
Add qemu_get_counted_string to read a string prefixed by a count byte Create MigrationIncomingState
socket shutdown
Provide runtime Target page information
Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source Return path: Source handling of return path
qemu_loadvm errors and debug
ram_debug_dump_bitmap: Dump a migration bitmap as text Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
migrate_init: Call from savevm
Allow savevm handlers to state whether they could go into postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
MIG_STATE_POSTCOPY_ACTIVE: Add new migration state qemu_savevm_state_complete: Postcopy changes
Postcopy page-map-incoming (PMI) structure
Postcopy: Maintain sentmap and calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: Create a fault handler thread before marking the ram as
userfault
Page request: Add MIG_RPCOMM_REQPAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue
Add assertion to check migration_dirty_pages postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration qemu_ram_block_from_host
Don't sync dirty bitmaps in postcopy
Host page!=target page: Cleanup bitmaps
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_ram_handle_{run,end} commands End of migration for postcopy

 Makefile.objs                    |    2 +-
 arch_init.c                      |  739 +++++++++++++++++++++++++--
 docs/migration.txt               |  189 +++++++
 exec.c                           |   76 ++-
 hmp-commands.hx                  |   15 +
 hmp.c                            |    7 +
 hmp.h                            |    1 +
 include/exec/cpu-common.h        |    8 +-
 include/migration/migration.h    |  130 +++++
 include/migration/postcopy-ram.h |  106 ++++
 include/migration/qemu-file.h    |   47 ++
 include/migration/vmstate.h      |    2 +-
 include/qemu/sockets.h           |    1 +
 include/qemu/typedefs.h          |    9 +-
 include/sysemu/sysemu.h          |   43 +-
 migration-fd.c                   |   24 +-
 migration-rdma.c                 |    4 +-
 migration.c                      |  693 +++++++++++++++++++++++++-
 postcopy-ram.c                   | 1016 ++++++++++++++++++++++++++++++++++++++
 qapi-schema.json                 |   14 +-
 qemu-file.c                      |  598 +++++++++++++++++++++-
 qmp-commands.hx                  |   19 +
 savevm.c                         |  881 +++++++++++++++++++++++++++++++--
 tests/Makefile                   |    2 +-
 tests/test-vmstate.c             |   74 +--
 util/qemu-sockets.c              |   28 ++
 26 files changed, 4550 insertions(+), 178 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 postcopy-ram.c
Signature · 1 line
1.9.3
#1 lore raw

PATCH v4 1/47 QEMUSizedBuffer based QEMUFile

From: "Dr. David Alan Gilbert" <[email protected]>

* Please comment on separate thread for this QEMUSizedBuffer patch *

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

For fixes/tweeks I've done:

---
 include/migration/qemu-file.h |  28 +++
 include/qemu/typedefs.h       |   1 +
 qemu-file.c                   | 456 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 485 insertions(+)
include/migration/qemu-file.h +28 -0
index c90f529..6ef8ebc 100644
@@ -25,6 +25,8 @@
 #define QEMU_FILE_H 1
 #include "exec/cpu-common.h"
 
+#include <stdint.h>
+
 /* This function writes a chunk of data to a file at the given position.
  * The pos argument can be ignored if the file is only being used for
  * streaming.  The handler should try to write all of the data it can.
@@ -94,11 +96,21 @@ typedef struct QEMUFileOps {
     QEMURamSaveFunc *save_page;
 } QEMUFileOps;
 
+struct QEMUSizedBuffer {
+    struct iovec *iov;
+    size_t n_iov;
+    size_t size; /* total allocated size in all iov's */
+    size_t used; /* number of used bytes */
+};
+
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
+
 QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
 QEMUFile *qemu_fopen(const char *filename, const char *mode);
 QEMUFile *qemu_fdopen(int fd, const char *mode);
 QEMUFile *qemu_fopen_socket(int fd, const char *mode);
 QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input);
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);
 void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size);
 bool qemu_file_mode_is_not_valid(const char *mode);
 
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len);
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *);
+void qsb_free(QEMUSizedBuffer *);
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t length);
+size_t qsb_get_length(const QEMUSizedBuffer *qsb);
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
+                       uint8_t *buf);
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
+                     off_t pos, size_t count);
+
+
+/*
+ * For use on files opened with qemu_bufopen
+ */
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f);
+
 static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v)
 {
     qemu_put_byte(f, (int)v);
include/qemu/typedefs.h +1 -0
index 5f20b0e..db1153a 100644
@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;
 typedef struct PCIEAERErr PCIEAERErr;
 typedef struct PCIEPort PCIEPort;
 typedef struct PCIESlot PCIESlot;
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct MSIMessage MSIMessage;
 typedef struct SerialState SerialState;
 typedef struct PCMCIACardState PCMCIACardState;
qemu-file.c +456 -0

459 lines not on this page · raw message

Signature · 1 line
1.9.3
#2 lore raw

Re: [Qemu-devel] [PATCH v4 01/47] QEMUSizedBuffer based QEMUFile

Quoted 41 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

* Please comment on separate thread for this QEMUSizedBuffer patch *

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

For fixes/tweeks I've done:

---
  include/migration/qemu-file.h |  28 +++
  include/qemu/typedefs.h       |   1 +
  qemu-file.c                   | 456 ++++++++++++++++++++++++++++++++++++++++++
  3 files changed, 485 insertions(+)
include/migration/qemu-file.h +2 -0
index c90f529..6ef8ebc 100644
@@ -25,6 +25,8 @@
  #define QEMU_FILE_H 1
  #include "exec/cpu-common.h"
 
+#include <stdint.h>
+
  /* This function writes a chunk of data to a file at the given position.
   * The pos argument can be ignored if the file is only being used for
   * streaming.  The handler should try to write all of the data it can.
@@ -94,11 +96,21 @@ typedef struct QEMUFileOps {
      QEMURamSaveFunc *save_page;
  } QEMUFileOps;

include/migration/qemu-file.h typedef struct QEMUFileOps { · line 107

Quoted 10 lines
include/migration/qemu-file.h +9 -0
 
+struct QEMUSizedBuffer {
+    struct iovec *iov;
+    size_t n_iov;
+    size_t size; /* total allocated size in all iov's */
+    size_t used; /* number of used bytes */
+};
+
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
+

There is a redefinition of typedef ‘QEMUSizedBuffer’ in 'include/qemu/typedefs.h:68', when i compile qemu, it complains;)

Quoted 37 lines
include/migration/qemu-file.h +17 -0
  QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
  QEMUFile *qemu_fopen(const char *filename, const char *mode);
  QEMUFile *qemu_fdopen(int fd, const char *mode);
  QEMUFile *qemu_fopen_socket(int fd, const char *mode);
  QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input);
  int qemu_get_fd(QEMUFile *f);
  int qemu_fclose(QEMUFile *f);
  int64_t qemu_ftell(QEMUFile *f);
@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);
  void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size);
  bool qemu_file_mode_is_not_valid(const char *mode);
 
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len);
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *);
+void qsb_free(QEMUSizedBuffer *);
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t length);
+size_t qsb_get_length(const QEMUSizedBuffer *qsb);
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
+                       uint8_t *buf);
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
+                     off_t pos, size_t count);
+
+
+/*
+ * For use on files opened with qemu_bufopen
+ */
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f);
+
  static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v)
  {
      qemu_put_byte(f, (int)v);
include/qemu/typedefs.h +0 -0
index 5f20b0e..db1153a 100644

include/qemu/typedefs.h typedef struct PCIEAERLog PCIEAERLog; · line 63

Quoted 4 lines
include/qemu/typedefs.h +1 -0
@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;
  typedef struct PCIEAERErr PCIEAERErr;
  typedef struct PCIEPort PCIEPort;
  typedef struct PCIESlot PCIESlot;
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;

Here!, see above comment. Thanks.

Quoted 468 lines
include/qemu/typedefs.h +0 -0
  typedef struct MSIMessage MSIMessage;
  typedef struct SerialState SerialState;
  typedef struct PCMCIACardState PCMCIACardState;
qemu-file.c +456 -0
index a8e3912..ccc516c 100644
@@ -878,3 +878,459 @@ uint64_t qemu_get_be64(QEMUFile *f)
      v |= qemu_get_be32(f);
      return v;
  }
+
+#define QSB_CHUNK_SIZE      (1 << 10)
+#define QSB_MAX_CHUNK_SIZE  (16 * QSB_CHUNK_SIZE)
+
+/**
+ * Create a QEMUSizedBuffer
+ * This type of buffer uses scatter-gather lists internally and
+ * can grow to any size. Any data array in the scatter-gather list
+ * can hold different amount of bytes.
+ *
+ * @buffer: Optional buffer to copy into the QSB
+ * @len: size of initial buffer; if @buffer is given, buffer must
+ *       hold at least len bytes
+ *
+ * Returns a pointer to a QEMUSizedBuffer or NULL on allocation failure
+ */
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len)
+{
+    QEMUSizedBuffer *qsb;
+    size_t alloc_len, num_chunks, i, to_copy;
+    size_t chunk_size = (len > QSB_MAX_CHUNK_SIZE)
+                        ? QSB_MAX_CHUNK_SIZE
+                        : QSB_CHUNK_SIZE;
+
+    num_chunks = DIV_ROUND_UP(len ? len : QSB_CHUNK_SIZE, chunk_size);
+    alloc_len = num_chunks * chunk_size;
+
+    qsb = g_try_new0(QEMUSizedBuffer, 1);
+    if (!qsb) {
+        return NULL;
+    }
+
+    qsb->iov = g_try_new0(struct iovec, num_chunks);
+    if (!qsb->iov) {
+        g_free(qsb);
+        return NULL;
+    }
+
+    qsb->n_iov = num_chunks;
+
+    for (i = 0; i < num_chunks; i++) {
+        qsb->iov[i].iov_base = g_try_malloc0(chunk_size);
+        if (!qsb->iov[i].iov_base) {
+            /* qsb_free is safe since g_free can cope with NULL */
+            qsb_free(qsb);
+            return NULL;
+        }
+
+        qsb->iov[i].iov_len = chunk_size;
+        if (buffer) {
+            to_copy = (len - qsb->used) > chunk_size
+                      ? chunk_size : (len - qsb->used);
+            memcpy(qsb->iov[i].iov_base, &buffer[qsb->used], to_copy);
+            qsb->used += to_copy;
+        }
+    }
+
+    qsb->size = alloc_len;
+
+    return qsb;
+}
+
+/**
+ * Free the QEMUSizedBuffer
+ *
+ * @qsb: The QEMUSizedBuffer to free
+ */
+void qsb_free(QEMUSizedBuffer *qsb)
+{
+    size_t i;
+
+    if (!qsb) {
+        return;
+    }
+
+    for (i = 0; i < qsb->n_iov; i++) {
+        g_free(qsb->iov[i].iov_base);
+    }
+    g_free(qsb->iov);
+    g_free(qsb);
+}
+
+/**
+ * Get the number of used bytes in the QEMUSizedBuffer
+ *
+ * @qsb: A QEMUSizedBuffer
+ *
+ * Returns the number of bytes currently used in this buffer
+ */
+size_t qsb_get_length(const QEMUSizedBuffer *qsb)
+{
+    return qsb->used;
+}
+
+/**
+ * Set the length of the buffer; the primary usage of this
+ * function is to truncate the number of used bytes in the buffer.
+ * The size will not be extended beyond the current number of
+ * allocated bytes in the QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_len: The new length of bytes in the buffer
+ *
+ * Returns the number of bytes the buffer was truncated or extended
+ * to.
+ */
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t new_len)
+{
+    if (new_len <= qsb->size) {
+        qsb->used = new_len;
+    } else {
+        qsb->used = qsb->size;
+    }
+    return qsb->used;
+}
+
+/**
+ * Get the iovec that holds the data for a given position @pos.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @pos: The index of a byte in the buffer
+ * @d_off: Pointer to an offset that this function will indicate
+ *         at what position within the returned iovec the byte
+ *         is to be found
+ *
+ * Returns the index of the iovec that holds the byte at the given
+ * index @pos in the byte stream; a negative number if the iovec
+ * for the given position @pos does not exist.
+ */
+static ssize_t qsb_get_iovec(const QEMUSizedBuffer *qsb,
+                             off_t pos, off_t *d_off)
+{
+    ssize_t i;
+    off_t curr = 0;
+
+    if (pos > qsb->used) {
+        return -1;
+    }
+
+    for (i = 0; i < qsb->n_iov; i++) {
+        if (curr + qsb->iov[i].iov_len > pos) {
+            *d_off = pos - curr;
+            return i;
+        }
+        curr += qsb->iov[i].iov_len;
+    }
+    return -1;
+}
+
+/*
+ * Convert the QEMUSizedBuffer into a flat buffer.
+ *
+ * Note: If at all possible, try to avoid this function since it
+ *       may unnecessarily copy memory around.
+ *
+ * @qsb: pointer to QEMUSizedBuffer
+ * @start: offset to start at
+ * @count: number of bytes to copy
+ * @buf: a pointer to a buffer to write into (at least @count bytes)
+ *
+ * Returns the number of bytes copied into the output buffer
+ */
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *qsb, off_t start,
+                       size_t count, uint8_t *buffer)
+{
+    const struct iovec *iov;
+    size_t to_copy, all_copy;
+    ssize_t index;
+    off_t s_off;
+    off_t d_off = 0;
+    char *s;
+
+    if (start > qsb->used) {
+        return 0;
+    }
+
+    all_copy = qsb->used - start;
+    if (all_copy > count) {
+        all_copy = count;
+    } else {
+        count = all_copy;
+    }
+
+    index = qsb_get_iovec(qsb, start, &s_off);
+    if (index < 0) {
+        return 0;
+    }
+
+    while (all_copy > 0) {
+        iov = &qsb->iov[index];
+
+        s = iov->iov_base;
+
+        to_copy = iov->iov_len - s_off;
+        if (to_copy > all_copy) {
+            to_copy = all_copy;
+        }
+        memcpy(&buffer[d_off], &s[s_off], to_copy);
+
+        d_off += to_copy;
+        all_copy -= to_copy;
+
+        s_off = 0;
+        index++;
+    }
+
+    return count;
+}
+
+/**
+ * Grow the QEMUSizedBuffer to the given size and allocate
+ * memory for it.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_size: The new size of the buffer
+ *
+ * Return:
+ *    a negative error code in case of memory allocation failure
+ * or
+ *    the new size of the buffer. The returned size may be greater or equal
+ *    to @new_size.
+ */
+static ssize_t qsb_grow(QEMUSizedBuffer *qsb, size_t new_size)
+{
+    size_t needed_chunks, i;
+
+    if (qsb->size < new_size) {
+        struct iovec *new_iov;
+        size_t size_diff = new_size - qsb->size;
+        size_t chunk_size = (size_diff > QSB_MAX_CHUNK_SIZE)
+                             ? QSB_MAX_CHUNK_SIZE : QSB_CHUNK_SIZE;
+
+        needed_chunks = DIV_ROUND_UP(size_diff, chunk_size);
+
+        new_iov = g_try_malloc_n(qsb->n_iov + needed_chunks,
+                                 sizeof(struct iovec));
+        if (new_iov == NULL) {
+            return -ENOMEM;
+        }
+
+        /* Allocate new chunks as needed into new_iov */
+        for (i = qsb->n_iov; i < qsb->n_iov + needed_chunks; i++) {
+            new_iov[i].iov_base = g_try_malloc0(chunk_size);
+            new_iov[i].iov_len = chunk_size;
+            if (!new_iov[i].iov_base) {
+                size_t j;
+
+                /* Free previously allocated new chunks */
+                for (j = qsb->n_iov; j < i; j++) {
+                    g_free(new_iov[j].iov_base);
+                }
+                g_free(new_iov);
+
+                return -ENOMEM;
+            }
+        }
+
+        /*
+         * Now we can't get any allocation errors, copy over to new iov
+         * and switch.
+         */
+        for (i = 0; i < qsb->n_iov; i++) {
+            new_iov[i] = qsb->iov[i];
+        }
+
+        qsb->n_iov += needed_chunks;
+        g_free(qsb->iov);
+        qsb->iov = new_iov;
+        qsb->size += (needed_chunks * chunk_size);
+    }
+
+    return qsb->size;
+}
+
+/**
+ * Write into the QEMUSizedBuffer at a given position and a given
+ * number of bytes. This function will automatically grow the
+ * QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @source: A byte array to copy data from
+ * @pos: The position within the @qsb to write data to
+ * @size: The number of bytes to copy into the @qsb
+ *
+ * Returns @size or a negative error code in case of memory allocation failure,
+ *           or with an invalid 'pos'
+ */
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *source,
+                     off_t pos, size_t count)
+{
+    ssize_t rc = qsb_grow(qsb, pos + count);
+    size_t to_copy;
+    size_t all_copy = count;
+    const struct iovec *iov;
+    ssize_t index;
+    char *dest;
+    off_t d_off, s_off = 0;
+
+    if (rc < 0) {
+        return rc;
+    }
+
+    if (pos + count > qsb->used) {
+        qsb->used = pos + count;
+    }
+
+    index = qsb_get_iovec(qsb, pos, &d_off);
+    if (index < 0) {
+        return -EINVAL;
+    }
+
+    while (all_copy > 0) {
+        iov = &qsb->iov[index];
+
+        dest = iov->iov_base;
+
+        to_copy = iov->iov_len - d_off;
+        if (to_copy > all_copy) {
+            to_copy = all_copy;
+        }
+
+        memcpy(&dest[d_off], &source[s_off], to_copy);
+
+        s_off += to_copy;
+        all_copy -= to_copy;
+
+        d_off = 0;
+        index++;
+    }
+
+    return count;
+}
+
+/**
+ * Create a deep copy of the given QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ *
+ * Returns a clone of @qsb or NULL on allocation failure
+ */
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *qsb)
+{
+    QEMUSizedBuffer *out = qsb_create(NULL, qsb_get_length(qsb));
+    size_t i;
+    ssize_t res;
+    off_t pos = 0;
+
+    if (!out) {
+        return NULL;
+    }
+
+    for (i = 0; i < qsb->n_iov; i++) {
+        res =  qsb_write_at(out, qsb->iov[i].iov_base,
+                            pos, qsb->iov[i].iov_len);
+        if (res < 0) {
+            qsb_free(out);
+            return NULL;
+        }
+        pos += res;
+    }
+
+    return out;
+}
+
+typedef struct QEMUBuffer {
+    QEMUSizedBuffer *qsb;
+    QEMUFile *file;
+} QEMUBuffer;
+
+static int buf_get_buffer(void *opaque, uint8_t *buf, int64_t pos, int size)
+{
+    QEMUBuffer *s = opaque;
+    ssize_t len = qsb_get_length(s->qsb) - pos;
+
+    if (len <= 0) {
+        return 0;
+    }
+
+    if (len > size) {
+        len = size;
+    }
+    return qsb_get_buffer(s->qsb, pos, len, buf);
+}
+
+static int buf_put_buffer(void *opaque, const uint8_t *buf,
+                          int64_t pos, int size)
+{
+    QEMUBuffer *s = opaque;
+
+    return qsb_write_at(s->qsb, buf, pos, size);
+}
+
+static int buf_close(void *opaque)
+{
+    QEMUBuffer *s = opaque;
+
+    qsb_free(s->qsb);
+
+    g_free(s);
+
+    return 0;
+}
+
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f)
+{
+    QEMUBuffer *p;
+
+    qemu_fflush(f);
+
+    p = f->opaque;
+
+    return p->qsb;
+}
+
+static const QEMUFileOps buf_read_ops = {
+    .get_buffer = buf_get_buffer,
+    .close =      buf_close,
+};
+
+static const QEMUFileOps buf_write_ops = {
+    .put_buffer = buf_put_buffer,
+    .close =      buf_close,
+};
+
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input)
+{
+    QEMUBuffer *s;
+
+    if (mode == NULL || (mode[0] != 'r' && mode[0] != 'w') ||
+        mode[1] != '\0') {
+        error_report("qemu_bufopen: Argument validity check failed");
+        return NULL;
+    }
+
+    s = g_malloc0(sizeof(QEMUBuffer));
+    if (mode[0] == 'r') {
+        s->qsb = input;
+    }
+
+    if (s->qsb == NULL) {
+        s->qsb = qsb_create(NULL, 0);
+    }
+    if (!s->qsb) {
+        g_free(s);
+        error_report("qemu_bufopen: qsb_create failed");
+        return NULL;
+    }
+
+
+    if (mode[0] == 'r') {
+        s->file = qemu_fopen_ops(s, &buf_read_ops);
+    } else {
+        s->file = qemu_fopen_ops(s, &buf_write_ops);
+    }
+    return s->file;
+}
Reviewed-by on v4 1/47
#3 lore raw

Re: [Qemu-devel] [PATCH v4 01/47] QEMUSizedBuffer based QEMUFile

In reply to Dr. David Alan Gilbert (git)
Quoted 12 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

* Please comment on separate thread for this QEMUSizedBuffer patch *

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

Quoted 7 lines

For fixes/tweeks I've done:

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#4 lore raw

PATCH v4 2/47 Tests: QEMUSizedBuffer/QEMUBuffer

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

* Please comment on separate thread for this QEMUSziedBuffer patch *

Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.

Only some of the tests are changed so that the fd backed QEMUFile is still tested.

---
 tests/Makefile       |  2 +-
 tests/test-vmstate.c | 74 +++++++++++++++++++++++++++-------------------------
 2 files changed, 39 insertions(+), 37 deletions(-)
tests/Makefile +1 -1
index 834279c..d004618 100644
@@ -260,7 +260,7 @@ tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
 	libqemuutil.a libqemustub.a
 tests/test-vmstate$(EXESUF): tests/test-vmstate.o \
 	vmstate.o qemu-file.o \
-	libqemuutil.a
+	libqemuutil.a libqemustub.a
 
 tests/test-qapi-types.c tests/test-qapi-types.h :\
 $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py
tests/test-vmstate.c +38 -36

143 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v4 2/47
#5 lore raw

Re: [Qemu-devel] [PATCH v4 02/47] Tests: QEMUSizedBuffer/QEMUBuffer

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

* Please comment on separate thread for this QEMUSziedBuffer patch *

Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.

Quoted 5 lines

Only some of the tests are changed so that the fd backed QEMUFile is still tested.

Looks ok, although I do wonder if the removal of some of the test paths for regular files is a concern.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#6 lore raw

PATCH v4 3/47 Start documenting how postcopy works.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 docs/migration.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
docs/migration.txt +189 -0
index 0492a45..a07b744 100644
@@ -294,3 +294,192 @@ save/send this state when we are in the middle of a pio operation
 (that is what ide_drive_pio_state_needed() checks).  If DRQ_STAT is
 not enabled, the values on that fields are garbage and don't need to
 be sent.
+
+= Return path =
+
+In most migration scenarios there is only a single data path that runs
+from the source VM to the destination, typically along a single fd (although
+possibly with another fd or similar for some fast way of throwing pages across).
+
+However, some uses need two way communication; in particular the Postcopy destination
+needs to be able to request pages on demand from the source.
+
+For these scenarios there is a 'return path' from the destination to the source;
+qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return
+path.
+
+  Source side
+     Forward path - written by migration thread
+     Return path  - opened by main thread, read by return-path thread
+
+  Destination side
+     Forward path - read by main thread
+     Return path  - opened by main thread, written by main thread AND postcopy
+                    thread (protected by rp_mutex)
+
+= Postcopy =
+'Postcopy' migration is a way to deal with migrations that refuse to converge;
+its plus side is that there is an upper bound on the amount of migration traffic
+and time it takes, the down side is that during the postcopy phase, a failure of
+*either* side or the network connection causes the guest to be lost.
+
+In postcopy the destination CPUs are started before all the memory has been
+transferred, and accesses to pages that are yet to be transferred cause
+a fault that's translated by QEMU into a request to the source QEMU.
+
+Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
+doesn't finish in a given time the switch is made to postcopy.
+
+=== Enabling postcopy ===
+
+To enable postcopy (prior to the start of migration):
+
+migrate_set_capability x-postcopy-ram on
+
+The migration will still start in precopy mode, however issuing:
+
+migrate_start_postcopy
+
+will now cause the transition from precopy to postcopy.
+It can be issued immediately after migration is started or any
+time later on.  Issuing it after the end of a migration is harmless.
+
+=== Postcopy device transfer ===
+
+Loading of device data may cause the device emulation to access guest RAM
+that may trigger faults that have to be resolved by the source, as such
+the migration stream has to be able to respond with page data *during* the
+device load, and hence the device data has to be read from the stream completely
+before the device load begins to free the stream up.  This is achieved by
+'packaging' the device data into a blob that's read in one go.
+
+Source behaviour
+
+Until postcopy is entered the migration stream is identical to normal
+precopy, except for the addition of a 'postcopy advise' command at
+the beginning, to tell the destination that postcopy might happen.
+When postcopy starts the source sends the page discard data and then
+forms the 'package' containing:
+
+   Command: 'postcopy ram listen'
+   The device state
+      A series of sections, identical to the precopy streams device state stream
+      containing everything except postcopiable devices (i.e. RAM)
+   Command: 'postcopy ram run'
+
+The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the
+contents are formatted in the same way as the main migration stream.
+
+Destination behaviour
+
+Initially the destination looks the same as precopy, with a single thread
+reading the migration stream; the 'postcopy advise' and 'discard' commands
+are processed to change the way RAM is managed, but don't affect the stream
+processing.
+
+------------------------------------------------------------------------------
+                        1      2   3     4 5                      6   7
+main -----DISCARD-CMD_PACKAGED ( LISTEN  DEVICE     DEVICE DEVICE RUN )
+thread                             |       |
+                                   |     (page request)
+                                   |        \___
+                                   v            \
+listen thread:                     --- page -- page -- page -- page -- page --
+
+                                   a   b        c
+------------------------------------------------------------------------------
+
+On receipt of CMD_PACKAGED (1)
+   All the data associated with the package - the ( ... ) section in the
+diagram - is read into memory (into a QEMUSizedBuffer), and the main thread
+recurses into qemu_loadvm_state_main to process the contents of the package (2)
+which contains commands (3,6) and devices (4...)
+
+On receipt of 'postcopy ram listen' - 3 -(i.e. the 1st command in the package)
+a new thread (a) is started that takes over servicing the migration stream,
+while the main thread carries on loading the package.   It loads normal
+background page data (b) but if during a device load a fault happens (5) the
+returned page (c) is loaded by the listen thread allowing the main threads
+device load to carry on.
+
+The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination
+CPUs start running.
+At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour
+and is no longer used by migration, while the listen thread carries
+on servicing page data until the end of migration.
+
+=== Postcopy states ===
+
+Postcopy moves through a series of states (see postcopy_ram_state)
+from ADVISE->LISTEN->RUNNING->END
+
+  Advise: Set at the start of migration if postcopy is enabled, even
+          if it hasn't had the start command; here the destination
+          checks that its OS has the support needed for postcopy, and performs
+          setup to ensure the RAM mappings are suitable for later postcopy.
+          (Triggered by reception of POSTCOPY_RAM_ADVISE command)
+
+  Listen: The first command in the package, POSTCOPY_RAM_LISTEN, switches
+          the destination state to Listen, and starts a new thread
+          (the 'listen thread') which takes over the job of receiving
+          pages off the migration stream, while the main thread carries
+          on processing the blob.  With this thread able to process page
+          reception, the destination now 'sensitises' the RAM to detect
+          any access to missing pages (on Linux using the 'userfault'
+          system).
+
+  Running: POSTCOPY_RAM_RUN causes the destination to synchronise all
+          state and start the CPUs and IO devices running.  The main
+          thread now finishes processing the migration package and
+          now carries on as it would for normal precopy migration
+          (although it can't do the cleanup it would do as it
+          finishes a normal migration).
+
+  End: The listen thread can now quit, and perform the cleanup of migration
+          state, the migration is now complete.
+
+=== Source side page maps ===
+
+The source side keeps two bitmaps during postcopy; 'the migration bitmap'
+and 'sent map'.  The 'migration bitmap' is basically the same as in
+the precopy case, and holds a bit to indicate that page is 'dirty' -
+i.e. needs sending.  During the precopy phase this is updated as the CPU
+dirties pages, however during postcopy the CPUs are stopped and nothing
+should dirty anything any more.
+
+The 'sent map' is used for the transition to postcopy. It is a bitmap that
+has a bit set whenever a page is sent to the destination, however during
+the transition to postcopy mode it is masked against the migration bitmap
+(sentmap &= migrationbitmap) to generate a bitmap recording pages that
+have been previously been sent but are now dirty again.  This masked
+sentmap is sent to the destination which discards those now dirty pages
+before starting the CPUs.
+
+Note that once in postcopy mode, the sent map is still updated; however,
+its contents are not necessarily consistent with the pages already sent
+due to the masking with the migration bitmap.
+
+=== Destination side page maps ===
+
+(Needs to be changed so we can update both easily - at the moment updates are done
+ with a lock)
+The destination keeps a 'requested map' and a 'received map'.
+Both maps are initially 0, as pages are received the bits are set in 'received map'.
+Incoming requests from the kernel cause the bit to be set in the 'requested map'.
+When a page is received that is marked as 'requested' the kernel is notified.
+If the kernel requests a page that has already been 'received' the kernel is notified
+without re-requesting.
+
+This leads to three valid page states:
+page states:
+    missing (!rc,!rq)  - page not yet received or requested
+    received (rc,!rq)  - Page received
+    requested (!rc,rq) - page requested but not yet received
+
+state transitions:
+      received -> missing   (only during setup/discard)
+
+      missing -> received   (normal incoming page)
+      requested -> received (incoming page previously requested)
+      missing -> requested  (userfault request)
+
Signature · 1 line
1.9.3
Reviewed-by on v4 3/47
#7 lore raw

Re: [Qemu-devel] [PATCH v4 03/47] Start documenting how postcopy works.

Quoted 3 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#8 lore raw

PATCH v4 4/47 qemu_ram_foreach_block: pass up error value, and down the ramblock name

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.

Pass the name of the ramblock to the function; helps in debugging.

---
 exec.c                    | 10 ++++++++--
 include/exec/cpu-common.h |  4 ++--
 migration-rdma.c          |  4 ++--
 3 files changed, 12 insertions(+), 6 deletions(-)
exec.c +8 -2
index 759055d..7989d19 100644
@@ -2867,12 +2867,18 @@ bool cpu_physical_memory_is_io(hwaddr phys_addr)
              memory_region_is_romd(mr));
 }
 
-void qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
+int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
 {
     RAMBlock *block;
+    int ret;
 
     QTAILQ_FOREACH(block, &ram_list.blocks, next) {
-        func(block->host, block->offset, block->length, opaque);
+        ret = func(block->idstr, block->host, block->offset, block->length,
+                   opaque);
+        if (ret) {
+            return ret;
+        }
     }
+    return 0;
 }
 #endif
include/exec/cpu-common.h +2 -2
index e3ec4c8..8042f50 100644
@@ -118,10 +118,10 @@ void cpu_flush_icache_range(hwaddr start, int len);
 extern struct MemoryRegion io_mem_rom;
 extern struct MemoryRegion io_mem_notdirty;
 
-typedef void (RAMBlockIterFunc)(void *host_addr,
+typedef int (RAMBlockIterFunc)(const char *block_name, void *host_addr,
     ram_addr_t offset, ram_addr_t length, void *opaque);
 
-void qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
+int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
 
 #endif
 
migration-rdma.c +2 -2
index b32dbdf..c0e52ed 100644
@@ -595,10 +595,10 @@ static int __qemu_rdma_add_block(RDMAContext *rdma, void *host_addr,
  * in advanced before the migration starts. This tells us where the RAM blocks
  * are so that we can register them individually.
  */
-static void qemu_rdma_init_one_block(void *host_addr,
+static int qemu_rdma_init_one_block(const char *block_name, void *host_addr,
     ram_addr_t block_offset, ram_addr_t length, void *opaque)
 {
-    __qemu_rdma_add_block(opaque, host_addr, block_offset, length);
+    return __qemu_rdma_add_block(opaque, host_addr, block_offset, length);
 }
 
 /*
Signature · 1 line
1.9.3
Reviewed-by on v4 4/47
#9 lore raw

Re: [Qemu-devel] [PATCH v4 04/47] qemu_ram_foreach_block: pass up error value, and down the ramblock name

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

check the return value of the function it calls and error if it's non-0 Fixup qemu_rdma_init_one_block that is the only current caller,
and __qemu_rdma_add_block the only function it calls using it.

Pass the name of the ramblock to the function; helps in debugging.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#10 lore raw

PATCH v4 5/47 improve DPRINTF macros, add to savevm

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Improve the existing DPRINTF macros in migration.c and arch_init by:
1) Making them go to stderr rather than stdout (so you can run with
-nographic and redirect your debug to a file)
2) Making them print the ms time with each debug - useful for
debugging latency issues

Add the same macro to savevm

---
 arch_init.c |  5 ++++-
 migration.c | 12 ++++++++++++
 savevm.c    | 10 ++++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)
arch_init.c +4 -1
index c974f3f..772de36 100644
@@ -53,9 +53,12 @@
 #include "hw/acpi/acpi.h"
 #include "qemu/host-utils.h"
 
+// #define DEBUG_ARCH_INIT
 #ifdef DEBUG_ARCH_INIT
 #define DPRINTF(fmt, ...) \
-    do { fprintf(stdout, "arch_init: " fmt, ## __VA_ARGS__); } while (0)
+    do { fprintf(stderr,  "arch_init@%" PRId64 " " fmt "\n", \
+                          qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \
+                          ## __VA_ARGS__); } while (0)
 #else
 #define DPRINTF(fmt, ...) \
     do { } while (0)
migration.c +12 -0
index 8d675b3..e241370 100644
@@ -26,6 +26,18 @@
 #include "qmp-commands.h"
 #include "trace.h"
 
+//#define DEBUG_MIGRATION
+
+#ifdef DEBUG_MIGRATION
+#define DPRINTF(fmt, ...) \
+    do { fprintf(stderr, "migration@%" PRId64 " " fmt "\n", \
+                          qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \
+                          ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
 enum {
     MIG_STATE_ERROR = -1,
     MIG_STATE_NONE,
savevm.c +10 -0
index e19ae0a..c3a1f68 100644
@@ -43,6 +43,16 @@
 #include "block/snapshot.h"
 #include "block/qapi.h"
 
+#ifdef DEBUG_SAVEVM
+#define DPRINTF(fmt, ...) \
+    do { fprintf(stderr, "savevm@%" PRId64 " " fmt "\n", \
+                          qemu_clock_get_ms(QEMU_CLOCK_REALTIME), \
+                          ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
 
 #ifndef ETH_P_RARP
 #define ETH_P_RARP 0x8035
Signature · 1 line
1.9.3
Reviewed-by on v4 5/47
#11 lore raw

Re: [Qemu-devel] [PATCH v4 05/47] improve DPRINTF macros, add to savevm

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Improve the existing DPRINTF macros in migration.c and arch_init by:
1) Making them go to stderr rather than stdout (so you can run with
-nographic and redirect your debug to a file)
2) Making them print the ms time with each debug - useful for
debugging latency issues

Quoted 3 lines

Add the same macro to savevm

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#12 lore raw

PATCH v4 6/47 Add qemu_get_counted_string to read a string prefixed by a count byte

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

and use it in loadvm_state.

---
 include/migration/qemu-file.h |  2 ++
 qemu-file.c                   | 15 +++++++++++++++
 savevm.c                      | 18 ++++++++++--------
 3 files changed, 27 insertions(+), 8 deletions(-)
include/migration/qemu-file.h +2 -0
index 6ef8ebc..a8cac7a 100644
@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 {
     qemu_get_be64s(f, (uint64_t *)pv);
 }
+
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf);
 #endif
qemu-file.c +15 -0
index ccc516c..a057b3e 100644
@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)
     return v;
 }
 
+/*
+ * Get a string whose length is determined by a single preceding byte
+ * A preallocated 256 byte buffer must be passed in.
+ * Returns: 0 on success and a 0 terminated string in the buffer
+ */
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf)
+{
+    unsigned int len = qemu_get_byte(f);
+    int res = qemu_get_buffer(f, buf, len);
+
+    buf[len] = 0;
+
+    return res != len;
+}
+
 #define QSB_CHUNK_SIZE      (1 << 10)
 #define QSB_MAX_CHUNK_SIZE  (16 * QSB_CHUNK_SIZE)
 
savevm.c +10 -8
index c3a1f68..cb6f0de 100644
@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f)
 
     v = qemu_get_be32(f);
     if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        fprintf(stderr, "SaveVM v2 format is obsolete and don't work anymore\n");
+        error_report("SaveVM v2 format is obsolete and don't work anymore");
         return -ENOTSUP;
     }
     if (v != QEMU_VM_FILE_VERSION) {
@@ -918,31 +918,33 @@ int qemu_loadvm_state(QEMUFile *f)
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
         SaveStateEntry *se;
-        char idstr[257];
-        int len;
+        char idstr[256];
 
         switch (section_type) {
         case QEMU_VM_SECTION_START:
         case QEMU_VM_SECTION_FULL:
             /* Read section start */
             section_id = qemu_get_be32(f);
-            len = qemu_get_byte(f);
-            qemu_get_buffer(f, (uint8_t *)idstr, len);
-            idstr[len] = 0;
+            if (qemu_get_counted_string(f, (uint8_t *)idstr)) {
+                error_report("Unable to read ID string for section %u",
+                            section_id);
+                return -EINVAL;
+            }
             instance_id = qemu_get_be32(f);
             version_id = qemu_get_be32(f);
 
             /* Find savevm section */
             se = find_se(idstr, instance_id);
             if (se == NULL) {
-                fprintf(stderr, "Unknown savevm section or instance '%s' %d\n", idstr, instance_id);
+                error_report("Unknown savevm section or instance '%s' %d",
+                             idstr, instance_id);
                 ret = -EINVAL;
                 goto out;
             }
 
             /* Validate version */
             if (version_id > se->version_id) {
-                fprintf(stderr, "savevm: unsupported version %d for '%s' v%d\n",
+                error_report("savevm: unsupported version %d for '%s' v%d",
                         version_id, idstr, se->version_id);
                 ret = -EINVAL;
                 goto out;
Signature · 1 line
1.9.3
#13 lore raw

Re: [Qemu-devel] [PATCH v4 06/47] Add qemu_get_counted_string to read a string prefixed by a count byte

Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

and use it in loadvm_state.

---
 include/migration/qemu-file.h |  2 ++
 qemu-file.c                   | 15 +++++++++++++++
 savevm.c                      | 18 ++++++++++--------
 3 files changed, 27 insertions(+), 8 deletions(-)
include/migration/qemu-file.h +0 -0
index 6ef8ebc..a8cac7a 100644

include/migration/qemu-file.h static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) · line 304

Quoted 5 lines
include/migration/qemu-file.h +2 -0
@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 {
     qemu_get_be64s(f, (uint64_t *)pv);
 }
+
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf);

I'd suggest writing the prototype as

int qemu_get_counted_string(QEMUFile *f, uint8_t buf[256]);

The compiled code will be identical, of course, but it helps to document what the function expects.

Quoted 32 lines
include/migration/qemu-file.h +0 -0
 #endif
qemu-file.c +15 -0
index ccc516c..a057b3e 100644
@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)
     return v;
 }
 
+/*
+ * Get a string whose length is determined by a single preceding byte
+ * A preallocated 256 byte buffer must be passed in.
+ * Returns: 0 on success and a 0 terminated string in the buffer
+ */
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf)
+{
+    unsigned int len = qemu_get_byte(f);
+    int res = qemu_get_buffer(f, buf, len);
+
+    buf[len] = 0;
+
+    return res != len;
+}
+
 #define QSB_CHUNK_SIZE      (1 << 10)
 #define QSB_MAX_CHUNK_SIZE  (16 * QSB_CHUNK_SIZE)
 
savevm.c +0 -0
index c3a1f68..cb6f0de 100644

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 911

Quoted 5 lines
savevm.c +1 -1
@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f)
 
     v = qemu_get_be32(f);
     if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        fprintf(stderr, "SaveVM v2 format is obsolete and don't work anymore\n");
+        error_report("SaveVM v2 format is obsolete and don't work anymore");

These changes of fprintf() to error_report() look like an unrelated cleanup.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#14 lore raw

Re: [Qemu-devel] [PATCH v4 06/47] Add qemu_get_counted_string to read a string prefixed by a count byte

Quoted 25 lines · * David Gibson ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

and use it in loadvm_state.

---
 include/migration/qemu-file.h |  2 ++
 qemu-file.c                   | 15 +++++++++++++++
 savevm.c                      | 18 ++++++++++--------
 3 files changed, 27 insertions(+), 8 deletions(-)
include/migration/qemu-file.h +0 -0
index 6ef8ebc..a8cac7a 100644

include/migration/qemu-file.h static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv) · line 304

Quoted 5 lines
include/migration/qemu-file.h +2 -0
@@ -300,4 +300,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 {
     qemu_get_be64s(f, (uint64_t *)pv);
 }
+
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf);

I'd suggest writing the prototype as

Quoted 4 lines

int qemu_get_counted_string(QEMUFile *f, uint8_t buf[256]);

The compiled code will be identical, of course, but it helps to document what the function expects.

Good idea; done.

Quoted 33 lines
Quoted 33 lines
include/migration/qemu-file.h +0 -0
 #endif
qemu-file.c +15 -0
index ccc516c..a057b3e 100644
@@ -879,6 +879,21 @@ uint64_t qemu_get_be64(QEMUFile *f)
     return v;
 }
 
+/*
+ * Get a string whose length is determined by a single preceding byte
+ * A preallocated 256 byte buffer must be passed in.
+ * Returns: 0 on success and a 0 terminated string in the buffer
+ */
+int qemu_get_counted_string(QEMUFile *f, uint8_t *buf)
+{
+    unsigned int len = qemu_get_byte(f);
+    int res = qemu_get_buffer(f, buf, len);
+
+    buf[len] = 0;
+
+    return res != len;
+}
+
 #define QSB_CHUNK_SIZE      (1 << 10)
 #define QSB_MAX_CHUNK_SIZE  (16 * QSB_CHUNK_SIZE)
 
savevm.c +0 -0
index c3a1f68..cb6f0de 100644
@@ -908,7 +908,7 @@ int qemu_loadvm_state(QEMUFile *f)
 
Quoted 7 lines

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 911

Quoted 4 lines
savevm.c +1 -1
     v = qemu_get_be32(f);
     if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        fprintf(stderr, "SaveVM v2 format is obsolete and don't work anymore\n");
+        error_report("SaveVM v2 format is obsolete and don't work anymore");

These changes of fprintf() to error_report() look like an unrelated cleanup.

Not quite; with the use of qemu_get_counted_string it can return an error,
so I check it, and use error_report; while I was there I converted a couple of the surrounding fprintf's to error_report at the same time.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#15 lore raw

PATCH v4 7/47 Create MigrationIncomingState

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.

allocate MIS in process_incoming_migration_co

---
 include/migration/migration.h |  9 +++++++++
 include/qemu/typedefs.h       |  2 ++
 migration.c                   | 28 ++++++++++++++++++++++++++++
 savevm.c                      |  2 ++
 4 files changed, 41 insertions(+)
include/migration/migration.h +9 -0
index 3cb5ba8..8a36255 100644
@@ -41,6 +41,15 @@ struct MigrationParams {
 
 typedef struct MigrationState MigrationState;
 
+/* State for the incoming migration */
+struct MigrationIncomingState {
+    QEMUFile *file;
+};
+
+MigrationIncomingState *migration_incoming_get_current(void);
+MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);
+void migration_incoming_state_destroy(void);
+
 struct MigrationState
 {
     int64_t bandwidth_limit;
include/qemu/typedefs.h +2 -0
index db1153a..0f79b5c 100644
@@ -14,6 +14,7 @@ typedef struct Visitor Visitor;
 
 struct Monitor;
 typedef struct Monitor Monitor;
+typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct MigrationParams MigrationParams;
 
 typedef struct Property Property;
@@ -44,6 +45,7 @@ typedef struct PixelFormat PixelFormat;
 typedef struct QemuConsole QemuConsole;
 typedef struct CharDriverState CharDriverState;
 typedef struct MACAddr MACAddr;
+typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct NetClientState NetClientState;
 typedef struct I2CBus I2CBus;
 typedef struct ISABus ISABus;
migration.c +28 -0
index e241370..ac46ddb 100644
@@ -65,6 +65,7 @@ static NotifierList migration_state_notifiers =
    migrations at once.  For now we don't need to add
    dynamic creation of migration */
 
+/* For outgoing */
 MigrationState *migrate_get_current(void)
 {
     static MigrationState current_migration = {
@@ -77,6 +78,28 @@ MigrationState *migrate_get_current(void)
     return &current_migration;
 }
 
+/* For incoming */
+static MigrationIncomingState *mis_current;
+
+MigrationIncomingState *migration_incoming_get_current(void)
+{
+    return mis_current;
+}
+
+MigrationIncomingState *migration_incoming_state_init(QEMUFile* f)
+{
+    mis_current = g_malloc0(sizeof(MigrationIncomingState));
+    mis_current->file = f;
+
+    return mis_current;
+}
+
+void migration_incoming_state_destroy(void)
+{
+    g_free(mis_current);
+    mis_current = NULL;
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
@@ -106,9 +129,14 @@ static void process_incoming_migration_co(void *opaque)
     Error *local_err = NULL;
     int ret;
 
+    migration_incoming_state_init(f);
+
     ret = qemu_loadvm_state(f);
+
     qemu_fclose(f);
     free_xbzrle_decoded_buf();
+    migration_incoming_state_destroy();
+
     if (ret < 0) {
         error_report("load of migration failed: %s", strerror(-ret));
         exit(EXIT_FAILURE);
savevm.c +2 -0
index cb6f0de..a0c3b40 100644
@@ -1244,9 +1244,11 @@ int load_vmstate(const char *name)
     }
 
     qemu_system_reset(VMRESET_SILENT);
+    migration_incoming_state_init(f);
     ret = qemu_loadvm_state(f);
 
     qemu_fclose(f);
+    migration_incoming_state_destroy();
     if (ret < 0) {
         error_report("Error %d while loading VM state", ret);
         return ret;
Signature · 1 line
1.9.3
#16 lore raw

Re: [Qemu-devel] [PATCH v4 07/47] Create MigrationIncomingState

Quoted 21 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.

allocate MIS in process_incoming_migration_co

---
 include/migration/migration.h |  9 +++++++++
 include/qemu/typedefs.h       |  2 ++
 migration.c                   | 28 ++++++++++++++++++++++++++++
 savevm.c                      |  2 ++
 4 files changed, 41 insertions(+)
include/migration/migration.h +0 -0
index 3cb5ba8..8a36255 100644

include/migration/migration.h struct MigrationParams { · line 50

Quoted 10 lines
include/migration/migration.h +7 -0
@@ -41,6 +41,15 @@ struct MigrationParams {
 
 typedef struct MigrationState MigrationState;
 
+/* State for the incoming migration */
+struct MigrationIncomingState {
+    QEMUFile *file;
+};
+
+MigrationIncomingState *migration_incoming_get_current(void);
+MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);

Hrm. I'd prefer to see this called migration_incoming_state_new(), since it allocates a new structure, rather than just initializing an already allocated one.

I guess you're trying to match migrate_init() in name, so i guess migrate_incoming_init() would work as well.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#17 lore raw

Re: [Qemu-devel] [PATCH v4 07/47] Create MigrationIncomingState

Quoted 37 lines · * David Gibson ([email protected]) wrote:
Quoted 21 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

There are currently lots of pieces of incoming migration state scattered around, and postcopy is adding more, and it seems better to try and keep it together.

allocate MIS in process_incoming_migration_co

---
 include/migration/migration.h |  9 +++++++++
 include/qemu/typedefs.h       |  2 ++
 migration.c                   | 28 ++++++++++++++++++++++++++++
 savevm.c                      |  2 ++
 4 files changed, 41 insertions(+)
include/migration/migration.h +0 -0
index 3cb5ba8..8a36255 100644

include/migration/migration.h struct MigrationParams { · line 50

Quoted 10 lines
include/migration/migration.h +7 -0
@@ -41,6 +41,15 @@ struct MigrationParams {
 
 typedef struct MigrationState MigrationState;
 
+/* State for the incoming migration */
+struct MigrationIncomingState {
+    QEMUFile *file;
+};
+
+MigrationIncomingState *migration_incoming_get_current(void);
+MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);

Hrm. I'd prefer to see this called migration_incoming_state_new(), since it allocates a new structure, rather than just initializing an already allocated one.

Quoted 2 lines

I guess you're trying to match migrate_init() in name, so i guess migrate_incoming_init() would work as well.

No, you're right the 1st time, _new is more consistent; fixed.

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#18 lore raw

PATCH v4 8/47 socket shutdown

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add QEMUFile interface to allow a socket to be 'shut down' - i.e. any reads/writes will fail (and any blocking read/write will be woken).

Add qemu_socket wrapper to let OS dependencies be extracted out.

---
 include/migration/qemu-file.h | 10 ++++++++++
 include/qemu/sockets.h        |  1 +
 qemu-file.c                   | 27 +++++++++++++++++++++++++--
 util/qemu-sockets.c           | 28 ++++++++++++++++++++++++++++
 4 files changed, 64 insertions(+), 2 deletions(-)
include/migration/qemu-file.h +10 -0
index a8cac7a..935cf42 100644
@@ -84,6 +84,14 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                size_t size,
                                int *bytes_sent);
 
+/*
+ * Stop any read or write (depending on flags) on the underlying
+ * transport on the QEMUFile.
+ * Existing blocking reads/writes must be woken
+ * Returns 0 on success, -err on error
+ */
+typedef int (QEMUFileShutdownFunc)(void *opaque, bool rd, bool wr);
+
 typedef struct QEMUFileOps {
     QEMUFilePutBufferFunc *put_buffer;
     QEMUFileGetBufferFunc *get_buffer;
@@ -94,6 +102,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
 struct QEMUSizedBuffer {
@@ -178,6 +187,7 @@ void qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);
 int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
+void qemu_file_shutdown(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
include/qemu/sockets.h +1 -0
index fdbb196..ea8ffc6 100644
@@ -41,6 +41,7 @@ int socket_set_nodelay(int fd);
 void qemu_set_block(int fd);
 void qemu_set_nonblock(int fd);
 int socket_set_fast_reuse(int fd);
+int socket_shutdown(int fd, bool rd, bool wr);
 int send_all(int fd, const void *buf, int len1);
 int recv_all(int fd, void *buf, int len1, bool single_read);
 
qemu-file.c +25 -2
index a057b3e..14dcf34 100644
@@ -90,6 +90,14 @@ static int socket_close(void *opaque)
     return 0;
 }
 
+/* qemufile_ to disambiguate from the qemu-sockets.c code which it uses */
+static int qemufile_socket_shutdown(void *opaque, bool rd, bool wr)
+{
+    QEMUFileSocket *s = opaque;
+
+    return socket_shutdown(s->fd, rd, wr);
+}
+
 static int stdio_get_fd(void *opaque)
 {
     QEMUFileStdio *s = opaque;
@@ -337,15 +345,30 @@ QEMUFile *qemu_fdopen(int fd, const char *mode)
 static const QEMUFileOps socket_read_ops = {
     .get_fd =     socket_get_fd,
     .get_buffer = socket_get_buffer,
-    .close =      socket_close
+    .close =      socket_close,
+    .shut_down       = qemufile_socket_shutdown
+
 };
 
 static const QEMUFileOps socket_write_ops = {
     .get_fd =     socket_get_fd,
     .writev_buffer = socket_writev_buffer,
-    .close =      socket_close
+    .close =      socket_close,
+    .shut_down       = qemufile_socket_shutdown
+
 };
 
+/*
+ * Stop a file from being read/written - not all backing files can do this
+ * typically only sockets can.  The caller should make sure they only
+ * call this for things that can.
+ */
+void qemu_file_shutdown(QEMUFile *f)
+{
+    assert(f->ops->shut_down);
+    f->ops->shut_down(f, true, true);
+}
+
 bool qemu_file_mode_is_not_valid(const char *mode)
 {
     if (mode == NULL ||
util/qemu-sockets.c +28 -0
index 1eef590..830e6d7 100644
@@ -981,3 +981,31 @@ int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp)
     qemu_opts_del(opts);
     return fd;
 }
+
+int socket_shutdown(int fd, bool rd, bool wr)
+{
+    int how = 0;
+
+#ifndef WIN32
+    if (rd) {
+        how = SHUT_RD;
+    }
+
+    if (wr) {
+        how = rd ? SHUT_RDWR : SHUT_WR;
+    }
+
+#else
+    /* Untested */
+    if (rd) {
+        how = SD_RECEIVE;
+    }
+
+    if (wr) {
+        how = rd ? SD_BOTH : SD_SEND;
+    }
+
+#endif
+
+    return shutdown(fd, how);
+}
Signature · 1 line
1.9.3
#19 lore raw

Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 11 lines
util/qemu-sockets.c +11 -0
+#ifndef WIN32
+    if (rd) {
+        how = SHUT_RD;
+    }
+
+    if (wr) {
+        how = rd ? SHUT_RDWR : SHUT_WR;
+    }
+
+#else
+    /* Untested */

util/qemu-sockets.c int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp) · line 1009

Quoted 10 lines
util/qemu-sockets.c +10 -0
+    if (rd) {
+        how = SD_RECEIVE;
+    }
+
+    if (wr) {
+        how = rd ? SD_BOTH : SD_SEND;
+    }
+
+#endif
+

These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.

Paolo

#20 lore raw

Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown

Quoted 18 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 17 lines
(quoted diff) +17 -0
+#ifndef WIN32
+    if (rd) {
+        how = SHUT_RD;
+    }
+
+    if (wr) {
+        how = rd ? SHUT_RDWR : SHUT_WR;
+    }
+
+#else
+    /* Untested */
+    if (rd) {
+        how = SD_RECEIVE;
+    }
+
+    if (wr) {
+        how = rd ? SD_BOTH : SD_SEND;
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+    }
+
+#endif
+

These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.

OK, something like this? (the qemu-file.c abstraction is still needed to cover QEMUFile's that aren't simple sockets, but I've removed the second layer in util/qemu-sockets.c).

include/qemu/sockets.h +7 -0
@@ -44,6 +44,13 @@ int socket_set_fast_reuse(int fd);
 int send_all(int fd, const void *buf, int len1);
 int recv_all(int fd, void *buf, int len1, bool single_read);
 
+#ifdef WIN32
+/* Windows has different names for the same constants with the same values */
+#define SHUT_RD   0
+#define SHUT_WR   1
+#define SHUT_RDWR 2
+#endif
+
 /* callback function for nonblocking connect
  * valid fd on success, negative error code on failure
  */
qemu-file.c +7 -0
@@ -90,6 +90,13 @@ static int socket_close(void *opaque)
     return 0;
 }
 
+static int socket_shutdown(void *opaque, bool rd, bool wr)
+{
+    QEMUFileSocket *s = opaque;
+
+    return shutdown(s->fd, rd ? (wr ? SHUT_RDWR : SHUT_RD) : SHUT_WR);
+}
+
 static int stdio_get_fd(void *opaque)
 {
     QEMUFileStdio *s = opaque;

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#21 lore raw

Re: [Qemu-devel] [PATCH v4 08/47] socket shutdown

Il 07/10/2014 12:00, Dr. David Alan Gilbert ha scritto:

Quoted 28 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 18 lines

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 17 lines
(quoted diff) +17 -0
+#ifndef WIN32
+    if (rd) {
+        how = SHUT_RD;
+    }
+
+    if (wr) {
+        how = rd ? SHUT_RDWR : SHUT_WR;
+    }
+
+#else
+    /* Untested */
+    if (rd) {
+        how = SD_RECEIVE;
+    }
+
+    if (wr) {
+        how = rd ? SD_BOTH : SD_SEND;
Quoted 8 lines

qemu-file.c static int socket_close(void *opaque)

Quoted 4 lines
qemu-file.c +4 -0
+    }
+
+#endif
+

These are the same on Windows and non-Windows actually. Just #define SHUT_* to 0/1/2 and avoid the wrapper.

Quoted 3 lines

OK, something like this? (the qemu-file.c abstraction is still needed to cover QEMUFile's that aren't simple sockets, but I've removed the second layer in util/qemu-sockets.c).

Yes. Or just pass SHUT_* directly to socket_shutdown, of course.

Paolo

Quoted 37 lines
include/qemu/sockets.h +7 -0
@@ -44,6 +44,13 @@ int socket_set_fast_reuse(int fd);
 int send_all(int fd, const void *buf, int len1);
 int recv_all(int fd, void *buf, int len1, bool single_read);
 
+#ifdef WIN32
+/* Windows has different names for the same constants with the same values */
+#define SHUT_RD   0
+#define SHUT_WR   1
+#define SHUT_RDWR 2
+#endif
+
 /* callback function for nonblocking connect
  * valid fd on success, negative error code on failure
  */
qemu-file.c +7 -1
@@ -90,6 +90,13 @@ static int socket_close(void *opaque)
     return 0;
 }
 
+static int socket_shutdown(void *opaque, bool rd, bool wr)
+{
+    QEMUFileSocket *s = opaque;
+
+    return shutdown(s->fd, rd ? (wr ? SHUT_RDWR : SHUT_RD) : SHUT_WR);
+}
+
 static int stdio_get_fd(void *opaque)
 {
     QEMUFileStdio *s = opaque;
--

Dr. David Alan Gilbert / [email protected] / Manchester, UK

#22 lore raw

PATCH v4 9/47 Provide runtime Target page information

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into arch_init.

Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.

---
 exec.c                  | 10 ++++++++++
 include/sysemu/sysemu.h |  1 +
 2 files changed, 11 insertions(+)
exec.c +10 -0
index 7989d19..65ee612 100644
@@ -2881,4 +2881,14 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
     }
     return 0;
 }
+
+/*
+ * Allows code that needs to deal with migration bitmaps etc to still be built
+ * target independent.
+ */
+size_t qemu_target_page_bits(void)
+{
+    return TARGET_PAGE_BITS;
+}
+
 #endif
include/sysemu/sysemu.h +1 -0
index d8539fd..6e5953d 100644
@@ -68,6 +68,7 @@ int qemu_reset_requested_get(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_devices_reset(void);
 void qemu_system_reset(bool report);
+size_t qemu_target_page_bits(void);
 
 void qemu_add_exit_notifier(Notifier *notify);
 void qemu_remove_exit_notifier(Notifier *notify);
Signature · 1 line
1.9.3
#23 lore raw

PATCH v4 10/47 Return path: Open a return path on QEMUFile for sockets

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

---
 include/migration/qemu-file.h |  7 +++++
 qemu-file.c                   | 73 +++++++++++++++++++++++++++++++++++++------
 2 files changed, 70 insertions(+), 10 deletions(-)
include/migration/qemu-file.h +7 -0
index 935cf42..210e9c3 100644
@@ -85,6 +85,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
                                int *bytes_sent);
 
 /*
+ * Return a QEMUFile for comms in the opposite direction
+ */
+typedef QEMUFile *(QEMURetPathFunc)(void *opaque);
+
+/*
  * Stop any read or write (depending on flags) on the underlying
  * transport on the QEMUFile.
  * Existing blocking reads/writes must be woken
@@ -102,6 +107,7 @@ typedef struct QEMUFileOps {
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
+    QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
 } QEMUFileOps;
 
@@ -188,6 +194,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);
 void qemu_file_set_error(QEMUFile *f, int ret);
 void qemu_file_shutdown(QEMUFile *f);
+QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
qemu-file.c +63 -10

100 lines not on this page · raw message

Signature · 1 line
1.9.3
#24 lore raw

Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.

I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#25 lore raw

Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets

Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

Quoted 7 lines

This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.

I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.

Can you give an example of a transport where it would be a problem, so I can look at how that works?

It's a little tricky since, on the destination, at the time we create the connection we don't know that we're going to need the return path.

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#26 lore raw

Re: [Qemu-devel] [PATCH v4 10/47] Return path: Open a return path on QEMUFile for sockets

Quoted 13 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.

Wire it up for 'socket' QEMUFile's using a dup'd fd.

This doesn't seem like the right abstraction to me. In particular I can't really see how you'd implement this for anything other than socket.

Quoted 6 lines
Quoted 3 lines

I'd suggest instead creating new "open" helper functions (within the QEMUFile code) that open both a forward and return path simultaneously.

Can you give an example of a transport where it would be a problem, so I can look at how that works?

pipe

socket routed through some external transport / encoding layer that's one way only.

Quoted 2 lines

It's a little tricky since, on the destination, at the time we create the connection we don't know that we're going to need the return path.

Creating it and not using it shouldn't be a problem though, should it? As long as you just fall back to precopy rather than failing the migration if you're unable to open it.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#27 lore raw

PATCH v4 11/47 Return path: socket_writev_buffer: Block even on non-blocking fd's

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

---
 qemu-file.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
qemu-file.c +35 -4
index 7393415..57eabd8 100644
@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
     }
-    return len;
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
Signature · 1 line
1.9.3
#28 lore raw

Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 34 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

---
 qemu-file.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
qemu-file.c +11 -3
index 7393415..57eabd8 100644
@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {

qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109

Quoted 10 lines
qemu-file.c +10 -0
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.

Is that safe? This gives the caller no means to detect a partially completed send.

Quoted 19 lines
qemu-file.c +14 -1
+                 */
+                return -err;
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
     }
-    return len;
+
+    return offset;
 }
 
 static int socket_get_fd(void *opaque)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#29 lore raw

Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 40 lines · * David Gibson ([email protected]) wrote:
Quoted 39 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

---
 qemu-file.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
qemu-file.c +16 -3
index 7393415..57eabd8 100644
@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
Quoted 8 lines

qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109

Quoted 5 lines
qemu-file.c +5 -0
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.

Is that safe? This gives the caller no means to detect a partially completed send.

Well I'm returning the -err, so the caller knows something has gone wrong - it just doesn't know whether it managed to send some part of the data before the failure.

Dave

Quoted 20 lines
Quoted 20 lines
qemu-file.c +14 -1
+                 */
+                return -err;
 
+            }
+
+            /* Emulate blocking */
+            GPollFD pfd;
+
+            pfd.fd = s->fd;
+            pfd.events = G_IO_OUT | G_IO_ERR;
+            pfd.revents = 0;
+            g_poll(&pfd, 1 /* 1 fd */, -1 /* no timeout */);
+        }
     }
-    return len;
+
+    return offset;
 }
 
Quoted 7 lines
Quoted 1 line

static int socket_get_fd(void *opaque)

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#30 lore raw

Re: [Qemu-devel] [PATCH v4 11/47] Return path: socket_writev_buffer: Block even on non-blocking fd's

Quoted 50 lines · Dr. David Alan Gilbert
Quoted 40 lines · * David Gibson ([email protected]) wrote:
Quoted 39 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block.

---
 qemu-file.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
qemu-file.c +16 -3
index 7393415..57eabd8 100644
@@ -85,12 +85,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     QEMUFileSocket *s = opaque;
     ssize_t len;
     ssize_t size = iov_size(iov, iovcnt);
+    ssize_t offset = 0;
+    int     err;
 
-    len = iov_send(s->fd, iov, iovcnt, 0, size);
-    if (len < size) {
-        len = -socket_error();
+    while (size > 0) {
+        len = iov_send(s->fd, iov, iovcnt, offset, size);
+
+        if (len > 0) {
+            size -= len;
+            offset += len;
+        }
+
+        if (size > 0) {
+            err = socket_error();
+
+            if (err != EAGAIN) {
+                error_report("socket_writev_buffer: Got err=%d for (%zd/%zd)",
+                             err, size, len);
Quoted 8 lines

qemu-file.c static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt, · line 109

Quoted 5 lines
qemu-file.c +5 -0
+                /*
+                 * If I've already sent some but only just got the error, I
+                 * could return the amount validly sent so far and wait for the
+                 * next call to report the error, but I'd rather flag the error
+                 * immediately.

Is that safe? This gives the caller no means to detect a partially completed send.

Quoted 3 lines

Well I'm returning the -err, so the caller knows something has gone wrong - it just doesn't know whether it managed to send some part of the data before the failure.

Right. Which seems like it could be pretty important.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#31 lore raw

PATCH v4 12/47 Handle bi-directional communication for fd migration

In reply to Dr. David Alan Gilbert (git)

From: Cristian Klein <[email protected]>

---
 migration-fd.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
migration-fd.c +22 -2
index d2e523a..129da99 100644
@@ -31,13 +31,29 @@
     do { } while (0)
 #endif
 
+static bool fd_is_socket(int fd)
+{
+    struct stat stat;
+    int ret = fstat(fd, &stat);
+    if (ret == -1) {
+        /* When in doubt say no */
+        return false;
+    }
+    return S_ISSOCK(stat.st_mode);
+}
+
 void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp)
 {
     int fd = monitor_get_fd(cur_mon, fdname, errp);
     if (fd == -1) {
         return;
     }
-    s->file = qemu_fdopen(fd, "wb");
+
+    if (fd_is_socket(fd)) {
+        s->file = qemu_fopen_socket(fd, "wb");
+    } else {
+        s->file = qemu_fdopen(fd, "wb");
+    }
 
     migrate_fd_connect(s);
 }
@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)
     DPRINTF("Attempting to start an incoming migration via fd\n");
 
     fd = strtol(infd, NULL, 0);
-    f = qemu_fdopen(fd, "rb");
+    if (fd_is_socket(fd)) {
+        f = qemu_fopen_socket(fd, "rb");
+    } else {
+        f = qemu_fdopen(fd, "rb");
+    }
     if(f == NULL) {
         error_setg_errno(errp, errno, "failed to open the source descriptor");
         return;
Signature · 1 line
1.9.3
#32 lore raw

Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration

Quoted 1 line · Dr. David Alan Gilbert (git)

From: Cristian Klein <[email protected]>

This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.

Quoted 54 lines
---
 migration-fd.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
migration-fd.c +22 -2
index d2e523a..129da99 100644
@@ -31,13 +31,29 @@
     do { } while (0)
 #endif
 
+static bool fd_is_socket(int fd)
+{
+    struct stat stat;
+    int ret = fstat(fd, &stat);
+    if (ret == -1) {
+        /* When in doubt say no */
+        return false;
+    }
+    return S_ISSOCK(stat.st_mode);
+}
+
 void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp)
 {
     int fd = monitor_get_fd(cur_mon, fdname, errp);
     if (fd == -1) {
         return;
     }
-    s->file = qemu_fdopen(fd, "wb");
+
+    if (fd_is_socket(fd)) {
+        s->file = qemu_fopen_socket(fd, "wb");
+    } else {
+        s->file = qemu_fdopen(fd, "wb");
+    }
 
     migrate_fd_connect(s);
 }
@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)
     DPRINTF("Attempting to start an incoming migration via fd\n");
 
     fd = strtol(infd, NULL, 0);
-    f = qemu_fdopen(fd, "rb");
+    if (fd_is_socket(fd)) {
+        f = qemu_fopen_socket(fd, "rb");
+    } else {
+        f = qemu_fdopen(fd, "rb");
+    }
     if(f == NULL) {
         error_setg_errno(errp, errno, "failed to open the source descriptor");
         return;
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#33 lore raw

Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration

Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)

From: Cristian Klein <[email protected]>

This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.

“”"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu’s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. “”"

If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.

Cristian

Quoted 61 lines
Quoted 55 lines

---
migration-fd.c | 24 ++++++++++++++++++++++--

1 file changed, 22 insertions(+), 2 deletions(-)
migration-fd.c +22 -2
index d2e523a..129da99 100644
@@ -31,13 +31,29 @@
    do { } while (0)
 #endif
 
+static bool fd_is_socket(int fd)
+{
+    struct stat stat;
+    int ret = fstat(fd, &stat);
+    if (ret == -1) {
+        /* When in doubt say no */
+        return false;
+    }
+    return S_ISSOCK(stat.st_mode);
+}
+
 void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp)
 {
    int fd = monitor_get_fd(cur_mon, fdname, errp);
    if (fd == -1) {
        return;
    }
-    s->file = qemu_fdopen(fd, "wb");
+
+    if (fd_is_socket(fd)) {
+        s->file = qemu_fopen_socket(fd, "wb");
+    } else {
+        s->file = qemu_fdopen(fd, "wb");
+    }
 
    migrate_fd_connect(s);
 }
@@ -58,7 +74,11 @@ void fd_start_incoming_migration(const char *infd, Error **errp)
    DPRINTF("Attempting to start an incoming migration via fd\n");
 
    fd = strtol(infd, NULL, 0);
-    f = qemu_fdopen(fd, "rb");
+    if (fd_is_socket(fd)) {
+        f = qemu_fopen_socket(fd, "rb");
+    } else {
+        f = qemu_fdopen(fd, "rb");
+    }
    if(f == NULL) {
        error_setg_errno(errp, errno, "failed to open the source descriptor");
        return;
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#34 lore raw

Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration

Quoted 8 lines · Cristian Klein
Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)

From: Cristian Klein <[email protected]>

This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.

Quoted 6 lines

“”"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu’s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. “”"

If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.

That would help, yes.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#35 lore raw

Re: [Qemu-devel] [PATCH v4 12/47] Handle bi-directional communication for fd migration

Quoted 9 lines · * David Gibson ([email protected]) wrote:
Quoted 8 lines · Cristian Klein
Quoted 5 lines · David Gibson
Quoted 1 line · Dr. David Alan Gilbert (git)

From: Cristian Klein <[email protected]>

This patch really, really requires a rationale in the commit message. The reason it's necessary is certainly not obvious.

Quoted 8 lines
Quoted 6 lines

??????"
libvirt prefers opening the TCP connection itself, for two reasons. First, connection failed errors can be detected easier, without having to parse qemu???s error output. Second, libvirt might be asked to secure the transfer by tunnelling the communication through an TLS layer. Therefore, libvirt opens the TCP connection itself and passes an FD to qemu using QMP and a POSIX-specific mechanism. Hence, in order to make the reverse-path work in such cases, qemu needs to distinguish if the transmitted FD is a socket (reverse-path available) or not (reverse-path might not be available) and use the corresponding abstraction. ??????"

If the above message is clarifies the purpose of this commit, feel free to add it in the next version of the patch.

That would help, yes.

I've added that text into the commit message.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#36 lore raw

PATCH v4 13/47 Migration commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

---
 include/migration/migration.h |  1 +
 include/sysemu/sysemu.h       |  9 ++++++++
 savevm.c                      | 48 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+)
include/migration/migration.h +1 -0
index 8a36255..e23947a 100644
@@ -33,6 +33,7 @@
 #define QEMU_VM_SECTION_END          0x03
 #define QEMU_VM_SECTION_FULL         0x04
 #define QEMU_VM_SUBSECTION           0x05
+#define QEMU_VM_COMMAND              0x06
 
 struct MigrationParams {
     bool blk;
include/sysemu/sysemu.h +9 -0
index 6e5953d..eed7e77 100644
@@ -82,6 +82,13 @@ void do_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+/* Subcommands for QEMU_VM_COMMAND */
+enum qemu_vm_cmd {
+    QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */
+
+    QEMU_VM_CMD_AFTERLASTVALID
+};
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -89,6 +96,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
+                              uint16_t len, uint8_t *data);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +48 -0

66 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v4 13/47
#37 lore raw

Re: [Qemu-devel] [PATCH v4 13/47] Migration commands

Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.

For use in postcopy.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#38 lore raw

PATCH v4 14/47 Return path: Control commands

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add two src->dest commands:
* OPENRP - To request that the destination open the return path
* REQACK - Request an acknowledge from the destination

---
 include/migration/migration.h |  2 ++
 include/sysemu/sysemu.h       |  4 +++
 savevm.c                      | 57 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+)
include/migration/migration.h +2 -0
index e23947a..173775b 100644
@@ -45,6 +45,8 @@ typedef struct MigrationState MigrationState;
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
+
+    QEMUFile *return_path;
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
include/sysemu/sysemu.h +4 -0
index eed7e77..ad96f2a 100644
@@ -85,6 +85,8 @@ void qemu_announce_self(void);
 /* Subcommands for QEMU_VM_COMMAND */
 enum qemu_vm_cmd {
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */
+    QEMU_VM_CMD_OPENRP,        /* Tell the dest to open the Return path */
+    QEMU_VM_CMD_REQACK,        /* Request an ACK on the RP */
 
     QEMU_VM_CMD_AFTERLASTVALID
 };
@@ -98,6 +100,8 @@ void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
+void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
+void qemu_savevm_send_openrp(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +57 -0

83 lines not on this page · raw message

Signature · 1 line
1.9.3
#39 lore raw

Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */

+ QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */

OPEN_RETURN_PATH?

Quoted 1 line

+ QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */

SEND_ACK or ACK_REQUESTED?

Quoted 1 line

QEMU_VM_CMD_AFTERLASTVALID

Pleaseseparatewords. Is this enum actually used at all?

Please avoid the difference between QEMU_VM_CMD and MIG_RPCOMM_.

Perhaps MIG_CMD and MIG_RPCMD_?

Paolo

#40 lore raw

Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands

Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */

+ QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */

OPEN_RETURN_PATH?

Quoted 1 line

+ QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */

SEND_ACK or ACK_REQUESTED?

Quoted 7 lines
Quoted 1 line

QEMU_VM_CMD_AFTERLASTVALID

Pleaseseparatewords. Is this enum actually used at all?

Please avoid the difference between QEMU_VM_CMD and MIG_RPCOMM_.

Perhaps MIG_CMD and MIG_RPCMD_?

Almost, I went with:

    MIG_CMD_INVALID = 0,       /* Must be 0 */
    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
    MIG_CMD_SEND_ACK,          /* Request an ACK on the RP */
    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */

MIG_CMD_POSTCOPY_ADVISE = 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */

MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */

and

    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
    MIG_RP_CMD_ACK,          /* data (seq: be32 ) */
    MIG_RP_CMD_REQ_PAGES,    /* data (start: be64, len: be64) */

the only oddity I get from that is from the 'SEND_ACK' you suggested; since all my functions to send commands are send_ I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#41 lore raw

Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands

Quoted 38 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */

+ QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */

OPEN_RETURN_PATH?

Quoted 1 line

+ QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */

SEND_ACK or ACK_REQUESTED?

Quoted 7 lines
Quoted 1 line

QEMU_VM_CMD_AFTERLASTVALID

Pleaseseparatewords. Is this enum actually used at all?

Please avoid the difference between QEMU_VM_CMD and MIG_RPCOMM_.

Perhaps MIG_CMD and MIG_RPCMD_?

Almost, I went with:

    MIG_CMD_INVALID = 0,       /* Must be 0 */
    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
    MIG_CMD_SEND_ACK,          /* Request an ACK on the RP */
    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */

MIG_CMD_POSTCOPY_ADVISE = 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */

MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */

and

Quoted 8 lines
    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
    MIG_RP_CMD_ACK,          /* data (seq: be32 ) */
    MIG_RP_CMD_REQ_PAGES,    /* data (start: be64, len: be64) */

the only oddity I get from that is from the 'SEND_ACK' you suggested; since all my functions to send commands are send_ I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.

Perhaps ping/pong?

Paolo

#42 lore raw

Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands

Quoted 46 lines · Paolo Bonzini
Quoted 43 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */

+ QEMU_VM_CMD_OPENRP, /* Tell the dest to open the Return path */

OPEN_RETURN_PATH?

Quoted 1 line

+ QEMU_VM_CMD_REQACK, /* Request an ACK on the RP */

SEND_ACK or ACK_REQUESTED?

Quoted 7 lines
Quoted 1 line

QEMU_VM_CMD_AFTERLASTVALID

Pleaseseparatewords. Is this enum actually used at all?

Please avoid the difference between QEMU_VM_CMD and MIG_RPCOMM_.

Perhaps MIG_CMD and MIG_RPCMD_?

Almost, I went with:

    MIG_CMD_INVALID = 0,       /* Must be 0 */
    MIG_CMD_OPEN_RETURN_PATH,  /* Tell the dest to open the Return path */
    MIG_CMD_SEND_ACK,          /* Request an ACK on the RP */
    MIG_CMD_PACKAGED,          /* Send a wrapped stream within this stream */

MIG_CMD_POSTCOPY_ADVISE = 20, /* Prior to any page transfers, just
warn we might want to do PC */
MIG_CMD_POSTCOPY_LISTEN, /* Start listening for incoming
pages as it's running. */
MIG_CMD_POSTCOPY_RUN, /* Start execution */
MIG_CMD_POSTCOPY_END, /* Postcopy is finished. */

MIG_CMD_POSTCOPY_RAM_DISCARD, /* A list of pages to discard that
were previously sent during precopy but are dirty. */

and

    MIG_RP_CMD_INVALID = 0,  /* Must be 0 */
    MIG_RP_CMD_SHUT,         /* sibling will not send any more RP messages */
    MIG_RP_CMD_ACK,          /* data (seq: be32 ) */
    MIG_RP_CMD_REQ_PAGES,    /* data (start: be64, len: be64) */
Quoted 5 lines
Quoted 3 lines

the only oddity I get from that is from the 'SEND_ACK' you suggested; since all my functions to send commands are send_ I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.

Perhaps ping/pong?

I like that idea. Calling it "send_ack" looks like it's just asking for confusing names somewhere.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#43 lore raw

Re: [Qemu-devel] [PATCH v4 14/47] Return path: Control commands

In reply to Paolo Bonzini
Quoted 5 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines

the only oddity I get from that is from the 'SEND_ACK' you suggested; since all my functions to send commands are send_ I currently have
'qemu_savevm_send_send_ack' which while consistent looks a bit odd.

Perhaps ping/pong?

Done (although I'm sure I'll find 'ack' in the comments somewhere).

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#44 lore raw

PATCH v4 15/47 Return path: Send responses from destination to source

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send an 'ack' message
Use it in the CMD_REQACK handler

---
 include/migration/migration.h | 18 ++++++++++++++++++
 migration.c                   | 41 +++++++++++++++++++++++++++++++++++++++++
 savevm.c                      |  2 +-
 3 files changed, 60 insertions(+), 1 deletion(-)
include/migration/migration.h +18 -0
index 173775b..12e640d 100644
@@ -40,6 +40,13 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Commands sent on the return path from destination to source*/
+enum mig_rpcomm_cmd {
+    MIG_RPCOMM_INVALID = 0,  /* Must be 0 */
+    MIG_RPCOMM_SHUT,         /* sibling will not send any more RP messages */
+    MIG_RPCOMM_ACK,          /* data (seq: be32 ) */
+    MIG_RPCOMM_AFTERLASTVALID
+};
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -47,6 +54,7 @@ struct MigrationIncomingState {
     QEMUFile *file;
 
     QEMUFile *return_path;
+    QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
@@ -168,6 +176,16 @@ int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
 
+/* Sending on the return path - generic and then for each message type */
+void migrate_send_rp_message(MigrationIncomingState *mis,
+                             enum mig_rpcomm_cmd cmd,
+                             uint16_t len, uint8_t *data);
+void migrate_send_rp_shut(MigrationIncomingState *mis,
+                          uint32_t value);
+void migrate_send_rp_ack(MigrationIncomingState *mis,
+                         uint32_t value);
+
+
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags);
migration.c +41 -0

53 lines not on this page · raw message

savevm.c +1 -1
index 793384a..8eebbfd 100644
@@ -969,7 +969,7 @@ static int loadvm_process_command(QEMUFile *f)
                          tmp32);
             return -1;
         }
-        /* migrate_send_rp_ack(mis, tmp32); TODO: gets added later */
+        migrate_send_rp_ack(mis, tmp32);
         break;
 
     default:
Signature · 1 line
1.9.3
Reviewed-by on v4 15/47
#45 lore raw

Re: [Qemu-devel] [PATCH v4 15/47] Return path: Send responses from destination to source

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send an 'ack' message
Use it in the CMD_REQACK handler

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#46 lore raw

PATCH v4 16/47 Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
 include/migration/migration.h |  10 +++
 migration.c                   | 181 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 190 insertions(+), 1 deletion(-)
include/migration/migration.h +10 -0
index 12e640d..b87c289 100644
@@ -47,6 +47,14 @@ enum mig_rpcomm_cmd {
     MIG_RPCOMM_ACK,          /* data (seq: be32 ) */
     MIG_RPCOMM_AFTERLASTVALID
 };
+
+/* Source side RP state */
+struct MigrationRetPathState {
+    uint32_t      latest_ack;
+    QemuThread    rp_thread;
+    bool          error;
+};
+
 typedef struct MigrationState MigrationState;
 
 /* State for the incoming migration */
@@ -69,9 +77,11 @@ struct MigrationState
     QemuThread thread;
     QEMUBH *cleanup_bh;
     QEMUFile *file;
+    QEMUFile *return_path;
 
     int state;
     MigrationParams params;
+    struct MigrationRetPathState rp_state;
     double mbps;
     int64_t total_time;
     int64_t downtime;
migration.c +180 -1

212 lines not on this page · raw message

Signature · 1 line
1.9.3
#47 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

include/migration/migration.h enum mig_rpcomm_cmd { · line 55

Quoted 5 lines
include/migration/migration.h +5 -0
+/* Source side RP state */
+struct MigrationRetPathState {
+    uint32_t      latest_ack;
+    QemuThread    rp_thread;
+    bool          error;

Should the QemuFile be in here?

include/migration/migration.h enum mig_rpcomm_cmd { · line 57

Quoted 2 lines
include/migration/migration.h +2 -0
+};
+

Also please do not abbrev words, and add a typedef that matches the struct if it is useful. If it is not, just embed the struct without giving the type a name (struct { } rp_state).

migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264

Quoted 12 lines
migration.c +12 -0
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}

Should CANCELLING also be considered active? It is on the source->dest path.

migration.c int64_t migrate_xbzrle_cache_size(void) · line 829

Quoted 13 lines
migration.c +12 -0
+static void await_outgoing_return_path_close(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->return_path) {
+        qemu_file_shutdown(ms->return_path);
+    }

As mentioned early, I think it's simpler to let these function handle themselves the case where there is no return path, and call them unconditionally.

Paolo

#48 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 8 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

include/migration/migration.h enum mig_rpcomm_cmd { · line 55

Quoted 5 lines
include/migration/migration.h +5 -0
+/* Source side RP state */
+struct MigrationRetPathState {
+    uint32_t      latest_ack;
+    QemuThread    rp_thread;
+    bool          error;

Should the QemuFile be in here?

Yes, done.

Quoted 6 lines

include/migration/migration.h enum mig_rpcomm_cmd { · line 57

Quoted 2 lines
include/migration/migration.h +2 -0
+};
+

Also please do not abbrev words, and add a typedef that matches the struct if it is useful. If it is not, just embed the struct without giving the type a name (struct { } rp_state).

Done.

Quoted 16 lines

migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264

Quoted 13 lines
migration.c +12 -0
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}

Should CANCELLING also be considered active? It is on the source->dest path.

Hmm, possibly - but my intention here was just to round up all of the places that already checked for ACTIVE+SETUP so that I could add POSTCOPY_ACTIVE; only one of those places also checked for CANCELLING, so I left it out.

Quoted 8 lines
Quoted 8 lines
migration.c +8 -0
+static void await_outgoing_return_path_close(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
Quoted 8 lines

migration.c int64_t migrate_xbzrle_cache_size(void) · line 829

Quoted 4 lines
migration.c +4 -0
+     */
+    if (qemu_file_get_error(ms->file) && ms->return_path) {
+        qemu_file_shutdown(ms->return_path);
+    }

As mentioned early, I think it's simpler to let these function handle themselves the case where there is no return path, and call them unconditionally.

I still need to think about that.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#49 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 15 lines · Dr. David Alan Gilbert
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
migration.c +7 -0
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

migration.c MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) · line 264

Quoted 5 lines
migration.c +5 -0
+    default:
+        return false;
+
+    }
+}

Should CANCELLING also be considered active? It is on the source->dest path.

Quoted 3 lines

Hmm, possibly - but my intention here was just to round up all of the places that already checked for ACTIVE+SETUP so that I could add POSTCOPY_ACTIVE; only one of those places also checked for CANCELLING, so I left it out.

Ok, I would need to check the callers... There may be bugs waiting to be fixed. :) For now I guess it's ok as is.

Thanks for answering my comments!

Paolo

#50 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)
Quoted 189 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

---
  include/migration/migration.h |  10 +++
  migration.c                   | 181 +++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 190 insertions(+), 1 deletion(-)
include/migration/migration.h +10 -0
index 12e640d..b87c289 100644
@@ -47,6 +47,14 @@ enum mig_rpcomm_cmd {
      MIG_RPCOMM_ACK,          /* data (seq: be32 ) */
      MIG_RPCOMM_AFTERLASTVALID
  };
+
+/* Source side RP state */
+struct MigrationRetPathState {
+    uint32_t      latest_ack;
+    QemuThread    rp_thread;
+    bool          error;
+};
+
  typedef struct MigrationState MigrationState;
 
  /* State for the incoming migration */
@@ -69,9 +77,11 @@ struct MigrationState
      QemuThread thread;
      QEMUBH *cleanup_bh;
      QEMUFile *file;
+    QEMUFile *return_path;
 
      int state;
      MigrationParams params;
+    struct MigrationRetPathState rp_state;
      double mbps;
      int64_t total_time;
      int64_t downtime;
migration.c +110 -1
index 5ba8f3e..ee6db1d 100644
@@ -246,6 +246,23 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
      return head;
  }
 
+/*
+ * Return true if we're already in the middle of a migration
+ * (i.e. any of the active or setup states)
+ */
+static bool migration_already_active(MigrationState *ms)
+{
+    switch (ms->state) {
+    case MIG_STATE_ACTIVE:
+    case MIG_STATE_SETUP:
+        return true;
+
+    default:
+        return false;
+
+    }
+}
+
  static void get_xbzrle_cache_stats(MigrationInfo *info)
  {
      if (migrate_use_xbzrle()) {
@@ -371,6 +388,21 @@ static void migrate_set_state(MigrationState *s, int old_state, int new_state)
      }
  }
 
+static void migrate_fd_cleanup_src_rp(MigrationState *ms)
+{
+    QEMUFile *rp = ms->return_path;
+
+    /*
+     * When stuff goes wrong (e.g. failing destination) on the rp, it can get
+     * cleaned up from a few threads; make sure not to do it twice in parallel
+     */
+    rp = atomic_cmpxchg(&ms->return_path, rp, NULL);
+    if (rp) {
+        DPRINTF("cleaning up return path\n");
+        qemu_fclose(rp);
+    }
+}
+
  static void migrate_fd_cleanup(void *opaque)
  {
      MigrationState *s = opaque;
@@ -378,6 +410,8 @@ static void migrate_fd_cleanup(void *opaque)
      qemu_bh_delete(s->cleanup_bh);
      s->cleanup_bh = NULL;
 
+    migrate_fd_cleanup_src_rp(s);
+
      if (s->file) {
          trace_migrate_fd_cleanup();
          qemu_mutex_unlock_iothread();
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
      int old_state ;
      trace_migrate_fd_cancel();
 
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);
+    }
+
      do {
          old_state = s->state;
          if (old_state != MIG_STATE_SETUP && old_state != MIG_STATE_ACTIVE) {
@@ -655,8 +694,148 @@ int64_t migrate_xbzrle_cache_size(void)
      return s->xbzrle_cache_size;
  }
 
-/* migration thread support */
+/*
+ * Something bad happened to the RP stream, mark an error
+ * The caller shall print something to indicate why
+ */
+static void source_return_path_bad(MigrationState *s)
+{
+    s->rp_state.error = true;
+    migrate_fd_cleanup_src_rp(s);
+}
 
+/*
+ * Handles messages sent on the return path towards the source VM
+ *
+ */
+static void *source_return_path_thread(void *opaque)
+{
+    MigrationState *ms = opaque;
+    QEMUFile *rp = ms->return_path;
+    uint16_t expected_len, header_len, header_com;
+    const int max_len = 512;
+    uint8_t buf[max_len];
+    uint32_t tmp32;
+    int res;
+
+    DPRINTF("RP: %s entry", __func__);
+    while (rp && !qemu_file_get_error(rp) &&
+        migration_already_active(ms)) {
+        DPRINTF("RP: %s top of loop", __func__);
+        header_com = qemu_get_be16(rp);
+        header_len = qemu_get_be16(rp);
+
+        switch (header_com) {
+        case MIG_RPCOMM_SHUT:
+        case MIG_RPCOMM_ACK:
+            expected_len = 4;
+            break;
+
+        default:
+            error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
+                    header_com, header_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        if (header_len > expected_len) {
+            error_report("RP: Received command 0x%04x with"
+                    "incorrect length %d expecting %d",
+                    header_com, header_len,
+                    expected_len);
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* We know we've got a valid header by this point */
+        res = qemu_get_buffer(rp, buf, header_len);
+        if (res != header_len) {
+            DPRINTF("RP: Failed to read command data");
+            source_return_path_bad(ms);
+            goto out;
+        }
+
+        /* OK, we have the command and the data */
+        switch (header_com) {
+        case MIG_RPCOMM_SHUT:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            if (tmp32) {
+                error_report("RP: Sibling indicated error %d", tmp32);
+                source_return_path_bad(ms);
+            } else {
+                DPRINTF("RP: SHUT received");
+            }
+            /*

migration.c int64_t migrate_xbzrle_cache_size(void) · line 778

Quoted 10 lines
migration.c +10 -0
+             * We'll let the main thread deal with closing the RP
+             * we could do a shutdown(2) on it, but we're the only user
+             * anyway, so there's nothing gained.
+             */
+            goto out;
+
+        case MIG_RPCOMM_ACK:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            DPRINTF("RP: Received ACK 0x%x", tmp32);
+            atomic_xchg(&ms->rp_state.latest_ack, tmp32);

I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)

Quoted 64 lines
migration.c +60 -0
+            break;
+
+        default:
+            /* This shouldn't happen because we should catch this above */
+            DPRINTF("RP: Bad header_com in dispatch");
+        }
+        /* Latest command processed, now leave a gap for the next one */
+        header_com = MIG_RPCOMM_INVALID;
+    }
+    if (rp && qemu_file_get_error(rp)) {
+        DPRINTF("%s: rp bad at end", __func__);
+        source_return_path_bad(ms);
+    }
+
+    DPRINTF("%s: Bottom exit", __func__);
+
+out:
+    return NULL;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);
+    if (!ms->return_path) {
+        return -1;
+    }
+
+    DPRINTF("%s: starting thread", __func__);
+    qemu_thread_create(&ms->rp_state.rp_thread, "return path",
+                       source_return_path_thread, ms, QEMU_THREAD_JOINABLE);
+
+    DPRINTF("%s: continuing", __func__);
+
+    return 0;
+}
+
+__attribute__ (( unused )) /* Until later in patch series */
+static void await_outgoing_return_path_close(MigrationState *ms)
+{
+    /*
+     * If this is a normal exit then the destination will send a SHUT and the
+     * rp_thread will exit, however if there's an error we need to cause
+     * it to exit, which we can do by a shutdown.
+     * (canceling must also shutdown to stop us getting stuck here if
+     * the destination died at just the wrong place)
+     */
+    if (qemu_file_get_error(ms->file) && ms->return_path) {
+        qemu_file_shutdown(ms->return_path);
+    }
+    DPRINTF("%s: Joining", __func__);
+    qemu_thread_join(&ms->rp_state.rp_thread);
+    DPRINTF("%s: Exit", __func__);
+}
+
+/*
+ * Master migration thread on the source VM.
+ * It drives the migration and pumps the data down the outgoing channel.
+ */
  static void *migration_thread(void *opaque)
  {
      MigrationState *s = opaque;
#51 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 6 lines · * zhanghailiang ([email protected]) wrote:

migration.c int64_t migrate_xbzrle_cache_size(void) · line 778

Quoted 4 lines
migration.c +4 -0
+        case MIG_RPCOMM_ACK:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            DPRINTF("RP: Received ACK 0x%x", tmp32);
+            atomic_xchg(&ms->rp_state.latest_ack, tmp32);

I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)

Nothing currently; I've used the REQ/ACK as debug at the moment; I was thinking
that someone might want to wait on an ack being received before carrying on; but hadn't actually needed it in postcopy.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#52 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 7 lines · Dr. David Alan Gilbert
Quoted 5 lines · * zhanghailiang ([email protected]) wrote:
Quoted 4 lines
migration.c +4 -0
+        case MIG_RPCOMM_ACK:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            DPRINTF("RP: Received ACK 0x%x", tmp32);
+            atomic_xchg(&ms->rp_state.latest_ack, tmp32);
Quoted 6 lines
Quoted 1 line

I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)

Nothing currently; I've used the REQ/ACK as debug at the moment; I was thinking
that someone might want to wait on an ack being received before carrying on; but hadn't actually needed it in postcopy.

OK, i see, Thanks.

Quoted 7 lines

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

.

#53 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

In reply to zhanghailiang
Quoted 2 lines · zhanghailiang
Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[snip]

Quoted 6 lines

migration.c int64_t migrate_xbzrle_cache_size(void) · line 778

Quoted 4 lines
migration.c +4 -0
+        case MIG_RPCOMM_ACK:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            DPRINTF("RP: Received ACK 0x%x", tmp32);
+            atomic_xchg(&ms->rp_state.latest_ack, tmp32);

I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)

Also, you don't appear to use tmp32 after that point, so what's the reason for the exchange, rather than just an assignment?

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#54 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 10 lines · * David Gibson ([email protected]) wrote:
Quoted 2 lines · zhanghailiang
Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

[snip]

Quoted 5 lines
Quoted 4 lines
migration.c +4 -0
+        case MIG_RPCOMM_ACK:
+            tmp32 = be32_to_cpup((uint32_t *)buf);
+            DPRINTF("RP: Received ACK 0x%x", tmp32);
+            atomic_xchg(&ms->rp_state.latest_ack, tmp32);
Quoted 4 lines
Quoted 1 line

I didn't see *ms->rp_state.latest_ack* been used elsewhere, what's it used for?;)

Also, you don't appear to use tmp32 after that point, so what's the reason for the exchange, rather than just an assignment?

I've killed the 'latest_ack' off; I've kept the DPRINTF (and might turn it into a trace).

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#55 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

In reply to Dr. David Alan Gilbert (git)
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

[snip]

migration.c static void migrate_fd_cancel(MigrationState *s) · line 453

Quoted 7 lines
migration.c +3 -0
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
     int old_state ;
     trace_migrate_fd_cancel();
 
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);

Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.

[snip]

migration.c int64_t migrate_xbzrle_cache_size(void) · line 803

Quoted 5 lines
migration.c +5 -0
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);

So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#56 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

[snip]

Quoted 4 lines
migration.c +0 -0
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
     int old_state ;
     trace_migrate_fd_cancel();
 
Quoted 6 lines

migration.c static void migrate_fd_cancel(MigrationState *s) · line 453

Quoted 3 lines
migration.c +3 -0
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);

Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.

The rp thread might be in the middle of a blocking read()/recv() so I'm doing a shutdown() to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.

Quoted 11 lines

[snip]

migration.c int64_t migrate_xbzrle_cache_size(void) · line 803

Quoted 5 lines
migration.c +5 -0
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);

So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.

However, since the implementation of the get_return_path is a method on the particular implementation, and it can be different for a qemu_file opened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#57 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 20 lines · Dr. David Alan Gilbert
Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

[snip]

Quoted 4 lines
migration.c +0 -0
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
     int old_state ;
     trace_migrate_fd_cancel();
 
Quoted 6 lines

migration.c static void migrate_fd_cancel(MigrationState *s) · line 453

Quoted 3 lines
migration.c +3 -0
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);

Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.

Quoted 3 lines

The rp thread might be in the middle of a blocking read()/recv() so I'm doing a shutdown() to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.

Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise pthread_cancel() should kill a thread, even if it's blocked at the moment.

Quoted 12 lines
Quoted 11 lines

[snip]

migration.c int64_t migrate_xbzrle_cache_size(void) · line 803

Quoted 5 lines
migration.c +5 -0
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);

So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.

Quoted 4 lines

However, since the implementation of the get_return_path is a method on the particular implementation, and it can be different for a qemu_file opened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).

So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.

The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.

The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.

The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#58 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 18 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

[snip]

migration.c static void migrate_fd_cancel(MigrationState *s) · line 453

Quoted 7 lines
migration.c +3 -0
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
     int old_state ;
     trace_migrate_fd_cancel();
 
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);

Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.

Quoted 7 lines
Quoted 3 lines

The rp thread might be in the middle of a blocking read()/recv() so I'm doing a shutdown() to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.

Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise pthread_cancel() should kill a thread, even if it's blocked at the moment.

It was Paolo's idea to use shutdown() and I agree - it works well; I'd originally thought about using pthread_cancel but it seemed to be generally disliked - you have to be very careful to either know exactly the points at which it might be killed (if you use the deferred version) or be prepared to deal with your thread disappearing at any time and ensure your data structures are always consistent. In addition there was some concern that there was no Windows equivalent to pthread_cancel.

Quoted 17 lines
Quoted 12 lines
Quoted 11 lines

[snip]

migration.c int64_t migrate_xbzrle_cache_size(void) · line 803

Quoted 5 lines
migration.c +5 -0
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);

So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.

Quoted 4 lines

However, since the implementation of the get_return_path is a method on the particular implementation, and it can be different for a qemu_file opened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).

Quoted 4 lines

So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.

Yes, but hey we've got FILE* in C anyway, so it might be bad, but it's not inconsitent.

Quoted 2 lines

The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.

I think the bit that's specific, is as you say that I don't know whether I need it until later.

Quoted 5 lines

The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.

I'm OK with some transports not supporting this; I check for it and error out. At a higher level I do send an 'open_return_path' command from src->dest early on to say I'm going to want a return path, I guess a pipe might be able to open that fd then and pass it back over the original fd? But that might be hairy.

Quoted 5 lines

The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.

I'd be OK with setting some flag or similar at the same time if that would help; but I don't think there's a safe posix'y way of killing a thread that might be stuck in a recv()/read() other than shutdown().

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#59 lore raw

Re: [Qemu-devel] [PATCH v4 16/47] Return path: Source handling of return path

Quoted 30 lines · Dr. David Alan Gilbert
Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 18 lines · * David Gibson ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Open a return path, and handle messages that are received upon it.

[snip]

migration.c static void migrate_fd_cancel(MigrationState *s) · line 453

Quoted 7 lines
migration.c +3 -0
@@ -414,6 +448,11 @@ static void migrate_fd_cancel(MigrationState *s)
     int old_state ;
     trace_migrate_fd_cancel();
 
+    if (s->return_path) {
+        /* shutdown the rp socket, so causing the rp thread to shutdown */
+        qemu_file_shutdown(s->return_path);

Terminating the rp thread via shutting down its file seems roundabout, and kind of dependent on the socket file implementation.

Quoted 7 lines
Quoted 3 lines

The rp thread might be in the middle of a blocking read()/recv() so I'm doing a shutdown() to cause those to exit; once I have to do that anyway it didn't seem necessary to add anything etra.

Hm. I don't recall, does the rp thread need to do some cleanup at this point? Otherwise pthread_cancel() should kill a thread, even if it's blocked at the moment.

Quoted 7 lines

It was Paolo's idea to use shutdown() and I agree - it works well; I'd originally thought about using pthread_cancel but it seemed to be generally disliked - you have to be very careful to either know exactly the points at which it might be killed (if you use the deferred version) or be prepared to deal with your thread disappearing at any time and ensure your data structures are always consistent. In addition there was some concern that there was no Windows equivalent to pthread_cancel.

Hmm, yeah all right.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines

[snip]

migration.c int64_t migrate_xbzrle_cache_size(void) · line 803

Quoted 5 lines
migration.c +5 -0
+__attribute__ (( unused )) /* Until later in patch series */
+static int open_outgoing_return_path(MigrationState *ms)
+{
+
+    ms->return_path = qemu_file_get_return_path(ms->file);

So, another reason this get_return_path abstraction doesn't seem right to me, is that it's not obvious that for non-socket file types, the source and destination side "get return path" operations would necessarily be the same.

However, since the implementation of the get_return_path is a method on the particular implementation, and it can be different for a qemu_file opened for read or write, then that non-socket file type could implement it how it likes including something like shutdown).

Quoted 7 lines
Quoted 4 lines

So, I'm a little less bothered by this since I realised that QemuFile is basically only used for migration streams, not for other file type operations. The fact that that makes QemuFile a really bad name is a different matter.

Yes, but hey we've got FILE* in C anyway, so it might be bad, but it's not inconsitent.

Uh.. not following the analogy here, sorry.

Quoted 12 lines
Quoted 2 lines

The return path operation is quite specific to a migration stream, and doesn't really belong with a "file" abstraction.

I think the bit that's specific, is as you say that I don't know whether I need it until later.

Quoted 5 lines

The case I've been considering where it's not easy to see how to abstract this is that of a pipe - in that case it will be necessary to open a second pipe from destination to source, which probably needs some preliminary work when first opening the connection, and therefore can't easily be encapsulated into a "get return path" callback.

Quoted 5 lines

I'm OK with some transports not supporting this; I check for it and error out. At a higher level I do send an 'open_return_path' command from src->dest early on to say I'm going to want a return path, I guess a pipe might be able to open that fd then and pass it back over the original fd? But that might be hairy.

You can pass fds over Unix sockets, but not over pipes AFAIK.

Quoted 12 lines
Quoted 5 lines

The abstraction of the shutdown is another question again - I can't think of any other file type which has an operation similar in effect to shutdown(), so it seems really socket specific. Which is another reason I'm not convinced telling the rp thread to die via its stream is a good idea.

I'd be OK with setting some flag or similar at the same time if that would help; but I don't think there's a safe posix'y way of killing a thread that might be stuck in a recv()/read() other than shutdown().

Dave

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#60 lore raw

PATCH v4 17/47 qemu_loadvm errors and debug

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Flip many fprintf's to error_report
Add lots of DPRINTF debug in qemu_loadvm*

---
 savevm.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
savevm.c +17 -6
index 8eebbfd..2c0d61a 100644
@@ -719,6 +719,8 @@ int qemu_savevm_state_iterate(QEMUFile *f)
         trace_savevm_section_end(se->idstr, se->section_id);
 
         if (ret < 0) {
+            DPRINTF("%s: setting error state after iterate on id=%d/%s",
+                    __func__, se->section_id, se->idstr);
             qemu_file_set_error(f, ret);
         }
         if (ret <= 0) {
@@ -1019,6 +1021,7 @@ int qemu_loadvm_state(QEMUFile *f)
         SaveStateEntry *se;
         char idstr[256];
 
+        DPRINTF("qemu_loadvm_state loop: section_type=%d", section_type);
         switch (section_type) {
         case QEMU_VM_SECTION_START:
         case QEMU_VM_SECTION_FULL:
@@ -1032,6 +1035,9 @@ int qemu_loadvm_state(QEMUFile *f)
             instance_id = qemu_get_be32(f);
             version_id = qemu_get_be32(f);
 
+            DPRINTF("qemu_loadvm_state loop START/FULL: id=%d(%s)",
+                    section_id, idstr);
+
             /* Find savevm section */
             se = find_se(idstr, instance_id);
             if (se == NULL) {
@@ -1059,8 +1065,9 @@ int qemu_loadvm_state(QEMUFile *f)
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
-                fprintf(stderr, "qemu: warning: error while loading state for instance 0x%x of device '%s'\n",
-                        instance_id, idstr);
+                error_report("qemu: error while loading state for"
+                             "instance 0x%x of device '%s'",
+                             instance_id, idstr);
                 goto out;
             }
             break;
@@ -1068,23 +1075,25 @@ int qemu_loadvm_state(QEMUFile *f)
         case QEMU_VM_SECTION_END:
             section_id = qemu_get_be32(f);
 
+            DPRINTF("QEMU_VM_SECTION_PART/END entry for id=%d", section_id);
             QLIST_FOREACH(le, &loadvm_handlers, entry) {
                 if (le->section_id == section_id) {
                     break;
                 }
             }
             if (le == NULL) {
-                fprintf(stderr, "Unknown savevm section %d\n", section_id);
+                error_report("Unknown savevm section %d", section_id);
                 ret = -EINVAL;
                 goto out;
             }
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
-                fprintf(stderr, "qemu: warning: error while loading state section id %d\n",
-                        section_id);
+                error_report("qemu: error while loading state section"
+                             " id %d (%s)", section_id, le->se->idstr);
                 goto out;
             }
+            DPRINTF("QEMU_VM_SECTION_PART/END done for id=%d", section_id);
             break;
         case QEMU_VM_COMMAND:
             ret = loadvm_process_command(f);
@@ -1093,11 +1102,12 @@ int qemu_loadvm_state(QEMUFile *f)
             }
             break;
         default:
-            fprintf(stderr, "Unknown savevm section type %d\n", section_type);
+            error_report("Unknown savevm section type %d", section_type);
             ret = -EINVAL;
             goto out;
         }
     }
+    DPRINTF("qemu_loadvm_state loop: exited loop");
 
     cpu_synchronize_all_post_init();
 
@@ -1113,6 +1123,7 @@ out:
         ret = qemu_file_get_error(f);
     }
 
+    DPRINTF("qemu_loadvm_state out: ret=%d", ret);
     return ret;
 }
 
Signature · 1 line
1.9.3
Reviewed-by on v4 17/47
#61 lore raw

Re: [Qemu-devel] [PATCH v4 17/47] qemu_loadvm errors and debug

Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Flip many fprintf's to error_report
Add lots of DPRINTF debug in qemu_loadvm*

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#62 lore raw

PATCH v4 18/47 ram_debug_dump_bitmap: Dump a migration bitmap as text

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +39 -0
index 772de36..6970733 100644
@@ -769,6 +769,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128l;
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found |= (thisbit ^ expected);
+        }
+        if (found) {
+            linebuf[curb] = '\0';
+            fprintf(stderr,  "0x%08" PRIx64 " : %s\n", cur, linebuf);
+        }
+    }
+}
+
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
     RAMBlock *block;
include/migration/migration.h +1 -0
index b87c289..ff47987 100644
@@ -157,6 +157,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);
 double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
Signature · 1 line
1.9.3
#63 lore raw

Re: [Qemu-devel] [PATCH v4 18/47] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 20 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +1 -0
index 772de36..6970733 100644
@@ -769,6 +769,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
+/*

arch_init.c static void reset_ram_globals(void) · line 782

Quoted 10 lines
arch_init.c +10 -0
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128l;

I don't think there's any point to the 'l' there. "long" isn't necessarily correct for an int64_t, and normal type promotion should get this right anyway.

Assuming the user has a >128 character wide terminal seems a little obnoxious, too.

Quoted 10 lines
arch_init.c +10 -0
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*

arch_init.c static void reset_ram_globals(void) · line 802

Quoted 10 lines
arch_init.c +10 -0
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found |= (thisbit ^ expected);

I guess this will have the right result with the obvious encoding of a bool, but I don't think it's conceptually correct. It should be logical, not bitwise operations so:

	found = found || (thisbit != expected);
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#64 lore raw

Re: [Qemu-devel] [PATCH v4 18/47] ram_debug_dump_bitmap: Dump a migration bitmap as text

Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Misses out lines that are all the expected value so the output can be quite compact depending on the circumstance.

---
 arch_init.c                   | 39 +++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h |  1 +
 2 files changed, 40 insertions(+)
arch_init.c +0 -0
index 772de36..6970733 100644
@@ -769,6 +769,45 @@ static void reset_ram_globals(void)
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
 
Quoted 15 lines

arch_init.c static void reset_ram_globals(void) · line 782

Quoted 11 lines
arch_init.c +11 -0
+/*
+ * 'expected' is the value you expect the bitmap mostly to be full
+ * of and it won't bother printing lines that are all this value
+ * if 'todump' is null the migration bitmap is dumped.
+ */
+void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
+{
+    int64_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    int64_t cur;
+    int64_t linelen = 128l;

I don't think there's any point to the 'l' there. "long" isn't necessarily correct for an int64_t, and normal type promotion should get this right anyway.

Fixed.

Quoted 2 lines

Assuming the user has a >128 character wide terminal seems a little obnoxious, too.

This is a debug routine primarily to help me, but it seems right to leave it in for the next person who has to debug it; it's easy enough for them to tweak it to whatever their preference is.

Quoted 24 lines
Quoted 10 lines
arch_init.c +10 -0
+    char linebuf[129];
+
+    if (!todump) {
+        todump = migration_bitmap;
+    }
+
+    for (cur = 0; cur < ram_pages; cur += linelen) {
+        int64_t curb;
+        bool found = false;
+        /*

arch_init.c static void reset_ram_globals(void) · line 802

Quoted 10 lines
arch_init.c +10 -0
+         * Last line; catch the case where the line length
+         * is longer than remaining ram
+         */
+        if (cur+linelen > ram_pages) {
+            linelen = ram_pages - cur;
+        }
+        for (curb = 0; curb < linelen; curb++) {
+            bool thisbit = test_bit(cur+curb, todump);
+            linebuf[curb] = thisbit ? '1' : '.';
+            found |= (thisbit ^ expected);

I guess this will have the right result with the obvious encoding of a bool, but I don't think it's conceptually correct. It should be logical, not bitwise operations so:

Quoted 1 line
	found = found || (thisbit != expected);

Fixed.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#65 lore raw

PATCH v4 19/47 Rework loadvm path for subloops

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.

---
 savevm.c | 136 +++++++++++++++++++++++++++++++++++++++------------------------
 1 file changed, 84 insertions(+), 52 deletions(-)
savevm.c +84 -52
index 2c0d61a..7236232 100644
@@ -915,6 +915,26 @@ static SaveStateEntry *find_se(const char *idstr, int instance_id)
     return NULL;
 }
 
+/* These are ORable flags */
+const int LOADVM_EXITCODE_QUITLOOP     =  1;
+const int LOADVM_EXITCODE_QUITPARENT   =  2;
+const int LOADVM_EXITCODE_KEEPHANDLERS =  4;
+
+typedef struct LoadStateEntry {
+    QLIST_ENTRY(LoadStateEntry) entry;
+    SaveStateEntry *se;
+    int section_id;
+    int version_id;
+} LoadStateEntry;
+
+typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
+
+static LoadStateEntry_Head loadvm_handlers =
+ QLIST_HEAD_INITIALIZER(loadvm_handlers);
+
+static int qemu_loadvm_state_main(QEMUFile *f,
+                                  LoadStateEntry_Head *loadvm_handlers);
+
 static int loadvm_process_command_simple_lencheck(const char *name,
                                                   unsigned int actual,
                                                   unsigned int expected)
@@ -931,8 +951,11 @@ static int loadvm_process_command_simple_lencheck(const char *name,
 /*
  * Process an incoming 'QEMU_VM_COMMAND'
  * negative return on error (will issue error message)
+ * 0   just a normal return
+ * 1   All good, but exit the loop
  */
-static int loadvm_process_command(QEMUFile *f)
+static int loadvm_process_command(QEMUFile *f,
+                                  LoadStateEntry_Head *loadvm_handlers)
 {
     MigrationIncomingState *mis = migration_incoming_get_current();
     uint16_t com;
@@ -982,39 +1005,13 @@ static int loadvm_process_command(QEMUFile *f)
     return 0;
 }
 
-typedef struct LoadStateEntry {
-    QLIST_ENTRY(LoadStateEntry) entry;
-    SaveStateEntry *se;
-    int section_id;
-    int version_id;
-} LoadStateEntry;
-
-int qemu_loadvm_state(QEMUFile *f)
+static int qemu_loadvm_state_main(QEMUFile *f,
+                                  LoadStateEntry_Head *loadvm_handlers)
 {
-    QLIST_HEAD(, LoadStateEntry) loadvm_handlers =
-        QLIST_HEAD_INITIALIZER(loadvm_handlers);
-    LoadStateEntry *le, *new_le;
+    LoadStateEntry *le;
     uint8_t section_type;
-    unsigned int v;
     int ret;
-
-    if (qemu_savevm_state_blocked(NULL)) {
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v != QEMU_VM_FILE_MAGIC) {
-        return -EINVAL;
-    }
-
-    v = qemu_get_be32(f);
-    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
-        error_report("SaveVM v2 format is obsolete and don't work anymore");
-        return -ENOTSUP;
-    }
-    if (v != QEMU_VM_FILE_VERSION) {
-        return -ENOTSUP;
-    }
+    int exitcode = 0;
 
     while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
         uint32_t instance_id, version_id, section_id;
@@ -1043,16 +1040,14 @@ int qemu_loadvm_state(QEMUFile *f)
             if (se == NULL) {
                 error_report("Unknown savevm section or instance '%s' %d",
                              idstr, instance_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Validate version */
             if (version_id > se->version_id) {
                 error_report("savevm: unsupported version %d for '%s' v%d",
                         version_id, idstr, se->version_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             /* Add entry */
@@ -1061,14 +1056,14 @@ int qemu_loadvm_state(QEMUFile *f)
             le->se = se;
             le->section_id = section_id;
             le->version_id = version_id;
-            QLIST_INSERT_HEAD(&loadvm_handlers, le, entry);
+            QLIST_INSERT_HEAD(loadvm_handlers, le, entry);
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
                 error_report("qemu: error while loading state for"
                              "instance 0x%x of device '%s'",
                              instance_id, idstr);
-                goto out;
+                return ret;
             }
             break;
         case QEMU_VM_SECTION_PART:
@@ -1076,47 +1071,84 @@ int qemu_loadvm_state(QEMUFile *f)
             section_id = qemu_get_be32(f);
 
             DPRINTF("QEMU_VM_SECTION_PART/END entry for id=%d", section_id);
-            QLIST_FOREACH(le, &loadvm_handlers, entry) {
+            QLIST_FOREACH(le, loadvm_handlers, entry) {
                 if (le->section_id == section_id) {
                     break;
                 }
             }
             if (le == NULL) {
                 error_report("Unknown savevm section %d", section_id);
-                ret = -EINVAL;
-                goto out;
+                return -EINVAL;
             }
 
             ret = vmstate_load(f, le->se, le->version_id);
             if (ret < 0) {
                 error_report("qemu: error while loading state section"
                              " id %d (%s)", section_id, le->se->idstr);
-                goto out;
+                return ret;
             }
             DPRINTF("QEMU_VM_SECTION_PART/END done for id=%d", section_id);
             break;
         case QEMU_VM_COMMAND:
-            ret = loadvm_process_command(f);
-            if (ret < 0) {
-                goto out;
+            ret = loadvm_process_command(f, loadvm_handlers);
+            DPRINTF("%s QEMU_VM_COMMAND ret: %d", __func__, ret);
+            if ((ret < 0) || (ret & LOADVM_EXITCODE_QUITLOOP)) {
+                return ret;
             }
+            exitcode |= ret; /* Lets us pass flags up to the parent */
             break;
         default:
             error_report("Unknown savevm section type %d", section_type);
-            ret = -EINVAL;
-            goto out;
+            return -EINVAL;
         }
     }
     DPRINTF("qemu_loadvm_state loop: exited loop");
 
-    cpu_synchronize_all_post_init();
+    if (exitcode & LOADVM_EXITCODE_QUITPARENT) {
+        DPRINTF("loadvm_handlers_state_main: End of loop with QUITPARENT");
+        exitcode &= ~LOADVM_EXITCODE_QUITPARENT;
+        exitcode &= LOADVM_EXITCODE_QUITLOOP;
+    }
+
+    return exitcode;
+}
+
+int qemu_loadvm_state(QEMUFile *f)
+{
+    LoadStateEntry *le, *new_le;
+    unsigned int v;
+    int ret;
+
+    if (qemu_savevm_state_blocked(NULL)) {
+        return -EINVAL;
+    }
+
+    v = qemu_get_be32(f);
+    if (v != QEMU_VM_FILE_MAGIC) {
+        return -EINVAL;
+    }
 
-    ret = 0;
+    v = qemu_get_be32(f);
+    if (v == QEMU_VM_FILE_VERSION_COMPAT) {
+        error_report("SaveVM v2 format is obsolete and don't work anymore");
+        return -ENOTSUP;
+    }
+    if (v != QEMU_VM_FILE_VERSION) {
+        return -ENOTSUP;
+    }
+
+    QLIST_INIT(&loadvm_handlers);
+    ret = qemu_loadvm_state_main(f, &loadvm_handlers);
 
-out:
-    QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) {
-        QLIST_REMOVE(le, entry);
-        g_free(le);
+    if (ret == 0) {
+        cpu_synchronize_all_post_init();
+    }
+
+    if ((ret < 0) || !(ret & LOADVM_EXITCODE_KEEPHANDLERS)) {
+        QLIST_FOREACH_SAFE(le, &loadvm_handlers, entry, new_le) {
+            QLIST_REMOVE(le, entry);
+            g_free(le);
+        }
     }
 
     if (ret == 0) {
Signature · 1 line
1.9.3
#66 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines

+/* These are ORable flags */

... make them an "enum".

savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 920

Quoted 2 lines
savevm.c +2 -0
+const int LOADVM_EXITCODE_QUITLOOP     =  1;
+const int LOADVM_EXITCODE_QUITPARENT   =  2;

LOADVM_QUIT_ALL, LOADVM_QUIT respectively?

Review

Quoted 2 lines
(quoted diff) +2 -0
+const int LOADVM_EXITCODE_KEEPHANDLERS =  4;
+

Is it more common to drop or keep handlers?

In either case, please add a comment to the three constants that details how to use them. In particular, please document why you should drop (resp. keep) handlers...

Is it by chance that they are only used in savevm.c? Should they be moved to a header file?

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110

Quoted 5 lines
savevm.c +4 -0
+    if (exitcode & LOADVM_EXITCODE_QUITPARENT) {
+        DPRINTF("loadvm_handlers_state_main: End of loop with QUITPARENT");
+        exitcode &= ~LOADVM_EXITCODE_QUITPARENT;
+        exitcode &= LOADVM_EXITCODE_QUITLOOP;

Either you want |=, or the first &= is useless.

Paolo

#67 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Quoted 5 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 2 lines

+/* These are ORable flags */

... make them an "enum".

OK, will do - I'd generally tended to avoid using enum for things that were ORable where the combinations weren't themselves members of the enum; but I can do that.

Quoted 10 lines

savevm.c static SaveStateEntry *find_se(const char *idstr, int instance_id) · line 920

Quoted 2 lines
savevm.c +2 -0
+const int LOADVM_EXITCODE_QUITLOOP     =  1;
+const int LOADVM_EXITCODE_QUITPARENT   =  2;

LOADVM_QUIT_ALL, LOADVM_QUIT respectively?

Review

Quoted 2 lines
(quoted diff) +2 -0
+const int LOADVM_EXITCODE_KEEPHANDLERS =  4;
+

Is it more common to drop or keep handlers?

I'ts more common to drop them.

Quoted 3 lines

In either case, please add a comment to the three constants that details how to use them. In particular, please document why you should drop (resp. keep) handlers...

Does this make it clearer:

/* ORable flags that control the (potentially nested) loadvm_state loops */ enum LoadVMExitCodes {
/* Quit the loop level that received this command */
LOADVM_QUIT_LOOP = 1,
/* Quit this loop and our parent */
LOADVM_QUIT_PARENT = 2,
/*
* Keep the LoadStateEntry handler list after the loop exits,
* because they're being used in another thread.
*/
LOADVM_KEEP_HANDLERS = 4,
};

Quoted 2 lines

Is it by chance that they are only used in savevm.c? Should they be moved to a header file?

They're local.

Quoted 6 lines

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110

Quoted 4 lines
savevm.c +4 -0
+    if (exitcode & LOADVM_EXITCODE_QUITPARENT) {
+        DPRINTF("loadvm_handlers_state_main: End of loop with QUITPARENT");
+        exitcode &= ~LOADVM_EXITCODE_QUITPARENT;
+        exitcode &= LOADVM_EXITCODE_QUITLOOP;

Either you want |=, or the first &= is useless.

Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.

The idea is we have:
1 outer loadvm_state loop
2 receives packaged command
3 inner_loadvm_state loop
4 receives handle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits

so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#68 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Il 07/10/2014 10:58, Dr. David Alan Gilbert ha scritto:

Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110

Quoted 5 lines
savevm.c +4 -0
+    if (exitcode & LOADVM_EXITCODE_QUITPARENT) {
+        DPRINTF("loadvm_handlers_state_main: End of loop with QUITPARENT");
+        exitcode &= ~LOADVM_EXITCODE_QUITPARENT;
+        exitcode &= LOADVM_EXITCODE_QUITLOOP;

Either you want |=, or the first &= is useless.

Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.

The idea is we have:
1 outer loadvm_state loop
2 receives packaged command
3 inner_loadvm_state loop
4 receives handle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits

Quoted 3 lines

so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.

Yes, that was my understanding as well.

We have only two nested loops, but if we had three, should it be QUIT_PARENT or QUIT_ALL?

Paolo

#69 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Quoted 21 lines · * Paolo Bonzini ([email protected]) wrote:

Il 07/10/2014 10:58, Dr. David Alan Gilbert ha scritto:

Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

savevm.c int qemu_loadvm_state(QEMUFile *f) · line 1110

Quoted 5 lines
savevm.c +4 -0
+    if (exitcode & LOADVM_EXITCODE_QUITPARENT) {
+        DPRINTF("loadvm_handlers_state_main: End of loop with QUITPARENT");
+        exitcode &= ~LOADVM_EXITCODE_QUITPARENT;
+        exitcode &= LOADVM_EXITCODE_QUITLOOP;

Either you want |=, or the first &= is useless.

Ooh nicely spotted; yes that should be |= - now I need to figure out why this didn't break things.

The idea is we have:
1 outer loadvm_state loop
2 receives packaged command
3 inner_loadvm_state loop
4 receives handle_listen
5 < QUITPARENT
6 < QUITLOOP
7 < QUITLOOP
8 exits

Quoted 8 lines
Quoted 3 lines

so QUITPARENT causes it's parent to exit, and to do that the inner loop transforms QUITPARENT into QUITLOOP as it's exit.

Yes, that was my understanding as well.

We have only two nested loops, but if we had three, should it be QUIT_PARENT or QUIT_ALL?

The answer probably depends on why you've got 3 nested loops; either way is a bit of guesswork about what some potential future user wants to do.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#70 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

In reply to Dr. David Alan Gilbert (git)
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 7 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.

Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.

It's also not clear from this patch alone why the lifetime of loadvm_handlers now needs to exceed that of qemu_loadvm_state().

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#71 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Quoted 15 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 7 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.

Quoted 7 lines

Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.

It's also not clear from this patch alone why the lifetime of loadvm_handlers now needs to exceed that of qemu_loadvm_state().

OK, how about if I reworked that last sentence to be:

loadvm_handlers is made module-global to survive beyond the lifetime of the outer qemu_loadvm_state since it may still be in use by a subloop in the postcopy listen thread.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#72 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Quoted 21 lines · Dr. David Alan Gilbert
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 7 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.

Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.

Quoted 8 lines
Quoted 2 lines

It's also not clear from this patch alone why the lifetime of loadvm_handlers now needs to exceed that of qemu_loadvm_state().

OK, how about if I reworked that last sentence to be:

loadvm_handlers is made module-global to survive beyond the lifetime of the outer qemu_loadvm_state since it may still be in use by a subloop in the postcopy listen thread.

Yeah, that's better. A global seems ugly though. Would it be better to dynamically allocate the list head and pass a pointer into the listen thread, or even to pass the list head by value into the listen thread.

The individual list elements need to be cleaned up at some point anyway, so I don't think that introduces any lifetime questions that weren't already there.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#73 lore raw

Re: [Qemu-devel] [PATCH v4 19/47] Rework loadvm path for subloops

Quoted 31 lines · * David Gibson ([email protected]) wrote:
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 6 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.

Quoted 7 lines

Split the core of qemu_loadvm_state out so we can use it for both.

Allow the inner loadvm loop to quit and signal whether the parent should.

loadvm_handlers is made static since it's lifetime is greater than the outer qemu_loadvm_state.

Maybe it's just me, but "made static" to me indicates either a change from fully-global to module-global, or (function) local automatic to local static, not a change from function local-automatic to module-global as here.

Quoted 8 lines
Quoted 2 lines

It's also not clear from this patch alone why the lifetime of loadvm_handlers now needs to exceed that of qemu_loadvm_state().

OK, how about if I reworked that last sentence to be:

loadvm_handlers is made module-global to survive beyond the lifetime of the outer qemu_loadvm_state since it may still be in use by a subloop in the postcopy listen thread.

Quoted 8 lines

Yeah, that's better. A global seems ugly though. Would it be better to dynamically allocate the list head and pass a pointer into the listen thread, or even to pass the list head by value into the listen thread.

The individual list elements need to be cleaned up at some point anyway, so I don't think that introduces any lifetime questions that weren't already there.

I've moved the loadvm_handlers out into the MigrationIncomingState structure, and free them when that is deallocated at the end of migration.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#74 lore raw

PATCH v4 20/47 Add migration-capability boolean for postcopy-ram.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h | 1 +
 migration.c                   | 9 +++++++++
 qapi-schema.json              | 6 +++++-
 3 files changed, 15 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index ff47987..0d9f62d 100644
@@ -173,6 +173,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy_ram(void);
 bool migrate_rdma_pin_all(void);
 bool migrate_zero_blocks(void);
 
migration.c +9 -0
index ee6db1d..527423e 100644
@@ -658,6 +658,15 @@ bool migrate_rdma_pin_all(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
 }
 
+bool migrate_postcopy_ram(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM];
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
qapi-schema.json +5 -1
index 2e9e261..cdf5290 100644
@@ -494,10 +494,14 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has been
+#          migrated, pulling the remaining pages along as needed. NOTE: If the
+#          migration fails during postcopy the VM will fail.  (since 2.2)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] }
 
 ##
 # @MigrationCapabilityStatus
Signature · 1 line
1.9.3
#75 lore raw

Re: [Qemu-devel] [PATCH v4 20/47] Add migration-capability boolean for postcopy-ram.

Quoted 12 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h | 1 +
 migration.c                   | 9 +++++++++
 qapi-schema.json              | 6 +++++-
 3 files changed, 15 insertions(+), 1 deletion(-)

#

qapi-schema.json line 504

Quoted 9 lines
qapi-schema.json +5 -1
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has been
+#          migrated, pulling the remaining pages along as needed. NOTE: If the
+#          migration fails during postcopy the VM will fail.  (since 2.2)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] }

Can we wrap this to keep things in 80 columns? Also, the question was raised on the libvirt list on whether the interface is stable enough to name this 'postcopy-ram' from the get-go (rather than marking the interface experimental), so that libvirt can start using it sooner.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#76 lore raw

Re: [Qemu-devel] [PATCH v4 20/47] Add migration-capability boolean for postcopy-ram.

Quoted 12 lines · * Eric Blake ([email protected]) wrote:
Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h | 1 +
 migration.c                   | 9 +++++++++
 qapi-schema.json              | 6 +++++-
 3 files changed, 15 insertions(+), 1 deletion(-)
Quoted 12 lines

qapi-schema.json line 504

Quoted 10 lines

#

qapi-schema.json +5 -1
+# @x-postcopy-ram: Start executing on the migration target before all of RAM has been
+#          migrated, pulling the remaining pages along as needed. NOTE: If the
+#          migration fails during postcopy the VM will fail.  (since 2.2)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'x-postcopy-ram'] }

Can we wrap this to keep things in 80 columns?

Done.

Quoted 4 lines

Also, the question was
raised on the libvirt list on whether the interface is stable enough to name this 'postcopy-ram' from the get-go (rather than marking the interface experimental), so that libvirt can start using it sooner.

I'm still nervous about that, what I intend to do is add one patch at the end of the series that removes the x- so that can get discussed separately.

While I'm confident that the interface to libvirt is stable, removing the x- declares that the whole thing is stable and I then have to maintain migration compatibility; and it seemed sensible to let people try it for a release; however if libvirt have no way to support QEMUs ability to have experimental features, I guess no one is actually going to try it, which is very disappointing.

Dave

Quoted 4 lines
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#77 lore raw

PATCH v4 21/47 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add state variable showing current incoming postcopy state.

---
 include/migration/migration.h |   8 +
 include/sysemu/sysemu.h       |  20 +++
 savevm.c                      | 335 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 363 insertions(+)
include/migration/migration.h +8 -0
index 0d9f62d..2c078c4 100644
@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    volatile enum {
+        POSTCOPY_RAM_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+        POSTCOPY_RAM_INCOMING_ADVISE,
+        POSTCOPY_RAM_INCOMING_LISTENING,
+        POSTCOPY_RAM_INCOMING_RUNNING,
+        POSTCOPY_RAM_INCOMING_END
+    } postcopy_ram_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/sysemu/sysemu.h +20 -0
index ad96f2a..102dd93 100644
@@ -88,6 +88,16 @@ enum qemu_vm_cmd {
     QEMU_VM_CMD_OPENRP,        /* Tell the dest to open the Return path */
     QEMU_VM_CMD_REQACK,        /* Request an ACK on the RP */
 
+    QEMU_VM_CMD_POSTCOPY_RAM_ADVISE = 20,  /* Prior to any page transfers, just
+                                              warn we might want to do PC */
+    QEMU_VM_CMD_POSTCOPY_RAM_DISCARD,      /* A list of pages to discard that
+                                              were previously sent during
+                                              precopy but are dirty. */
+    QEMU_VM_CMD_POSTCOPY_RAM_LISTEN,       /* Start listening for incoming
+                                              pages as it's running. */
+    QEMU_VM_CMD_POSTCOPY_RAM_RUN,          /* Start execution */
+    QEMU_VM_CMD_POSTCOPY_RAM_END,          /* Postcopy is finished. */
+
     QEMU_VM_CMD_AFTERLASTVALID
 };
 
@@ -102,6 +112,16 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_openrp(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+void qemu_savevm_send_postcopy_ram_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_end(QEMUFile *f, uint8_t status);
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +335 -0

365 lines not on this page · raw message

Signature · 1 line
1.9.3
#78 lore raw

Re: [Qemu-devel] [PATCH v4 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 3 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add state variable showing current incoming postcopy state.

This appears to implement a lot more than just adding a state variable...

Quoted 12 lines
---
 include/migration/migration.h |   8 +
 include/sysemu/sysemu.h       |  20 +++
 savevm.c                      | 335 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 363 insertions(+)
include/migration/migration.h +0 -0
index 0d9f62d..2c078c4 100644

include/migration/migration.h typedef struct MigrationState MigrationState; · line 64

Quoted 4 lines
include/migration/migration.h +1 -0
@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;
 struct MigrationIncomingState {
     QEMUFile *file;
 
+    volatile enum {

What's the reason for the volatile? I think that really needs a comment.

Quoted 73 lines
include/migration/migration.h +7 -0
+        POSTCOPY_RAM_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+        POSTCOPY_RAM_INCOMING_ADVISE,
+        POSTCOPY_RAM_INCOMING_LISTENING,
+        POSTCOPY_RAM_INCOMING_RUNNING,
+        POSTCOPY_RAM_INCOMING_END
+    } postcopy_ram_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };
include/sysemu/sysemu.h +20 -0
index ad96f2a..102dd93 100644
@@ -88,6 +88,16 @@ enum qemu_vm_cmd {
     QEMU_VM_CMD_OPENRP,        /* Tell the dest to open the Return path */
     QEMU_VM_CMD_REQACK,        /* Request an ACK on the RP */
 
+    QEMU_VM_CMD_POSTCOPY_RAM_ADVISE = 20,  /* Prior to any page transfers, just
+                                              warn we might want to do PC */
+    QEMU_VM_CMD_POSTCOPY_RAM_DISCARD,      /* A list of pages to discard that
+                                              were previously sent during
+                                              precopy but are dirty. */
+    QEMU_VM_CMD_POSTCOPY_RAM_LISTEN,       /* Start listening for incoming
+                                              pages as it's running. */
+    QEMU_VM_CMD_POSTCOPY_RAM_RUN,          /* Start execution */
+    QEMU_VM_CMD_POSTCOPY_RAM_END,          /* Postcopy is finished. */
+
     QEMU_VM_CMD_AFTERLASTVALID
 };
 
@@ -102,6 +112,16 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_openrp(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_advise(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist);
+
+void qemu_savevm_send_postcopy_ram_listen(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_run(QEMUFile *f);
+void qemu_savevm_send_postcopy_ram_end(QEMUFile *f, uint8_t status);
+
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
savevm.c +10 -0
index 7236232..b942e8c 100644
@@ -39,6 +39,7 @@
 #include "exec/memory.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "qemu/bitops.h"
 #include "qemu/iov.h"
 #include "block/snapshot.h"
 #include "block/qapi.h"
@@ -624,6 +625,92 @@ void qemu_savevm_send_openrp(QEMUFile *f)
 {
     qemu_savevm_command_send(f, QEMU_VM_CMD_OPENRP, 0, NULL);
 }
+
+/* Send prior to any RAM transfer */
+void qemu_savevm_send_postcopy_ram_advise(QEMUFile *f)
+{
+    DPRINTF("send postcopy-ram-advise");
+    uint64_t tmp[2];
+    tmp[0] = cpu_to_be64(sysconf(_SC_PAGESIZE));
+    tmp[1] = cpu_to_be64(1ul << qemu_target_page_bits());
+

savevm.c void qemu_savevm_send_openrp(QEMUFile *f) · line 646

Quoted 10 lines
savevm.c +10 -0
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_ADVISE, 16,
+                             (uint8_t *)tmp);
+}
+
+/* Prior to running, to cause pages that have been dirtied after precopy
+ * started to be discarded on the destination.
+ * CMD_POSTCOPY_RAM_DISCARD consist of:
+ *  3 byte header (filled in by qemu_savevm_send_postcopy_ram_discard)
+ *      byte   version (0)
+ *      byte   offset into the 1st data word containing 1st page of RAMBlock

I'm not able to follow what that description means.

Quoted 110 lines
savevm.c +103 -0
+ *      byte   Length of name field
+ *  n x byte   RAM block name (NOT 0 terminated)
+ *  n x
+ *      be64   Page addresses for start of an invalidation range
+ *      be32   mask of 32 pages, '1' to discard'
+ *
+ *  Hopefully this is pretty sparse so we don't get too many entries,
+ *  and using the mask should deal with most pagesize differences
+ *  just ending up as a single full mask
+ *
+ * The mask is always 32bits irrespective of the long size
+ *
+ *  name:  RAMBlock name that these entries are part of
+ *  len: Number of page entries
+ *  addrlist: 'len' addresses
+ *  masklist: 'len' masks (corresponding to the addresses)
+ */
+void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
+                                           uint16_t len, uint8_t offset,
+                                           uint64_t *addrlist,
+                                           uint32_t *masklist)
+{
+    uint8_t *buf;
+    uint16_t tmplen;
+    uint16_t t;
+
+    DPRINTF("send postcopy-ram-discard");
+    buf = g_malloc0(len*12 + strlen(name) + 3);
+    buf[0] = 0; /* Version */
+    buf[1] = offset;
+    assert(strlen(name) < 256);
+    buf[2] = strlen(name);
+    memcpy(buf+3, name, strlen(name));
+    tmplen = 3+strlen(name);
+
+    for (t = 0; t < len; t++) {
+        cpu_to_be64w((uint64_t *)(buf + tmplen), addrlist[t]);
+        tmplen += 8;
+        cpu_to_be32w((uint32_t *)(buf + tmplen), masklist[t]);
+        tmplen += 4;
+    }
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_DISCARD,
+                             tmplen, buf);
+    g_free(buf);
+}
+
+/* Get the destination into a state where it can receive page data. */
+void qemu_savevm_send_postcopy_ram_listen(QEMUFile *f)
+{
+    DPRINTF("send postcopy-ram-listen");
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_LISTEN, 0, NULL);
+}
+
+/* Kick the destination into running */
+void qemu_savevm_send_postcopy_ram_run(QEMUFile *f)
+{
+    DPRINTF("send postcopy-ram-run");
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_RUN, 0, NULL);
+}
+
+/* End of postcopy - with a status byte; 0 is good, anything else is a fail */
+void qemu_savevm_send_postcopy_ram_end(QEMUFile *f, uint8_t status)
+{
+    DPRINTF("send postcopy-ram-end");
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_END, 1, &status);
+}
+
 bool qemu_savevm_state_blocked(Error **errp)
 {
     SaveStateEntry *se;
@@ -935,6 +1022,220 @@ static LoadStateEntry_Head loadvm_handlers =
 static int qemu_loadvm_state_main(QEMUFile *f,
                                   LoadStateEntry_Head *loadvm_handlers);
 
+/* ------ incoming postcopy-ram messages ------ */
+/* 'advise' arrives before any RAM transfers just to tell us that a postcopy
+ * *might* happen - it might be skipped if precopy transferred everything
+ * quickly.
+ */
+static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,
+                                             uint64_t remote_hps,
+                                             uint64_t remote_tps)
+{
+    DPRINTF("%s", __func__);
+    if (mis->postcopy_ram_state != POSTCOPY_RAM_INCOMING_NONE) {
+        error_report("CMD_POSTCOPY_RAM_ADVISE in wrong postcopy state (%d)",
+                     mis->postcopy_ram_state);
+        return -1;
+    }
+
+    if (remote_hps != sysconf(_SC_PAGESIZE))  {
+        /*
+         * Some combinations of mismatch are probably possible but it gets
+         * a bit more complicated.  In particular we need to place whole
+         * host pages on the dest at once, and we need to ensure that we
+         * handle dirtying to make sure we never end up sending part of
+         * a hostpage on it's own.
+         */
+        error_report("Postcopy needs matching host page sizes (s=%d d=%d)",
+                     (int)remote_hps, (int)sysconf(_SC_PAGESIZE));
+        return -1;
+    }
+
+    if (remote_tps != (1ul << qemu_target_page_bits())) {
+        /*
+         * Again, some differences could be dealt with, but for now keep it
+         * simple.
+         */
+        error_report("Postcopy needs matching target page sizes (s=%d d=%d)",
+                     (int)remote_tps, 1 << qemu_target_page_bits());

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1070

Quoted 10 lines
savevm.c +10 -0
+        return -1;
+    }
+
+    mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_ADVISE;
+
+    /*
+     * Postcopy will be sending lots of small messages along the return path
+     * that it needs quick answers to.
+     */
+    socket_set_nodelay(qemu_get_fd(mis->return_path));

So, here you break the QEMUFile abstraction and assume you have a socket.

Quoted 52 lines
savevm.c +52 -0
+    return 0;
+}
+
+/* After postcopy we will be told to throw some pages away since they're
+ * dirty and will have to be demand fetched.  Must happen before CPU is
+ * started.
+ * There can be 0..many of these messages, each encoding multiple pages.
+ * Bits set in the message represent a page in the source VMs bitmap, but
+ * since the guest/target page sizes can be different on s/d then we have
+ * to convert.
+ */
+static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
+                                              uint16_t len)
+{
+    int tmp;
+    unsigned int first_bit_offset;
+    char ramid[256];
+
+    DPRINTF("%s", __func__);
+
+    if (mis->postcopy_ram_state != POSTCOPY_RAM_INCOMING_ADVISE) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD in wrong postcopy state (%d)",
+                     mis->postcopy_ram_state);
+        return -1;
+    }
+    /* We're expecting a
+     *    3 byte header,
+     *    a RAM ID string
+     *    then at least 1 12 byte chunks
+    */
+    if (len < 16) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }
+
+    tmp = qemu_get_byte(mis->file);
+    if (tmp != 0) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid version (%d)", tmp);
+        return -1;
+    }
+    first_bit_offset = qemu_get_byte(mis->file);
+
+    if (qemu_get_counted_string(mis->file, (uint8_t *)ramid)) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD Failed to read RAMBlock ID");
+        return -1;
+    }
+
+    len -= 3+strlen(ramid);
+    if (len % 12) {
+        error_report("CMD_POSTCOPY_RAM_DISCARD invalid length (%d)", len);
+        return -1;
+    }

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1133

Quoted 10 lines
savevm.c +10 -0
+    while (len) {
+        uint64_t startaddr;
+        uint32_t mask;
+        /*
+         * We now have pairs of address, mask
+         *   The mask is 32 bits of bitmask starting at 'startaddr'-offset
+         *   RAMBlock; e.g. if the RAMBlock started at 8k where TPS=4k
+         *   then first_bit_offset=2 and the 1st 2 bits of the mask
+         *   aren't relevant to this RAMBlock, and bit 2 corresponds
+         *   to the 1st page of this RAMBlock

Um.. yeah.. can't make much snse of this comment either.

Quoted 46 lines
savevm.c +46 -0
+         */
+        startaddr = qemu_get_be64(mis->file);
+        mask = qemu_get_be32(mis->file);
+
+        len -= 12;
+
+        while (mask) {
+            /* mask= .....?10...0 */
+            /*             ^fs    */
+            int firstset = ctz32(mask);
+
+            /* tmp32=.....?11...1 */
+            /*             ^fs    */
+            uint32_t tmp32 = mask | ((((uint32_t)1)<<firstset)-1);
+
+            /* mask= .?01..10...0 */
+            /*         ^fz ^fs    */
+            int firstzero = cto32(tmp32);
+
+            if ((startaddr == 0) && (firstset < first_bit_offset)) {
+                error_report("CMD_POSTCOPY_RAM_DISCARD bad data; bit set"
+                               " prior to block; block=%s offset=%d"
+                               " firstset=%d\n", ramid, first_bit_offset,
+                               firstzero);
+                return -1;
+            }
+
+            /*
+             * we know there must be at least 1 bit set due to the loop entry
+             * If there is no 0 firstzero will be 32
+             */
+            /* TODO - ram_discard_range gets added in a later patch
+            int ret = ram_discard_range(mis, ramid,
+                                startaddr + firstset - first_bit_offset,
+                                startaddr + (firstzero - 1) - first_bit_offset);
+            ret = -1;
+            if (ret) {
+                return ret;
+            }
+            */
+
+            /* mask= .?0000000000 */
+            /*         ^fz ^fs    */
+            if (firstzero != 32) {
+                mask &= (((uint32_t)-1) << firstzero);
+            } else {

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1189

Quoted 10 lines
savevm.c +10 -0
+                mask = 0;
+            }
+        }
+    }
+    DPRINTF("%s finished", __func__);
+
+    return 0;
+}
+
+/* After this message we must be able to immediately receive page data */

The purpose of the listen message from a protocol point of view isn't really clear to me. I understand why the destination needs to set up the postcopy handling before processing the device data, but why does it need an assertion from the source to start this, rather than just an internal detail on the load path.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#79 lore raw

Re: [Qemu-devel] [PATCH v4 21/47] Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

Quoted 6 lines · * David Gibson ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add state variable showing current incoming postcopy state.

This appears to implement a lot more than just adding a state variable...

It's clearer with the title line; I've reworded it to:

Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.

The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy

Quoted 15 lines
Quoted 15 lines
---
 include/migration/migration.h |   8 +
 include/sysemu/sysemu.h       |  20 +++
 savevm.c                      | 335 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 363 insertions(+)
include/migration/migration.h +0 -0
index 0d9f62d..2c078c4 100644
@@ -61,6 +61,14 @@ typedef struct MigrationState MigrationState;
 struct MigrationIncomingState {
     QEMUFile *file;
 
Quoted 3 lines

include/migration/migration.h typedef struct MigrationState MigrationState; · line 64

Quoted 1 line
include/migration/migration.h +1 -0
+    volatile enum {

What's the reason for the volatile? I think that really needs a comment.

I've removed it and replaced it by atomic_ functions to access it; the state is accessed from multiple threads.

Quoted 10 lines
Quoted 10 lines
include/migration/migration.h +7 -0
+        POSTCOPY_RAM_INCOMING_NONE = 0,  /* Initial state - no postcopy */
+        POSTCOPY_RAM_INCOMING_ADVISE,
+        POSTCOPY_RAM_INCOMING_LISTENING,
+        POSTCOPY_RAM_INCOMING_RUNNING,
+        POSTCOPY_RAM_INCOMING_END
+    } postcopy_ram_state;
+
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
 };

<snip>

Quoted 29 lines
Quoted 29 lines
savevm.c +14 -0
index 7236232..b942e8c 100644
@@ -39,6 +39,7 @@
 #include "exec/memory.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "qemu/bitops.h"
 #include "qemu/iov.h"
 #include "block/snapshot.h"
 #include "block/qapi.h"
@@ -624,6 +625,92 @@ void qemu_savevm_send_openrp(QEMUFile *f)
 {
     qemu_savevm_command_send(f, QEMU_VM_CMD_OPENRP, 0, NULL);
 }
+
+/* Send prior to any RAM transfer */
+void qemu_savevm_send_postcopy_ram_advise(QEMUFile *f)
+{
+    DPRINTF("send postcopy-ram-advise");
+    uint64_t tmp[2];
+    tmp[0] = cpu_to_be64(sysconf(_SC_PAGESIZE));
+    tmp[1] = cpu_to_be64(1ul << qemu_target_page_bits());
+
+    qemu_savevm_command_send(f, QEMU_VM_CMD_POSTCOPY_RAM_ADVISE, 16,
+                             (uint8_t *)tmp);
+}
+
Quoted 8 lines

savevm.c void qemu_savevm_send_openrp(QEMUFile *f) · line 646

Quoted 6 lines
savevm.c +6 -0
+/* Prior to running, to cause pages that have been dirtied after precopy
+ * started to be discarded on the destination.
+ * CMD_POSTCOPY_RAM_DISCARD consist of:
+ *  3 byte header (filled in by qemu_savevm_send_postcopy_ram_discard)
+ *      byte   version (0)
+ *      byte   offset into the 1st data word containing 1st page of RAMBlock

I'm not able to follow what that description means.

I've reworded it as:
* byte offset to be subtracted from each page address to deal with
* RAMBlocks that don't start on a mask word boundary.

(I've tried reworking this protocol a few times, the offset still seems to work out easiest, otherwise you end up having to treat the address entries as potentially signed).

<snip>

Quoted 8 lines

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1070

Quoted 5 lines
savevm.c +5 -0
+    /*
+     * Postcopy will be sending lots of small messages along the return path
+     * that it needs quick answers to.
+     */
+    socket_set_nodelay(qemu_get_fd(mis->return_path));

So, here you break the QEMUFile abstraction and assume you have a socket.

Ah yes; I put that in to see if it would help. Hmm; I've taken it out for now, I need to see if there's a sensible way to add the abstration.

Quoted 11 lines

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1133

Quoted 10 lines
savevm.c +10 -0
+    while (len) {
+        uint64_t startaddr;
+        uint32_t mask;
+        /*
+         * We now have pairs of address, mask
+         *   The mask is 32 bits of bitmask starting at 'startaddr'-offset
+         *   RAMBlock; e.g. if the RAMBlock started at 8k where TPS=4k
+         *   then first_bit_offset=2 and the 1st 2 bits of the mask
+         *   aren't relevant to this RAMBlock, and bit 2 corresponds
+         *   to the 1st page of this RAMBlock
Quoted 1 line

Um.. yeah.. can't make much snse of this comment either.

Well, at least it's the one that corresponds to the comment above you couldn't understand either.
I've reworded it to:
* We now have pairs of address, mask
* Each word of mask is 32 bits, where each bit corresponds to one
* target page.
* RAMBlocks don't necessarily start on word boundaries,
* and the offset in the header indicates the offset into the 1st
* mask word that corresponds to the 1st page of the RAMBlock.

<snip>

Quoted 7 lines

savevm.c static LoadStateEntry_Head loadvm_handlers = · line 1189

Quoted 1 line
savevm.c +1 -0
+/* After this message we must be able to immediately receive page data */

The purpose of the listen message from a protocol point of view isn't really clear to me. I understand why the destination needs to set up the postcopy handling before processing the device data, but why does it need an assertion from the source to start this, rather than just an internal detail on the load path.

The device migration format isn't structured well enough to allow the receiver to know the size of the device data without parsing it all. The parsing isn't structured, so there's no way to do that without calling all the device code that may read from guest memory. We solve that by sending all the device data in the CMD_PACKAGED.

While we could assume that a CMD_PACKAGED is always for postcopy data, instead I don't special case the processing at all of the data inside the package, keeping that general. Then the listen message is used inside that package to start the listener thread off and set up the other associated state.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#80 lore raw

PATCH v4 22/47 QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

---
 include/sysemu/sysemu.h |  4 +++
 savevm.c                | 82 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/sysemu/sysemu.h +4 -0
index 102dd93..ef98fa9 100644
@@ -87,6 +87,7 @@ enum qemu_vm_cmd {
     QEMU_VM_CMD_INVALID = 0,   /* Must be 0 */
     QEMU_VM_CMD_OPENRP,        /* Tell the dest to open the Return path */
     QEMU_VM_CMD_REQACK,        /* Request an ACK on the RP */
+    QEMU_VM_CMD_PACKAGED,      /* Send a wrapped stream within this stream */
 
     QEMU_VM_CMD_POSTCOPY_RAM_ADVISE = 20,  /* Prior to any page transfers, just
                                               warn we might want to do PC */
@@ -101,6 +102,8 @@ enum qemu_vm_cmd {
     QEMU_VM_CMD_AFTERLASTVALID
 };
 
+#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24)
+
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
@@ -112,6 +115,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
 void qemu_savevm_send_openrp(QEMUFile *f);
+void qemu_savevm_send_packaged(QEMUFile *f, const QEMUSizedBuffer *qsb);
 void qemu_savevm_send_postcopy_ram_advise(QEMUFile *f);
 void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
                                            uint16_t len, uint8_t offset,
savevm.c +82 -0

100 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v4 22/47
#81 lore raw

Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Though one minor comment:

[snip]

Quoted 4 lines

+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that

I'm not quite understanding what that comment's getting at.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#82 lore raw

Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 17 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Though one minor comment:

Quoted 7 lines

[snip]

Quoted 4 lines

+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that

I'm not quite understanding what that comment's getting at.

We have these VM Commands; and they are a command type, and a length:
CMD_whatever
length: whatever
data for whatever

This comment is describing that, to make things easier for this code it's ended up as:

CMD_PACKAGED
CMD length: 4 <--- i.e. just enough to hold the next 'length' field
package length
---------------
The package

Which is a little different, hence i thought it needed the comment.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#83 lore raw

Re: [Qemu-devel] [PATCH v4 22/47] QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream

Quoted 34 lines · Dr. David Alan Gilbert
Quoted 17 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

QEMU_VM_CMD_PACKAGED is a migration command that allows a chunk of migration stream to be sent in one go, and be received by a separate instance of the loadvm loop while not interacting with the migration stream.

This is used by postcopy to load device state (from the package) while loading memory pages from the main stream.

Though one minor comment:

Quoted 7 lines

[snip]

Quoted 4 lines

+/* We have a buffer of data to send; we don't want that all to be loaded
+ * by the command itself, so the command contains just the length of the
+ * extra buffer that we then send straight after it.
+ * TODO: Must be a better way to organise that

I'm not quite understanding what that comment's getting at.

We have these VM Commands; and they are a command type, and a length:
CMD_whatever
length: whatever
data for whatever

This comment is describing that, to make things easier for this code it's ended up as:

Quoted 7 lines

CMD_PACKAGED
CMD length: 4 <--- i.e. just enough to hold the next 'length' field
package length
---------------
The package

Which is a little different, hence i thought it needed the comment.

Ah.. right. That seems.. gratuitously easy to get wrong further down the track. Why not just use the cmd length as the package length.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#84 lore raw

PATCH v4 23/47 migrate_init: Call from savevm

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

---
 include/migration/migration.h | 1 +
 include/qemu/typedefs.h       | 1 +
 migration.c                   | 2 +-
 savevm.c                      | 2 ++
 4 files changed, 5 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 2c078c4..3aeae47 100644
@@ -140,6 +140,7 @@ int migrate_fd_close(MigrationState *s);
 
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
+MigrationState *migrate_init(const MigrationParams *params);
 bool migration_in_setup(MigrationState *);
 bool migration_has_finished(MigrationState *);
 bool migration_has_failed(MigrationState *);
include/qemu/typedefs.h +1 -0
index 0f79b5c..8539de6 100644
@@ -16,6 +16,7 @@ struct Monitor;
 typedef struct Monitor Monitor;
 typedef struct MigrationIncomingState MigrationIncomingState;
 typedef struct MigrationParams MigrationParams;
+typedef struct MigrationState MigrationState;
 
 typedef struct Property Property;
 typedef struct PropertyInfo PropertyInfo;
migration.c +1 -1
index 527423e..3a45b2a 100644
@@ -488,7 +488,7 @@ bool migration_has_failed(MigrationState *s)
             s->state == MIG_STATE_ERROR);
 }
 
-static MigrationState *migrate_init(const MigrationParams *params)
+MigrationState *migrate_init(const MigrationParams *params)
 {
     MigrationState *s = migrate_get_current();
     int64_t bandwidth_limit = s->bandwidth_limit;
savevm.c +2 -0
index bffe890..a368a25 100644
@@ -949,6 +949,8 @@ static int qemu_savevm_state(QEMUFile *f)
         .blk = 0,
         .shared = 0
     };
+    MigrationState *ms = migrate_init(&params);
+    ms->file = f;
 
     if (qemu_savevm_state_blocked(NULL)) {
         return -EINVAL;
Signature · 1 line
1.9.3
#85 lore raw

Re: [Qemu-devel] [PATCH v4 23/47] migrate_init: Call from savevm

Quoted 31 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

---
  include/migration/migration.h | 1 +
  include/qemu/typedefs.h       | 1 +
  migration.c                   | 2 +-
  savevm.c                      | 2 ++
  4 files changed, 5 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
index 2c078c4..3aeae47 100644
@@ -140,6 +140,7 @@ int migrate_fd_close(MigrationState *s);
 
  void add_migration_state_change_notifier(Notifier *notify);
  void remove_migration_state_change_notifier(Notifier *notify);
+MigrationState *migrate_init(const MigrationParams *params);
  bool migration_in_setup(MigrationState *);
  bool migration_has_finished(MigrationState *);
  bool migration_has_failed(MigrationState *);
include/qemu/typedefs.h +0 -0
index 0f79b5c..8539de6 100644

include/qemu/typedefs.h struct Monitor; · line 20

Quoted 5 lines
include/qemu/typedefs.h +1 -0
@@ -16,6 +16,7 @@ struct Monitor;
  typedef struct Monitor Monitor;
  typedef struct MigrationIncomingState MigrationIncomingState;
  typedef struct MigrationParams MigrationParams;
+typedef struct MigrationState MigrationState;
 

Er, another redefinition, when compile, it complains there is a redefinition of typedef ‘MigrationState’ in 'include/migration/migration.h:59', is this a problem?

Quoted 29 lines
include/qemu/typedefs.h +0 -0
  typedef struct Property Property;
  typedef struct PropertyInfo PropertyInfo;
migration.c +1 -1
index 527423e..3a45b2a 100644
@@ -488,7 +488,7 @@ bool migration_has_failed(MigrationState *s)
              s->state == MIG_STATE_ERROR);
  }
 
-static MigrationState *migrate_init(const MigrationParams *params)
+MigrationState *migrate_init(const MigrationParams *params)
  {
      MigrationState *s = migrate_get_current();
      int64_t bandwidth_limit = s->bandwidth_limit;
savevm.c +2 -0
index bffe890..a368a25 100644
@@ -949,6 +949,8 @@ static int qemu_savevm_state(QEMUFile *f)
          .blk = 0,
          .shared = 0
      };
+    MigrationState *ms = migrate_init(&params);
+    ms->file = f;
 
      if (qemu_savevm_state_blocked(NULL)) {
          return -EINVAL;
Reviewed-by on v4 23/47
#86 lore raw

Re: [Qemu-devel] [PATCH v4 23/47] migrate_init: Call from savevm

In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#87 lore raw

PATCH v4 24/47 Allow savevm handlers to state whether they could go into postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts

---
 arch_init.c                 |  7 +++++++
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  4 +++-
 migration.c                 |  9 ++++++++-
 savevm.c                    | 23 +++++++++++++++++++----
 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +7 -0
index 6970733..44072d8 100644
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,
 };
 
 void ram_mig_init(void)
include/migration/vmstate.h +1 -1
index 9a001bd..4991935 100644
@@ -54,7 +54,7 @@ typedef struct SaveVMHandlers {
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
     uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
-
+    bool (*can_postcopy)(void *opaque);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
include/sysemu/sysemu.h +3 -1
index ef98fa9..e7ff3d0 100644
@@ -110,7 +110,9 @@ void qemu_savevm_state_begin(QEMUFile *f,
 int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
migration.c +8 -1
index 3a45b2a..bca397d 100644
@@ -865,8 +865,15 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->file)) {
-            pending_size = qemu_savevm_state_pending(s->file, max_size);
+            uint64_t pend_post, pend_nonpost;
+            DPRINTF("iterate\n");
+            qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+                                      &pend_post);
+            pending_size = pend_nonpost + pend_post;
             trace_migrate_pending(pending_size, max_size);
+            DPRINTF("pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64
+                    " nonpost=%" PRIu64 ")\n",
+                    pending_size, max_size, pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
savevm.c +19 -4
index a368a25..1642a59 100644
@@ -911,10 +911,18 @@ void qemu_savevm_state_complete(QEMUFile *f)
     qemu_fflush(f);
 }
 
-uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
+/* Give an estimate of the amount left to be transferred,
+ * the result is split into the amount for units that can and
+ * for units that can't do postcopy.
+ */
+void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
+                               uint64_t *res_non_postcopiable,
+                               uint64_t *res_postcopiable)
 {
     SaveStateEntry *se;
-    uint64_t ret = 0;
+    uint64_t res_nonpc = 0;
+    uint64_t res_pc = 0;
+    uint64_t tmp;
 
     QTAILQ_FOREACH(se, &savevm_handlers, entry) {
         if (!se->ops || !se->ops->save_live_pending) {
@@ -925,9 +933,16 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
                 continue;
             }
         }
-        ret += se->ops->save_live_pending(f, se->opaque, max_size);
+        tmp = se->ops->save_live_pending(f, se->opaque, max_size);
+
+        if (se->ops->can_postcopy(se->opaque)) {
+            res_pc += tmp;
+        } else {
+            res_nonpc += tmp;
+        }
     }
-    return ret;
+    *res_non_postcopiable = res_nonpc;
+    *res_postcopiable = res_pc;
 }
 
 void qemu_savevm_state_cancel(void)
Signature · 1 line
1.9.3
#88 lore raw

Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy

Quoted 32 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts

---
 arch_init.c                 |  7 +++++++
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  4 +++-
 migration.c                 |  9 ++++++++-
 savevm.c                    | 23 +++++++++++++++++++----
 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +6 -0
index 6970733..44072d8 100644
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,

arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208

Quoted 4 lines
arch_init.c +1 -0
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,

Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?

On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#89 lore raw

Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy

Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts

---
 arch_init.c                 |  7 +++++++
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  4 +++-
 migration.c                 |  9 ++++++++-
 savevm.c                    | 23 +++++++++++++++++++----
 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +6 -0
index 6970733..44072d8 100644
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,

arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208

Quoted 4 lines
arch_init.c +1 -0
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,

Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?

Quoted 4 lines

On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.

It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.

Dave

Quoted 5 lines
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#90 lore raw

Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy

Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts

---
 arch_init.c                 |  7 +++++++
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  4 +++-
 migration.c                 |  9 ++++++++-
 savevm.c                    | 23 +++++++++++++++++++----
 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +6 -0
index 6970733..44072d8 100644
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,

arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208

Quoted 4 lines
arch_init.c +1 -0
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,

Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?

Quoted 7 lines
Quoted 4 lines

On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.

It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.

Ok. I still wonder if it might be better to instead extend the save_live_pending callback in order to return both non-postcopyable and postcopyable quantites. It allows for the case of a postcopyable device which has some non-postcopyable data - and with any postcopyable device other than RAM, it seems likely that there will need to be some precopied metadata at least. Plus it avoids adding another callback.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#91 lore raw

Re: [Qemu-devel] [PATCH v4 24/47] Allow savevm handlers to state whether they could go into postcopy

Quoted 51 lines · * David Gibson ([email protected]) wrote:
Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * David Gibson ([email protected]) wrote:
Quoted 32 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Use that to split the qemu_savevm_state_pending counts into postcopiable and non-postcopiable amounts

---
 arch_init.c                 |  7 +++++++
 include/migration/vmstate.h |  2 +-
 include/sysemu/sysemu.h     |  4 +++-
 migration.c                 |  9 ++++++++-
 savevm.c                    | 23 +++++++++++++++++++----
 5 files changed, 38 insertions(+), 7 deletions(-)
arch_init.c +6 -0
index 6970733..44072d8 100644
@@ -1192,6 +1192,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+/* RAM's always up for postcopying */
+static bool ram_can_postcopy(void *opaque)
+{
+    return true;
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,

arch_init.c static SaveVMHandlers savevm_ram_handlers = { · line 1208

Quoted 4 lines
arch_init.c +1 -0
@@ -1199,6 +1205,7 @@ static SaveVMHandlers savevm_ram_handlers = {
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
+    .can_postcopy = ram_can_postcopy,

Is there actually any plausible device for which you'd need a callback here, rather than just having a static bool?

Quoted 7 lines
Quoted 4 lines

On the other hand, it does seem kind of plausible that there might be situations in which some data from a device must be pre-copied, but more can be post-copied, which would necessitate extending the per-handler callback to return quantities for both.

It's cheap enough and I couldn't make a strong argument about any possible device, so I just used the function.

Quoted 7 lines

Ok. I still wonder if it might be better to instead extend the save_live_pending callback in order to return both non-postcopyable and postcopyable quantites. It allows for the case of a postcopyable device which has some non-postcopyable data - and with any postcopyable device other than RAM, it seems likely that there will need to be some precopied metadata at least. Plus it avoids adding another callback.

There are two separate suggestions there - which I'll address in opposite order:
1) Extending save_live_pending callback to avoid a new callback can_postcopy is used for a few diferent decisions; not just where to add the pending value to; so I don't think extending s_l_p saves the need for the other callback

2) Allowing a device to do both pre and postcopy
Yeh I can see you could theoretically have a device where that would be useful; but for reasonably small chunks of metadata it already gets the chance to send those during the _begin call. I'll make the change to save_live_pending's parameters to let this work.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#92 lore raw

PATCH v4 25/47 postcopy: OS support test

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 Makefile.objs                    |   2 +-
 include/migration/postcopy-ram.h |  19 +++++
 postcopy-ram.c                   | 160 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   6 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 postcopy-ram.c
Makefile.objs +1 -1
index 97db978..fa0a3a0 100644
@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o
 common-obj-$(CONFIG_RDMA) += migration-rdma.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o
-common-obj-y += page_cache.o xbzrle.o
+common-obj-y += page_cache.o xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
 
include/migration/postcopy-ram.h new file +19 -0
new file mode 100644
index 0000000..dcd1afa
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return 0 if the host supports everything we need to do postcopy-ram */
+int postcopy_ram_hosttest(void);
+
+#endif
postcopy-ram.c new file +160 -0

160 lines not on this page · raw message

savevm.c +6 -0
index 1642a59..a0cb88b 100644
@@ -33,6 +33,7 @@
 #include "qemu/timer.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/sockets.h"
 #include "qemu/queue.h"
 #include "sysemu/cpus.h"
@@ -1087,6 +1088,11 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    /* Check this host can do it  */
+    if (postcopy_ram_hosttest()) {
+        return -1;
+    }
+
     if (remote_hps != sysconf(_SC_PAGESIZE))  {
         /*
          * Some combinations of mismatch are probably possible but it gets
Signature · 1 line
1.9.3
#93 lore raw

Re: [Qemu-devel] [PATCH v4 25/47] postcopy: OS support test

Quoted 43 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 Makefile.objs                    |   2 +-
 include/migration/postcopy-ram.h |  19 +++++
 postcopy-ram.c                   | 160 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   6 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 postcopy-ram.c
Makefile.objs +1 -1
index 97db978..fa0a3a0 100644
@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o
 common-obj-$(CONFIG_RDMA) += migration-rdma.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o
-common-obj-y += page_cache.o xbzrle.o
+common-obj-y += page_cache.o xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
 
include/migration/postcopy-ram.h new file +7 -0
new file mode 100644
index 0000000..dcd1afa
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>

include/migration/postcopy-ram.h line 17

Quoted 10 lines
include/migration/postcopy-ram.h new file +10 -0
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return 0 if the host supports everything we need to do postcopy-ram */
+int postcopy_ram_hosttest(void);

Maybe postcopy_supported_by_host() would be a bit clearer?

[snip]

include/migration/postcopy-ram.h line 19

Quoted 2 lines
include/migration/postcopy-ram.h new file +2 -0
+#ifdef HOST_X86_64
+ /* NOTE: These are Andrea's 3.15.0 world */

I thought the usual approach in qemu was to import the updated headers first in a separate patch, rather than embeddeding new defines.

Quoted 20 lines
include/migration/postcopy-ram.h new file +20 -0
+#ifndef MADV_USERFAULT
+#define MADV_USERFAULT   18
+#define MADV_NOUSERFAULT 19
+#endif
+
+#ifndef __NR_remap_anon_pages
+#define __NR_remap_anon_pages 321
+#endif
+
+#ifndef __NR_userfaultfd
+#define __NR_userfaultfd 322
+#endif
+
+#endif
+
+#ifndef USERFAULTFD_PROTOCOL
+#define USERFAULTFD_PROTOCOL (uint64_t)0xaa
+#endif
+
+#endif
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#94 lore raw

Re: [Qemu-devel] [PATCH v4 25/47] postcopy: OS support test

Quoted 48 lines · * David Gibson ([email protected]) wrote:
Quoted 47 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Provide a check to see if the OS we're running on has all the bits needed for postcopy.

Creates postcopy-ram.c which will get most of the other helpers we need.

---
 Makefile.objs                    |   2 +-
 include/migration/postcopy-ram.h |  19 +++++
 postcopy-ram.c                   | 160 +++++++++++++++++++++++++++++++++++++++
 savevm.c                         |   6 ++
 4 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 postcopy-ram.c
Makefile.objs +1 -1
index 97db978..fa0a3a0 100644
@@ -54,7 +54,7 @@ common-obj-y += qemu-file.o
 common-obj-$(CONFIG_RDMA) += migration-rdma.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o
-common-obj-y += page_cache.o xbzrle.o
+common-obj-y += page_cache.o xbzrle.o postcopy-ram.o
 
 common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
 
include/migration/postcopy-ram.h new file +11 -0
new file mode 100644
index 0000000..dcd1afa
@@ -0,0 +1,19 @@
+/*
+ * Postcopy migration for RAM
+ *
+ * Copyright 2013 Red Hat, Inc. and/or its affiliates
+ *
+ * Authors:
+ *  Dave Gilbert  <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
Quoted 8 lines

include/migration/postcopy-ram.h line 17

Quoted 6 lines
include/migration/postcopy-ram.h new file +6 -0
+ */
+#ifndef QEMU_POSTCOPY_RAM_H
+#define QEMU_POSTCOPY_RAM_H
+
+/* Return 0 if the host supports everything we need to do postcopy-ram */
+int postcopy_ram_hosttest(void);

Maybe postcopy_supported_by_host() would be a bit clearer?

I went with postcopy_ram_supported_by_host
(and flipped the sense so it returns true if it's supported)

Quoted 6 lines

[snip]

Quoted 2 lines

+#ifdef HOST_X86_64
+ /* NOTE: These are Andrea's 3.15.0 world */

I thought the usual approach in qemu was to import the updated headers first in a separate patch, rather than embeddeding new defines.

Yes, those will sort themselves out when the syscalls land in the kernel and then we can import the headers.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#95 lore raw

PATCH v4 26/47 migrate_start_postcopy: Command to trigger transition to postcopy

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

---
 hmp-commands.hx               | 15 +++++++++++++++
 hmp.c                         |  7 +++++++
 hmp.h                         |  1 +
 include/migration/migration.h |  3 +++
 migration.c                   | 22 ++++++++++++++++++++++
 qapi-schema.json              |  8 ++++++++
 qmp-commands.hx               | 19 +++++++++++++++++++
 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
index 0b1a4f7..63cd23a 100644
@@ -985,6 +985,21 @@ Enable/Disable the usage of a capability @var{capability} for migration.
 ETEXI
 
     {
+        .name       = "migrate_start_postcopy",
+        .args_type  = "",
+        .params     = "",
+        .help       = "Switch migration to postcopy mode",
+        .mhandler.cmd = hmp_migrate_start_postcopy,
+    },
+
+STEXI
+@item migrate_start_postcopy
+@findex migrate_start_postcopy
+Switch in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+ETEXI
+
+    {
         .name       = "client_migrate_info",
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
index 63d7686..07acb8d 100644
@@ -1079,6 +1079,13 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
+{
+    Error *err = NULL;
+    qmp_migrate_start_postcopy(&err);
+    hmp_handle_error(mon, &err);
+}
+
 void hmp_set_password(Monitor *mon, const QDict *qdict)
 {
     const char *protocol  = qdict_get_str(qdict, "protocol");
hmp.h +1 -0
index 4bb5dca..da1334f 100644
@@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
include/migration/migration.h +3 -0
index 3aeae47..b74121e 100644
@@ -100,6 +100,9 @@ struct MigrationState
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
+
+    /* Flag set once the migration has been asked to enter postcopy */
+    volatile bool start_postcopy;
 };
 
 void process_incoming_migration(QEMUFile *f);
migration.c +22 -0
index bca397d..7757acc 100644
@@ -379,6 +379,28 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
     }
 }
 
+void qmp_migrate_start_postcopy(Error **errp)
+{
+    MigrationState *s = migrate_get_current();
+
+    if (!migrate_postcopy_ram()) {
+        error_setg(errp, "Enable postcopy with migration_set_capability before"
+                         " the start of migration");
+        return;
+    }
+
+    if (s->state == MIG_STATE_NONE) {
+        error_setg(errp, "Postcopy must be started after migration has been"
+                         " started");
+        return;
+    }
+    /*
+     * we don't error if migration has finished since that would be racy
+     * with issuing this command.
+     */
+    s->start_postcopy = true;
+}
+
 /* shared migration helpers */
 
 static void migrate_set_state(MigrationState *s, int old_state, int new_state)
qapi-schema.json +8 -0
index cdf5290..792dd63 100644
@@ -541,6 +541,14 @@
 { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
 
 ##
+# @migrate-start-postcopy
+#
+# Switch migration to postcopy mode
+#
+# Since: 2.2
+{ 'command': 'migrate-start-postcopy' }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
qmp-commands.hx +19 -0
index 1abd619..b4dd291 100644
@@ -685,6 +685,25 @@ Example:
 
 EQMP
     {
+        .name       = "migrate-start-postcopy",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_migrate_start_postcopy,
+    },
+
+SQMP
+migrate-start-postcopy
+----------------------
+
+Switch an in-progress migration to postcopy mode. Ignored after the end of
+migration (or once already in postcopy).
+
+Example:
+-> { "execute": "migrate-start-postcopy" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
Signature · 1 line
1.9.3
Reviewed-by on v4 26/47
#96 lore raw

Re: [Qemu-devel] [PATCH v4 26/47] migrate_start_postcopy: Command to trigger transition to postcopy

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:

migrate_start_postcopy

Quoted 8 lines

command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).

Issuing the command before migration has started will error, and issuing after it has finished is ignored.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#97 lore raw

PATCH v4 27/47 MIG_STATE_POSTCOPY_ACTIVE: Add new migration state

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIG_STATE_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

---
 include/migration/migration.h |  2 ++
 migration.c                   | 58 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 54 insertions(+), 6 deletions(-)
include/migration/migration.h +2 -0
index b74121e..2ff9d35 100644
@@ -147,6 +147,8 @@ MigrationState *migrate_init(const MigrationParams *params);
 bool migration_in_setup(MigrationState *);
 bool migration_has_finished(MigrationState *);
 bool migration_has_failed(MigrationState *);
+/* True if outgoing migration has entered postcopy phase */
+bool migration_postcopy_phase(MigrationState *);
 MigrationState *migrate_get_current(void);
 
 uint64_t ram_bytes_remaining(void);
migration.c +52 -6

118 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v4 27/47
#98 lore raw

Re: [Qemu-devel] [PATCH v4 27/47] MIG_STATE_POSTCOPY_ACTIVE: Add new migration state

Quoted 8 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

'MIG_STATE_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy

'migration_postcopy_phase' is provided for other sections to know if they're in postcopy.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#99 lore raw

PATCH v4 28/47 qemu_savevm_state_complete: Postcopy changes

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.

We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the _complete call.

---
 include/sysemu/sysemu.h |  1 +
 savevm.c                | 52 ++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)
include/sysemu/sysemu.h +1 -0
index e7ff3d0..46665ce 100644
@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
                                uint64_t *res_non_postcopiable,
                                uint64_t *res_postcopiable);
+void qemu_savevm_state_postcopy_complete(QEMUFile *f);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
savevm.c +51 -1

74 lines not on this page · raw message

Signature · 1 line
1.9.3
#100 lore raw

Re: [Qemu-devel] [PATCH v4 28/47] qemu_savevm_state_complete: Postcopy changes

Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.

We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the _complete call.

So, we should probably rename qemu_savevm_state_complete() to reflect the fact that it's no longer actually a completion, but just the transition from pre-copy to post-copy phases. A good, brief name doesn't immediately occur to me, unfortunately.

Quoted 31 lines
---
 include/sysemu/sysemu.h |  1 +
 savevm.c                | 52 ++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)
include/sysemu/sysemu.h +1 -0
index e7ff3d0..46665ce 100644
@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
                                uint64_t *res_non_postcopiable,
                                uint64_t *res_postcopiable);
+void qemu_savevm_state_postcopy_complete(QEMUFile *f);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
savevm.c +5 -0
index a0cb88b..7c4541d 100644
@@ -854,10 +854,51 @@ int qemu_savevm_state_iterate(QEMUFile *f)
     return ret;
 }
 
+/*
+ * Calls the complete routines just for those devices that are postcopiable;
+ * causing the last few pages to be sent immediately and doing any associated
+ * cleanup.
+ * Note postcopy also calls the plain qemu_savevm_state_complete to complete

savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 871

Quoted 10 lines
savevm.c +10 -0
+ * all the other devices, but that happens at the point we switch to postcopy.
+ */
+void qemu_savevm_state_postcopy_complete(QEMUFile *f)
+{
+    SaveStateEntry *se;
+    int ret;
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        if (!se->ops || !se->ops->save_live_complete ||
+            !se->ops->can_postcopy) {

So, you check for the presence of a can_postcopy callback, but you don't ever actually invoke it.

savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 884

Quoted 13 lines
savevm.c +13 -0
+            continue;
+        }
+        if (se->ops && se->ops->is_active) {
+            if (!se->ops->is_active(se->opaque)) {
+                continue;
+            }
+        }
+        trace_savevm_section_start(se->idstr, se->section_id);
+        /* Section type */
+        qemu_put_byte(f, QEMU_VM_SECTION_END);
+        qemu_put_be32(f, se->section_id);
+
+        ret = se->ops->save_live_complete(f, se->opaque);

I'm wondering if it might be clearer not to overload the save_live_complete hook, but instead allow both "execution transition" (old complete) and "final complete" (postcopy complete) hooks (expecting only one to be non-NULL in most cases).

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#101 lore raw

Re: [Qemu-devel] [PATCH v4 28/47] qemu_savevm_state_complete: Postcopy changes

Quoted 13 lines · * David Gibson ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip:
a) Finishing postcopiable iterative devices - they'll carry on
b) The termination byte on the end of the stream.

We then also add:
qemu_savevm_state_postcopy_complete
which is called at the end of a postcopy migration to call the complete methods on devices skipped in the _complete call.

Quoted 4 lines

So, we should probably rename qemu_savevm_state_complete() to reflect the fact that it's no longer actually a completion, but just the transition from pre-copy to post-copy phases. A good, brief name doesn't immediately occur to me, unfortunately.

Well it's still completion in the non-postcopy case; if you do think of a good obvious name then I'd be happy to change it. (Another way would be to add aparameter to qemu_savevm_state_complete to make it do one or the other, but some of the conditions in it were already a bit hairy).

Quoted 36 lines
Quoted 36 lines
---
 include/sysemu/sysemu.h |  1 +
 savevm.c                | 52 ++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)
include/sysemu/sysemu.h +1 -0
index e7ff3d0..46665ce 100644
@@ -113,6 +113,7 @@ void qemu_savevm_state_cancel(void);
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
                                uint64_t *res_non_postcopiable,
                                uint64_t *res_postcopiable);
+void qemu_savevm_state_postcopy_complete(QEMUFile *f);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_reqack(QEMUFile *f, uint32_t value);
savevm.c +10 -0
index a0cb88b..7c4541d 100644
@@ -854,10 +854,51 @@ int qemu_savevm_state_iterate(QEMUFile *f)
     return ret;
 }
 
+/*
+ * Calls the complete routines just for those devices that are postcopiable;
+ * causing the last few pages to be sent immediately and doing any associated
+ * cleanup.
+ * Note postcopy also calls the plain qemu_savevm_state_complete to complete
+ * all the other devices, but that happens at the point we switch to postcopy.
+ */
+void qemu_savevm_state_postcopy_complete(QEMUFile *f)
+{
+    SaveStateEntry *se;
Quoted 8 lines

savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 871

Quoted 5 lines
savevm.c +5 -0
+    int ret;
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        if (!se->ops || !se->ops->save_live_complete ||
+            !se->ops->can_postcopy) {

So, you check for the presence of a can_postcopy callback, but you don't ever actually invoke it.

Thanks, fixed.

Quoted 14 lines

savevm.c int qemu_savevm_state_iterate(QEMUFile *f) · line 884

Quoted 13 lines
savevm.c +13 -0
+            continue;
+        }
+        if (se->ops && se->ops->is_active) {
+            if (!se->ops->is_active(se->opaque)) {
+                continue;
+            }
+        }
+        trace_savevm_section_start(se->idstr, se->section_id);
+        /* Section type */
+        qemu_put_byte(f, QEMU_VM_SECTION_END);
+        qemu_put_be32(f, se->section_id);
+
+        ret = se->ops->save_live_complete(f, se->opaque);
Quoted 4 lines

I'm wondering if it might be clearer not to overload the save_live_complete hook, but instead allow both "execution transition" (old complete) and "final complete" (postcopy complete) hooks (expecting only one to be non-NULL in most cases).

Note that I only call save_live_complete once for any one device. Non-postcopied devices get done in the original loop, postcopied devices in the new ones, so from the point of view of a device it's still a complete.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#102 lore raw

PATCH v4 29/47 Postcopy page-map-incoming (PMI) structure

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.

---
 include/migration/migration.h    |  19 ++++
 include/migration/postcopy-ram.h |  12 +++
 include/qemu/typedefs.h          |   1 +
 postcopy-ram.c                   | 220 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 252 insertions(+)
include/migration/migration.h +19 -0
index 2ff9d35..1405a15 100644
@@ -57,6 +57,24 @@ struct MigrationRetPathState {
 
 typedef struct MigrationState MigrationState;
 
+/* Postcopy page-map-incoming - data about each page on the inbound side */
+
+typedef enum {
+   POSTCOPY_PMI_MISSING,   /* page hasn't yet been received */
+   POSTCOPY_PMI_REQUESTED, /* Kernel asked for a page, but we've not got it */
+   POSTCOPY_PMI_RECEIVED   /* We've got the page */
+} PostcopyPMIState;
+
+struct PostcopyPMI {
+    QemuMutex      mutex;
+    unsigned long *received_map;  /* Pages that we have received */
+    unsigned long *requested_map; /* Pages that we're sending a request for */
+    unsigned long  host_mask;     /* A mask with enough bits set to cover one
+                                     host page in the PMI */
+    unsigned long  host_bits;     /* The number of bits in the map representing
+                                     one host page */
+};
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -71,6 +89,7 @@ struct MigrationIncomingState {
 
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
+    PostcopyPMI    postcopy_pmi;
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
include/migration/postcopy-ram.h +12 -0
index dcd1afa..addb88a 100644
@@ -13,7 +13,19 @@
 #ifndef QEMU_POSTCOPY_RAM_H
 #define QEMU_POSTCOPY_RAM_H
 
+#include "migration/migration.h"
+
 /* Return 0 if the host supports everything we need to do postcopy-ram */
 int postcopy_ram_hosttest(void);
 
+/*
+ * In 'advise' mode record that a page has been received.
+ */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index);
+
+void postcopy_pmi_destroy(MigrationIncomingState *mis);
+void postcopy_pmi_discard_range(MigrationIncomingState *mis,
+                                size_t start, size_t npages);
+void postcopy_pmi_dump(MigrationIncomingState *mis);
 #endif
include/qemu/typedefs.h +1 -0
index 8539de6..61b330c 100644
@@ -77,6 +77,7 @@ typedef struct QEMUSGList QEMUSGList;
 typedef struct SHPCDevice SHPCDevice;
 typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
+typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
 typedef struct AdapterInfo AdapterInfo;
 
postcopy-ram.c +220 -0

237 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v4 29/47
#103 lore raw

Re: [Qemu-devel] [PATCH v4 29/47] Postcopy page-map-incoming (PMI) structure

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.

Though there are a couple of minor comments below:

Quoted 109 lines
---
 include/migration/migration.h    |  19 ++++
 include/migration/postcopy-ram.h |  12 +++
 include/qemu/typedefs.h          |   1 +
 postcopy-ram.c                   | 220 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 252 insertions(+)
include/migration/migration.h +19 -0
index 2ff9d35..1405a15 100644
@@ -57,6 +57,24 @@ struct MigrationRetPathState {
 
 typedef struct MigrationState MigrationState;
 
+/* Postcopy page-map-incoming - data about each page on the inbound side */
+
+typedef enum {
+   POSTCOPY_PMI_MISSING,   /* page hasn't yet been received */
+   POSTCOPY_PMI_REQUESTED, /* Kernel asked for a page, but we've not got it */
+   POSTCOPY_PMI_RECEIVED   /* We've got the page */
+} PostcopyPMIState;
+
+struct PostcopyPMI {
+    QemuMutex      mutex;
+    unsigned long *received_map;  /* Pages that we have received */
+    unsigned long *requested_map; /* Pages that we're sending a request for */
+    unsigned long  host_mask;     /* A mask with enough bits set to cover one
+                                     host page in the PMI */
+    unsigned long  host_bits;     /* The number of bits in the map representing
+                                     one host page */
+};
+
 /* State for the incoming migration */
 struct MigrationIncomingState {
     QEMUFile *file;
@@ -71,6 +89,7 @@ struct MigrationIncomingState {
 
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
+    PostcopyPMI    postcopy_pmi;
 };
 
 MigrationIncomingState *migration_incoming_get_current(void);
include/migration/postcopy-ram.h +12 -0
index dcd1afa..addb88a 100644
@@ -13,7 +13,19 @@
 #ifndef QEMU_POSTCOPY_RAM_H
 #define QEMU_POSTCOPY_RAM_H
 
+#include "migration/migration.h"
+
 /* Return 0 if the host supports everything we need to do postcopy-ram */
 int postcopy_ram_hosttest(void);
 
+/*
+ * In 'advise' mode record that a page has been received.
+ */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index);
+
+void postcopy_pmi_destroy(MigrationIncomingState *mis);
+void postcopy_pmi_discard_range(MigrationIncomingState *mis,
+                                size_t start, size_t npages);
+void postcopy_pmi_dump(MigrationIncomingState *mis);
 #endif
include/qemu/typedefs.h +1 -0
index 8539de6..61b330c 100644
@@ -77,6 +77,7 @@ typedef struct QEMUSGList QEMUSGList;
 typedef struct SHPCDevice SHPCDevice;
 typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
+typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
 typedef struct AdapterInfo AdapterInfo;
 
postcopy-ram.c +14 -0
index bba5c71..210585c 100644
@@ -23,6 +23,9 @@
 #include "qemu-common.h"
 #include "migration/migration.h"
 #include "migration/postcopy-ram.h"
+#include "sysemu/sysemu.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
 
 //#define DEBUG_POSTCOPY
 
@@ -82,6 +85,216 @@
 #if defined(__linux__) && defined(MADV_USERFAULT) && \
                           defined(__NR_remap_anon_pages)
 
+/* ---------------------------------------------------------------------- */
+/* Postcopy pagemap-inbound (pmi) - data structures that record the       */
+/* state of each page used by the inbound postcopy                        */
+/* It's a pair of bitmaps (of the same structure as the migration bitmaps)*/
+/* holding one bit per target-page, although all operations work on host  */
+/* pages.                                                                 */
+__attribute__ (( unused )) /* Until later in patch series */
+static void postcopy_pmi_init(MigrationIncomingState *mis, size_t ram_pages)
+{
+    unsigned int tpb = qemu_target_page_bits();
+    unsigned long host_bits;

postcopy-ram.c line 108

Quoted 10 lines
postcopy-ram.c +10 -0
+
+    qemu_mutex_init(&mis->postcopy_pmi.mutex);
+    mis->postcopy_pmi.received_map = bitmap_new(ram_pages);
+    mis->postcopy_pmi.requested_map = bitmap_new(ram_pages);
+    bitmap_clear(mis->postcopy_pmi.received_map, 0, ram_pages);
+    bitmap_clear(mis->postcopy_pmi.requested_map, 0, ram_pages);
+    /*
+     * Each bit in the map represents one 'target page' which is no bigger
+     * than a host page but can be smaller.  It's useful to have some
+     * convenience masks for later

So, there's no inherent reason a target page couldn't be bigger than a host page. It's fair enough not to handle that case for now, but something somewhere should probably verify that it's no the case.

Quoted 17 lines
postcopy-ram.c +17 -0
+     */
+
+    /*
+     * The number of bits one host page takes up in the bitmap
+     * e.g. on a 64k host page, 4k Target page, host_bits=64/4=16
+     */
+    host_bits = sysconf(_SC_PAGESIZE) / (1ul << tpb);
+    /* Should be a power of 2 */
+    assert(host_bits && !(host_bits & (host_bits - 1)));
+    /*
+     * If the host_bits isn't a division of the number of bits in long
+     * then the code gets a lot more complex; disallow for now
+     * (I'm not aware of a system where it's true anyway)
+     */
+    assert(((sizeof(long) * 8) % host_bits) == 0);
+
+    mis->postcopy_pmi.host_bits = host_bits;

postcopy-ram.c line 135

Quoted 10 lines
postcopy-ram.c +10 -0
+    /* A mask, starting at bit 0, containing host_bits continuous set bits */
+    mis->postcopy_pmi.host_mask =  (1ul << host_bits) - 1;
+
+    assert((ram_pages % host_bits) == 0);
+}
+
+void postcopy_pmi_destroy(MigrationIncomingState *mis)
+{
+    if (mis->postcopy_pmi.received_map) {
+        g_free(mis->postcopy_pmi.received_map);

g_free() is safe to call on NULL anyway, isn't it?

Quoted 67 lines
postcopy-ram.c +67 -0
+        mis->postcopy_pmi.received_map = NULL;
+    }
+    if (mis->postcopy_pmi.requested_map) {
+        g_free(mis->postcopy_pmi.requested_map);
+        mis->postcopy_pmi.requested_map = NULL;
+    }
+    qemu_mutex_destroy(&mis->postcopy_pmi.mutex);
+}
+
+/*
+ * Mark a set of pages in the PMI as being clear; this is used by the discard
+ * at the start of postcopy, and before the postcopy stream starts.
+ */
+void postcopy_pmi_discard_range(MigrationIncomingState *mis,
+                                size_t start, size_t npages)
+{
+    bitmap_clear(mis->postcopy_pmi.received_map, start, npages);
+}
+
+/*
+ * Test a host-page worth of bits in the map starting at bitmap_index
+ * The bits should all be consistent
+ */
+static bool test_hpbits(MigrationIncomingState *mis,
+                        size_t bitmap_index, unsigned long *map)
+{
+    long masked;
+
+    assert((bitmap_index & (mis->postcopy_pmi.host_bits-1)) == 0);
+
+    masked = (map[BIT_WORD(bitmap_index)] >>
+               (bitmap_index % BITS_PER_LONG)) &
+             mis->postcopy_pmi.host_mask;
+
+    assert((masked == 0) || (masked == mis->postcopy_pmi.host_mask));
+    return !!masked;
+}
+
+/*
+ * Set host-page worth of bits in the map starting at bitmap_index
+ */
+static void set_hpbits(MigrationIncomingState *mis,
+                       size_t bitmap_index, unsigned long *map)
+{
+    assert((bitmap_index & (mis->postcopy_pmi.host_bits-1)) == 0);
+
+    map[BIT_WORD(bitmap_index)] |= mis->postcopy_pmi.host_mask <<
+                                    (bitmap_index % BITS_PER_LONG);
+}
+
+/*
+ * Clear host-page worth of bits in the map starting at bitmap_index
+ */
+static void clear_hpbits(MigrationIncomingState *mis,
+                         size_t bitmap_index, unsigned long *map)
+{
+    assert((bitmap_index & (mis->postcopy_pmi.host_bits-1)) == 0);
+
+    map[BIT_WORD(bitmap_index)] &= ~(mis->postcopy_pmi.host_mask <<
+                                    (bitmap_index % BITS_PER_LONG));
+}
+
+/*
+ * Retrieve the state of the given page
+ * Note: This version for use by callers already holding the lock
+ */
+static PostcopyPMIState postcopy_pmi_get_state_nolock(

postcopy-ram.c line 212

Quoted 10 lines
postcopy-ram.c +10 -0
+                            MigrationIncomingState *mis,
+                            size_t bitmap_index)
+{
+    bool received, requested;
+
+    received = test_hpbits(mis, bitmap_index, mis->postcopy_pmi.received_map);
+    requested = test_hpbits(mis, bitmap_index, mis->postcopy_pmi.requested_map);
+
+    if (received) {
+        assert(!requested);

Clearing the requested bit when you set the received bit seems a bit pointless. (requested && received) isn't meaningfully different from (!requested && received) but there seems no reason to go to extra trouble to avoid that state, and having the record might be interesting for gathering statistics.

Quoted 66 lines
postcopy-ram.c +66 -0
+        return POSTCOPY_PMI_RECEIVED;
+    } else {
+        return requested ? POSTCOPY_PMI_REQUESTED : POSTCOPY_PMI_MISSING;
+    }
+}
+
+/* Retrieve the state of the given page */
+__attribute__ (( unused )) /* Until later in patch series */
+static PostcopyPMIState postcopy_pmi_get_state(MigrationIncomingState *mis,
+                                               size_t bitmap_index)
+{
+    PostcopyPMIState ret;
+    qemu_mutex_lock(&mis->postcopy_pmi.mutex);
+    ret = postcopy_pmi_get_state_nolock(mis, bitmap_index);
+    qemu_mutex_unlock(&mis->postcopy_pmi.mutex);
+
+    return ret;
+}
+
+/*
+ * Set the page state to the given state if the previous state was as expected
+ * Return the actual previous state.
+ */
+__attribute__ (( unused )) /* Until later in patch series */
+static PostcopyPMIState postcopy_pmi_change_state(MigrationIncomingState *mis,
+                                           size_t bitmap_index,
+                                           PostcopyPMIState expected_state,
+                                           PostcopyPMIState new_state)
+{
+    PostcopyPMIState old_state;
+
+    qemu_mutex_lock(&mis->postcopy_pmi.mutex);
+    old_state = postcopy_pmi_get_state_nolock(mis, bitmap_index);
+
+    if (old_state == expected_state) {
+        switch (new_state) {
+        case POSTCOPY_PMI_MISSING:
+          assert(0); /* This shouldn't actually happen - use discard_range */
+          break;
+
+        case POSTCOPY_PMI_REQUESTED:
+          assert(old_state == POSTCOPY_PMI_MISSING);
+          set_hpbits(mis, bitmap_index, mis->postcopy_pmi.requested_map);
+          break;
+
+        case POSTCOPY_PMI_RECEIVED:
+          assert(old_state == POSTCOPY_PMI_MISSING ||
+                 old_state == POSTCOPY_PMI_REQUESTED);
+          set_hpbits(mis, bitmap_index, mis->postcopy_pmi.received_map);
+          clear_hpbits(mis, bitmap_index, mis->postcopy_pmi.requested_map);
+          break;
+        }
+    }
+
+    qemu_mutex_unlock(&mis->postcopy_pmi.mutex);
+    return old_state;
+}
+
+/*
+ * Useful when debugging postcopy, although if it failed early the
+ * received map can be quite sparse and thus big when dumped.
+ */
+void postcopy_pmi_dump(MigrationIncomingState *mis)
+{
+    fprintf(stderr, "postcopy_pmi_dump: requested\n");
+    ram_debug_dump_bitmap(mis->postcopy_pmi.requested_map, false);

postcopy-ram.c line 288

Quoted 10 lines
postcopy-ram.c +10 -0
+    fprintf(stderr, "postcopy_pmi_dump: received\n");
+    ram_debug_dump_bitmap(mis->postcopy_pmi.received_map, true);
+    fprintf(stderr, "postcopy_pmi_dump: end\n");
+}
+
+/* Called by ram_load prior to mapping the page */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index)
+{
+    if (mis->postcopy_ram_state == POSTCOPY_RAM_INCOMING_ADVISE) {

A silent no-op if you're not in the expected migration phase doesn't seem right. Should this be an assert() instead?

Quoted 25 lines
postcopy-ram.c +16 -0
+        /*
+         * If we're in precopy-advise mode we need to track received pages even
+         * though we don't need to place pages atomically yet.
+         * In advise mode there's only a single thread, so don't need locks
+         */
+        set_bit(bitmap_index, mis->postcopy_pmi.received_map);
+    }
+}
+
 int postcopy_ram_hosttest(void)
 {
     /* TODO: Needs guarding with CONFIG_ once we have libc's that have the defs
@@ -156,5 +369,12 @@ int postcopy_ram_hosttest(void)
     return -1;
 }
 
+/* Called by ram_load prior to mapping the page */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index)
+{
+    /* We don't support postcopy so don't care */
+}
+
 #endif
 
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#104 lore raw

Re: [Qemu-devel] [PATCH v4 29/47] Postcopy page-map-incoming (PMI) structure

Quoted 12 lines · * David Gibson ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it.

Though there are a couple of minor comments below:

<snip>

Quoted 17 lines
Quoted 17 lines
postcopy-ram.c +17 -0
+/* ---------------------------------------------------------------------- */
+/* Postcopy pagemap-inbound (pmi) - data structures that record the       */
+/* state of each page used by the inbound postcopy                        */
+/* It's a pair of bitmaps (of the same structure as the migration bitmaps)*/
+/* holding one bit per target-page, although all operations work on host  */
+/* pages.                                                                 */
+__attribute__ (( unused )) /* Until later in patch series */
+static void postcopy_pmi_init(MigrationIncomingState *mis, size_t ram_pages)
+{
+    unsigned int tpb = qemu_target_page_bits();
+    unsigned long host_bits;
+
+    qemu_mutex_init(&mis->postcopy_pmi.mutex);
+    mis->postcopy_pmi.received_map = bitmap_new(ram_pages);
+    mis->postcopy_pmi.requested_map = bitmap_new(ram_pages);
+    bitmap_clear(mis->postcopy_pmi.received_map, 0, ram_pages);
+    bitmap_clear(mis->postcopy_pmi.requested_map, 0, ram_pages);
Quoted 8 lines

postcopy-ram.c line 108

Quoted 4 lines
postcopy-ram.c +4 -0
+    /*
+     * Each bit in the map represents one 'target page' which is no bigger
+     * than a host page but can be smaller.  It's useful to have some
+     * convenience masks for later

So, there's no inherent reason a target page couldn't be bigger than a host page. It's fair enough not to handle that case for now, but something somewhere should probably verify that it's no the case.

I've added a guard for this in the host test.

Quoted 21 lines
Quoted 21 lines
postcopy-ram.c +21 -0
+     */
+
+    /*
+     * The number of bits one host page takes up in the bitmap
+     * e.g. on a 64k host page, 4k Target page, host_bits=64/4=16
+     */
+    host_bits = sysconf(_SC_PAGESIZE) / (1ul << tpb);
+    /* Should be a power of 2 */
+    assert(host_bits && !(host_bits & (host_bits - 1)));
+    /*
+     * If the host_bits isn't a division of the number of bits in long
+     * then the code gets a lot more complex; disallow for now
+     * (I'm not aware of a system where it's true anyway)
+     */
+    assert(((sizeof(long) * 8) % host_bits) == 0);
+
+    mis->postcopy_pmi.host_bits = host_bits;
+    /* A mask, starting at bit 0, containing host_bits continuous set bits */
+    mis->postcopy_pmi.host_mask =  (1ul << host_bits) - 1;
+
+    assert((ram_pages % host_bits) == 0);
Quoted 8 lines

postcopy-ram.c line 135

Quoted 6 lines
postcopy-ram.c +6 -0
+}
+
+void postcopy_pmi_destroy(MigrationIncomingState *mis)
+{
+    if (mis->postcopy_pmi.received_map) {
+        g_free(mis->postcopy_pmi.received_map);

g_free() is safe to call on NULL anyway, isn't it?

It is; fixed.

Quoted 13 lines
Quoted 13 lines
postcopy-ram.c +13 -0
+/*
+ * Retrieve the state of the given page
+ * Note: This version for use by callers already holding the lock
+ */
+static PostcopyPMIState postcopy_pmi_get_state_nolock(
+                            MigrationIncomingState *mis,
+                            size_t bitmap_index)
+{
+    bool received, requested;
+
+    received = test_hpbits(mis, bitmap_index, mis->postcopy_pmi.received_map);
+    requested = test_hpbits(mis, bitmap_index, mis->postcopy_pmi.requested_map);
+
Quoted 8 lines

postcopy-ram.c line 212

Quoted 2 lines
postcopy-ram.c +2 -0
+    if (received) {
+        assert(!requested);

Clearing the requested bit when you set the received bit seems a bit pointless. (requested && received) isn't meaningfully different from (!requested && received) but there seems no reason to go to extra trouble to avoid that state, and having the record might be interesting for gathering statistics.

Hmm yes I think you're right; but I want to think about it to convince me a bit more; this code originally started off as two really seprate bitmaps and has slowly morphed into really representing 3 states. I've added it to a TODO.

Quoted 8 lines

postcopy-ram.c line 288

Quoted 5 lines
postcopy-ram.c +5 -0
+/* Called by ram_load prior to mapping the page */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index)
+{
+    if (mis->postcopy_ram_state == POSTCOPY_RAM_INCOMING_ADVISE) {

A silent no-op if you're not in the expected migration phase doesn't seem right. Should this be an assert() instead?

No. This routine is called by the RAM code prior to doing anything with the page, but it does it in all postcopy states; it's just that we only care about it in the ADVISE state (it makes things a little cleaner - the RAM code no longer has to know about the postcopy stages).

Dave

Quoted 16 lines
Quoted 16 lines
postcopy-ram.c +9 -0
+        /*
+         * If we're in precopy-advise mode we need to track received pages even
+         * though we don't need to place pages atomically yet.
+         * In advise mode there's only a single thread, so don't need locks
+         */
+        set_bit(bitmap_index, mis->postcopy_pmi.received_map);
+    }
+}
+
 int postcopy_ram_hosttest(void)
 {
     /* TODO: Needs guarding with CONFIG_ once we have libc's that have the defs
@@ -156,5 +369,12 @@ int postcopy_ram_hosttest(void)
     return -1;
 }
 
Quoted 15 lines
Quoted 9 lines
postcopy-ram.c +7 -0
+/* Called by ram_load prior to mapping the page */
+void postcopy_hook_early_receive(MigrationIncomingState *mis,
+                                 size_t bitmap_index)
+{
+    /* We don't support postcopy so don't care */
+}
+
 #endif
 
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#105 lore raw

PATCH v4 30/47 Postcopy: Maintain sentmap and calculate discard

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

---
 arch_init.c                      | 271 ++++++++++++++++++++++++++++++++++++++-
 include/migration/migration.h    |  12 ++
 include/migration/postcopy-ram.h |  34 +++++
 include/qemu/typedefs.h          |   1 +
 migration.c                      |   2 +
 postcopy-ram.c                   | 111 ++++++++++++++++
 savevm.c                         |   3 -
 7 files changed, 428 insertions(+), 6 deletions(-)
arch_init.c +268 -3
index 44072d8..030d189 100644
@@ -40,6 +40,7 @@
 #include "hw/audio/audio.h"
 #include "sysemu/kvm.h"
 #include "migration/migration.h"
+#include "migration/postcopy-ram.h"
 #include "hw/i386/smbios.h"
 #include "exec/address-spaces.h"
 #include "hw/audio/pcspk.h"
@@ -415,9 +416,15 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t **current_data,
     return bytes_sent;
 }
 
+/* mr: The region to search for dirty pages in
+ * start: Start address (typically so we can continue from previous page)
+ * bitoffset: Pointer into which to store the offset into the dirty map
+ *            at which the bit was found.
+ */
 static inline
 ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
-                                                 ram_addr_t start)
+                                                 ram_addr_t start,
+                                                 unsigned long *bitoffset)
 {
     unsigned long base = mr->ram_addr >> TARGET_PAGE_BITS;
     unsigned long nr = base + (start >> TARGET_PAGE_BITS);
@@ -436,6 +443,7 @@ ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
         clear_bit(next, migration_bitmap);
         migration_dirty_pages--;
     }
+    *bitoffset = next;
     return (next - base) << TARGET_PAGE_BITS;
 }
 
@@ -564,6 +572,19 @@ static void migration_bitmap_sync(void)
     }
 }
 
+static RAMBlock *ram_find_block(const char *id)
+{
+    RAMBlock *block;
+
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+        if (!strcmp(id, block->idstr)) {
+            return block;
+        }
+    }
+
+    return NULL;
+}
+
 /*
  * ram_save_page: Send the given page to the stream
  *
@@ -652,13 +673,14 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage)
     bool complete_round = false;
     int bytes_sent = 0;
     MemoryRegion *mr;
+    unsigned long bitoffset;
 
     if (!block)
         block = QTAILQ_FIRST(&ram_list.blocks);
 
     while (true) {
         mr = block->mr;
-        offset = migration_bitmap_find_and_reset_dirty(mr, offset);
+        offset = migration_bitmap_find_and_reset_dirty(mr, offset, &bitoffset);
         if (complete_round && block == last_seen_block &&
             offset >= last_offset) {
             break;
@@ -676,6 +698,11 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage)
 
             /* if page is unmodified, continue to the next */
             if (bytes_sent > 0) {
+                MigrationState *s = migrate_get_current();
+                if (s->sentmap) {
+                    set_bit(bitoffset, s->sentmap);
+                }
+
                 last_sent_block = block;
                 break;
             }
@@ -735,12 +762,19 @@ void free_xbzrle_decoded_buf(void)
 
 static void migration_end(void)
 {
+    MigrationState *s = migrate_get_current();
+
     if (migration_bitmap) {
         memory_global_dirty_log_stop();
         g_free(migration_bitmap);
         migration_bitmap = NULL;
     }
 
+    if (s->sentmap) {
+        g_free(s->sentmap);
+        s->sentmap = NULL;
+    }
+
     XBZRLE_cache_lock();
     if (XBZRLE.cache) {
         cache_fini(XBZRLE.cache);
@@ -808,6 +842,232 @@ void ram_debug_dump_bitmap(unsigned long *todump, bool expected)
     }
 }
 
+/* **** functions for postcopy ***** */
+
+/*
+ * A helper to get 32 bits from a bit map; trivial for HOST_LONG_BITS=32
+ * messier for 64; the bitmaps are actually long's that are 32 or 64bit
+ */
+static uint32_t get_32bits_map(unsigned long *map, int64_t start)
+{
+#if HOST_LONG_BITS == 64
+    uint64_t tmp64;
+
+    tmp64 = map[start / 64];
+    return (start & 32) ? (tmp64 >> 32) : (tmp64 & 0xffffffffu);
+#elif HOST_LONG_BITS == 32
+    /*
+     * Irrespective of host endianness, sentmap[n] is for pages earlier
+     * than sentmap[n+1] so we can't just cast up
+     */
+    return map[start / 32];
+#else
+#error "Host long other than 64/32 not supported"
+#endif
+}
+
+/*
+ * A helper to put 32 bits into a bit map; trivial for HOST_LONG_BITS=32
+ * messier for 64; the bitmaps are actually long's that are 32 or 64bit
+ */
+__attribute__ (( unused )) /* Until later in patch series */
+static void put_32bits_map(unsigned long *map, int64_t start,
+                           uint32_t v)
+{
+#if HOST_LONG_BITS == 64
+    uint64_t tmp64 = v;
+    uint64_t mask = 0xffffffffu;
+
+    if (start & 32) {
+        tmp64 = tmp64 << 32;
+        mask =  mask << 32;
+    }
+
+    map[start / 64] = (map[start / 64] & ~mask) | tmp64;
+#elif HOST_LONG_BITS == 32
+    /*
+     * Irrespective of host endianness, sentmap[n] is for pages earlier
+     * than sentmap[n+1] so we can't just cast up
+     */
+    map[start / 32] = v;
+#else
+#error "Host long other than 64/32 not supported"
+#endif
+}
+
+/*
+ * When working on 32bit chunks of a bitmap where the only valid section
+ * is between start..end (inclusive), generate a mask with only those
+ * valid bits set for the current 32bit word within that bitmask.
+ */
+static int make_32bit_mask(unsigned long start, unsigned long end,
+                           unsigned long cur32)
+{
+    unsigned long first32, last32;
+    uint32_t mask = ~(uint32_t)0;
+    first32 = start / 32;
+    last32 = end / 32;
+
+    if ((cur32 == first32) && (start & 31)) {
+        /* e.g. (start & 31) = 3
+         *         1 << .    -> 2^3
+         *         . - 1     -> 2^3 - 1 i.e. mask 2..0
+         *         ~.        -> mask 31..3
+         */
+        mask &= ~((((uint32_t)1) << (start & 31)) - 1);
+    }
+
+    if ((cur32 == last32) && ((end & 31) != 31)) {
+        /* e.g. (end & 31) = 3
+         *            .   +1 -> 4
+         *         1 << .    -> 2^4
+         *         . -1      -> 2^4 - 1
+         *                   = mask set 3..0
+         */
+        mask &= (((uint32_t)1) << ((end & 31) + 1)) - 1;
+    }
+
+    return mask;
+}
+
+/*
+ * Callback from ram_postcopy_each_ram_discard for each RAMBlock
+ * start,end: Indexes into the bitmap for the first and last bit
+ *            representing the named block
+ */
+static int pc_send_discard_bm_ram(MigrationState *ms,
+                                  PostcopyDiscardState *pds,
+                                  unsigned long start, unsigned long end)
+{
+    /*
+     * There is no guarantee that start, end are on convenient 32bit multiples
+     * (We always send 32bit chunks over the wire, irrespective of long size)
+     */
+    unsigned long first32, last32, cur32;
+    first32 = start / 32;
+    last32 = end / 32;
+
+    for (cur32 = first32; cur32 <= last32; cur32++) {
+        /* Deal with start/end not on alignment */
+        uint32_t mask = make_32bit_mask(start, end, cur32);
+
+        uint32_t data = get_32bits_map(ms->sentmap, cur32 * 32);
+        data &= mask;
+
+        if (data) {
+            postcopy_discard_send_chunk(ms, pds, (cur32-first32) * 32, data);
+        }
+    }
+
+    return 0;
+}
+
+/*
+ * Utility for the outgoing postcopy code.
+ *   Calls postcopy_send_discard_bm_ram for each RAMBlock
+ *   passing it bitmap indexes and name.
+ * Returns: 0 on success
+ * (qemu_ram_foreach_block ends up passing unscaled lengths
+ *  which would mean postcopy code would have to deal with target page)
+ */
+static int pc_each_ram_discard(MigrationState *ms)
+{
+    struct RAMBlock *block;
+    int ret;
+
+    QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+        unsigned long first = block->offset >> TARGET_PAGE_BITS;
+        unsigned long last = (block->offset + (block->length-1))
+                                >> TARGET_PAGE_BITS;
+        PostcopyDiscardState *pds = postcopy_discard_send_init(ms,
+                                                               first & 31,
+                                                               block->idstr);
+
+        /*
+         * Postcopy sends chunks of bitmap over the wire, but it
+         * just needs indexes at this point, avoids it having
+         * target page specific code.
+         */
+        ret = pc_send_discard_bm_ram(ms, pds, first, last);
+        postcopy_discard_send_finish(ms, pds);
+        if (ret) {
+            return ret;
+        }
+    }
+
+    return 0;
+}
+
+/*
+ * Transmit the set of pages to be discarded after precopy to the target
+ * these are pages that have been sent previously but have been dirtied
+ * Hopefully this is pretty sparse
+ */
+int ram_postcopy_send_discard_bitmap(MigrationState *ms)
+{
+    /* This should be our last sync, the src is now paused */
+    migration_bitmap_sync();
+
+    /*
+     * Update the sentmap to be  sentmap&=dirty
+     */
+    bitmap_and(ms->sentmap, ms->sentmap, migration_bitmap,
+               last_ram_offset() >> TARGET_PAGE_BITS);
+
+
+    DPRINTF("Dumping merged sentmap");
+#ifdef DEBUG_POSTCOPY
+    ram_debug_dump_bitmap(ms->sentmap, false);
+#endif
+
+    return pc_each_ram_discard(ms);
+}
+
+/*
+ * At the start of the postcopy phase of migration, any now-dirty
+ * precopied pages are discarded.
+ *
+ * start..end is an inclusive range of bits indexed in the source
+ *    VMs bitmap for this RAMBlock, source_target_page_bits tells
+ *    us what one of those bits represents.
+ *
+ * start/end are offsets from the start of the bitmap for RAMBlock 'block_name'
+ *
+ * Returns 0 on success.
+ */
+int ram_discard_range(MigrationIncomingState *mis,
+                      const char *block_name,
+                      uint64_t start, uint64_t end)
+{
+    assert(end >= start);
+
+    RAMBlock *rb = ram_find_block(block_name);
+
+    if (!rb) {
+        error_report("ram_discard_range: Failed to find block '%s'",
+                     block_name);
+        return -1;
+    }
+
+    uint64_t index_offset = rb->offset >> TARGET_PAGE_BITS;
+    postcopy_pmi_discard_range(mis, start + index_offset, (end - start) + 1);
+
+    /* +1 gives the byte after the end of the last page to be discarded */
+    ram_addr_t end_offset = (end+1) << TARGET_PAGE_BITS;
+    uint8_t *host_startaddr = rb->host + (start << TARGET_PAGE_BITS);
+    uint8_t *host_endaddr;
+
+    if (end_offset <= rb->length) {
+        host_endaddr   = rb->host + (end_offset-1);
+        return postcopy_ram_discard_range(mis, host_startaddr, host_endaddr);
+    } else {
+        error_report("ram_discard_range: Overrun block '%s' (%" PRIu64
+                     "/%" PRIu64 "/%zu)",
+                     block_name, start, end, rb->length);
+        return -1;
+    }
+}
+
 static int ram_save_setup(QEMUFile *f, void *opaque)
 {
     RAMBlock *block;
@@ -846,7 +1106,6 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 
         acct_clear();
     }
-
     qemu_mutex_lock_iothread();
     qemu_mutex_lock_ramlist();
     bytes_transferred = 0;
@@ -856,6 +1115,12 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     migration_bitmap = bitmap_new(ram_bitmap_pages);
     bitmap_set(migration_bitmap, 0, ram_bitmap_pages);
 
+    if (migrate_postcopy_ram()) {
+        MigrationState *s = migrate_get_current();
+        s->sentmap = bitmap_new(ram_bitmap_pages);
+        bitmap_clear(s->sentmap, 0, ram_bitmap_pages);
+    }
+
     /*
      * Count the total number of pages used by ram blocks not including any
      * gaps due to alignment or unplugs.
include/migration/migration.h +12 -0
index 1405a15..73d338e 100644
@@ -122,6 +122,13 @@ struct MigrationState
 
     /* Flag set once the migration has been asked to enter postcopy */
     volatile bool start_postcopy;
+
+    /* bitmap of pages that have been sent at least once
+     * only maintained and used in postcopy at the moment
+     * where it's used to send the dirtymap at the start
+     * of the postcopy phase
+     */
+    unsigned long *sentmap;
 };
 
 void process_incoming_migration(QEMUFile *f);
@@ -191,6 +198,11 @@ double xbzrle_mig_cache_miss_rate(void);
 
 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size);
 void ram_debug_dump_bitmap(unsigned long *todump, bool expected);
+/* For outgoing discard bitmap */
+int ram_postcopy_send_discard_bitmap(MigrationState *ms);
+/* For incoming postcopy discard */
+int ram_discard_range(MigrationIncomingState *mis, const char *block_name,
+                      uint64_t start, uint64_t end);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +34 -0
index addb88a..2a39a03 100644
@@ -28,4 +28,38 @@ void postcopy_pmi_destroy(MigrationIncomingState *mis);
 void postcopy_pmi_discard_range(MigrationIncomingState *mis,
                                 size_t start, size_t npages);
 void postcopy_pmi_dump(MigrationIncomingState *mis);
+
+/*
+ * Discard the contents of memory start..end inclusive.
+ * We can assume that if we've been called postcopy_ram_hosttest returned true
+ */
+int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
+                               uint8_t *end);
+
+
+/*
+ * Called at the start of each RAMBlock by the bitmap code
+ * offset is the bit within the first 32bit chunk of mask
+ * that represents the first page of the RAM Block
+ * Returns a new PDS
+ */
+PostcopyDiscardState *postcopy_discard_send_init(MigrationState *ms,
+                                                 uint8_t offset,
+                                                 const char *name);
+
+/*
+ * Called by the bitmap code for each chunk to discard
+ * May send a discard message, may just leave it queued to
+ * be sent later
+ */
+void postcopy_discard_send_chunk(MigrationState *ms, PostcopyDiscardState *pds,
+                                unsigned long pos, uint32_t bitmap);
+
+/*
+ * Called at the end of each RAMBlock by the bitmap code
+ * Sends any outstanding discard messages, frees the PDS
+ */
+void postcopy_discard_send_finish(MigrationState *ms,
+                                  PostcopyDiscardState *pds);
+
 #endif
include/qemu/typedefs.h +1 -0
index 61b330c..79f57c0 100644
@@ -63,6 +63,7 @@ typedef struct PCIEAERLog PCIEAERLog;
 typedef struct PCIEAERErr PCIEAERErr;
 typedef struct PCIEPort PCIEPort;
 typedef struct PCIESlot PCIESlot;
+typedef struct PostcopyDiscardState PostcopyDiscardState;
 typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct MSIMessage MSIMessage;
 typedef struct SerialState SerialState;
migration.c +2 -0
index 29ee740..db860c9 100644
@@ -22,6 +22,7 @@
 #include "block/block.h"
 #include "qemu/sockets.h"
 #include "migration/block.h"
+#include "migration/postcopy-ram.h"
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
@@ -947,6 +948,7 @@ static void *migration_thread(void *opaque)
             } else {
                 int ret;
 
+                DPRINTF("done iterating\n");
                 qemu_mutex_lock_iothread();
                 start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
postcopy-ram.c +111 -0

128 lines not on this page · raw message

savevm.c +0 -3
index 7c4541d..7f9e0b2 100644
@@ -1267,15 +1267,12 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,
              * we know there must be at least 1 bit set due to the loop entry
              * If there is no 0 firstzero will be 32
              */
-            /* TODO - ram_discard_range gets added in a later patch
             int ret = ram_discard_range(mis, ramid,
                                 startaddr + firstset - first_bit_offset,
                                 startaddr + (firstzero - 1) - first_bit_offset);
-            ret = -1;
             if (ret) {
                 return ret;
             }
-            */
 
             /* mask= .?0000000000 */
             /*         ^fz ^fs    */
Signature · 1 line
1.9.3
#106 lore raw

Re: [Qemu-devel] [PATCH v4 30/47] Postcopy: Maintain sentmap and calculate discard

Quoted 10 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

I find this one really hard to wrap my head around, and I'm having trouble putting my finger on why.

I do wonder if the "base + tiny bitmap" encodinng for the discard list over the wire is the best choice. It seems to involve a bunch of rather tedious code rejigging the bitmap into 32-bit chunks, and a bunch of rather hard to follow code moving back and forth between that encoding and simple address or page ranges for handling the actual discards. It also involves sending the bit offsets for the start of each ram block over the wire, which feels like it should be an internal detail.

Would just a simple list of start..end or start/len pairs end up simpler overall? Converting the bitmap used to track it on the source into ranges would be a little fiddly, but I suspect less so than the code to split into 32-bit pieces.

It might also be a bit more robust against possible future options for source host vs. dest host vs. target page size, since the source can construct it in terms if its granularity constraints, and destination can round each chunk out to its own granularity.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#107 lore raw

Re: [Qemu-devel] [PATCH v4 30/47] Postcopy: Maintain sentmap and calculate discard

Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.

Quoted 6 lines
Quoted 3 lines

Maintain a 'sentmap' of pages that have already been sent. Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.

I find this one really hard to wrap my head around, and I'm having trouble putting my finger on why.

It seems to turn out fairly compact, and a lot of the time the bitmap we're trying to transfer is very sparse, but seems to be a little clumpy.

Quoted 8 lines

I do wonder if the "base + tiny bitmap" encodinng for the discard list over the wire is the best choice. It seems to involve a bunch of rather tedious code rejigging the bitmap into 32-bit chunks, and a bunch of rather hard to follow code moving back and forth between that encoding and simple address or page ranges for handling the actual discards. It also involves sending the bit offsets for the start of each ram block over the wire, which feels like it should be an internal detail.

Yes the fiddling into 32bit chunks is a bit messier than it was originally; the problem here is that the migration bitmap (for no apparent reason) uses 'long' so the complexity all comes from having the internal structure be flexible and wanting not to pass that flexibility onto the wire.

Quoted 4 lines

Would just a simple list of start..end or start/len pairs end up simpler overall? Converting the bitmap used to track it on the source into ranges would be a little fiddly, but I suspect less so than the code to split into 32-bit pieces.

In the end you have to traverse that bitmap somewhere, since I already had the data in the form of a bitmap it seemed reasonable to take advantage of it as a compact representation.

Quoted 4 lines

It might also be a bit more robust against possible future options for source host vs. dest host vs. target page size, since the source can construct it in terms if its granularity constraints, and destination can round each chunk out to its own granularity.

That already happens here as the destination reconsistutes the addresses from the bitmap.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#108 lore raw

PATCH v4 31/47 postcopy: Incoming initialisation

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 arch_init.c                      |  11 ++++
 include/migration/migration.h    |   1 +
 include/migration/postcopy-ram.h |  12 +++++
 migration.c                      |   1 +
 postcopy-ram.c                   | 110 ++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |   4 ++
 6 files changed, 138 insertions(+), 1 deletion(-)
arch_init.c +11 -0
index 030d189..4a03171 100644
@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
     }
 }
 
+/*
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}
+
 static int ram_load(QEMUFile *f, void *opaque, int version_id)
 {
     ram_addr_t addr;
include/migration/migration.h +1 -0
index 73d338e..be63c89 100644
@@ -203,6 +203,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms);
 /* For incoming postcopy discard */
 int ram_discard_range(MigrationIncomingState *mis, const char *block_name,
                       uint64_t start, uint64_t end);
+int ram_postcopy_incoming_init(MigrationIncomingState *mis);
 
 /**
  * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +12 -0
index 2a39a03..8f237a2 100644
@@ -19,6 +19,18 @@
 int postcopy_ram_hosttest(void);
 
 /*
+ * Initialise postcopy-ram, setting the RAM to a state where we can go into
+ * postcopy later; must be called prior to any precopy.
+ * called from arch_init's similarly named ram_postcopy_incoming_init
+ */
+int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages);
+
+/*
+ * At the end of a migration where postcopy_ram_incoming_init was called.
+ */
+int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis);
+
+/*
  * In 'advise' mode record that a page has been received.
  */
 void postcopy_hook_early_receive(MigrationIncomingState *mis,
migration.c +1 -0
index db860c9..63d70b6 100644
@@ -99,6 +99,7 @@ MigrationIncomingState *migration_incoming_state_init(QEMUFile* f)
 
 void migration_incoming_state_destroy(void)
 {
+    postcopy_pmi_destroy(mis_current);
     g_free(mis_current);
     mis_current = NULL;
 }
postcopy-ram.c +109 -1

128 lines not on this page · raw message

savevm.c +4 -0
index 7f9e0b2..54bdb26 100644
@@ -1166,6 +1166,10 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (ram_postcopy_incoming_init(mis)) {
+        return -1;
+    }
+
     mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_ADVISE;
 
     /*
Signature · 1 line
1.9.3
#109 lore raw

Re: [Qemu-devel] [PATCH v4 31/47] postcopy: Incoming initialisation

Quoted 20 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 arch_init.c                      |  11 ++++
 include/migration/migration.h    |   1 +
 include/migration/postcopy-ram.h |  12 +++++
 migration.c                      |   1 +
 postcopy-ram.c                   | 110 ++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |   4 ++
 6 files changed, 138 insertions(+), 1 deletion(-)
arch_init.c +0 -0
index 030d189..4a03171 100644
@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
     }
 }
 

arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) · line 1357

Quoted 10 lines
arch_init.c +10 -0
+/*
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}

Um.. yeah. I'm sure ram_postcopy_incoming_init versus postcopy_ram_incoming_init won't get confusing o_O.

[snip]

Quoted 22 lines
arch_init.c +22 -0
+/*
+ * Setup an area of RAM so that it *can* be used for postcopy later; this
+ * must be done right at the start prior to pre-copy.
+ * opaque should be the MIS.
+ */
+static int init_area(const char *block_name, void *host_addr,
+                     ram_addr_t offset, ram_addr_t length, void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+
+    DPRINTF("init_area: %s: %p offset=%zx length=%zd(%zx)",
+            block_name, host_addr, offset, length, length);
+    /*
+     * We need the whole of RAM to be truly empty for postcopy, so things
+     * like ROMs and any data tables built during init must be zero'd
+     * - we're going to get the copy from the source anyway.
+     */
+    if (postcopy_ram_discard_range(mis, host_addr, (host_addr + length - 1))) {
+        return -1;
+    }
+
+    /*

arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)

Quoted 10 lines
arch_init.c +10 -0
+     * We also need the area to be normal 4k pages, not huge pages
+     * (otherwise we can't be sure we can use remap_anon_pages to put
+     * a 4k page in later).  THP might come along and map a 2MB page
+     * and when it's partially accessed in precopy it might not break
+     * it down, but leave a 2MB zero'd page.
+     */
+    if (madvise(host_addr, length, MADV_NOHUGEPAGE)) {
+        perror("init_area: NOHUGEPAGE");
+        return -1;
+    }

I'm assuming this is because remap_anon_pages() can't automatically split a THP itself. It's not immediately obvious to me why it can't though.

Also.. what effect will this have on an actual hugetlbfs memory region? If there's code to handle that case I haven't spotted it yet.

arch_init.c

Quoted 12 lines
arch_init.c +12 -0
+
+    return 0;
+}
+
+/*
+ * At the end of migration, undo the effects of init_area
+ * opaque should be the MIS.
+ */
+static int cleanup_area(const char *block_name, void *host_addr,
+                        ram_addr_t offset, ram_addr_t length, void *opaque)
+{
+    /* Turn off userfault here as well? */

This comment appears to be obsoleted by the code below.

Quoted 90 lines
arch_init.c +65 -0
+
+    DPRINTF("cleanup_area: %s: %p offset=%zx length=%zd(%zx)",
+            block_name, host_addr, offset, length, length);
+    /*
+     * We turned off hugepage for the precopy stage with postcopy enabled
+     * we can turn it back on now.
+     */
+    if (madvise(host_addr, length, MADV_HUGEPAGE)) {
+        perror("init_area: HUGEPAGE");
+        return -1;
+    }
+
+    /*
+     * We can also turn off userfault now since we should have all the
+     * pages.   It can be useful to leave it on to debug postcopy
+     * if you're not sure it's always getting every page.
+     */
+    if (madvise(host_addr, length, MADV_NOUSERFAULT)) {
+        perror("init_area: NOUSERFAULT");
+        return -1;
+    }
+
+    return 0;
+}
+
+/*
+ * Initialise postcopy-ram, setting the RAM to a state where we can go into
+ * postcopy later; must be called prior to any precopy.
+ * called from arch_init's similarly named ram_postcopy_incoming_init
+ */
+int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages)
+{
+    postcopy_pmi_init(mis, ram_pages);
+
+    if (qemu_ram_foreach_block(init_area, mis)) {
+        return -1;
+    }
+
+    return 0;
+}
+
+/*
+ * At the end of a migration where postcopy_ram_incoming_init was called.
+ */
+int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
+{
+    /* TODO: Join the fault thread once we're sure it will exit */
+    if (qemu_ram_foreach_block(cleanup_area, mis)) {
+        return -1;
+    }
+
+    return 0;
+}
+
 #else
 /* No target OS support, stubs just fail */
@@ -404,6 +501,17 @@ void postcopy_hook_early_receive(MigrationIncomingState *mis,
     /* We don't support postcopy so don't care */
 }
 
+int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages)
+{
+    error_report("postcopy_ram_incoming_init: No OS support");
+    return -1;
+}
+
+int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
+{
+    assert(0);
+}
+
 void postcopy_pmi_destroy(MigrationIncomingState *mis)
 {
     /* Called in normal cleanup path - so it's OK */
savevm.c +4 -0
index 7f9e0b2..54bdb26 100644
@@ -1166,6 +1166,10 @@ static int loadvm_postcopy_ram_handle_advise(MigrationIncomingState *mis,
         return -1;
     }
 
+    if (ram_postcopy_incoming_init(mis)) {
+        return -1;
+    }
+
     mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_ADVISE;
 
     /*
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#110 lore raw

Re: [Qemu-devel] [PATCH v4 31/47] postcopy: Incoming initialisation

Quoted 21 lines · * David Gibson ([email protected]) wrote:
Quoted 20 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 arch_init.c                      |  11 ++++
 include/migration/migration.h    |   1 +
 include/migration/postcopy-ram.h |  12 +++++
 migration.c                      |   1 +
 postcopy-ram.c                   | 110 ++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |   4 ++
 6 files changed, 138 insertions(+), 1 deletion(-)
arch_init.c +0 -0
index 030d189..4a03171 100644
@@ -1345,6 +1345,17 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
     }
 }
 
Quoted 13 lines

arch_init.c void ram_handle_compressed(void *host, uint8_t ch, uint64_t size) · line 1357

Quoted 10 lines
arch_init.c +10 -0
+/*
+ * Allocate data structures etc needed by incoming migration with postcopy-ram
+ * postcopy-ram's similarly names postcopy_ram_incoming_init does the work
+ */
+int ram_postcopy_incoming_init(MigrationIncomingState *mis)
+{
+    size_t ram_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+    return postcopy_ram_incoming_init(mis, ram_pages);
+}

Um.. yeah. I'm sure ram_postcopy_incoming_init versus postcopy_ram_incoming_init won't get confusing o_O.

agreed; that's why I put the comments on. My problem here is that:
1) last_ram_offset() comes from code that's poisoned so it can't be built in a target independent file
2) I'd managed so far (with a couple of hacks) to keep postcopy_ram.c target independent.
3) ram_ is the prefix for external names in arch_init.c
4) postcopy_ram_ is the prefix for external names in postcopy_ram.c

If I threw in the towel and made postcopy_ram target dependent it would remove the need for that wrapper; it might be the best bet. (Other naming suggestions also welcome)

Quoted 29 lines

[snip]

Quoted 28 lines
arch_init.c +28 -0
+/*
+ * Setup an area of RAM so that it *can* be used for postcopy later; this
+ * must be done right at the start prior to pre-copy.
+ * opaque should be the MIS.
+ */
+static int init_area(const char *block_name, void *host_addr,
+                     ram_addr_t offset, ram_addr_t length, void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+
+    DPRINTF("init_area: %s: %p offset=%zx length=%zd(%zx)",
+            block_name, host_addr, offset, length, length);
+    /*
+     * We need the whole of RAM to be truly empty for postcopy, so things
+     * like ROMs and any data tables built during init must be zero'd
+     * - we're going to get the copy from the source anyway.
+     */
+    if (postcopy_ram_discard_range(mis, host_addr, (host_addr + length - 1))) {
+        return -1;
+    }
+
+    /*
+     * We also need the area to be normal 4k pages, not huge pages
+     * (otherwise we can't be sure we can use remap_anon_pages to put
+     * a 4k page in later).  THP might come along and map a 2MB page
+     * and when it's partially accessed in precopy it might not break
+     * it down, but leave a 2MB zero'd page.
+     */
Quoted 8 lines

arch_init.c

Quoted 4 lines
arch_init.c +4 -0
+    if (madvise(host_addr, length, MADV_NOHUGEPAGE)) {
+        perror("init_area: NOHUGEPAGE");
+        return -1;
+    }

I'm assuming this is because remap_anon_pages() can't automatically split a THP itself. It's not immediately obvious to me why it can't though.

No, I think this restriction stems from two things:
1) remap_anon_pages not allowing us to map into an area that's already got a page present - it's a good protection mechanism against us doing something stupid and receiving a page that we already received and the destination is busy accessing.

2) We wouldn't want THP to decide to convert a page that we'd only partially received into a HP because we wouldn't then receive userfault messages for it.

(Although it might be best to check with Andrea).
(1) might disappear with the modifications to replace remap_anon_pages that Andrea is working on.

Quoted 2 lines

Also.. what effect will this have on an actual hugetlbfs memory region? If there's code to handle that case I haven't spotted it yet.

I wouldn't expect this code to work with hugetlbfs mappings.

Quoted 6 lines
Quoted 6 lines
arch_init.c +6 -0
+
+    return 0;
+}
+
+/*
+ * At the end of migration, undo the effects of init_area
Quoted 8 lines

arch_init.c

Quoted 6 lines

+ * opaque should be the MIS.
+ */

arch_init.c +4 -0
+static int cleanup_area(const char *block_name, void *host_addr,
+                        ram_addr_t offset, ram_addr_t length, void *opaque)
+{
+    /* Turn off userfault here as well? */

This comment appears to be obsoleted by the code below.

Thanks; I've squashed it.

Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#111 lore raw

PATCH v4 32/47 postcopy: ram_enable_notify to switch on userfault

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h    |  2 ++
 include/migration/postcopy-ram.h |  6 +++++
 postcopy-ram.c                   | 49 +++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |  9 ++++++++
 4 files changed, 65 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index be63c89..b01cc17 100644
@@ -87,6 +87,8 @@ struct MigrationIncomingState {
         POSTCOPY_RAM_INCOMING_END
     } postcopy_ram_state;
 
+    /* For the kernel to send us notifications */
+    int            userfault_fd;
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
     PostcopyPMI    postcopy_pmi;
include/migration/postcopy-ram.h +6 -0
index 8f237a2..413b670 100644
@@ -19,6 +19,12 @@
 int postcopy_ram_hosttest(void);
 
 /*
+ * Make all of RAM sensitive to accesses to areas that haven't yet been written
+ * and wire up anything necessary to deal with it.
+ */
+int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+
+/*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
  * postcopy later; must be called prior to any precopy.
  * called from arch_init's similarly named ram_postcopy_incoming_init
postcopy-ram.c +48 -1

63 lines not on this page · raw message

savevm.c +9 -0
index 54bdb26..859c96f 100644
@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis)
 
     mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_LISTENING;
 
+    /*
+     * Sensitise RAM - can now generate requests for blocks that don't exist
+     * However, at this point the CPU shouldn't be running, and the IO
+     * shouldn't be doing anything yet so don't actually expect requests
+     */
+    if (postcopy_ram_enable_notify(mis)) {
+        return -1;
+    }
+
     /* TODO start up the postcopy listening thread */
     return 0;
 }
Signature · 1 line
1.9.3
#112 lore raw

Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 499

Quoted 3 lines
postcopy-ram.c +3 -0
+static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
+                                       ram_addr_t offset, ram_addr_t length,
+                                       void *opaque)

Weird name, and I'm not referring to the British -ise. :)

Perhaps ram_block_enable_userfault or ram_block_enable_notify? It helps clarity to limit the use of the "postcopy_ram_" prefix for static function.

Paolo

#113 lore raw

Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

Quoted 9 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 499

Quoted 3 lines
postcopy-ram.c +3 -0
+static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
+                                       ram_addr_t offset, ram_addr_t length,
+                                       void *opaque)

Weird name, and I'm not referring to the British -ise. :)

Perhaps ram_block_enable_userfault or ram_block_enable_notify? It helps clarity to limit the use of the "postcopy_ram_" prefix for static function.

Yep, that's fair enough; I'll make it ram_block_enable_notify.

Dave

Quoted 1 line

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#114 lore raw

Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

In reply to Dr. David Alan Gilbert (git)
Quoted 45 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h    |  2 ++
 include/migration/postcopy-ram.h |  6 +++++
 postcopy-ram.c                   | 49 +++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |  9 ++++++++
 4 files changed, 65 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index be63c89..b01cc17 100644
@@ -87,6 +87,8 @@ struct MigrationIncomingState {
         POSTCOPY_RAM_INCOMING_END
     } postcopy_ram_state;
 
+    /* For the kernel to send us notifications */
+    int            userfault_fd;
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
     PostcopyPMI    postcopy_pmi;
include/migration/postcopy-ram.h +6 -0
index 8f237a2..413b670 100644
@@ -19,6 +19,12 @@
 int postcopy_ram_hosttest(void);
 
 /*
+ * Make all of RAM sensitive to accesses to areas that haven't yet been written
+ * and wire up anything necessary to deal with it.
+ */
+int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+
+/*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
  * postcopy later; must be called prior to any precopy.
  * called from arch_init's similarly named ram_postcopy_incoming_init
postcopy-ram.c +0 -0
index 8eccf26..925ac77 100644

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494

Quoted 10 lines
postcopy-ram.c +7 -0
@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
     return 0;
 }
 
+/*
+ * Mark the given area of RAM as requiring notification to unwritten areas
+ * Used as a  callback on qemu_ram_foreach_block.
+ *   host_addr: Base of area to mark
+ *   offset: Offset in the whole ram arena
+ *   length: Length of the section
+ *   opaque: Unused
                ^^^^^^

This appears to be wrong - opaque is used to find the MIS.

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502

Quoted 8 lines
postcopy-ram.c +8 -0
+ * Returns 0 on success
+ */
+static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
+                                       ram_addr_t offset, ram_addr_t length,
+                                       void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+    uint64_t tokern[2];

"tokern"?

Quoted 66 lines
postcopy-ram.c +33 -1
+
+    if (madvise(host_addr, length, MADV_USERFAULT)) {
+        perror("postcopy_ram_sensitise_area madvise");
+        return -1;
+    }
+
+    /* Now tell our userfault_fd that it's responsible for this area */
+    tokern[0] = (uint64_t)(uintptr_t)host_addr | 1; /* 1 means register area */
+    tokern[1] = (uint64_t)(uintptr_t)host_addr + length;
+    if (write(mis->userfault_fd, tokern, 16) != 16) {
+        perror("postcopy_ram_sensitise_area write");
+        madvise(host_addr, length, MADV_NOUSERFAULT);
+        return -1;
+    }
+
+    return 0;
+}
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    /* Mark so that we get notified of accesses to unwritten areas */
+    if (qemu_ram_foreach_block(postcopy_ram_sensitise_area, mis)) {
+        return -1;
+    }
+
+    return 0;
+}
+
 #else
 /* No target OS support, stubs just fail */
-
 int postcopy_ram_hosttest(void)
 {
     error_report("postcopy_ram_hosttest: No OS support");
@@ -528,6 +570,11 @@ int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
 {
     assert(0);
 }
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    assert(0);
+}
 #endif
 
 /* ------------------------------------------------------------------------- */
savevm.c +9 -0
index 54bdb26..859c96f 100644
@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis)
 
     mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_LISTENING;
 
+    /*
+     * Sensitise RAM - can now generate requests for blocks that don't exist
+     * However, at this point the CPU shouldn't be running, and the IO
+     * shouldn't be doing anything yet so don't actually expect requests
+     */
+    if (postcopy_ram_enable_notify(mis)) {
+        return -1;
+    }
+
     /* TODO start up the postcopy listening thread */
     return 0;
 }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#115 lore raw

Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

Quoted 58 lines · * David Gibson ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h    |  2 ++
 include/migration/postcopy-ram.h |  6 +++++
 postcopy-ram.c                   | 49 +++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |  9 ++++++++
 4 files changed, 65 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index be63c89..b01cc17 100644
@@ -87,6 +87,8 @@ struct MigrationIncomingState {
         POSTCOPY_RAM_INCOMING_END
     } postcopy_ram_state;
 
+    /* For the kernel to send us notifications */
+    int            userfault_fd;
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
     PostcopyPMI    postcopy_pmi;
include/migration/postcopy-ram.h +6 -0
index 8f237a2..413b670 100644
@@ -19,6 +19,12 @@
 int postcopy_ram_hosttest(void);
 
 /*
+ * Make all of RAM sensitive to accesses to areas that haven't yet been written
+ * and wire up anything necessary to deal with it.
+ */
+int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+
+/*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
  * postcopy later; must be called prior to any precopy.
  * called from arch_init's similarly named ram_postcopy_incoming_init
postcopy-ram.c +0 -0
index 8eccf26..925ac77 100644

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494

Quoted 10 lines
postcopy-ram.c +7 -0
@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
     return 0;
 }
 
+/*
+ * Mark the given area of RAM as requiring notification to unwritten areas
+ * Used as a  callback on qemu_ram_foreach_block.
+ *   host_addr: Base of area to mark
+ *   offset: Offset in the whole ram arena
+ *   length: Length of the section
+ *   opaque: Unused
                ^^^^^^
Quoted 1 line

This appears to be wrong - opaque is used to find the MIS.

Fixed.

Quoted 11 lines

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502

Quoted 9 lines
postcopy-ram.c +8 -0
+ * Returns 0 on success
+ */
+static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
+                                       ram_addr_t offset, ram_addr_t length,
+                                       void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+    uint64_t tokern[2];

"tokern"?

Now "to_kernel"

Dave

Quoted 55 lines
Quoted 55 lines
postcopy-ram.c +33 -1
+
+    if (madvise(host_addr, length, MADV_USERFAULT)) {
+        perror("postcopy_ram_sensitise_area madvise");
+        return -1;
+    }
+
+    /* Now tell our userfault_fd that it's responsible for this area */
+    tokern[0] = (uint64_t)(uintptr_t)host_addr | 1; /* 1 means register area */
+    tokern[1] = (uint64_t)(uintptr_t)host_addr + length;
+    if (write(mis->userfault_fd, tokern, 16) != 16) {
+        perror("postcopy_ram_sensitise_area write");
+        madvise(host_addr, length, MADV_NOUSERFAULT);
+        return -1;
+    }
+
+    return 0;
+}
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    /* Mark so that we get notified of accesses to unwritten areas */
+    if (qemu_ram_foreach_block(postcopy_ram_sensitise_area, mis)) {
+        return -1;
+    }
+
+    return 0;
+}
+
 #else
 /* No target OS support, stubs just fail */
-
 int postcopy_ram_hosttest(void)
 {
     error_report("postcopy_ram_hosttest: No OS support");
@@ -528,6 +570,11 @@ int postcopy_ram_discard_range(MigrationIncomingState *mis, uint8_t *start,
 {
     assert(0);
 }
+
+int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+{
+    assert(0);
+}
 #endif
 
 /* ------------------------------------------------------------------------- */
savevm.c +0 -0
index 54bdb26..859c96f 100644
@@ -1304,6 +1304,15 @@ static int loadvm_postcopy_ram_handle_listen(MigrationIncomingState *mis)
 
     mis->postcopy_ram_state = POSTCOPY_RAM_INCOMING_LISTENING;
 
Quoted 18 lines
Quoted 12 lines
savevm.c +9 -0
+    /*
+     * Sensitise RAM - can now generate requests for blocks that don't exist
+     * However, at this point the CPU shouldn't be running, and the IO
+     * shouldn't be doing anything yet so don't actually expect requests
+     */
+    if (postcopy_ram_enable_notify(mis)) {
+        return -1;
+    }
+
     /* TODO start up the postcopy listening thread */
     return 0;
 }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#116 lore raw

Re: [Qemu-devel] [PATCH v4 32/47] postcopy: ram_enable_notify to switch on userfault

Quoted 75 lines · Dr. David Alan Gilbert
Quoted 58 lines · * David Gibson ([email protected]) wrote:
Quoted 45 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h    |  2 ++
 include/migration/postcopy-ram.h |  6 +++++
 postcopy-ram.c                   | 49 +++++++++++++++++++++++++++++++++++++++-
 savevm.c                         |  9 ++++++++
 4 files changed, 65 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
index be63c89..b01cc17 100644
@@ -87,6 +87,8 @@ struct MigrationIncomingState {
         POSTCOPY_RAM_INCOMING_END
     } postcopy_ram_state;
 
+    /* For the kernel to send us notifications */
+    int            userfault_fd;
     QEMUFile *return_path;
     QemuMutex      rp_mutex;    /* We send replies from multiple threads */
     PostcopyPMI    postcopy_pmi;
include/migration/postcopy-ram.h +6 -0
index 8f237a2..413b670 100644
@@ -19,6 +19,12 @@
 int postcopy_ram_hosttest(void);
 
 /*
+ * Make all of RAM sensitive to accesses to areas that haven't yet been written
+ * and wire up anything necessary to deal with it.
+ */
+int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+
+/*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
  * postcopy later; must be called prior to any precopy.
  * called from arch_init's similarly named ram_postcopy_incoming_init
postcopy-ram.c +0 -0
index 8eccf26..925ac77 100644

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 494

Quoted 10 lines
postcopy-ram.c +7 -0
@@ -485,9 +485,51 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
     return 0;
 }
 
+/*
+ * Mark the given area of RAM as requiring notification to unwritten areas
+ * Used as a  callback on qemu_ram_foreach_block.
+ *   host_addr: Base of area to mark
+ *   offset: Offset in the whole ram arena
+ *   length: Length of the section
+ *   opaque: Unused
                ^^^^^^
Quoted 1 line

This appears to be wrong - opaque is used to find the MIS.

Fixed.

Quoted 11 lines

postcopy-ram.c int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) · line 502

Quoted 9 lines
postcopy-ram.c +8 -0
+ * Returns 0 on success
+ */
+static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
+                                       ram_addr_t offset, ram_addr_t length,
+                                       void *opaque)
+{
+    MigrationIncomingState *mis = opaque;
+    uint64_t tokern[2];

"tokern"?

Quoted 1 line

Now "to_kernel"

Ah! I thought it was just mispelled "token".

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#117 lore raw

PATCH v4 33/47 Postcopy: Postcopy startup in migration thread

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +3 -0
index b01cc17..f401775 100644
@@ -125,6 +125,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     volatile bool start_postcopy;
 
+    /* Flag set once the migration thread is running (and needs joining) */
+    volatile bool started_migration_thread;
+
     /* bitmap of pages that have been sent at least once
      * only maintained and used in postcopy at the moment
      * where it's used to send the dirtymap at the start
migration.c +182 -19

274 lines not on this page · raw message

Signature · 1 line
1.9.3
#118 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 15 lines

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +0 -0
index b01cc17..f401775 100644

include/migration/migration.h struct MigrationState · line 129

Quoted 5 lines
include/migration/migration.h +2 -0
@@ -125,6 +125,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     volatile bool start_postcopy;
 
+    /* Flag set once the migration thread is running (and needs joining) */
+    volatile bool started_migration_thread;

volatile almost never does what you think it does. :)

In this case, I think only one thread reads/writes the variable so "volatile" is unnecessary.

Otherwise, you would need to add actual memory barriers, atomic operations, or synchronization primitives.

For start_postcopy, it is okay because it is just a hint to the compiler and the processor will eventually see the assignment. For this case QEMU has atomic_read/atomic_set (corresponding to __ATOMIC_RELAXED in C/C++1x), so you could use those as well.

Quoted 47 lines
include/migration/migration.h +0 -0
     /* bitmap of pages that have been sent at least once
      * only maintained and used in postcopy at the moment
      * where it's used to send the dirtymap at the start
migration.c +12 -3
index 63d70b6..1731017 100644
@@ -475,7 +475,10 @@ static void migrate_fd_cleanup(void *opaque)
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
-        qemu_thread_join(&s->thread);
+        if (s->started_migration_thread) {
+            qemu_thread_join(&s->thread);
+            s->started_migration_thread = false;
+        }
         qemu_mutex_lock_iothread();
 
         qemu_fclose(s->file);
@@ -872,7 +875,6 @@ out:
     return NULL;
 }
 
-__attribute__ (( unused )) /* Until later in patch series */
 static int open_outgoing_return_path(MigrationState *ms)
 {
 
@@ -890,7 +892,6 @@ static int open_outgoing_return_path(MigrationState *ms)
     return 0;
 }
 
-__attribute__ (( unused )) /* Until later in patch series */
 static void await_outgoing_return_path_close(MigrationState *ms)
 {
     /*
@@ -908,6 +909,97 @@ static void await_outgoing_return_path_close(MigrationState *ms)
     DPRINTF("%s: Exit", __func__);
 }
 
+/* Switch from normal iteration to postcopy
+ * Returns non-0 on error
+ */
+static int postcopy_start(MigrationState *ms)
+{
+    int ret;
+    const QEMUSizedBuffer *qsb;
+    migrate_set_state(ms, MIG_STATE_ACTIVE, MIG_STATE_POSTCOPY_ACTIVE);

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 929

Quoted 10 lines
migration.c +10 -0
+
+    DPRINTF("postcopy_start\n");
+    qemu_mutex_lock_iothread();
+    DPRINTF("postcopy_start: setting run state\n");
+    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+
+    if (ret < 0) {
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;

Please use "goto" for error returns, like

fail_locked:

    qemu_mutex_unlock_iothread();
fail:
    migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
    return -1;
Quoted 7 lines
migration.c +7 -0
+    }
+
+    /*
+     * in Finish migrate and with the io-lock held everything should
+     * be quiet, but we've potentially still got dirty pages and we
+     * need to tell the destination to throw any pages it's already received
+     * that are dirty

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 946

Quoted 10 lines
migration.c +10 -0
+     */
+    if (ram_postcopy_send_discard_bitmap(ms)) {
+        DPRINTF("postcopy send discard bitmap failed\n");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    DPRINTF("postcopy_start: sending req 2\n");
+    qemu_savevm_send_reqack(ms->file, 2);

Perhaps move it below qemu_file_set_rate_limit, and add trace_qemu_savevm_send_reqack?

Also what is 2/3/4? Is this just for debugging or is it part of the protocol?

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 959

Quoted 13 lines
migration.c +13 -0
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);
+    DPRINTF("postcopy_start: do state_complete\n");
+
+    /*
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */

The comments in the code are very nice. Thanks. This is a huge improvement from the last version I received.

Quoted 14 lines
migration.c +14 -0
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    qemu_savevm_state_complete(fb);
+    DPRINTF("postcopy_start: sending req 3\n");
+    qemu_savevm_send_reqack(fb, 3);
+
+    qemu_savevm_send_postcopy_ram_run(fb);
+

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 983

Quoted 10 lines
migration.c +10 -0
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
+    if (qsb_get_length(qsb) > MAX_VM_CMD_PACKAGED_SIZE) {
+        DPRINTF("postcopy_start: Unreasonably large packaged state: %lu\n",
+                (unsigned long)(qsb_get_length(qsb)));
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        qemu_fclose(fb);

Close fb above migrate_set_state, and use goto as above. Or just have three labels.

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 996

Quoted 13 lines
migration.c +13 -0
+        return -1;
+    }
+    qemu_savevm_send_packaged(ms->file, qsb);
+    qemu_fclose(fb);
+
+    qemu_mutex_unlock_iothread();
+
+    DPRINTF("postcopy_start not finished sending ack\n");
+    qemu_savevm_send_reqack(ms->file, 4);
+
+    ret = qemu_file_get_error(ms->file);
+    if (ret) {
+        error_report("postcopy_start: Migration stream errored");

This should have been reported already.

Quoted 30 lines
migration.c +14 -0
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+    }
+
+    return ret;
+}
+
 /*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
@@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
+
     bool old_vm_running = false;
 
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationPhase current_active_type = MIG_STATE_ACTIVE;
+
     qemu_savevm_state_begin(s->file, &s->params);
 
+    if (migrate_postcopy_ram()) {
+        /* Now tell the dest that it should open it's end so it can reply */
+        qemu_savevm_send_openrp(s->file);

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1036

Quoted 10 lines
migration.c +10 -0
+
+        /* And ask it to send an ack that will make stuff easier to debug */
+        qemu_savevm_send_reqack(s->file, 1);
+
+        /* Tell the destination that we *might* want to do postcopy later;
+         * if the other end can't do postcopy it should fail now, nice and
+         * early.
+         */
+        qemu_savevm_send_postcopy_ram_advise(s->file);
+    }

Should this be done here or in the save_state_begin function for RAM? In general, I'm curious if there are parts of postcopy_start that could/should be changed into new save state functions (with postcopy_start just iterating on all devices).

Quoted 67 lines
migration.c +41 -11
     s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
+    current_active_type = MIG_STATE_ACTIVE;
     migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ACTIVE);
 
     DPRINTF("setup complete\n");
@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)
                     " nonpost=%" PRIu64 ")\n",
                     pending_size, max_size, pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
+                /* Still a significant amount to transfer */
+
+                current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                if (migrate_postcopy_ram() &&
+                    s->state != MIG_STATE_POSTCOPY_ACTIVE &&
+                    pend_nonpost == 0 && s->start_postcopy) {
+
+                    if (!postcopy_start(s)) {
+                        current_active_type = MIG_STATE_POSTCOPY_ACTIVE;
+                    }
+
+                    continue;
+                }
+                /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
                 int ret;
 
-                DPRINTF("done iterating\n");
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
+                DPRINTF("done iterating pending size %" PRIu64 "\n",
+                        pending_size);
+
+                if (s->state == MIG_STATE_ACTIVE) {
+                    qemu_mutex_lock_iothread();
+                    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                    qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+                    old_vm_running = runstate_is_running();
+
+                    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+                    if (ret >= 0) {
+                        qemu_file_set_rate_limit(s->file, INT64_MAX);
+                        qemu_savevm_state_complete(s->file);
+                    }
+                    qemu_mutex_unlock_iothread();
+
+                    if (ret < 0) {
+                        migrate_set_state(s, current_active_type,
+                                          MIG_STATE_ERROR);
+                        break;
+                    }
+                } else if (s->state == MIG_STATE_POSTCOPY_ACTIVE) {
+                    DPRINTF("postcopy end\n");
+
+                    qemu_savevm_state_postcopy_complete(s->file);
+                    DPRINTF("postcopy end after complete\n");
 
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete(s->file);
                 }
-                qemu_mutex_unlock_iothread();
 
-                if (ret < 0) {

migration.c static void *migration_thread(void *opaque) · line 1114

Quoted 10 lines
migration.c +8 -2
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
-                    break;
+                /*
+                 * If rp was opened we must clean up the thread before
+                 * cleaning everything else up.
+                 * Postcopy opens rp if enabled (even if it's not avtivated)
+                 */
+                if (migrate_postcopy_ram()) {
+                    DPRINTF("before rp close");
+                    await_outgoing_return_path_close(s);

Should this be done even if there is an error? Perhaps move it altogether out of the big migration thread while() loop?

migration.c static void *migration_thread(void *opaque) · line 1121

Quoted 9 lines
migration.c +3 -2
+                    DPRINTF("after rp close");
                 }
-
                 if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COMPLETED);
+                    migrate_set_state(s, current_active_type,
+                                      MIG_STATE_COMPLETED);
                     break;
                 }

This "else" is huge, can you extract it into its own function?

Quoted 39 lines
migration.c +16 -1
             }
         }
 
         if (qemu_file_get_error(s->file)) {
-            migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
+            migrate_set_state(s, current_active_type, MIG_STATE_ERROR);
+            DPRINTF("migration_thread: file is in error state\n");
             break;
         }
         current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque)
         }
     }
 
+    DPRINTF("migration_thread: After loop");
     qemu_mutex_lock_iothread();
     if (s->state == MIG_STATE_COMPLETED) {
         int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)
     /* Notify before starting migration thread */
     notifier_list_notify(&migration_state_notifiers, s);
 
+    /* Open the return path; currently for postcopy but other things might
+     * also want it.
+     */
+    if (migrate_postcopy_ram()) {
+        if (open_outgoing_return_path(s)) {
+            error_report("Unable to open return-path for postcopy");
+            migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ERROR);
+            migrate_fd_cleanup(s);
+            return;
+        }
+    }
+
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
+    s->started_migration_thread = true;
 }
#119 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Quoted 19 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 18 lines

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +0 -0
index b01cc17..f401775 100644
@@ -125,6 +125,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     volatile bool start_postcopy;
 
Quoted 4 lines

include/migration/migration.h struct MigrationState · line 129

Quoted 2 lines
include/migration/migration.h +2 -0
+    /* Flag set once the migration thread is running (and needs joining) */
+    volatile bool started_migration_thread;

volatile almost never does what you think it does. :)

True.

Quoted 2 lines

In this case, I think only one thread reads/writes the variable so "volatile" is unnecessary.

Lets just check that; so it's set by 'migrate_fd_connect' (from the main thread) when it spawns the thread, and it's cleared by migrate_fd_cleanup that's always run as a bh, so should always be in the main thread; so yes - always the same thread, that's nice and simple; volatile evaporated.

Quoted 5 lines

Otherwise, you would need to add actual memory barriers, atomic operations, or synchronization primitives.

For start_postcopy, it is okay because it is just a hint to the compiler and the processor will eventually see the assignment.

Yes, in this case my understanding is that it's necessary to stop the compiler potentially moving the check outside the loop.

Quoted 3 lines

For this case
QEMU has atomic_read/atomic_set (corresponding to __ATOMIC_RELAXED in C/C++1x), so you could use those as well.

Ah, so those look like they just volatile cast anyway.

(I've probably got some other flags I need to think about reading/writing atomically/safely).

Dave
(I'll take the other issues in this mail separately since there are quite a few). --
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#120 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Quoted 4 lines · Dr. David Alan Gilbert
Quoted 2 lines

For this case QEMU has atomic_read/atomic_set (corresponding to __ATOMIC_RELAXED in C/C++1x), so you could use those as well.

Ah, so those look like they just volatile cast anyway.

Yeah, but it explicitly shows that the assignment is a) for a multi-threaded operation b) using relaxed semantics. It attaches the information to the use instead of the variable; it just happens that volatile is the pre-C11 way to express those.

Paolo

#121 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Quoted 6 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Quoted 2 lines

For this case QEMU has atomic_read/atomic_set (corresponding to __ATOMIC_RELAXED in C/C++1x), so you could use those as well.

Quoted 6 lines
Quoted 1 line

Ah, so those look like they just volatile cast anyway.

Yeah, but it explicitly shows that the assignment is a) for a multi-threaded operation b) using relaxed semantics. It attaches the information to the use instead of the variable; it just happens that volatile is the pre-C11 way to express those.

OK, I'll use those anyway; Ideally what I'd have is a way to mark something so that it'd compile-time-fail if I didn't use an atomic_ on it, because it's the type of thing that I'm bound to forget somewhere.

Dave

Quoted 2 lines

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#122 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

In reply to Paolo Bonzini
Quoted 15 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 14 lines

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +0 -0
index b01cc17..f401775 100644

<snip>

Quoted 21 lines
Quoted 8 lines
include/migration/migration.h +8 -0
+/* Switch from normal iteration to postcopy
+ * Returns non-0 on error
+ */
+static int postcopy_start(MigrationState *ms)
+{
+    int ret;
+    const QEMUSizedBuffer *qsb;
+    migrate_set_state(ms, MIG_STATE_ACTIVE, MIG_STATE_POSTCOPY_ACTIVE);

include/migration/migration.h

Quoted 10 lines
include/migration/migration.h +10 -0
+
+    DPRINTF("postcopy_start\n");
+    qemu_mutex_lock_iothread();
+    DPRINTF("postcopy_start: setting run state\n");
+    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+
+    if (ret < 0) {
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;

Please use "goto" for error returns, like

Quoted 5 lines

fail_locked:

    qemu_mutex_unlock_iothread();
fail:
    migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
    return -1;

Done; they all end up unlocking, but I've got another label for a case that has to close the fb later.

Quoted 12 lines
Quoted 12 lines
include/migration/migration.h +12 -0
+    }
+
+    /*
+     * in Finish migrate and with the io-lock held everything should
+     * be quiet, but we've potentially still got dirty pages and we
+     * need to tell the destination to throw any pages it's already received
+     * that are dirty
+     */
+    if (ram_postcopy_send_discard_bitmap(ms)) {
+        DPRINTF("postcopy send discard bitmap failed\n");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
Quoted 8 lines

include/migration/migration.h

Quoted 5 lines
include/migration/migration.h +5 -0
+        return -1;
+    }
+
+    DPRINTF("postcopy_start: sending req 2\n");
+    qemu_savevm_send_reqack(ms->file, 2);

Perhaps move it below qemu_file_set_rate_limit, and add trace_qemu_savevm_send_reqack?

Trace added, and also moved as requested - was the request to move it just to elimintate the other DPRINTF?

Quoted 2 lines

Also what is 2/3/4? Is this just for debugging or is it part of the protocol?

Debug; they're very useful for matching the debug streams up, especially when the timers on the two hosts are very different.
(I'm up for suggestions on how to mark the 2/3/4 for debug more clearly, especially if it meant that it didn't make the ping (ne reqack) dedicated to debug).

Quoted 36 lines

include/migration/migration.h

Quoted 13 lines
include/migration/migration.h +13 -0
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);
+    DPRINTF("postcopy_start: do state_complete\n");
+
+    /*
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */

The comments in the code are very nice. Thanks. This is a huge improvement from the last version I received.

Quoted 19 lines
include/migration/migration.h +19 -0
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    qemu_savevm_state_complete(fb);
+    DPRINTF("postcopy_start: sending req 3\n");
+    qemu_savevm_send_reqack(fb, 3);
+
+    qemu_savevm_send_postcopy_ram_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
+    if (qsb_get_length(qsb) > MAX_VM_CMD_PACKAGED_SIZE) {
Quoted 8 lines

include/migration/migration.h

Quoted 5 lines
include/migration/migration.h +5 -0
+        DPRINTF("postcopy_start: Unreasonably large packaged state: %lu\n",
+                (unsigned long)(qsb_get_length(qsb)));
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        qemu_fclose(fb);

Close fb above migrate_set_state, and use goto as above. Or just have three labels.

Done, it's a separate label.

Quoted 16 lines
Quoted 4 lines
include/migration/migration.h +3 -0
+        return -1;
+    }
+    qemu_savevm_send_packaged(ms->file, qsb);

include/migration/migration.h

Quoted 10 lines
include/migration/migration.h +10 -0
+    qemu_fclose(fb);
+
+    qemu_mutex_unlock_iothread();
+
+    DPRINTF("postcopy_start not finished sending ack\n");
+    qemu_savevm_send_reqack(ms->file, 4);
+
+    ret = qemu_file_get_error(ms->file);
+    if (ret) {
+        error_report("postcopy_start: Migration stream errored");

This should have been reported already.

No, sorry - I don't trust qemu_file reporting errors by itself.

Dave
(Again, the rest of the comments on this patch can wait for another mail)

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#123 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Quoted 2 lines · Dr. David Alan Gilbert

Trace added, and also moved as requested - was the request to move it just to elimintate the other DPRINTF?

Yes.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Also what is 2/3/4? Is this just for debugging or is it part of the protocol?

Debug; they're very useful for matching the debug streams up, especially when the timers on the two hosts are very different.
(I'm up for suggestions on how to mark the 2/3/4 for debug more clearly, especially if it meant that it didn't make the ping (ne reqack) dedicated to debug).

No problem, as long as it's clear to the guy matching the code against the debug output.

Paolo

#124 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

In reply to Paolo Bonzini
Quoted 40 lines · * Paolo Bonzini ([email protected]) wrote:

Il 03/10/2014 19:47, Dr. David Alan Gilbert (git) ha scritto:

Quoted 39 lines

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
migration.c +15 -0
@@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;
+
     bool old_vm_running = false;
 
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationPhase current_active_type = MIG_STATE_ACTIVE;
+
     qemu_savevm_state_begin(s->file, &s->params);
 
+    if (migrate_postcopy_ram()) {
+        /* Now tell the dest that it should open it's end so it can reply */
+        qemu_savevm_send_openrp(s->file);
+
+        /* And ask it to send an ack that will make stuff easier to debug */
+        qemu_savevm_send_reqack(s->file, 1);
+
+        /* Tell the destination that we *might* want to do postcopy later;
+         * if the other end can't do postcopy it should fail now, nice and
+         * early.
Quoted 8 lines

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1036

Quoted 3 lines
migration.c +3 -0
+         */
+        qemu_savevm_send_postcopy_ram_advise(s->file);
+    }

Should this be done here or in the save_state_begin function for RAM? In general, I'm curious if there are parts of postcopy_start that could/should be changed into new save state functions (with postcopy_start just iterating on all devices).

The contents of this 'if' are generic to whatever is being postcopied, (and as per one of your other comments the _ram_ has been removed from the send_postcopy_ram_advise); so I think this is the right place for it.

Quoted 66 lines
Quoted 66 lines
migration.c +41 -10
     s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
+    current_active_type = MIG_STATE_ACTIVE;
     migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ACTIVE);
 
     DPRINTF("setup complete\n");
@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)
                     " nonpost=%" PRIu64 ")\n",
                     pending_size, max_size, pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
+                /* Still a significant amount to transfer */
+
+                current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                if (migrate_postcopy_ram() &&
+                    s->state != MIG_STATE_POSTCOPY_ACTIVE &&
+                    pend_nonpost == 0 && s->start_postcopy) {
+
+                    if (!postcopy_start(s)) {
+                        current_active_type = MIG_STATE_POSTCOPY_ACTIVE;
+                    }
+
+                    continue;
+                }
+                /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
                 int ret;
 
-                DPRINTF("done iterating\n");
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
+                DPRINTF("done iterating pending size %" PRIu64 "\n",
+                        pending_size);
+
+                if (s->state == MIG_STATE_ACTIVE) {
+                    qemu_mutex_lock_iothread();
+                    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                    qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+                    old_vm_running = runstate_is_running();
+
+                    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+                    if (ret >= 0) {
+                        qemu_file_set_rate_limit(s->file, INT64_MAX);
+                        qemu_savevm_state_complete(s->file);
+                    }
+                    qemu_mutex_unlock_iothread();
+
+                    if (ret < 0) {
+                        migrate_set_state(s, current_active_type,
+                                          MIG_STATE_ERROR);
+                        break;
+                    }
+                } else if (s->state == MIG_STATE_POSTCOPY_ACTIVE) {
+                    DPRINTF("postcopy end\n");
+
+                    qemu_savevm_state_postcopy_complete(s->file);
+                    DPRINTF("postcopy end after complete\n");
 
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete(s->file);
                 }
-                qemu_mutex_unlock_iothread();
 
Quoted 14 lines

migration.c static void *migration_thread(void *opaque) · line 1114

Quoted 11 lines
migration.c +8 -3
-                if (ret < 0) {
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
-                    break;
+                /*
+                 * If rp was opened we must clean up the thread before
+                 * cleaning everything else up.
+                 * Postcopy opens rp if enabled (even if it's not avtivated)
+                 */
+                if (migrate_postcopy_ram()) {
+                    DPRINTF("before rp close");
+                    await_outgoing_return_path_close(s);

Should this be done even if there is an error? Perhaps move it altogether out of the big migration thread while() loop?

Yes, I've made a note of that; I need to go and look at more error cases to see where it makes sense (e.g. the one above), however in the non-error case I do want it to wait here for the 'SHUT' from the destination to indicate that the destination believes migration completed correctly (or not), and that should happen before the state gets set to COMPLETED because we're waiting.

Quoted 11 lines

migration.c static void *migration_thread(void *opaque) · line 1121

Quoted 9 lines
migration.c +3 -2
+                    DPRINTF("after rp close");
                 }
-
                 if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COMPLETED);
+                    migrate_set_state(s, current_active_type,
+                                      MIG_STATE_COMPLETED);
                     break;
                 }

This "else" is huge, can you extract it into its own function?

Done, and all the changes inside this "else" I've moved into a separate commit that deals with the end rather than the start of postcopy.

Dave

Quoted 32 lines
Quoted 32 lines
migration.c +12 -1
             }
         }
 
         if (qemu_file_get_error(s->file)) {
-            migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
+            migrate_set_state(s, current_active_type, MIG_STATE_ERROR);
+            DPRINTF("migration_thread: file is in error state\n");
             break;
         }
         current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque)
         }
     }
 
+    DPRINTF("migration_thread: After loop");
     qemu_mutex_lock_iothread();
     if (s->state == MIG_STATE_COMPLETED) {
         int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)
     /* Notify before starting migration thread */
     notifier_list_notify(&migration_state_notifiers, s);
 
+    /* Open the return path; currently for postcopy but other things might
+     * also want it.
+     */
+    if (migrate_postcopy_ram()) {
+        if (open_outgoing_return_path(s)) {
+            error_report("Unable to open return-path for postcopy");
+            migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ERROR);
+            migrate_fd_cleanup(s);
+            return;
Quoted 9 lines

migration.c void migrate_fd_connect(MigrationState *s) · line 1211

Quoted 8 lines
migration.c +4 -0
+        }
+    }
+
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
+    s->started_migration_thread = true;
 }

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#125 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

In reply to Dr. David Alan Gilbert (git)
Quoted 85 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +3 -0
index b01cc17..f401775 100644
@@ -125,6 +125,9 @@ struct MigrationState
     /* Flag set once the migration has been asked to enter postcopy */
     volatile bool start_postcopy;
 
+    /* Flag set once the migration thread is running (and needs joining) */
+    volatile bool started_migration_thread;
+
     /* bitmap of pages that have been sent at least once
      * only maintained and used in postcopy at the moment
      * where it's used to send the dirtymap at the start
migration.c +29 -3
index 63d70b6..1731017 100644
@@ -475,7 +475,10 @@ static void migrate_fd_cleanup(void *opaque)
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
-        qemu_thread_join(&s->thread);
+        if (s->started_migration_thread) {
+            qemu_thread_join(&s->thread);
+            s->started_migration_thread = false;
+        }
         qemu_mutex_lock_iothread();
 
         qemu_fclose(s->file);
@@ -872,7 +875,6 @@ out:
     return NULL;
 }
 
-__attribute__ (( unused )) /* Until later in patch series */
 static int open_outgoing_return_path(MigrationState *ms)
 {
 
@@ -890,7 +892,6 @@ static int open_outgoing_return_path(MigrationState *ms)
     return 0;
 }
 
-__attribute__ (( unused )) /* Until later in patch series */
 static void await_outgoing_return_path_close(MigrationState *ms)
 {
     /*
@@ -908,6 +909,97 @@ static void await_outgoing_return_path_close(MigrationState *ms)
     DPRINTF("%s: Exit", __func__);
 }
 
+/* Switch from normal iteration to postcopy
+ * Returns non-0 on error
+ */
+static int postcopy_start(MigrationState *ms)
+{
+    int ret;
+    const QEMUSizedBuffer *qsb;
+    migrate_set_state(ms, MIG_STATE_ACTIVE, MIG_STATE_POSTCOPY_ACTIVE);
+
+    DPRINTF("postcopy_start\n");
+    qemu_mutex_lock_iothread();
+    DPRINTF("postcopy_start: setting run state\n");
+    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+
+    if (ret < 0) {
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    /*
+     * in Finish migrate and with the io-lock held everything should
+     * be quiet, but we've potentially still got dirty pages and we
+     * need to tell the destination to throw any pages it's already received
+     * that are dirty

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 946

Quoted 10 lines
migration.c +10 -0
+     */
+    if (ram_postcopy_send_discard_bitmap(ms)) {
+        DPRINTF("postcopy send discard bitmap failed\n");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    DPRINTF("postcopy_start: sending req 2\n");
+    qemu_savevm_send_reqack(ms->file, 2);

Are these reqacks just for debugging, or do they affect the protocol?

Quoted 69 lines
migration.c +57 -0
+    /*
+     * send rest of state - note things that are doing postcopy
+     * will notice we're in MIG_STATE_POSTCOPY_ACTIVE and not actually
+     * wrap their state up here
+     */
+    qemu_file_set_rate_limit(ms->file, INT64_MAX);
+    DPRINTF("postcopy_start: do state_complete\n");
+
+    /*
+     * We need to leave the fd free for page transfers during the
+     * loading of the device state, so wrap all the remaining
+     * commands and state into a package that gets sent in one go
+     */
+    QEMUFile *fb = qemu_bufopen("w", NULL);
+    if (!fb) {
+        error_report("Failed to create buffered file");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        return -1;
+    }
+
+    qemu_savevm_state_complete(fb);
+    DPRINTF("postcopy_start: sending req 3\n");
+    qemu_savevm_send_reqack(fb, 3);
+
+    qemu_savevm_send_postcopy_ram_run(fb);
+
+    /* <><> end of stuff going into the package */
+    qsb = qemu_buf_get(fb);
+
+    /* Now send that blob */
+    if (qsb_get_length(qsb) > MAX_VM_CMD_PACKAGED_SIZE) {
+        DPRINTF("postcopy_start: Unreasonably large packaged state: %lu\n",
+                (unsigned long)(qsb_get_length(qsb)));
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+        qemu_mutex_unlock_iothread();
+        qemu_fclose(fb);
+        return -1;
+    }
+    qemu_savevm_send_packaged(ms->file, qsb);
+    qemu_fclose(fb);
+
+    qemu_mutex_unlock_iothread();
+
+    DPRINTF("postcopy_start not finished sending ack\n");
+    qemu_savevm_send_reqack(ms->file, 4);
+
+    ret = qemu_file_get_error(ms->file);
+    if (ret) {
+        error_report("postcopy_start: Migration stream errored");
+        migrate_set_state(ms, MIG_STATE_POSTCOPY_ACTIVE, MIG_STATE_ERROR);
+    }
+
+    return ret;
+}
+
 /*
  * Master migration thread on the source VM.
  * It drives the migration and pumps the data down the outgoing channel.
@@ -915,16 +1007,36 @@ static void await_outgoing_return_path_close(MigrationState *ms)
 static void *migration_thread(void *opaque)
 {
     MigrationState *s = opaque;
+    /* Used by the bandwidth calcs, updated later */
     int64_t initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     int64_t setup_start = qemu_clock_get_ms(QEMU_CLOCK_HOST);
     int64_t initial_bytes = 0;
     int64_t max_size = 0;
     int64_t start_time = initial_time;

migration.c static void await_outgoing_return_path_close(MigrationState *ms) · line 1025

Quoted 10 lines
migration.c +6 -0
+
     bool old_vm_running = false;
 
+    /* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
+    enum MigrationPhase current_active_type = MIG_STATE_ACTIVE;
+
     qemu_savevm_state_begin(s->file, &s->params);
 
+    if (migrate_postcopy_ram()) {
+        /* Now tell the dest that it should open it's end so it can reply */

s/it's/its/

Quoted 18 lines
migration.c +13 -0
+        qemu_savevm_send_openrp(s->file);
+
+        /* And ask it to send an ack that will make stuff easier to debug */
+        qemu_savevm_send_reqack(s->file, 1);
+
+        /* Tell the destination that we *might* want to do postcopy later;
+         * if the other end can't do postcopy it should fail now, nice and
+         * early.
+         */
+        qemu_savevm_send_postcopy_ram_advise(s->file);
+    }
+
     s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
+    current_active_type = MIG_STATE_ACTIVE;
     migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ACTIVE);
 
     DPRINTF("setup complete\n");

migration.c static void *migration_thread(void *opaque) · line 1065

Quoted 9 lines
migration.c +6 -0
@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)
                     " nonpost=%" PRIu64 ")\n",
                     pending_size, max_size, pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
+                /* Still a significant amount to transfer */
+
+                current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                if (migrate_postcopy_ram() &&
+                    s->state != MIG_STATE_POSTCOPY_ACTIVE &&
+                    pend_nonpost == 0 && s->start_postcopy) {

Hrm. This is checking for pend_nonpost == 0, rather than just close to zero. IIUC this will only work if all "live sendable" state is also postcopyable. But if we have live sendable data that's not postcopyable - like the power hash page table - we'll need some threshold here, like we currently have for entering the stopped vm phase of a precopy migration.

Or am I missing something?

Quoted 109 lines
migration.c +61 -16
+
+                    if (!postcopy_start(s)) {
+                        current_active_type = MIG_STATE_POSTCOPY_ACTIVE;
+                    }
+
+                    continue;
+                }
+                /* Just another iteration step */
                 qemu_savevm_state_iterate(s->file);
             } else {
                 int ret;
 
-                DPRINTF("done iterating\n");
-                qemu_mutex_lock_iothread();
-                start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
-                qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
-                old_vm_running = runstate_is_running();
+                DPRINTF("done iterating pending size %" PRIu64 "\n",
+                        pending_size);
+
+                if (s->state == MIG_STATE_ACTIVE) {
+                    qemu_mutex_lock_iothread();
+                    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                    qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
+                    old_vm_running = runstate_is_running();
+
+                    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+                    if (ret >= 0) {
+                        qemu_file_set_rate_limit(s->file, INT64_MAX);
+                        qemu_savevm_state_complete(s->file);
+                    }
+                    qemu_mutex_unlock_iothread();
+
+                    if (ret < 0) {
+                        migrate_set_state(s, current_active_type,
+                                          MIG_STATE_ERROR);
+                        break;
+                    }
+                } else if (s->state == MIG_STATE_POSTCOPY_ACTIVE) {
+                    DPRINTF("postcopy end\n");
+
+                    qemu_savevm_state_postcopy_complete(s->file);
+                    DPRINTF("postcopy end after complete\n");
 
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
-                if (ret >= 0) {
-                    qemu_file_set_rate_limit(s->file, INT64_MAX);
-                    qemu_savevm_state_complete(s->file);
                 }
-                qemu_mutex_unlock_iothread();
 
-                if (ret < 0) {
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
-                    break;
+                /*
+                 * If rp was opened we must clean up the thread before
+                 * cleaning everything else up.
+                 * Postcopy opens rp if enabled (even if it's not avtivated)
+                 */
+                if (migrate_postcopy_ram()) {
+                    DPRINTF("before rp close");
+                    await_outgoing_return_path_close(s);
+                    DPRINTF("after rp close");
                 }
-
                 if (!qemu_file_get_error(s->file)) {
-                    migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COMPLETED);
+                    migrate_set_state(s, current_active_type,
+                                      MIG_STATE_COMPLETED);
                     break;
                 }
             }
         }
 
         if (qemu_file_get_error(s->file)) {
-            migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_ERROR);
+            migrate_set_state(s, current_active_type, MIG_STATE_ERROR);
+            DPRINTF("migration_thread: file is in error state\n");
             break;
         }
         current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1006,6 +1155,7 @@ static void *migration_thread(void *opaque)
         }
     }
 
+    DPRINTF("migration_thread: After loop");
     qemu_mutex_lock_iothread();
     if (s->state == MIG_STATE_COMPLETED) {
         int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1043,6 +1193,19 @@ void migrate_fd_connect(MigrationState *s)
     /* Notify before starting migration thread */
     notifier_list_notify(&migration_state_notifiers, s);
 
+    /* Open the return path; currently for postcopy but other things might
+     * also want it.
+     */
+    if (migrate_postcopy_ram()) {
+        if (open_outgoing_return_path(s)) {
+            error_report("Unable to open return-path for postcopy");
+            migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ERROR);
+            migrate_fd_cleanup(s);
+            return;
+        }
+    }
+
     qemu_thread_create(&s->thread, "migration", migration_thread, s,
                        QEMU_THREAD_JOINABLE);
+    s->started_migration_thread = true;
 }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#126 lore raw

Re: [Qemu-devel] [PATCH v4 33/47] Postcopy: Postcopy startup in migration thread

Quoted 16 lines · * David Gibson ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Rework the migration thread to setup and start postcopy.

---
 include/migration/migration.h |   3 +
 migration.c                   | 201 ++++++++++++++++++++++++++++++++++++++----
 2 files changed, 185 insertions(+), 19 deletions(-)
include/migration/migration.h +0 -0
index b01cc17..f401775 100644
Quoted 4 lines

include/migration/migration.h

Quoted 2 lines
include/migration/migration.h +2 -0
+    DPRINTF("postcopy_start: sending req 2\n");
+    qemu_savevm_send_reqack(ms->file, 2);

Are these reqacks just for debugging, or do they affect the protocol?

Just for debugging; comment added. They just make it easy to line traces up between the source and destination, and also make it easy to figure out how far stuff has got if it jams up.

Quoted 4 lines

include/migration/migration.h

Quoted 2 lines
include/migration/migration.h +2 -0
+    if (migrate_postcopy_ram()) {
+        /* Now tell the dest that it should open it's end so it can reply */

s/it's/its/

Fixed.

Quoted 35 lines
Quoted 18 lines
include/migration/migration.h +13 -0
+        qemu_savevm_send_openrp(s->file);
+
+        /* And ask it to send an ack that will make stuff easier to debug */
+        qemu_savevm_send_reqack(s->file, 1);
+
+        /* Tell the destination that we *might* want to do postcopy later;
+         * if the other end can't do postcopy it should fail now, nice and
+         * early.
+         */
+        qemu_savevm_send_postcopy_ram_advise(s->file);
+    }
+
     s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
+    current_active_type = MIG_STATE_ACTIVE;
     migrate_set_state(s, MIG_STATE_SETUP, MIG_STATE_ACTIVE);
 
     DPRINTF("setup complete\n");

include/migration/migration.h static void *migration_thread(void *opaque) · line 1065

Quoted 9 lines
include/migration/migration.h +6 -0
@@ -945,37 +1057,74 @@ static void *migration_thread(void *opaque)
                     " nonpost=%" PRIu64 ")\n",
                     pending_size, max_size, pend_post, pend_nonpost);
             if (pending_size && pending_size >= max_size) {
+                /* Still a significant amount to transfer */
+
+                current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+                if (migrate_postcopy_ram() &&
+                    s->state != MIG_STATE_POSTCOPY_ACTIVE &&
+                    pend_nonpost == 0 && s->start_postcopy) {

Hrm. This is checking for pend_nonpost == 0, rather than just close to zero. IIUC this will only work if all "live sendable" state is also postcopyable. But if we have live sendable data that's not postcopyable - like the power hash page table - we'll need some threshold here, like we currently have for entering the stopped vm phase of a precopy migration.

Quoted 1 line

Or am I missing something?

Hmm, I think you're right; I've changed this to:
pend_nonpost <= max_size

so that it's the same cut-off logic as the normal end-of-migrate; I think that will work; i.e. it gets small enough to be expected to complete quickly in the _complete phase that's at the start of postcopy.

Dave

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#127 lore raw

PATCH v4 34/47 Postcopy: Create a fault handler thread before marking the ram as userfault

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

---
 include/migration/migration.h |  3 +++
 postcopy-ram.c                | 23 +++++++++++++++++++++++
 2 files changed, 26 insertions(+)
include/migration/migration.h +3 -0
index f401775..cdd0e56 100644
@@ -87,6 +87,9 @@ struct MigrationIncomingState {
         POSTCOPY_RAM_INCOMING_END
     } postcopy_ram_state;
 
+    QemuThread     fault_thread;
+    QemuSemaphore  fault_thread_sem;
+
     /* For the kernel to send us notifications */
     int            userfault_fd;
     QEMUFile *return_path;
postcopy-ram.c +23 -0
index 925ac77..8b2a035 100644
@@ -518,8 +518,31 @@ static int postcopy_ram_sensitise_area(const char *block_name, void *host_addr,
     return 0;
 }
 
+/*
+ * Handle faults detected by the USERFAULT markings
+ */
+static void *postcopy_ram_fault_thread(void *opaque)
+{
+    MigrationIncomingState *mis = (MigrationIncomingState *)opaque;
+
+    fprintf(stderr, "postcopy_ram_fault_thread\n");
+    /* TODO: In later patch */
+    qemu_sem_post(&mis->fault_thread_sem);
+    while (1) {
+        /* TODO: In later patch */
+    }
+
+    return NULL;
+}
+
 int postcopy_ram_enable_notify(MigrationIncomingState *mis)
 {
+    /* Create the fault handler thread and wait for it to be ready */
+    qemu_sem_init(&mis->fault_thread_sem, 0);
+    qemu_thread_create(&mis->fault_thread, "postcopy/fault",
+                       postcopy_ram_fault_thread, mis, QEMU_THREAD_JOINABLE);
+    qemu_sem_wait(&mis->fault_thread_sem);
+
     /* Mark so that we get notified of accesses to unwritten areas */
     if (qemu_ram_foreach_block(postcopy_ram_sensitise_area, mis)) {
         return -1;
Signature · 1 line
1.9.3
Reviewed-by on v4 34/47
#128 lore raw

Re: [Qemu-devel] [PATCH v4 34/47] Postcopy: Create a fault handler thread before marking the ram as userfault

Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This could do with a bit more rationale in the commit message.

Also is there a reason not to fold this with the patch originally marking the RAM as userfault? IIRC that one wasn't partocularly long either.

Otherwise

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
author Applied
#129 lore raw

Re: [Qemu-devel] [PATCH v4 34/47] Postcopy: Create a fault handler thread before marking the ram as userfault

Quoted 8 lines · * David Gibson ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This could do with a bit more rationale in the commit message.

Also is there a reason not to fold this with the patch originally marking the RAM as userfault? IIRC that one wasn't partocularly long either.

I've merged it with 'ram_enable_notify to switch on userfault' and it's commit message now reads.

'Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)'

Dave

Quoted 10 lines

Otherwise

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#130 lore raw

PATCH v4 35/47 Page request: Add MIG_RPCOMM_REQPAGES reverse command

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source.

---
 include/migration/migration.h |  3 ++
 migration.c                   | 74 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
include/migration/migration.h +3 -0
index cdd0e56..5e0d30d 100644
@@ -45,6 +45,7 @@ enum mig_rpcomm_cmd {
     MIG_RPCOMM_INVALID = 0,  /* Must be 0 */
     MIG_RPCOMM_SHUT,         /* sibling will not send any more RP messages */
     MIG_RPCOMM_ACK,          /* data (seq: be32 ) */
+    MIG_RPCOMM_REQPAGES,     /* data (start: be64, len: be64) */
     MIG_RPCOMM_AFTERLASTVALID
 };
 
@@ -250,6 +251,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis,
                           uint32_t value);
 void migrate_send_rp_ack(MigrationIncomingState *mis,
                          uint32_t value);
+void migrate_send_rp_reqpages(MigrationIncomingState *mis, const char* rbname,
+                              ram_addr_t start, ram_addr_t len);
 
 
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
migration.c +74 -0

104 lines not on this page · raw message

Signature · 1 line
1.9.3
#131 lore raw

Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command

Quoted 57 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source.

---
 include/migration/migration.h |  3 ++
 migration.c                   | 74 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
include/migration/migration.h +3 -0
index cdd0e56..5e0d30d 100644
@@ -45,6 +45,7 @@ enum mig_rpcomm_cmd {
     MIG_RPCOMM_INVALID = 0,  /* Must be 0 */
     MIG_RPCOMM_SHUT,         /* sibling will not send any more RP messages */
     MIG_RPCOMM_ACK,          /* data (seq: be32 ) */
+    MIG_RPCOMM_REQPAGES,     /* data (start: be64, len: be64) */
     MIG_RPCOMM_AFTERLASTVALID
 };
 
@@ -250,6 +251,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis,
                           uint32_t value);
 void migrate_send_rp_ack(MigrationIncomingState *mis,
                          uint32_t value);
+void migrate_send_rp_reqpages(MigrationIncomingState *mis, const char* rbname,
+                              ram_addr_t start, ram_addr_t len);
 
 
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
migration.c +17 -0
index 1731017..cfdaa52 100644
@@ -144,6 +144,38 @@ void migrate_send_rp_ack(MigrationIncomingState *mis,
     migrate_send_rp_message(mis, MIG_RPCOMM_ACK, 4, (uint8_t *)&buf);
 }
 
+/* Request a range of pages from the source VM at the given
+ * start address.
+ *   rbname: Name of the RAMBlock to request the page in, if NULL it's the same
+ *           as the last request (a name must have been given previously)
+ *   Start: Address offset within the RB
+ *   Len: Length in bytes required - must be a multiple of pagesize
+ */
+void migrate_send_rp_reqpages(MigrationIncomingState *mis, const char *rbname,
+                              ram_addr_t start, ram_addr_t len)
+{
+    uint8_t bufc[16+1+255]; /* start (8 byte), len (8 byte), rbname upto 256 */
+    uint64_t *buf64 = (uint64_t *)bufc;
+    size_t msglen = 16; /* start + len */
+
+    assert(!(len & 1));
+    if (rbname) {
+        int rbname_len = strlen(rbname);

migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173

Quoted 10 lines
migration.c +10 -0
+        assert(rbname_len < 256);
+
+        len |= 1; /* Flag to say we've got a name */
+        bufc[msglen++] = rbname_len;
+        memcpy(bufc + msglen, rbname, rbname_len);
+        msglen += rbname_len;
+    }
+
+    buf64[0] = (uint64_t)start;
+    buf64[0] = cpu_to_be64(buf64[0]);

I think this would be clearer as well as less verbose, as just:

	buf64[0] = cpu_to_be64(start);
Quoted 27 lines
migration.c +16 -0
+    buf64[1] = (uint64_t)len;
+    buf64[1] = cpu_to_be64(buf64[1]);
+    migrate_send_rp_message(mis, MIG_RPCOMM_REQPAGES, msglen, bufc);
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s)
 }
 
 /*
+ * Process a request for pages received on the return path,
+ * We're allowed to send more than requested (e.g. to round to our page size)
+ * and we don't need to send pages that have already been sent.
+ */
+static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
+                                       ram_addr_t start, ram_addr_t len)
+{
+    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+}
+
+/*
  * Handles messages sent on the return path towards the source VM
  *
  */

migration.c static void *source_return_path_thread(void *opaque) · line 841

Quoted 4 lines
migration.c +1 -0
@@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)
     const int max_len = 512;
     uint8_t buf[max_len];
     uint32_t tmp32;
+    uint64_t tmp64a, tmp64b;

Hrm.. calling everything "tmp*" doesn't help readability.

Quoted 47 lines
migration.c +30 -0
+    char *tmpstr;
     int res;
 
     DPRINTF("RP: %s entry", __func__);
@@ -810,6 +855,11 @@ static void *source_return_path_thread(void *opaque)
             expected_len = 4;
             break;
 
+        case MIG_RPCOMM_REQPAGES:
+            /* 16 byte start/len _possibly_ plus an id str */
+            expected_len = 16 + 256;
+            break;
+
         default:
             error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
                     header_com, header_len);
@@ -857,6 +907,30 @@ static void *source_return_path_thread(void *opaque)
             atomic_xchg(&ms->rp_state.latest_ack, tmp32);
             break;
 
+        case MIG_RPCOMM_REQPAGES:
+            tmp64a = be64_to_cpup((uint64_t *)buf);  /* Start */
+            tmp64b = be64_to_cpup(((uint64_t *)buf)+1); /* Len */
+            tmpstr = NULL;
+            if (tmp64b & 1) {
+                tmp64b -= 1; /* Remove the flag */
+                /* Now we expect an idstr */
+                tmp32 = buf[16]; /* Length of the following idstr */
+                tmpstr = (char *)&buf[17];
+                buf[17+tmp32] = '\0';
+                expected_len = 16+1+tmp32;
+            } else {
+                expected_len = 16;
+            }
+            if (header_len != expected_len) {
+                error_report("RP: Received ReqPage with length %d expecting %d",
+                        header_len, expected_len);
+                source_return_path_bad(ms);
+            }
+            migrate_handle_rp_reqpages(ms, tmpstr,
+                                          (ram_addr_t)tmp64a,
+                                          (ram_addr_t)tmp64b);
+            break;
+
         default:
             /* This shouldn't happen because we should catch this above */
             DPRINTF("RP: Bad header_com in dispatch");
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#132 lore raw

Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command

Quoted 11 lines · * David Gibson ([email protected]) wrote:

migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source.

migration.c +2 -0
+    buf64[0] = (uint64_t)start;
+    buf64[0] = cpu_to_be64(buf64[0]);

I think this would be clearer as well as less verbose, as just:

	buf64[0] = cpu_to_be64(start);

I've gone with the halfway mark of:

buf64[0] = cpu_to_be64((uint64_t)start);

it jsut doesn't feel right passing something into a byteswap unless you know the size.

Quoted 25 lines
Quoted 25 lines
migration.c +16 -0
+    buf64[1] = (uint64_t)len;
+    buf64[1] = cpu_to_be64(buf64[1]);
+    migrate_send_rp_message(mis, MIG_RPCOMM_REQPAGES, msglen, bufc);
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s)
 }
 
 /*
+ * Process a request for pages received on the return path,
+ * We're allowed to send more than requested (e.g. to round to our page size)
+ * and we don't need to send pages that have already been sent.
+ */
+static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
+                                       ram_addr_t start, ram_addr_t len)
+{
+    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+}
+
+/*
  * Handles messages sent on the return path towards the source VM
  *
Quoted 8 lines

migration.c static void *source_return_path_thread(void *opaque) · line 841

Quoted 6 lines
migration.c +1 -0
  */
@@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)
     const int max_len = 512;
     uint8_t buf[max_len];
     uint32_t tmp32;
+    uint64_t tmp64a, tmp64b;

Hrm.. calling everything "tmp*" doesn't help readability.

True; most of the rest of those tmps are used by multiple commands and just read off the wire and immediately used.
They're now start/len for tmp64a/b.

Dave

Quoted 40 lines
Quoted 40 lines
migration.c +25 -0
+    char *tmpstr;
     int res;
 
     DPRINTF("RP: %s entry", __func__);
@@ -810,6 +855,11 @@ static void *source_return_path_thread(void *opaque)
             expected_len = 4;
             break;
 
+        case MIG_RPCOMM_REQPAGES:
+            /* 16 byte start/len _possibly_ plus an id str */
+            expected_len = 16 + 256;
+            break;
+
         default:
             error_report("RP: Received invalid cmd 0x%04x length 0x%04x",
                     header_com, header_len);
@@ -857,6 +907,30 @@ static void *source_return_path_thread(void *opaque)
             atomic_xchg(&ms->rp_state.latest_ack, tmp32);
             break;
 
+        case MIG_RPCOMM_REQPAGES:
+            tmp64a = be64_to_cpup((uint64_t *)buf);  /* Start */
+            tmp64b = be64_to_cpup(((uint64_t *)buf)+1); /* Len */
+            tmpstr = NULL;
+            if (tmp64b & 1) {
+                tmp64b -= 1; /* Remove the flag */
+                /* Now we expect an idstr */
+                tmp32 = buf[16]; /* Length of the following idstr */
+                tmpstr = (char *)&buf[17];
+                buf[17+tmp32] = '\0';
+                expected_len = 16+1+tmp32;
+            } else {
+                expected_len = 16;
+            }
+            if (header_len != expected_len) {
+                error_report("RP: Received ReqPage with length %d expecting %d",
+                        header_len, expected_len);
+                source_return_path_bad(ms);
+            }
Quoted 14 lines
Quoted 8 lines
migration.c +5 -0
+            migrate_handle_rp_reqpages(ms, tmpstr,
+                                          (ram_addr_t)tmp64a,
+                                          (ram_addr_t)tmp64b);
+            break;
+
         default:
             /* This shouldn't happen because we should catch this above */
             DPRINTF("RP: Bad header_com in dispatch");
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#133 lore raw

Re: [Qemu-devel] [PATCH v4 35/47] Page request: Add MIG_RPCOMM_REQPAGES reverse command

Quoted 13 lines · Dr. David Alan Gilbert
Quoted 11 lines · * David Gibson ([email protected]) wrote:

migration.c void migrate_send_rp_ack(MigrationIncomingState *mis, · line 173

Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Add MIG_RPCOMM_REQPAGES command on Return path for the postcopy destination to request a page from the source.

migration.c +2 -0
+    buf64[0] = (uint64_t)start;
+    buf64[0] = cpu_to_be64(buf64[0]);

I think this would be clearer as well as less verbose, as just:

	buf64[0] = cpu_to_be64(start);
Quoted 6 lines

I've gone with the halfway mark of:

buf64[0] = cpu_to_be64((uint64_t)start);

it jsut doesn't feel right passing something into a byteswap unless you know the size.

I've always thought of (this group of) byteswap functions as specifying the output size. It's a value parameter so integer promotion to the required type is pretty safe.

Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
migration.c +16 -0
+    buf64[1] = (uint64_t)len;
+    buf64[1] = cpu_to_be64(buf64[1]);
+    migrate_send_rp_message(mis, MIG_RPCOMM_REQPAGES, msglen, bufc);
+}
+
 void qemu_start_incoming_migration(const char *uri, Error **errp)
 {
     const char *p;
@@ -784,6 +816,17 @@ static void source_return_path_bad(MigrationState *s)
 }
 
 /*
+ * Process a request for pages received on the return path,
+ * We're allowed to send more than requested (e.g. to round to our page size)
+ * and we don't need to send pages that have already been sent.
+ */
+static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
+                                       ram_addr_t start, ram_addr_t len)
+{
+    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+}
+
+/*
  * Handles messages sent on the return path towards the source VM
  *
Quoted 8 lines

migration.c static void *source_return_path_thread(void *opaque) · line 841

Quoted 6 lines
migration.c +1 -0
  */
@@ -795,6 +838,8 @@ static void *source_return_path_thread(void *opaque)
     const int max_len = 512;
     uint8_t buf[max_len];
     uint32_t tmp32;
+    uint64_t tmp64a, tmp64b;

Hrm.. calling everything "tmp*" doesn't help readability.

Quoted 3 lines

True; most of the rest of those tmps are used by multiple commands and just read off the wire and immediately used.
They're now start/len for tmp64a/b.

Ok, great. I find it's usually best to declare appropriate variables for each case (or even use local blocks), rather than share. The compiler's smart enough to coalesce them, so there's no real cost.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#134 lore raw

PATCH v4 36/47 Page request: Process incoming page request

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.

---
 arch_init.c                   | 52 +++++++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h | 21 +++++++++++++++++
 include/qemu/typedefs.h       |  3 ++-
 migration.c                   | 34 +++++++++++++++++++++++++++-
 4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +52 -0
index 4a03171..72f9e17 100644
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
 }
 
 /*
+ * Queue the pages for transmission, e.g. a request from postcopy destination
+ *   ms: MigrationStatus in which the queue is held
+ *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
+ *   start: Offset from the start of the RAMBlock
+ *   len: Length (in bytes) to send
+ *   Return: 0 on success
+ */
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len)
+{
+    RAMBlock *ramblock;
+
+    if (!rbname) {
+        /* Reuse last RAMBlock */
+        ramblock = ms->last_req_rb;
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no previous block");
+            return -1;
+        }
+    } else {
+        ramblock = ram_find_block(rbname);
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no block '%s'", rbname);
+            return -1;
+        }
+    }
+    DPRINTF("ram_save_queue_pages: Block %s start %zx len %zx",
+                    ramblock->idstr, start, len);
+
+    if (start+len > ramblock->length) {
+        error_report("%s request overrun start=%zx len=%zx blocklen=%zx",
+                     __func__, start, len, ramblock->length);
+        return -1;
+    }
+
+    struct MigrationSrcPageRequest *new_entry =
+        g_malloc0(sizeof(struct MigrationSrcPageRequest));
+    new_entry->rb = ramblock;
+    new_entry->offset = start;
+    new_entry->len = len;
+    ms->last_req_rb = ramblock;
+
+    qemu_mutex_lock(&ms->src_page_req_mutex);
+    QSIMPLEQ_INSERT_TAIL(&ms->src_page_requests, new_entry, next_req);
+    qemu_mutex_unlock(&ms->src_page_req_mutex);
+
+    return 0;
+}
+
+/*
  * ram_find_and_save_block: Finds a page to send and sends it to f
  *
  * Returns:  The number of bytes written.
include/migration/migration.h +21 -0
index 5e0d30d..5bc01d5 100644
@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);
 MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);
 void migration_incoming_state_destroy(void);
 
+/*
+ * An outstanding page request, on the source, having been received
+ * and queued
+ */
+struct MigrationSrcPageRequest {
+    RAMBlock *rb;
+    hwaddr    offset;
+    hwaddr    len;
+
+    QSIMPLEQ_ENTRY(MigrationSrcPageRequest) next_req;
+};
+
 struct MigrationState
 {
     int64_t bandwidth_limit;
@@ -138,6 +150,12 @@ struct MigrationState
      * of the postcopy phase
      */
     unsigned long *sentmap;
+
+    /* Queue of outstanding page requests from the destination */
+    QemuMutex src_page_req_mutex;
+    QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests;
+    /* The RAMBlock used in the last src_page_request */
+    RAMBlock *last_req_rb;
 };
 
 void process_incoming_migration(QEMUFile *f);
@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len);
+
 #endif
include/qemu/typedefs.h +2 -1
index 79f57c0..24c2207 100644
@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;
 typedef struct QEMUFile QEMUFile;
 typedef struct QEMUBH QEMUBH;
 
+typedef struct AdapterInfo AdapterInfo;
 typedef struct AioContext AioContext;
 
 typedef struct Visitor Visitor;
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
 typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 
 #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
 //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
     migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
     s->state = MIG_STATE_SETUP;
     trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
     s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     return s;
 }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
 static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                        ram_addr_t start, ram_addr_t len)
 {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
 }
 
 /*
Signature · 1 line
1.9.3
#135 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 133 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.

---
  arch_init.c                   | 52 +++++++++++++++++++++++++++++++++++++++++++
  include/migration/migration.h | 21 +++++++++++++++++
  include/qemu/typedefs.h       |  3 ++-
  migration.c                   | 34 +++++++++++++++++++++++++++-
  4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +52 -0
index 4a03171..72f9e17 100644
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
  }
 
  /*
+ * Queue the pages for transmission, e.g. a request from postcopy destination
+ *   ms: MigrationStatus in which the queue is held
+ *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
+ *   start: Offset from the start of the RAMBlock
+ *   len: Length (in bytes) to send
+ *   Return: 0 on success
+ */
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len)
+{
+    RAMBlock *ramblock;
+
+    if (!rbname) {
+        /* Reuse last RAMBlock */
+        ramblock = ms->last_req_rb;
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no previous block");
+            return -1;
+        }
+    } else {
+        ramblock = ram_find_block(rbname);
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no block '%s'", rbname);
+            return -1;
+        }
+    }
+    DPRINTF("ram_save_queue_pages: Block %s start %zx len %zx",
+                    ramblock->idstr, start, len);
+
+    if (start+len > ramblock->length) {
+        error_report("%s request overrun start=%zx len=%zx blocklen=%zx",
+                     __func__, start, len, ramblock->length);
+        return -1;
+    }
+
+    struct MigrationSrcPageRequest *new_entry =
+        g_malloc0(sizeof(struct MigrationSrcPageRequest));
+    new_entry->rb = ramblock;
+    new_entry->offset = start;
+    new_entry->len = len;
+    ms->last_req_rb = ramblock;
+
+    qemu_mutex_lock(&ms->src_page_req_mutex);
+    QSIMPLEQ_INSERT_TAIL(&ms->src_page_requests, new_entry, next_req);
+    qemu_mutex_unlock(&ms->src_page_req_mutex);
+
+    return 0;
+}
+
+/*
   * ram_find_and_save_block: Finds a page to send and sends it to f
   *
   * Returns:  The number of bytes written.
include/migration/migration.h +21 -0
index 5e0d30d..5bc01d5 100644
@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);
  MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);
  void migration_incoming_state_destroy(void);
 
+/*
+ * An outstanding page request, on the source, having been received
+ * and queued
+ */
+struct MigrationSrcPageRequest {
+    RAMBlock *rb;
+    hwaddr    offset;
+    hwaddr    len;
+
+    QSIMPLEQ_ENTRY(MigrationSrcPageRequest) next_req;
+};
+
  struct MigrationState
  {
      int64_t bandwidth_limit;
@@ -138,6 +150,12 @@ struct MigrationState
       * of the postcopy phase
       */
      unsigned long *sentmap;
+
+    /* Queue of outstanding page requests from the destination */
+    QemuMutex src_page_req_mutex;
+    QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests;
+    /* The RAMBlock used in the last src_page_request */
+    RAMBlock *last_req_rb;
  };
 
  void process_incoming_migration(QEMUFile *f);
@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                               ram_addr_t offset, size_t size,
                               int *bytes_sent);
 
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len);
+
  #endif
include/qemu/typedefs.h +1 -0
index 79f57c0..24c2207 100644
@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;
  typedef struct QEMUFile QEMUFile;
  typedef struct QEMUBH QEMUBH;
 
+typedef struct AdapterInfo AdapterInfo;
  typedef struct AioContext AioContext;
 
  typedef struct Visitor Visitor;

include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85

Quoted 6 lines
include/qemu/typedefs.h +1 -1
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
  typedef struct PcGuestInfo PcGuestInfo;
  typedef struct PostcopyPMI PostcopyPMI;
  typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 

:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?

Quoted 68 lines
include/qemu/typedefs.h +0 -0
  #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
  #include "qemu/thread.h"
  #include "qmp-commands.h"
  #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
  //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
      migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
      if (s->file) {
          trace_migrate_fd_cleanup();
          qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
      s->state = MIG_STATE_SETUP;
      trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
      s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
      return s;
  }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
  static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                         ram_addr_t start, ram_addr_t len)
  {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
  }
 
  /*
#136 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 11 lines · * zhanghailiang ([email protected]) wrote:

include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85

Quoted 8 lines

typedef struct Visitor Visitor;

include/qemu/typedefs.h +1 -1
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
 typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 

:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?

Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)

What compiler and flags are you using?

Dave

Quoted 67 lines
Quoted 67 lines
include/qemu/typedefs.h +0 -0
 #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
 //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
     migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
     s->state = MIG_STATE_SETUP;
     trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
     s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     return s;
 }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
 static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                        ram_addr_t start, ram_addr_t len)
 {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
 }
 
Quoted 4 lines

migration.c static void source_return_path_bad(MigrationState *s) · line 861

Quoted 2 lines
migration.c +0 -0
 /*

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#137 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Il 08/10/2014 09:49, Dr. David Alan Gilbert ha scritto:

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?

Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)

What compiler and flags are you using?

I think it is visible with CentOS 6.

Paolo

#138 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

In reply to Dr. David Alan Gilbert
Quoted 14 lines · Dr. David Alan Gilbert
Quoted 11 lines · * zhanghailiang ([email protected]) wrote:

include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85

Quoted 8 lines

typedef struct Visitor Visitor;

include/qemu/typedefs.h +1 -1
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
  typedef struct PcGuestInfo PcGuestInfo;
  typedef struct PostcopyPMI PostcopyPMI;
  typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 

:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?

Quoted 6 lines

Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)

What compiler and flags are you using?

Dave

Hi Dave,

My compiler info:
gcc (SUSE Linux) 4.3.4

The configure info is:
#./configure --target-list=x86_64-softmmu --enable-debug --disable-gtk
...

CFLAGS            -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -g
QEMU_CFLAGS       -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all    -I/usr/include/libpng12   -I/usr/include/pixman-1
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g

...

Maybe its gcc's limitation, but why this redefinition need? After i remove one, it compiles successfully;)

Thanks,
zhanghailiang

Quoted 77 lines
Quoted 68 lines
Quoted 68 lines
include/qemu/typedefs.h +0 -0
  #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
  #include "qemu/thread.h"
  #include "qmp-commands.h"
  #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
  //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
      migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
      if (s->file) {
          trace_migrate_fd_cleanup();
          qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
      s->state = MIG_STATE_SETUP;
      trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
      s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
      return s;
  }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
  static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                         ram_addr_t start, ram_addr_t len)
  {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
  }
 
Quoted 4 lines

migration.c static void source_return_path_bad(MigrationState *s) · line 861

Quoted 2 lines
migration.c +0 -0
  /*

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

.

#139 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 30 lines · * zhanghailiang ([email protected]) wrote:
Quoted 14 lines · Dr. David Alan Gilbert
Quoted 11 lines · * zhanghailiang ([email protected]) wrote:

include/qemu/typedefs.h typedef struct FWCfgState FWCfgState; · line 85

Quoted 8 lines

typedef struct Visitor Visitor;

include/qemu/typedefs.h +1 -1
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
 typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 

:(, another redefinition, 'RAMBlock' also defined in 'include/exec/cpu-all.h:314', Am i miss something when compile qemu?

Quoted 6 lines

Interesting; I'm not seeing that problem at all (gcc 4.8.3-7)

What compiler and flags are you using?

Dave

Hi Dave,

My compiler info:
gcc (SUSE Linux) 4.3.4

The configure info is:
#./configure --target-list=x86_64-softmmu --enable-debug --disable-gtk
...

Quoted 7 lines
CFLAGS            -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -g
QEMU_CFLAGS       -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all    -I/usr/include/libpng12   -I/usr/include/pixman-1
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g

...

Maybe its gcc's limitation, but why this redefinition need? After i remove one, it compiles successfully;)

OK, thanks. I'll clean them up.

Dave

Quoted 83 lines

Thanks,
zhanghailiang

Quoted 77 lines
Quoted 68 lines
Quoted 68 lines
include/qemu/typedefs.h +0 -0
 #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
 //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
     migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
     s->state = MIG_STATE_SETUP;
     trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
     s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     return s;
 }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
 static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                        ram_addr_t start, ram_addr_t len)
 {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
 }
 
Quoted 4 lines

migration.c static void source_return_path_bad(MigrationState *s) · line 861

Quoted 2 lines
migration.c +0 -0
 /*

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#140 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

In reply to Dr. David Alan Gilbert (git)
Quoted 30 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.

---
 arch_init.c                   | 52 +++++++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h | 21 +++++++++++++++++
 include/qemu/typedefs.h       |  3 ++-
 migration.c                   | 34 +++++++++++++++++++++++++++-
 4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +9 -0
index 4a03171..72f9e17 100644
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
 }
 
 /*
+ * Queue the pages for transmission, e.g. a request from postcopy destination
+ *   ms: MigrationStatus in which the queue is held
+ *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
+ *   start: Offset from the start of the RAMBlock
+ *   len: Length (in bytes) to send
+ *   Return: 0 on success
+ */
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len)

arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681

Quoted 10 lines
arch_init.c +10 -0
+{
+    RAMBlock *ramblock;
+
+    if (!rbname) {
+        /* Reuse last RAMBlock */
+        ramblock = ms->last_req_rb;
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no previous block");
+            return -1;

This should be an assert() shouldn't it?

arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 689

Quoted 8 lines
arch_init.c +8 -0
+        }
+    } else {
+        ramblock = ram_find_block(rbname);
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no block '%s'", rbname);
+            return -1;
+        }

And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.

Quoted 158 lines
arch_init.c +25 -0
+    }
+    DPRINTF("ram_save_queue_pages: Block %s start %zx len %zx",
+                    ramblock->idstr, start, len);
+
+    if (start+len > ramblock->length) {
+        error_report("%s request overrun start=%zx len=%zx blocklen=%zx",
+                     __func__, start, len, ramblock->length);
+        return -1;
+    }
+
+    struct MigrationSrcPageRequest *new_entry =
+        g_malloc0(sizeof(struct MigrationSrcPageRequest));
+    new_entry->rb = ramblock;
+    new_entry->offset = start;
+    new_entry->len = len;
+    ms->last_req_rb = ramblock;
+
+    qemu_mutex_lock(&ms->src_page_req_mutex);
+    QSIMPLEQ_INSERT_TAIL(&ms->src_page_requests, new_entry, next_req);
+    qemu_mutex_unlock(&ms->src_page_req_mutex);
+
+    return 0;
+}
+
+/*
  * ram_find_and_save_block: Finds a page to send and sends it to f
  *
  * Returns:  The number of bytes written.
include/migration/migration.h +21 -0
index 5e0d30d..5bc01d5 100644
@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);
 MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);
 void migration_incoming_state_destroy(void);
 
+/*
+ * An outstanding page request, on the source, having been received
+ * and queued
+ */
+struct MigrationSrcPageRequest {
+    RAMBlock *rb;
+    hwaddr    offset;
+    hwaddr    len;
+
+    QSIMPLEQ_ENTRY(MigrationSrcPageRequest) next_req;
+};
+
 struct MigrationState
 {
     int64_t bandwidth_limit;
@@ -138,6 +150,12 @@ struct MigrationState
      * of the postcopy phase
      */
     unsigned long *sentmap;
+
+    /* Queue of outstanding page requests from the destination */
+    QemuMutex src_page_req_mutex;
+    QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests;
+    /* The RAMBlock used in the last src_page_request */
+    RAMBlock *last_req_rb;
 };
 
 void process_incoming_migration(QEMUFile *f);
@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len);
+
 #endif
include/qemu/typedefs.h +2 -1
index 79f57c0..24c2207 100644
@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;
 typedef struct QEMUFile QEMUFile;
 typedef struct QEMUBH QEMUBH;
 
+typedef struct AdapterInfo AdapterInfo;
 typedef struct AioContext AioContext;
 
 typedef struct Visitor Visitor;
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
 typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 
 #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
 //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
     migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
     s->state = MIG_STATE_SETUP;
     trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
     s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     return s;
 }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
 static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                        ram_addr_t start, ram_addr_t len)
 {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
 }
 
 /*
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#141 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 53 lines · * David Gibson ([email protected]) wrote:
Quoted 30 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.

---
 arch_init.c                   | 52 +++++++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h | 21 +++++++++++++++++
 include/qemu/typedefs.h       |  3 ++-
 migration.c                   | 34 +++++++++++++++++++++++++++-
 4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +9 -0
index 4a03171..72f9e17 100644
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
 }
 
 /*
+ * Queue the pages for transmission, e.g. a request from postcopy destination
+ *   ms: MigrationStatus in which the queue is held
+ *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
+ *   start: Offset from the start of the RAMBlock
+ *   len: Length (in bytes) to send
+ *   Return: 0 on success
+ */
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len)

arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681

Quoted 10 lines
arch_init.c +10 -0
+{
+    RAMBlock *ramblock;
+
+    if (!rbname) {
+        /* Reuse last RAMBlock */
+        ramblock = ms->last_req_rb;
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no previous block");
+            return -1;

This should be an assert() shouldn't it?

arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 689

Quoted 8 lines
arch_init.c +8 -0
+        }
+    } else {
+        ramblock = ram_find_block(rbname);
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no block '%s'", rbname);
+            return -1;
+        }
Quoted 2 lines

And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.

No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.

(On the incoming side it's a bit more OK since if you've not got a full running VM anyway yet then there's not much to lose).

Dave

Quoted 158 lines
Quoted 158 lines
arch_init.c +25 -0
+    }
+    DPRINTF("ram_save_queue_pages: Block %s start %zx len %zx",
+                    ramblock->idstr, start, len);
+
+    if (start+len > ramblock->length) {
+        error_report("%s request overrun start=%zx len=%zx blocklen=%zx",
+                     __func__, start, len, ramblock->length);
+        return -1;
+    }
+
+    struct MigrationSrcPageRequest *new_entry =
+        g_malloc0(sizeof(struct MigrationSrcPageRequest));
+    new_entry->rb = ramblock;
+    new_entry->offset = start;
+    new_entry->len = len;
+    ms->last_req_rb = ramblock;
+
+    qemu_mutex_lock(&ms->src_page_req_mutex);
+    QSIMPLEQ_INSERT_TAIL(&ms->src_page_requests, new_entry, next_req);
+    qemu_mutex_unlock(&ms->src_page_req_mutex);
+
+    return 0;
+}
+
+/*
  * ram_find_and_save_block: Finds a page to send and sends it to f
  *
  * Returns:  The number of bytes written.
include/migration/migration.h +21 -0
index 5e0d30d..5bc01d5 100644
@@ -102,6 +102,18 @@ MigrationIncomingState *migration_incoming_get_current(void);
 MigrationIncomingState *migration_incoming_state_init(QEMUFile *f);
 void migration_incoming_state_destroy(void);
 
+/*
+ * An outstanding page request, on the source, having been received
+ * and queued
+ */
+struct MigrationSrcPageRequest {
+    RAMBlock *rb;
+    hwaddr    offset;
+    hwaddr    len;
+
+    QSIMPLEQ_ENTRY(MigrationSrcPageRequest) next_req;
+};
+
 struct MigrationState
 {
     int64_t bandwidth_limit;
@@ -138,6 +150,12 @@ struct MigrationState
      * of the postcopy phase
      */
     unsigned long *sentmap;
+
+    /* Queue of outstanding page requests from the destination */
+    QemuMutex src_page_req_mutex;
+    QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests;
+    /* The RAMBlock used in the last src_page_request */
+    RAMBlock *last_req_rb;
 };
 
 void process_incoming_migration(QEMUFile *f);
@@ -273,4 +291,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len);
+
 #endif
include/qemu/typedefs.h +2 -1
index 79f57c0..24c2207 100644
@@ -8,6 +8,7 @@ typedef struct QEMUTimerListGroup QEMUTimerListGroup;
 typedef struct QEMUFile QEMUFile;
 typedef struct QEMUBH QEMUBH;
 
+typedef struct AdapterInfo AdapterInfo;
 typedef struct AioContext AioContext;
 
 typedef struct Visitor Visitor;
@@ -80,6 +81,6 @@ typedef struct FWCfgState FWCfgState;
 typedef struct PcGuestInfo PcGuestInfo;
 typedef struct PostcopyPMI PostcopyPMI;
 typedef struct Range Range;
-typedef struct AdapterInfo AdapterInfo;
+typedef struct RAMBlock RAMBlock;
 
 #endif /* QEMU_TYPEDEFS_H */
migration.c +33 -1
index cfdaa52..63d7699 100644
@@ -26,6 +26,8 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "exec/memory.h"
+#include "exec/address-spaces.h"
 
 //#define DEBUG_MIGRATION
 
@@ -504,6 +506,15 @@ static void migrate_fd_cleanup(void *opaque)
 
     migrate_fd_cleanup_src_rp(s);
 
+    /* This queue generally should be empty - but in the case of a failed
+     * migration might have some droppings in.
+     */
+    struct MigrationSrcPageRequest *mspr, *next_mspr;
+    QSIMPLEQ_FOREACH_SAFE(mspr, &s->src_page_requests, next_req, next_mspr) {
+        QSIMPLEQ_REMOVE_HEAD(&s->src_page_requests, next_req);
+        g_free(mspr);
+    }
+
     if (s->file) {
         trace_migrate_fd_cleanup();
         qemu_mutex_unlock_iothread();
@@ -610,6 +621,9 @@ MigrationState *migrate_init(const MigrationParams *params)
     s->state = MIG_STATE_SETUP;
     trace_migrate_set_state(MIG_STATE_SETUP);
 
+    qemu_mutex_init(&s->src_page_req_mutex);
+    QSIMPLEQ_INIT(&s->src_page_requests);
+
     s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     return s;
 }
@@ -823,7 +837,25 @@ static void source_return_path_bad(MigrationState *s)
 static void migrate_handle_rp_reqpages(MigrationState *ms, const char* rbname,
                                        ram_addr_t start, ram_addr_t len)
 {
-    DPRINTF("migrate_handle_rp_reqpages: at %zx for len %zx", start, len);
+    DPRINTF("migrate_handle_rp_reqpages: in %s start %zx len %zx",
+            rbname, start, len);
+
+    /* Round everything up to our host page size */
+    long our_host_ps = sysconf(_SC_PAGESIZE);
+    if (start & (our_host_ps-1)) {
+        long roundings = start & (our_host_ps-1);
+        start -= roundings;
+        len += roundings;
+    }
+    if (len & (our_host_ps-1)) {
+        long roundings = len & (our_host_ps-1);
+        len -= roundings;
+        len += our_host_ps;
+    }
+
+    if (ram_save_queue_pages(ms, rbname, start, len)) {
+        source_return_path_bad(ms);
+    }
 }
 
Quoted 7 lines
Quoted 1 line
migration.c +0 -0
 /*
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#142 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 54 lines · Dr. David Alan Gilbert
Quoted 53 lines · * David Gibson ([email protected]) wrote:
Quoted 30 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

On receiving MIG_RPCOMM_REQPAGES look up the address and queue the page.

---
 arch_init.c                   | 52 +++++++++++++++++++++++++++++++++++++++++++
 include/migration/migration.h | 21 +++++++++++++++++
 include/qemu/typedefs.h       |  3 ++-
 migration.c                   | 34 +++++++++++++++++++++++++++-
 4 files changed, 108 insertions(+), 2 deletions(-)
arch_init.c +9 -0
index 4a03171..72f9e17 100644
@@ -660,6 +660,58 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
 }
 
 /*
+ * Queue the pages for transmission, e.g. a request from postcopy destination
+ *   ms: MigrationStatus in which the queue is held
+ *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
+ *   start: Offset from the start of the RAMBlock
+ *   len: Length (in bytes) to send
+ *   Return: 0 on success
+ */
+int ram_save_queue_pages(MigrationState *ms, const char *rbname,
+                         ram_addr_t start, ram_addr_t len)

arch_init.c static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset, · line 681

Quoted 10 lines
arch_init.c +10 -0
+{
+    RAMBlock *ramblock;
+
+    if (!rbname) {
+        /* Reuse last RAMBlock */
+        ramblock = ms->last_req_rb;
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no previous block");
+            return -1;

This should be an assert() shouldn't it?

Quoted 8 lines
arch_init.c +8 -0
+        }
+    } else {
+        ramblock = ram_find_block(rbname);
+
+        if (!ramblock) {
+            error_report("ram_save_queue_pages no block '%s'", rbname);
+            return -1;
+        }
Quoted 8 lines
Quoted 2 lines

And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.

No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.

Ah, ok, that makes sense. Maybe adding something to the error message or a nearby comment indicating that if these happen it's certainly a bug, not the result of some external problem?

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#143 lore raw

Re: [Qemu-devel] [PATCH v4 36/47] Page request: Process incoming page request

Quoted 11 lines · * David Gibson ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 2 lines

And maybe this one too - I would have expected the rb names to have already been validated on the source machine at this stage.

No to both:
I've been trying to avoid asserts in migration outgoing code, because they shouldn't affect the state of your guest, so there's no reason to kill off what might still be a viable running guest just because migration failed.

Quoted 3 lines

Ah, ok, that makes sense. Maybe adding something to the error message or a nearby comment indicating that if these happen it's certainly a bug, not the result of some external problem?

Done.

Dave

Quoted 6 lines
Signature · 5 lines
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#144 lore raw

PATCH v4 37/47 Page request: Consume pages off the post-copy queue

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning.

Note:
a) After a queued page the linear walk carries on from after the
unqueued page; there is a reasonable chance that the destination was about to ask for other closeby pages anyway.

b) We have to be careful of any assumptions that the page walking
code makes, in particular it does some short cuts on its first linear walk that break as soon as we do a queued page.

---
 arch_init.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 125 insertions(+), 24 deletions(-)
arch_init.c +125 -24
index 72f9e17..a945990 100644
@@ -331,6 +331,7 @@ static RAMBlock *last_seen_block;
 /* This is the last block from where we have sent data */
 static RAMBlock *last_sent_block;
 static ram_addr_t last_offset;
+static bool last_was_from_queue;
 static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
 static uint32_t last_version;
@@ -460,6 +461,19 @@ static inline bool migration_bitmap_set_dirty(ram_addr_t addr)
     return ret;
 }
 
+static inline bool migration_bitmap_clear_dirty(ram_addr_t addr)
+{
+    bool ret;
+    int nr = addr >> TARGET_PAGE_BITS;
+
+    ret = test_and_clear_bit(nr, migration_bitmap);
+
+    if (ret) {
+        migration_dirty_pages--;
+    }
+    return ret;
+}
+
 static void migration_bitmap_sync_range(ram_addr_t start, ram_addr_t length)
 {
     ram_addr_t addr;
@@ -660,6 +674,39 @@ static int ram_save_page(QEMUFile *f, RAMBlock* block, ram_addr_t offset,
 }
 
 /*
+ * Unqueue a page from the queue fed by postcopy page requests
+ *
+ * Returns:   The RAMBlock* to transmit from (or NULL if the queue is empty)
+ *      ms:   MigrationState in
+ *  offset:   the byte offset within the RAMBlock for the start of the page
+ * bitoffset: global offset in the dirty/sent bitmaps
+ */
+static RAMBlock *ram_save_unqueue_page(MigrationState *ms, ram_addr_t *offset,
+                                       unsigned long *bitoffset)
+{
+    RAMBlock *result = NULL;
+    qemu_mutex_lock(&ms->src_page_req_mutex);
+    if (!QSIMPLEQ_EMPTY(&ms->src_page_requests)) {
+        struct MigrationSrcPageRequest *entry =
+                                    QSIMPLEQ_FIRST(&ms->src_page_requests);
+        result = entry->rb;
+        *offset = entry->offset;
+        *bitoffset = (entry->offset + entry->rb->offset) >> TARGET_PAGE_BITS;
+
+        if (entry->len > TARGET_PAGE_SIZE) {
+            entry->len -= TARGET_PAGE_SIZE;
+            entry->offset += TARGET_PAGE_SIZE;
+        } else {
+            QSIMPLEQ_REMOVE_HEAD(&ms->src_page_requests, next_req);
+            g_free(entry);
+        }
+    }
+    qemu_mutex_unlock(&ms->src_page_req_mutex);
+
+    return result;
+}
+
+/*
  * Queue the pages for transmission, e.g. a request from postcopy destination
  *   ms: MigrationStatus in which the queue is held
  *   rbname: The RAMBlock the request is for - may be NULL (to mean reuse last)
@@ -720,44 +767,97 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,
 
 static int ram_find_and_save_block(QEMUFile *f, bool last_stage)
 {
+    MigrationState *ms = migrate_get_current();
     RAMBlock *block = last_seen_block;
+    RAMBlock *tmpblock;
     ram_addr_t offset = last_offset;
+    ram_addr_t tmpoffset;
     bool complete_round = false;
     int bytes_sent = 0;
-    MemoryRegion *mr;
     unsigned long bitoffset;
+    unsigned long hps = sysconf(_SC_PAGESIZE);
 
-    if (!block)
+    if (!block) {
         block = QTAILQ_FIRST(&ram_list.blocks);
+        last_was_from_queue = false;
+    }
 
-    while (true) {
-        mr = block->mr;
-        offset = migration_bitmap_find_and_reset_dirty(mr, offset, &bitoffset);
-        if (complete_round && block == last_seen_block &&
-            offset >= last_offset) {
-            break;
+    while (true) { /* Until we send a block or run out of stuff to send */
+        tmpblock = NULL;
+
+        /*
+         * Don't break host-page chunks up with queue items
+         * so only unqueue if,
+         *   a) The last item came from the queue anyway
+         *   b) The last sent item was the last target-page in a host page
+         */
+        if (last_was_from_queue || (!last_sent_block) ||
+            ((last_offset & (hps - 1)) == (hps - TARGET_PAGE_SIZE))) {
+            tmpblock = ram_save_unqueue_page(ms, &tmpoffset, &bitoffset);
         }
-        if (offset >= block->length) {
-            offset = 0;
-            block = QTAILQ_NEXT(block, next);
-            if (!block) {
-                block = QTAILQ_FIRST(&ram_list.blocks);
-                complete_round = true;
-                ram_bulk_stage = false;
+
+        if (tmpblock) {
+            /* We've got a block from the postcopy queue */
+            DPRINTF("%s: Got postcopy item '%s' offset=%zx bitoffset=%zx",
+                    __func__, tmpblock->idstr, tmpoffset, bitoffset);
+            /* We're sending this page, and since it's postcopy nothing else
+             * will dirty it, and we must make sure it doesn't get sent again.
+             */
+            if (!migration_bitmap_clear_dirty(bitoffset << TARGET_PAGE_BITS)) {
+                DPRINTF("%s: Not dirty for postcopy %s/%zx bito=%zx (sent=%d)",
+                        __func__, tmpblock->idstr, tmpoffset, bitoffset,
+                        test_bit(bitoffset, ms->sentmap));
+                continue;
             }
+            /*
+             * As soon as we start servicing pages out of order, then we have
+             * to kill the bulk stage, since the bulk stage assumes
+             * in (migration_bitmap_find_and_reset_dirty) that every page is
+             * dirty, that's no longer true.
+             */
+            ram_bulk_stage = false;
+            /*
+             * We mustn't change block/offset unless it's to a valid one
+             * otherwise we can go down some of the exit cases in the normal
+             * path.
+             */
+            block = tmpblock;
+            offset = tmpoffset;
+            last_was_from_queue = true;
         } else {
-            bytes_sent = ram_save_page(f, block, offset, last_stage);
-
-            /* if page is unmodified, continue to the next */
-            if (bytes_sent > 0) {
-                MigrationState *s = migrate_get_current();
-                if (s->sentmap) {
-                    set_bit(bitoffset, s->sentmap);
+            MemoryRegion *mr;
+            /* priority queue empty, so just search for something dirty */
+            mr = block->mr;
+            offset = migration_bitmap_find_and_reset_dirty(mr, offset,
+                                                           &bitoffset);
+            if (complete_round && block == last_seen_block &&
+                offset >= last_offset) {
+                break;
+            }
+            if (offset >= block->length) {
+                offset = 0;
+                block = QTAILQ_NEXT(block, next);
+                if (!block) {
+                    block = QTAILQ_FIRST(&ram_list.blocks);
+                    complete_round = true;
+                    ram_bulk_stage = false;
                 }
+                continue; /* pick an offset in the new block */
+            }
+            last_was_from_queue = false;
+        }
 
-                last_sent_block = block;
-                break;
+        /* We have a page to send, so send it */
+        bytes_sent = ram_save_page(f, block, offset, last_stage);
+
+        /* if page is unmodified, continue to the next */
+        if (bytes_sent > 0) {
+            if (ms->sentmap) {
+                set_bit(bitoffset, ms->sentmap);
             }
+
+            last_sent_block = block;
+            break;
         }
     }
     last_seen_block = block;
@@ -851,6 +951,7 @@ static void reset_ram_globals(void)
     last_offset = 0;
     last_version = ram_list.version;
     ram_bulk_stage = true;
+    last_was_from_queue = false;
 }
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
Signature · 1 line
1.9.3