Skip to contentSkip to conversation

In memory QEMUFile

PATCH Superseded Dr. David Alan Gilbert (git) 3 participants 6 messages Aug 6, 2014 – Aug 7, 2014

v1 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

cover In memory QEMUFile

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile
written by Stefan Berger and Joel Schopp. I've made some minor fixes (and typo cleanups) and modified the existing test-vmstate to use it for some test cases.

While there's nothing other than test cases using it yet, I think
it's worth going in by itself, since I'm using it in two separate patchsets (postcopy and visitor/BER) and Sanidhya uses it in the periodic vmstate test world. In addition both microcheckpointing and COLO have similar but independent implementations (although they both have some extra-gotcha's so it might not be possible to reuse it), and there was another implementation of the same thing in the Yabusame Postcopy world. Thus it seems best to put in, if only to stop people writing yet another implementation.

Dr. David Alan Gilbert (2):
QEMUSizedBuffer based QEMUFile
Tests: QEMUSizedBuffer/QEMUBuffer

 include/migration/qemu-file.h |  28 +++
 include/qemu/typedefs.h       |   1 +
 qemu-file.c                   | 410 ++++++++++++++++++++++++++++++++++++++++++
 tests/Makefile                |   2 +-
 tests/test-vmstate.c          |  73 ++++----
 5 files changed, 477 insertions(+), 37 deletions(-)
Signature · 1 line
1.9.3
#1 lore raw

PATCH 1/2 QEMUSizedBuffer based QEMUFile

From: "Dr. David Alan Gilbert" <[email protected]>

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

For minor tweeks/rebase I've done to it:

---
 include/migration/qemu-file.h |  28 +++
 include/qemu/typedefs.h       |   1 +
 qemu-file.c                   | 410 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 439 insertions(+)
include/migration/qemu-file.h +28 -0
index c90f529..80af3ff 100644
@@ -25,6 +25,8 @@
 #define QEMU_FILE_H 1
 #include "exec/cpu-common.h"
 
+#include <stdint.h>
+
 /* This function writes a chunk of data to a file at the given position.
  * The pos argument can be ignored if the file is only being used for
  * streaming.  The handler should try to write all of the data it can.
@@ -94,11 +96,21 @@ typedef struct QEMUFileOps {
     QEMURamSaveFunc *save_page;
 } QEMUFileOps;
 
+struct QEMUSizedBuffer {
+    struct iovec *iov;
+    size_t n_iov;
+    size_t size; /* total allocated size in all iov's */
+    size_t used; /* number of used bytes */
+};
+
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
+
 QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
 QEMUFile *qemu_fopen(const char *filename, const char *mode);
 QEMUFile *qemu_fdopen(int fd, const char *mode);
 QEMUFile *qemu_fopen_socket(int fd, const char *mode);
 QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input);
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);
 void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size);
 bool qemu_file_mode_is_not_valid(const char *mode);
 
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len);
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *);
+void qsb_free(QEMUSizedBuffer *);
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t length);
+size_t qsb_get_length(const QEMUSizedBuffer *qsb);
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
+                       uint8_t **buf);
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
+                     off_t pos, size_t count);
+
+
+/*
+ * For use on files opened with qemu_bufopen
+ */
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f);
+
 static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v)
 {
     qemu_put_byte(f, (int)v);
include/qemu/typedefs.h +1 -0
index 5f20b0e..db1153a 100644
@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;
 typedef struct PCIEAERErr PCIEAERErr;
 typedef struct PCIEPort PCIEPort;
 typedef struct PCIESlot PCIESlot;
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct MSIMessage MSIMessage;
 typedef struct SerialState SerialState;
 typedef struct PCMCIACardState PCMCIACardState;
qemu-file.c +410 -0

413 lines not on this page · raw message

Signature · 1 line
1.9.3
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] QEMUSizedBuffer based QEMUFile

Quoted 21 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

For minor tweeks/rebase I've done to it:

---
 include/migration/qemu-file.h |  28 +++
 include/qemu/typedefs.h       |   1 +
 qemu-file.c                   | 410 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 439 insertions(+)
Quoted 8 lines

+/**
+ * Set the length of the buffer; the primary usage of this
+ * function is to truncate the number of used bytes in the buffer.
+ * The size will not be extended beyond the current number of
+ * allocated bytes in the QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_len : The new length of bytes in the buffer

No space before :

Quoted 8 lines

+/*
+ * Convert the QEMUSizedBuffer into a flat buffer.
+ *
+ * Note: If at all possible, try to avoid this function since it
+ * may unnecessarily copy memory around.
+ *
+ * @qsb: pointer to QEMUSizedBuffer
+ * @start : offset to start at

and again

Quoted 6 lines

+ * @count: number of bytes to copy
+ * @buf: a pointer to an optional buffer to write into; the pointer may
+ * point to NULL in which case the buffer will be allocated;
+ * if buffer is provided, it must be large enough to hold @count bytes
+ *
+ * Returns the number of bytes copied into the output buffer

spurious double space

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH 1/2] QEMUSizedBuffer based QEMUFile

Quoted 14 lines · * Eric Blake ([email protected]) wrote:
Quoted 13 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This is based on Stefan and Joel's patch that creates a QEMUFile that goes to a memory buffer; from:

http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html

Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be written to or read from.

Quoted 7 lines
Quoted 7 lines

For minor tweeks/rebase I've done to it:

---
 include/migration/qemu-file.h |  28 +++
 include/qemu/typedefs.h       |   1 +
 qemu-file.c                   | 410 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 439 insertions(+)

Spaces fixed in v2.

Dave

Quoted 24 lines
Quoted 9 lines

+/**
+ * Set the length of the buffer; the primary usage of this
+ * function is to truncate the number of used bytes in the buffer.
+ * The size will not be extended beyond the current number of
+ * allocated bytes in the QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_len : The new length of bytes in the buffer

No space before :

Quoted 8 lines

+/*
+ * Convert the QEMUSizedBuffer into a flat buffer.
+ *
+ * Note: If at all possible, try to avoid this function since it
+ * may unnecessarily copy memory around.
+ *
+ * @qsb: pointer to QEMUSizedBuffer
+ * @start : offset to start at

and again

Quoted 14 lines
Quoted 6 lines

+ * @count: number of bytes to copy
+ * @buf: a pointer to an optional buffer to write into; the pointer may
+ * point to NULL in which case the buffer will be allocated;
+ * if buffer is provided, it must be large enough to hold @count bytes
+ *
+ * Returns the number of bytes copied into the output buffer

spurious double space

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw

PATCH 2/2 Tests: QEMUSizedBuffer/QEMUBuffer

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Modify some of tests/test-vmstate.c to use the in memory file based on QEMUSizedBuffer to provide basic testing of QEMUSizedBuffer and the associated memory backed QEMUFile type.

Only some of the tests are changed so that the fd backed QEMUFile is still tested.

---
 tests/Makefile       |  2 +-
 tests/test-vmstate.c | 73 ++++++++++++++++++++++++++--------------------------
 2 files changed, 38 insertions(+), 37 deletions(-)
tests/Makefile +1 -1
index 4b2e1bb..f3d32ba 100644
@@ -253,7 +253,7 @@ tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
 	libqemuutil.a libqemustub.a
 tests/test-vmstate$(EXESUF): tests/test-vmstate.o \
 	vmstate.o qemu-file.o \
-	libqemuutil.a
+	libqemuutil.a libqemustub.a
 
 tests/test-qapi-types.c tests/test-qapi-types.h :\
 $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py
tests/test-vmstate.c +37 -36

142 lines not on this page · raw message

Signature · 1 line
1.9.3
#5 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 7 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Hi,
This patch-pair adds the QEMUSizedBuffer based in-memory QEMUFile
written by Stefan Berger and Joel Schopp. I've made some minor fixes (and typo cleanups) and modified the existing test-vmstate to use it for some test cases.

Nice to see these come back around. Still look good to me.