Migration: Remove copying of guest ram pages
v3 cover Migration: Remove copying of guest ram pages
In migration all data is copied to a static buffer in QEMUFile, this hurts our network bandwidth and CPU usage especially with large guests. We switched to iovec for storing different buffers to send (even a byte field is considered as a buffer) and use writev to send the iovec.
writev was chosen (as apposed to sendmsg) because it supprts non socket fds.
Guest memory pages are not copied by calling a new function qemu_put_buffer_no_copy.
The page header data and device state data are still copied into the static buffer. This data consists of a lot of bytes and integer fields and the static buffer is used to store it during batching.
Another improvement is changing qemu_putbe64/32/16 to create a single buffer instead of several byte sized buffer.
git repository: git://github.com/oritwas/qemu.git sendv_v2
Change from v2:
Always send data for the iovec even if writev_buffer is not implemented. Coalesce adjacent iovecs to create one big buffer from small adjacent buffer.
Changes from v1:
Use iov_send for socket.
Make writev_buffer optional and if it is not implemented use put_buffer
Future work: Make number of iovec changeable
Orit Wasserman (9):
Add QemuFileWritevBuffer QemuFileOps
Add socket_writev_buffer function
Update bytes_xfer in qemu_put_byte
Store the data to send also in iovec
Use writev ops if available
More optimized qemu_put_be64/32/16
Add qemu_put_buffer_no_copy
Use qemu_put_buffer_no_copy for guest memory pages coalesce adjacent iovecs
arch_init.c | 2 +- include/migration/qemu-file.h | 12 ++++ savevm.c | 127 ++++++++++++++++++++++++++++++++++-------- 3 files changed, 116 insertions(+), 25 deletions(-)
Signature · 1 line
1.7.11.7
PATCH v3 1/9 Add QemuFileWritevBuffer QemuFileOps
This will allow us to write an iovec
--- include/migration/qemu-file.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/migration/qemu-file.h +7 -0
@@ -51,11 +51,18 @@ typedef int (QEMUFileCloseFunc)(void *opaque);*/ typedef int (QEMUFileGetFD )(void *opaque); +/* +* This function writes an iovec to file. +*/ +typedef int (QEMUFileWritevBufferFunc )(void *opaque,struct iovec *iov, +int iovcnt); +typedef struct QEMUFileOps {QEMUFilePutBufferFunc *put_buffer;QEMUFileGetBufferFunc *get_buffer;QEMUFileCloseFunc *close;QEMUFileGetFD *get_fd; +QEMUFileWritevBufferFunc *writev_buffer; }QEMUFileOps ;QEMUFile *qemu_fopen_ops (void *opaque,const QEMUFileOps *ops);
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 1/9] Add QemuFileWritevBuffer QemuFileOps
Quoted 3 lines · Orit Wasserman
This will allow us to write an iovec
Signed-off-by Orit Wasserman <[email protected]>
PATCH v3 2/9 Add socket_writev_buffer function
In reply to Orit Wasserman--- savevm.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
savevm.c +14 -0
@@ -39,6 +39,7 @@#include "qmp-commands.h" #include "trace.h" #include "qemu/bitops.h" +#include "qemu/iov.h" #define SELF_ANNOUNCE_ROUNDS 5
@@ -171,6 +172,18 @@ static void coroutine_fn yield_until_fd_readable(int fd)qemu_coroutine_yield (); } +static int socket_writev_buffer (void *opaque,struct iovec *iov,int iovcnt) +{ +QEMUFileSocket *s = opaque; +ssize_t len; + + len =iov_send (s->fd, iov, iovcnt,0 ,iov_size (iov, iovcnt)); +if (len <iov_size (iov, iovcnt)) { + len = -socket_error (); + } +return len; +} +static int socket_get_fd (void *opaque) {QEMUFileSocket *s = opaque;
@@ -387,6 +400,7 @@ static const QEMUFileOps socket_read_ops = {static const QEMUFileOps socket_write_ops = { .get_fd = socket_get_fd, .put_buffer = socket_put_buffer, + .writev_buffer = socket_writev_buffer, .close = socket_close };
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 2/9] Add socket_writev_buffer function
Orit Wasserman <[email protected]> wrote:
D> Signed-off-by: Orit Wasserman <[email protected]>
Quoted 17 lines
--- savevm.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)savevm.c +1
-0@@ -39,6 +39,7 @@#include "qmp-commands.h" #include "trace.h" #include "qemu/bitops.h" +#include "qemu/iov.h" #define SELF_ANNOUNCE_ROUNDS 5
savevm.c static void coroutine_fn yield_until_fd_readable(int fd) · line 181
Quoted 10 lines
savevm.c +7
-0@@ -171,6 +172,18 @@ static void coroutine_fn yield_until_fd_readable(int fd)qemu_coroutine_yield (); } +static int socket_writev_buffer (void *opaque,struct iovec *iov,int iovcnt) +{ +QEMUFileSocket *s = opaque; +ssize_t len; + + len =iov_send (s->fd, iov, iovcnt,0 ,iov_size (iov, iovcnt)); +if (len <iov_size (iov, iovcnt)) {
Not the more expensive operation ever, but we can factor
iov_size(iov, iovcnt)
value into one variable.
Quoted 15 lines
savevm.c +6
-0+ len = -socket_error (); + } +return len; +} +static int socket_get_fd (void *opaque) {QEMUFileSocket *s = opaque;@@ -387,6 +400,7 @@ static const QEMUFileOps socket_read_ops = {static const QEMUFileOps socket_write_ops = { .get_fd = socket_get_fd, .put_buffer = socket_put_buffer, + .writev_buffer = socket_writev_buffer, .close = socket_close };
PATCH v3 3/9 Update bytes_xfer in qemu_put_byte
In reply to Orit Wasserman--- savevm.c | 2 ++ 1 file changed, 2 insertions(+)
savevm.c +2 -0
@@ -647,6 +647,8 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; + f->bytes_xfer +=1 ; +if (f->buf_index >=IO_BUF_SIZE ) {qemu_fflush (f); }
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 3/9] Update bytes_xfer in qemu_put_byte
Quoted 1 line · Orit Wasserman
Signed-off-by Orit Wasserman <[email protected]>
Re: [Qemu-devel] [PATCH v3 3/9] Update bytes_xfer in qemu_put_byte
In reply to Orit WassermanQuoted 10 lines · Orit Wasserman
Signed-off-by Orit Wasserman <[email protected]>--- savevm.c | 2 ++ 1 file changed, 2 insertions(+)savevm.c +0
-0
savevm.c void qemu_put_byte(QEMUFile *f, int v) · line 650
Quoted 4 lines
savevm.c +1
-0@@ -647,6 +647,8 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; + f->bytes_xfer +=1 ;
Why ' += 1' instead of the shorter '++'?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v3 3/9] Update bytes_xfer in qemu_put_byte
Quoted 12 lines · Eric Blake
Quoted 11 lines · Orit Wasserman
Signed-off-by Orit Wasserman <[email protected]>--- savevm.c | 2 ++ 1 file changed, 2 insertions(+)savevm.c +0
-0@@ -647,6 +647,8 @@ void qemu_put_byte(QEMUFile *f, int v)
Quoted 6 lines
savevm.c void qemu_put_byte(QEMUFile *f, int v) · line 650
Quoted 3 lines
savevm.c +1
-0f->buf[f->buf_index++] = v; f->is_write =1 ; + f->bytes_xfer +=1 ;Why ' += 1' instead of the shorter '++'?
It compiles to the same code but I can change it :)
PATCH v3 4/9 Store the data to send also in iovec
In reply to Orit WassermanAll data is still copied into the static buffer.
--- savevm.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
savevm.c +11 -2
@@ -114,6 +114,7 @@ void qemu_announce_self(void)/* savevm/loadvm support */ #define IO_BUF_SIZE 32768 +#define MAX_IOV_SIZE MIN(IOV_MAX, 64)struct QEMUFile {const QEMUFileOps *ops;
@@ -129,6 +130,9 @@ struct QEMUFile {int buf_size;/* 0 when writing */ uint8_t buf[IO_BUF_SIZE ]; +struct iovec iov[MAX_IOV_SIZE ]; +unsigned int iovcnt; +int last_error; };
@@ -527,6 +531,7 @@ static void qemu_fflush(QEMUFile *f) f->pos += f->buf_index; } f->buf_index =0 ; + f->iovcnt =0 ; }if (ret <0 ) {qemu_file_set_error (f, ret);
@@ -619,12 +624,14 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)if (l > size) l = size;memcpy (f->buf + f->buf_index, buf, l); + f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt++].iov_len = l; f->is_write =1 ; f->buf_index += l; f->bytes_xfer += l; buf += l; size -= l; -if (f->buf_index >=IO_BUF_SIZE ) { +if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);if (qemu_file_get_error (f)) {break ;
@@ -648,8 +655,10 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; -if (f->buf_index >=IO_BUF_SIZE ) { +if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f); } }
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 4/9] Store the data to send also in iovec
Quoted 38 lines · Orit Wasserman
All data is still copied into the static buffer.
Signed-off-by Orit Wasserman <[email protected]>--- savevm.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)savevm.c +5
-0@@ -114,6 +114,7 @@ void qemu_announce_self(void)/* savevm/loadvm support */ #define IO_BUF_SIZE 32768 +#define MAX_IOV_SIZE MIN(IOV_MAX, 64)struct QEMUFile {const QEMUFileOps *ops;@@ -129,6 +130,9 @@ struct QEMUFile {int buf_size;/* 0 when writing */ uint8_t buf[IO_BUF_SIZE ]; +struct iovec iov[MAX_IOV_SIZE ]; +unsigned int iovcnt; +int last_error; };@@ -527,6 +531,7 @@ static void qemu_fflush(QEMUFile *f) f->pos += f->buf_index; } f->buf_index =0 ; + f->iovcnt =0 ; }if (ret <0 ) {qemu_file_set_error (f, ret);
savevm.c void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size) · line 631
Quoted 8 lines
savevm.c +2
-0@@ -619,12 +624,14 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)if (l > size) l = size;memcpy (f->buf + f->buf_index, buf, l); + f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt++].iov_len = l; f->is_write =1 ; f->buf_index += l; f->bytes_xfer += l;
After reading the patch several times, we need to do a s/l/len/ on this function, 1 and l are too similar, and I was reading the wrong value :p
Quoted 9 lines
savevm.c +1
-1buf += l; size -= l; -if (f->buf_index >=IO_BUF_SIZE ) { +if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);if (qemu_file_get_error (f)) {break ;@@ -648,8 +655,10 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v;
savevm.c void qemu_put_byte(QEMUFile *f, int v) · line 664
Quoted 10 lines
savevm.c +3
-1f->is_write =1 ; f->bytes_xfer +=1 ; + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; -if (f->buf_index >=IO_BUF_SIZE ) { +if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f); } }
PATCH v3 5/9 Use writev ops if available
In reply to Orit WassermanUpdate qemu_fflush and stdio_close to use writev ops if they are available Use the buffers stored in the iovec.
--- savevm.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-)
savevm.c +23 -7
@@ -292,7 +292,7 @@ static int stdio_fclose(void *opaque)QEMUFileStdio *s = opaque;int ret =0 ; -if (s->file->ops->put_buffer) { +if (s->file->ops->put_buffer || s->file->ops->writev_buffer) {int fd =fileno (s->stdio_file);struct stat st;
@@ -515,24 +515,40 @@ static void qemu_file_set_error(QEMUFile *f, int ret) } } -/** Flushes QEMUFile buffer +/** +* Flushes QEMUFile buffer * +* If there is writev_buffer QEMUFileOps it uses it otherwise uses +* put_buffer ops. */ static void qemu_fflush (QEMUFile *f) {int ret =0 ; +int i =0 ; -if (!f->ops->put_buffer) { +if (!f->ops->writev_buffer && !f->ops->put_buffer) {return ; } -if (f->is_write && f->buf_index >0 ) { - ret = f->ops->put_buffer (f->opaque, f->buf, f->pos, f->buf_index); -if (ret >=0 ) { - f->pos += f->buf_index; + +if (f->is_write && f->iovcnt >0 ) { +if (f->ops->writev_buffer) { + ret = f->ops->writev_buffer (f->opaque, f->iov, f->iovcnt); +if (ret >=0 ) { + f->pos += ret; + } + }else { +for (i =0 ; i < f->iovcnt && ret >=0 ; i++) { + ret = f->ops->put_buffer (f->opaque, f->iov[i].iov_base, f->pos, + f->iov[i].iov_len); +if (ret >=0 ) { + f->pos += ret; + } + } } f->buf_index =0 ; f->iovcnt =0 ; } +if (ret <0 ) {qemu_file_set_error (f, ret); }
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 5/9] Use writev ops if available
Quoted 4 lines · Orit Wasserman
Update
qemu_fflushandstdio_closeto use writev ops if they are available Use the buffers stored in the iovec.Signed-off-by Orit Wasserman <[email protected]>
I think that qemu_fflush() should only be called for QEMUFiles opened in write mode, but that is independent of this series.
PATCH v3 6/9 More optimized qemu_put_be64/32/16
In reply to Orit Wassermanqemu_put_be functions used qemu_put_byte this caused lots of 1 bytes buffers in the iovec.
we move to use cpu_put_be64/32/16wu and put a single buffer per call.
--- savevm.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
savevm.c +12 -8
@@ -795,22 +795,26 @@ void qemu_file_reset_rate_limit(QEMUFile *f)void qemu_put_be16 (QEMUFile *f,unsigned int v) { -qemu_put_byte (f, v >>8 ); -qemu_put_byte (f, v); +uint16_t p; +cpu_to_be16wu (&p, v); + +qemu_put_buffer (f, (uint8_t *)&p,sizeof (p)); }void qemu_put_be32 (QEMUFile *f,unsigned int v) { -qemu_put_byte (f, v >>24 ); -qemu_put_byte (f, v >>16 ); -qemu_put_byte (f, v >>8 ); -qemu_put_byte (f, v); +uint32_t p; +cpu_to_be32wu (&p, v); + +qemu_put_buffer (f, (uint8_t *)&p,sizeof (p)); }void qemu_put_be64 (QEMUFile *f,uint64_t v) { -qemu_put_be32 (f, v >>32 ); -qemu_put_be32 (f, v); +uint64_t p; +cpu_to_be64wu (&p, v); + +qemu_put_buffer (f, (uint8_t *)&p,sizeof (p)); }unsigned int qemu_get_be16 (QEMUFile *f)
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 6/9] More optimized qemu_put_be64/32/16
Quoted 5 lines · Orit Wasserman
qemu_put_befunctions usedqemu_put_bytethis caused lots of 1 bytes buffers in the iovec.
we move to use cpu_put_be64/32/16wu and put a single buffer per call.Signed-off-by Orit Wasserman <[email protected]>
PATCH v3 7/9 Add qemu_put_buffer_no_copy
In reply to Orit WassermanThis allow us to add a buffer to the iovec to send without copying it into the static buffer.
--- include/migration/qemu-file.h | 5 +++++ savevm.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 33 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +5 -0
@@ -75,6 +75,11 @@ int qemu_fclose(QEMUFile *f);int64_t qemu_ftell (QEMUFile *f);void qemu_put_buffer (QEMUFile *f,const uint8_t *buf,int size);void qemu_put_byte (QEMUFile *f,int v); +/* +* put_buffer without copying the buffer. +* The buffer should be available till it is sent. +*/ +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size);static inline void qemu_put_ubyte (QEMUFile *f,unsigned int v) {
savevm.c +28 -9
@@ -621,6 +621,30 @@ int qemu_fclose(QEMUFile *f)return ret; } + +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size) +{ +if (f->last_error) { +return ; + } + +if (f->is_write ==0 && f->buf_index >0 ) { +fprintf (stderr, +"Attempted to write to buffer while read buffer is not empty\n" ); +abort (); + } + + f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt++].iov_len = size; + + f->is_write =1 ; + f->bytes_xfer += size; + +if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) { +qemu_fflush (f); + } +} +void qemu_put_buffer (QEMUFile *f,const uint8_t *buf,int size) {int l;
@@ -640,19 +664,14 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)if (l > size) l = size;memcpy (f->buf + f->buf_index, buf, l); - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; - f->iov[f->iovcnt++].iov_len = l; f->is_write =1 ; f->buf_index += l; - f->bytes_xfer += l; +qemu_put_buffer_no_copy (f, f->buf + (f->buf_index - l), l); +if (qemu_file_get_error (f)) { +break ; + } buf += l; size -= l; -if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) { -qemu_fflush (f); -if (qemu_file_get_error (f)) { -break ; - } - } } }
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 7/9] Add qemu_put_buffer_no_copy
Quoted 35 lines · Orit Wasserman
This allow us to add a buffer to the iovec to send without copying it into the static buffer.
Signed-off-by Orit Wasserman <[email protected]>--- include/migration/qemu-file.h | 5 +++++ savevm.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 33 insertions(+), 9 deletions(-)include/migration/qemu-file.h +5
-0@@ -75,6 +75,11 @@ int qemu_fclose(QEMUFile *f);int64_t qemu_ftell (QEMUFile *f);void qemu_put_buffer (QEMUFile *f,const uint8_t *buf,int size);void qemu_put_byte (QEMUFile *f,int v); +/* +* put_buffer without copying the buffer. +* The buffer should be available till it is sent. +*/ +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size);static inline void qemu_put_ubyte (QEMUFile *f,unsigned int v) {savevm.c +2
-0@@ -621,6 +621,30 @@ int qemu_fclose(QEMUFile *f)return ret; } + +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size)
savevm.c int qemu_fclose(QEMUFile *f) · line 635
Quoted 10 lines
savevm.c +10
-0+{ +if (f->last_error) { +return ; + } + +if (f->is_write ==0 && f->buf_index >0 ) { +fprintf (stderr, +"Attempted to write to buffer while read buffer is not empty\n" ); +abort (); + }
I don't understand this test at all (yes, I know that the test already existed).
We shouldn't never arrived qemu_put_buffer() with a QEMUFile with f->is_write == 0.
Change it for one assert()?
savevm.c int qemu_fclose(QEMUFile *f) · line 638
Quoted 2 lines
savevm.c +2
-0+ f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt++].iov_len = size;
This is clearly wrong, or I have completely missunderstood it (I will give a 50% to each posiblitiy).
Here we should be using "buf" and "size" passed as paramenters, f->buf and f->buf_index shouldn't be used, no?
savevm.c int qemu_fclose(QEMUFile *f) · line 639
Quoted 2 lines
savevm.c +2
-0+ + f->is_write =1 ;
is_write is completely redundant, and should just be a:
f->ops->put_buffer test (or now with writev). We only set it up when there is anything to be written?
But again, this is independent of this series.
Re: [Qemu-devel] [PATCH v3 7/9] Add qemu_put_buffer_no_copy
Quoted 55 lines · Juan Quintela
Quoted 35 lines · Orit Wasserman
This allow us to add a buffer to the iovec to send without copying it into the static buffer.
Signed-off-by Orit Wasserman <[email protected]>--- include/migration/qemu-file.h | 5 +++++ savevm.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 33 insertions(+), 9 deletions(-)include/migration/qemu-file.h +5
-0@@ -75,6 +75,11 @@ int qemu_fclose(QEMUFile *f);int64_t qemu_ftell (QEMUFile *f);void qemu_put_buffer (QEMUFile *f,const uint8_t *buf,int size);void qemu_put_byte (QEMUFile *f,int v); +/* +* put_buffer without copying the buffer. +* The buffer should be available till it is sent. +*/ +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size);static inline void qemu_put_ubyte (QEMUFile *f,unsigned int v) {savevm.c +2
-0@@ -621,6 +621,30 @@ int qemu_fclose(QEMUFile *f)return ret; } + +void qemu_put_buffer_no_copy (QEMUFile *f,const uint8_t *buf,int size)savevm.c int qemu_fclose(QEMUFile *f) · line 635
Quoted 10 lines
savevm.c +10
-0+{ +if (f->last_error) { +return ; + } + +if (f->is_write ==0 && f->buf_index >0 ) { +fprintf (stderr, +"Attempted to write to buffer while read buffer is not empty\n" ); +abort (); + }I don't understand this test at all (yes, I know that the test already existed).
We shouldn't never arrived
qemu_put_buffer()with a QEMUFile withf->is_write== 0.Change it for one
assert()?
Quoted 8 lines
savevm.c int qemu_fclose(QEMUFile *f) · line 638
Quoted 2 lines
savevm.c +2
-0+ f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt++].iov_len = size;This is clearly wrong, or I have completely missunderstood it (I will give a 50% to each posiblitiy).
Here we should be using "buf" and "size" passed as paramenters,
f->bufandf->buf_indexshouldn't be used, no?
you are right, it somehow moved to the next patch :( I will send a fixed v4 after you finish review the series ..
Quoted 11 lines
savevm.c int qemu_fclose(QEMUFile *f) · line 639
Quoted 3 lines
savevm.c +2
-0+ + f->is_write =1 ;
is_writeis completely redundant, and should just be a:
f->ops->put_buffertest (or now with writev). We only set it up when there is anything to be written?But again, this is independent of this series.
I agree, maybe we need a cleanup series and remove it. It will certainly simplify this code
Orit
PATCH v3 8/9 Use qemu_put_buffer_no_copy for guest memory pages
In reply to Orit WassermanThis will remove an unneeded copy of guest memory pages. For the page header and device state we still copy the data to the static buffer the other option is to allocate the memory on demand which is more expensive.
--- arch_init.c | 2 +- savevm.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch_init.c +1 -1
@@ -481,7 +481,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)/* XBZRLE overflow or normal page */ if (bytes_sent ==-1 ) { bytes_sent =save_block_hdr (f, block, offset, cont,RAM_SAVE_FLAG_PAGE ); -qemu_put_buffer (f, p,TARGET_PAGE_SIZE ); +qemu_put_buffer_no_copy (f, p,TARGET_PAGE_SIZE ); bytes_sent +=TARGET_PAGE_SIZE ; acct_info.norm_pages++; }
savevm.c +1 -1
@@ -634,7 +634,7 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; f->iov[f->iovcnt++].iov_len = size; f->is_write =1 ;
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 8/9] Use qemu_put_buffer_no_copy for guest memory pages
Quoted 6 lines · Orit Wasserman
This will remove an unneeded copy of guest memory pages. For the page header and device state we still copy the data to the static buffer the other option is to allocate the memory on demand which is more expensive.
Signed-off-by Orit Wasserman <[email protected]>
Okay so with this, the _nocopy can be later rewritten to do vmsplice to save another copy? Cool.
Quoted 33 lines
--- arch_init.c | 2 +- savevm.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch_init.c +1
-1@@ -481,7 +481,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)/* XBZRLE overflow or normal page */ if (bytes_sent ==-1 ) { bytes_sent =save_block_hdr (f, block, offset, cont,RAM_SAVE_FLAG_PAGE ); -qemu_put_buffer (f, p,TARGET_PAGE_SIZE ); +qemu_put_buffer_no_copy (f, p,TARGET_PAGE_SIZE ); bytes_sent +=TARGET_PAGE_SIZE ; acct_info.norm_pages++; }savevm.c +1
-1@@ -634,7 +634,7 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; f->iov[f->iovcnt++].iov_len = size; f->is_write =1 ;Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 8/9] Use qemu_put_buffer_no_copy for guest memory pages
In reply to Orit WassermanQuoted 16 lines · Orit Wasserman
This will remove an unneeded copy of guest memory pages. For the page header and device state we still copy the data to the static buffer the other option is to allocate the memory on demand which is more expensive.
Signed-off-by Orit Wasserman <[email protected]>--- arch_init.c | 2 +- savevm.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch_init.c +0
-0
arch_init.c static int ram_save_block(QEMUFile *f, bool last_stage) · line 487
Quoted 8 lines
arch_init.c +1
-1@@ -481,7 +481,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)/* XBZRLE overflow or normal page */ if (bytes_sent ==-1 ) { bytes_sent =save_block_hdr (f, block, offset, cont,RAM_SAVE_FLAG_PAGE ); -qemu_put_buffer (f, p,TARGET_PAGE_SIZE ); +qemu_put_buffer_no_copy (f, p,TARGET_PAGE_SIZE ); bytes_sent +=TARGET_PAGE_SIZE ; acct_info.norm_pages++; }
Once here, shouldn't we also change:
block-migration.c::blk_send()
qemu_put_buffer(f, blk->buf, BLOCK_SIZE);
to nocopy?
Again, this can be an additional patch.
savevm.c void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size) · line 638
Quoted 11 lines
savevm.c +1
-1@@ -634,7 +634,7 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; f->iov[f->iovcnt++].iov_len = size;
This bit should be in the previous patch, the error that I pointed?
Later, Juan.
Re: [Qemu-devel] [PATCH v3 8/9] Use qemu_put_buffer_no_copy for guest memory pages
Quoted 28 lines · Juan Quintela
Quoted 16 lines · Orit Wasserman
This will remove an unneeded copy of guest memory pages. For the page header and device state we still copy the data to the static buffer the other option is to allocate the memory on demand which is more expensive.
Signed-off-by Orit Wasserman <[email protected]>--- arch_init.c | 2 +- savevm.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch_init.c +0
-0arch_init.c static int ram_save_block(QEMUFile *f, bool last_stage) · line 487
Quoted 8 lines
arch_init.c +1
-1@@ -481,7 +481,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)/* XBZRLE overflow or normal page */ if (bytes_sent ==-1 ) { bytes_sent =save_block_hdr (f, block, offset, cont,RAM_SAVE_FLAG_PAGE ); -qemu_put_buffer (f, p,TARGET_PAGE_SIZE ); +qemu_put_buffer_no_copy (f, p,TARGET_PAGE_SIZE ); bytes_sent +=TARGET_PAGE_SIZE ; acct_info.norm_pages++; }Once here, shouldn't we also change:
Quoted 5 lines
block-migration.c::blk_send()
qemu_put_buffer(f, blk->buf, BLOCK_SIZE);to nocopy?
Sadly no, I looked at the code and I saw the buffer is freed immediately after call blk_send :(. look at mig_save_device_dirty.
Orit
Quoted 20 lines
Again, this can be an additional patch.
savevm.c void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size) · line 638
Quoted 11 lines
savevm.c +1
-1@@ -634,7 +634,7 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; f->iov[f->iovcnt++].iov_len = size;This bit should be in the previous patch, the error that I pointed?
Later, Juan.
Re: [Qemu-devel] [PATCH v3 8/9] Use qemu_put_buffer_no_copy for guest memory pages
Quoted 31 lines · Orit Wasserman
Quoted 30 lines · Juan Quintela
Quoted 16 lines · Orit Wasserman
This will remove an unneeded copy of guest memory pages. For the page header and device state we still copy the data to the static buffer the other option is to allocate the memory on demand which is more expensive.
Signed-off-by Orit Wasserman <[email protected]>--- arch_init.c | 2 +- savevm.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch_init.c +0
-0arch_init.c static int ram_save_block(QEMUFile *f, bool last_stage) · line 487
Quoted 8 lines
arch_init.c +1
-1@@ -481,7 +481,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)/* XBZRLE overflow or normal page */ if (bytes_sent ==-1 ) { bytes_sent =save_block_hdr (f, block, offset, cont,RAM_SAVE_FLAG_PAGE ); -qemu_put_buffer (f, p,TARGET_PAGE_SIZE ); +qemu_put_buffer_no_copy (f, p,TARGET_PAGE_SIZE ); bytes_sent +=TARGET_PAGE_SIZE ; acct_info.norm_pages++; }Once here, shouldn't we also change:
block-migration.c::blk_send()
Quoted 7 lines
Quoted 3 lines
qemu_put_buffer(f, blk->buf, BLOCK_SIZE);to nocopy?
Sadly no, I looked at the code and I saw the buffer is freed immediately after call
blk_send:(. look atmig_save_device_dirty.Orit
Hmm if _nocopy also implies it is not safe to free the buffer immediately, then I'd rather put this in a name e.g. _async instead of _nocopy.
Quoted 20 lines
Quoted 20 lines
Again, this can be an additional patch.
savevm.c void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size) · line 638
Quoted 11 lines
savevm.c +1
-1@@ -634,7 +634,7 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = f->buf + f->buf_index; + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; f->iov[f->iovcnt++].iov_len = size;This bit should be in the previous patch, the error that I pointed?
Later, Juan.
PATCH v3 9/9 coalesce adjacent iovecs
In reply to Orit WassermanThis way we send one big buffer instead of many small ones
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
savevm.c +17 -4
@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;
@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);
Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>
Why does this happen BTW?
Quoted 45 lines
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +17
-4@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
It happens in the last phase when we send the device state that consists of a lot bytes and int field that are written using qemu_put_byte/be16/...
Quoted 46 lines
Quoted 46 lines
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +17
-4@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 11 lines · Orit Wasserman
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
It happens in the last phase when we send the device state that consists of a lot bytes and int field that are written using qemu_put_byte/be16/...
Confused I thought device_state does not use _nocopy? My idea of using vmsplice relies exactly on this:
we can not splice device state ...
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +17
-4@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Orit Wasserman
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
Quoted 7 lines
Quoted 3 lines
It happens in the last phase when we send the device state that consists of a lot bytes and int field that are written using qemu_put_byte/be16/...
Confused I thought
device_statedoes not use_nocopy? My idea of using vmsplice relies exactly on this:
we can not splice device state ...
qemu_put_buffer calls qemu_put_buffer_no_copy (you know code reuse) So I guess we will need to add some flag if we want to use vmsplice or not or split no_copy into external and internal. It won't be a big change
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +17
-4@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 14 lines · Orit Wasserman
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · Orit Wasserman
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
It happens in the last phase when we send the device state that consists of a lot bytes and int field that are written using qemu_put_byte/be16/...
Quoted 7 lines
Quoted 3 lines
Confused I thought
device_statedoes not use_nocopy? My idea of using vmsplice relies exactly on this:
we can not splice device state ...
qemu_put_buffercallsqemu_put_buffer_no_copy(you know code reuse) So I guess we will need to add some flag if we want to use vmsplice or not or splitno_copyinto external and internal. It won't be a big change
Fair enough. Have time to look into this? Data copies is like 15% CPU, more with swap ...
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +17
-4@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + } f->is_write =1 ; f->bytes_xfer += size;@@ -690,8 +696,15 @@ void qemu_put_byte(QEMUFile *f, int v) f->buf[f->buf_index++] = v; f->is_write =1 ; f->bytes_xfer +=1 ; - f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); - f->iov[f->iovcnt++].iov_len =1 ; + +/* check for adjoint buffer and colace them */ +if (f->iovcnt >0 && f->buf + (f->buf_index -1 ) == + f->iov[f->iovcnt -1 ].iov_base + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len +=1 ; + }else { + f->iov[f->iovcnt].iov_base = f->buf + (f->buf_index -1 ); + f->iov[f->iovcnt++].iov_len =1 ; + }if (f->buf_index >=IO_BUF_SIZE || f->iovcnt >=MAX_IOV_SIZE ) {qemu_fflush (f);Signature · 1 line
1.7.11.7
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
In reply to Michael S. TsirkinQuoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Orit Wasserman
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
Quoted 8 lines
Quoted 4 lines
It happens in the last phase when we send the device state that consists of a lot
bytes and int field that are written using qemu_put_byte/be16/...Confused I thought
device_statedoes not use_nocopy? My idea of using vmsplice relies exactly on this:
we can not splice device state ...
As it is today, I am not sure that we can use vmsplice() because we are sending:
<header>
<page>
<header>
<page>
<header>
<page>
We can optimize at some pount to write a bigger/different header and sent a bunch of pages together, but just now we don't have that code.
Later, Juan.
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 31 lines · Juan Quintela
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Orit Wasserman
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>Why does this happen BTW?
Quoted 8 lines
Quoted 4 lines
It happens in the last phase when we send the device state that consists of a lot
bytes and int field that are written using qemu_put_byte/be16/...Confused I thought
device_statedoes not use_nocopy? My idea of using vmsplice relies exactly on this:
we can not splice device state ...As it is today, I am not sure that we can use
vmsplice()because we are sending:<header>
<page>
<header>
<page>
<header>
<page>
Quoted 4 lines
We can optimize at some pount to write a bigger/different header and sent a bunch of pages together, but just now we don't have that code.
Later, Juan.
Sending the page can do vmsplice, can't it? Multipage is likely a good idea anyway, e.g. RDMA wants to do this too.
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
<header>
<page>
<header>
<page>
<header>
<page>
Quoted 8 lines
Quoted 4 lines
We can optimize at some pount to write a bigger/different header and sent a bunch of pages together, but just now we don't have that code.
Later, Juan.
Sending the page can do vmsplice, can't it? Multipage is likely a good idea anyway, e.g. RDMA wants to do this too.
RDMA requires de pages lock into memory, no?
Later, Juan.
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
Quoted 14 lines · Juan Quintela
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines
<header>
<page>
<header>
<page>
<header>
<page>We can optimize at some pount to write a bigger/different header and sent a bunch of pages together, but just now we don't have that code.
Later, Juan.
Quoted 7 lines
Quoted 3 lines
Sending the page can do vmsplice, can't it? Multipage is likely a good idea anyway, e.g. RDMA wants to do this too.
RDMA requires de pages lock into memory, no?
Later, Juan.
That's a separate issue, but the point is it is asynchronous. It can be made to behave exactly the same implementing this API:
- send async: lock, send
- flush: poll for completion, unlock
Re: [Qemu-devel] [PATCH v3 9/9] coalesce adjacent iovecs
In reply to Orit WassermanQuoted 12 lines · Orit Wasserman
This way we send one big buffer instead of many small ones
Signed-off-by Orit Wasserman <[email protected]>--- savevm.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)savevm.c +0
-0
savevm.c void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size) · line 639
Quoted 8 lines
savevm.c +1
-2@@ -634,8 +634,14 @@ void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size)abort (); } - f->iov[f->iovcnt].iov_base = (uint8_t *)buf; - f->iov[f->iovcnt++].iov_len = size; +/* check for adjoint buffer and colace them */
s/colace/coalesce/
savevm.c void qemu_put_buffer_no_copy(QEMUFile *f, const uint8_t *buf, int size) · line 644
Quoted 7 lines
savevm.c +7
-0+if (f->iovcnt >0 && buf == f->iov[f->iovcnt -1 ].iov_base + + f->iov[f->iovcnt -1 ].iov_len) { + f->iov[f->iovcnt -1 ].iov_len += size; + }else { + f->iov[f->iovcnt].iov_base = (uint8_t *)buf; + f->iov[f->iovcnt++].iov_len = size; + }
Can we create a function for this?
Would help make sure that we don't forget to "fix" both places when we have changes.
Later, Juan.
Il 21/03/2013 17:05, Orit Wasserman ha scritto:
Quoted 16 lines
In migration all data is copied to a static buffer in QEMUFile, this hurts our network bandwidth and CPU usage especially with large guests. We switched to iovec for storing different buffers to send (even a byte field is considered as a buffer) and use writev to send the iovec.
writev was chosen (as apposed to sendmsg) because it supprts non socket fds.Guest memory pages are not copied by calling a new function
qemu_put_buffer_no_copy.
The page header data and device state data are still copied into the static buffer. This data consists of a lot of bytes and integer fields and the static buffer is used to store it during batching.
Another improvement is changing qemu_putbe64/32/16 to create a single buffer instead of several byte sized buffer.git repository:
git://github.com/oritwas/qemu.gitsendv_v2
Quoted 7 lines
Change from v2:
Always send data for the iovec even ifwritev_bufferis not implemented. Coalesce adjacent iovecs to create one big buffer from small adjacent buffer.Changes from v1:
Useiov_sendfor socket.
Makewritev_bufferoptional and if it is not implemented useput_buffer
I didn't review it too closely, so I don't feel like giving my Reviewed-by. Still, I must say it is a very nice and simple solution. Kudos!
Just one question: patch 6 is not strictly needed anymore now that you have coalescing, no? Should Juan leave it out?
Paolo
Quoted 25 lines · Paolo Bonzini
Il 21/03/2013 17:05, Orit Wasserman ha scritto:
Quoted 16 lines
In migration all data is copied to a static buffer in QEMUFile, this hurts our network bandwidth and CPU usage especially with large guests. We switched to iovec for storing different buffers to send (even a byte field is considered as a buffer) and use writev to send the iovec.
writev was chosen (as apposed to sendmsg) because it supprts non socket fds.Guest memory pages are not copied by calling a new function
qemu_put_buffer_no_copy.
The page header data and device state data are still copied into the static buffer. This data consists of a lot of bytes and integer fields and the static buffer is used to store it during batching.
Another improvement is changing qemu_putbe64/32/16 to create a single buffer instead of several byte sized buffer.git repository:
git://github.com/oritwas/qemu.gitsendv_v2Quoted 7 lines
Change from v2:
Always send data for the iovec even ifwritev_bufferis not implemented. Coalesce adjacent iovecs to create one big buffer from small adjacent buffer.Changes from v1:
Useiov_sendfor socket.
Makewritev_bufferoptional and if it is not implemented useput_buffer
Quoted 6 lines
I didn't review it too closely, so I don't feel like giving my Reviewed-by. Still, I must say it is a very nice and simple solution. Kudos!
Just one question: patch 6 is not strictly needed anymore now that you have coalescing, no? Should Juan leave it out?
true, Juan had a small fix so I'm sending v4 I will remove it.
Quoted 3 lines
Paolo
Quoted 14 lines · Orit Wasserman
In migration all data is copied to a static buffer in QEMUFile, this hurts our network bandwidth and CPU usage especially with large guests. We switched to iovec for storing different buffers to send (even a byte field is considered as a buffer) and use writev to send the iovec.
writev was chosen (as apposed to sendmsg) because it supprts non socket fds.Guest memory pages are not copied by calling a new function
qemu_put_buffer_no_copy.
The page header data and device state data are still copied into the static buffer. This data consists of a lot of bytes and integer fields and the static buffer is used to store it during batching.
Another improvement is changing qemu_putbe64/32/16 to create a single buffer instead of several byte sized buffer.
Quoted 1 line
git repository:
git://github.com/oritwas/qemu.gitsendv_v2
it is still sendv_v2 or sendv_v3 O:-)
Quoted 21 lines
Change from v2:
Always send data for the iovec even ifwritev_bufferis not implemented. Coalesce adjacent iovecs to create one big buffer from small adjacent buffer.Changes from v1:
Useiov_sendfor socket.
Makewritev_bufferoptional and if it is not implemented useput_bufferFuture work: Make number of iovec changeable
Orit Wasserman (9):
Add QemuFileWritevBuffer QemuFileOps
Addsocket_writev_bufferfunction
Updatebytes_xferinqemu_put_byte
Store the data to send also in iovec
Use writev ops if available
More optimized qemu_put_be64/32/16
Addqemu_put_buffer_no_copy
Useqemu_put_buffer_no_copyfor guest memory pages coalesce adjacent iovecs
Quoted 4 lines
arch_init.c | 2 +- include/migration/qemu-file.h | 12 ++++ savevm.c | 127 ++++++++++++++++++++++++++++++++++-------- 3 files changed, 116 insertions(+), 25 deletions(-)
Really nice series. Just a couple of small nits.
Later, Juan.