Skip to contentSkip to conversation

buffer_is_zero / migration optimizations

PATCH In review Peter Lieven 4 participants 31 messages Mar 26, 2013 – Apr 8, 2013

lore
Share & tools
lore
#0 lore raw

v5 cover buffer_is_zero / migration optimizations

this is v5 of my patch series with various optimizations in zero buffer checking and migration tweaks.

thanks especially to Eric Blake, Orit Wassermann and Paolo Bonzini for reviewing.

v5:
- move zero splat vector to a different patch
- fix indentation of can_user_buffer_find_nonzero_offset()
- do not unroll the first loop in buffer_find_nonzero_offset() to optimize it for zero page checking
- use an older unrolled version of find_next_bit() without SIMD instruction as there is no evidence that the vectorized version is better if not even worse and the code is easier to understand.
- added a word in the commit message of patch 8 about the skipped pages field in QMP MigrationStats.
- fixed the order of key-value pairs of MigrationStats in qapi-schema.json
- updated info about the performance benefit of is_zero_page() to the latest benchmark results in the commit message.

v4:
- do not inline buffer_find_nonzero_offset()
- inline can_usebuffer_find_nonzero_offset() correctly
- readd asserts in buffer_find_nonzero_offset() as profiling shows they do not hurt.
- change last occurences of scalar 8 by BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
- avoid deferencing p already in patch 5 where we know that the page (p) is zero
- explicitly set bytes_sent = 0 if we skip a zero page. bytes_sent was 0 before, but it was not obvious.
- add accounting information for skipped zero pages
- fix errors reported by checkpatch.pl

v3:
- remove asserts, inline functions and add a check function if buffer_find_nonzero_offset() can be used.
- use above check function in buffer_is_zero() and find_next_bit().
- use buffer_is_nonzero_offset() directly to find zero pages. we know that all requirements are met for memory pages.
- fix C89 violation in buffer_is_zero().
- avoid derefencing p in ram_save_block() if we already know the page is zero.
- fix initialization of last_offset in reset_ram_globals().
- avoid skipping pages with offset == 0 in bulk stage in migration_bitmap_find_and_reset_dirty().
- compared to v1 check for zero pages also after bulk ram migration as there are guests (e.g. Windows) which zero out large amount of memory while running.

v2:
- fix description, add trivial zero check and add asserts to buffer_find_nonzero_offset.
- add a constant for the unroll factor of buffer_find_nonzero_offset
- replace is_dup_page() by buffer_is_zero()
- added test results to xbzrle patch
- optimize descriptions

Peter Lieven (10):
move vector definitions to qemu-common.h
add a zero splat vector to qemu-common.h
cutils: add a function to find non-zero content in a buffer
buffer_is_zero: use vector optimizations if possible
bitops: unroll while loop in find_next_bit()
migration: search for zero instead of dup pages
migration: add an indicator for bulk state of ram migration
migration: do not sent zero pages in bulk stage
migration: do not search dirty pages in bulk stage
migration: use XBZRLE only after bulk stage

 arch_init.c                   |   74 +++++++++++++++++++----------------------
 hmp.c                         |    2 ++
 include/migration/migration.h |    2 ++
 include/qemu-common.h         |   37 +++++++++++++++++++++
 migration.c                   |    3 +-
 qapi-schema.json              |    8 +++--
 qmp-commands.hx               |    3 +-
 util/bitops.c                 |   18 +++++++++-
 util/cutils.c                 |   60 +++++++++++++++++++++++++++++++++
 9 files changed, 162 insertions(+), 45 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH v5 1/10 move vector definitions to qemu-common.h

vector optimizations will now be used at various places not just in is_dup_page() in arch_init.c

---
 arch_init.c           |   20 --------------------
 include/qemu-common.h |   21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 20 deletions(-)
arch_init.c +0 -20
index e8ade9e..35974c2 100644
@@ -116,26 +116,6 @@ const uint32_t arch_type = QEMU_ARCH;
 #define RAM_SAVE_FLAG_CONTINUE 0x20
 #define RAM_SAVE_FLAG_XBZRLE   0x40
 
-#ifdef __ALTIVEC__
-#include <altivec.h>
-#define VECTYPE        vector unsigned char
-#define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
-#define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
-/* altivec.h may redefine the bool macro as vector type.
- * Reset it to POSIX semantics. */
-#undef bool
-#define bool _Bool
-#elif defined __SSE2__
-#include <emmintrin.h>
-#define VECTYPE        __m128i
-#define SPLAT(p)       _mm_set1_epi8(*(p))
-#define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
-#else
-#define VECTYPE        unsigned long
-#define SPLAT(p)       (*(p) * (~0UL / 255))
-#define ALL_EQ(v1, v2) ((v1) == (v2))
-#endif
-
 
 static struct defconfig_file {
     const char *filename;
include/qemu-common.h +21 -0
index 2371132..d7ad3a7 100644
@@ -448,4 +448,25 @@ int uleb128_decode_small(const uint8_t *in, uint32_t *n);
 
 void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 
+/* vector definitions */
+#ifdef __ALTIVEC__
+#include <altivec.h>
+#define VECTYPE        vector unsigned char
+#define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
+#define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
+/* altivec.h may redefine the bool macro as vector type.
+ * Reset it to POSIX semantics. */
+#undef bool
+#define bool _Bool
+#elif defined __SSE2__
+#include <emmintrin.h>
+#define VECTYPE        __m128i
+#define SPLAT(p)       _mm_set1_epi8(*(p))
+#define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
+#else
+#define VECTYPE        unsigned long
+#define SPLAT(p)       (*(p) * (~0UL / 255))
+#define ALL_EQ(v1, v2) ((v1) == (v2))
+#endif
+
 #endif
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH v5 2/10 add a zero splat vector to qemu-common.h

In reply to Peter Lieven
---
 include/qemu-common.h |    3 +++
 1 file changed, 3 insertions(+)
include/qemu-common.h +3 -0
index d7ad3a7..9022646 100644
@@ -453,6 +453,7 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <altivec.h>
 #define VECTYPE        vector unsigned char
 #define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
+#define ZERO_SPLAT     vec_splat(vec_ld(0, 0), 0)
 #define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
 /* altivec.h may redefine the bool macro as vector type.
  * Reset it to POSIX semantics. */
@@ -462,10 +463,12 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <emmintrin.h>
 #define VECTYPE        __m128i
 #define SPLAT(p)       _mm_set1_epi8(*(p))
+#define ZERO_SPLAT     _mm_setzero_si128()
 #define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
 #else
 #define VECTYPE        unsigned long
 #define SPLAT(p)       (*(p) * (~0UL / 255))
+#define ZERO_SPLAT     0x0UL
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif
 
Signature · 1 line
1.7.9.5
#3 lore raw

Re: [Qemu-devel] [PATCHv5 02/10] add a zero splat vector to qemu-common.h

Il 26/03/2013 10:58, Peter Lieven ha scritto:

Quoted 19 lines
---
 include/qemu-common.h |    3 +++
 1 file changed, 3 insertions(+)
include/qemu-common.h +1 -0
index d7ad3a7..9022646 100644
@@ -453,6 +453,7 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <altivec.h>
 #define VECTYPE        vector unsigned char
 #define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
+#define ZERO_SPLAT     vec_splat(vec_ld(0, 0), 0)
 #define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
 /* altivec.h may redefine the bool macro as vector type.
  * Reset it to POSIX semantics. */
@@ -462,10 +463,12 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <emmintrin.h>

include/qemu-common.h void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size); · line 473

Quoted 10 lines
include/qemu-common.h +2 -0
 #define VECTYPE        __m128i
 #define SPLAT(p)       _mm_set1_epi8(*(p))
+#define ZERO_SPLAT     _mm_setzero_si128()
 #define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
 #else
 #define VECTYPE        unsigned long
 #define SPLAT(p)       (*(p) * (~0UL / 255))
+#define ZERO_SPLAT     0x0UL
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif

C trivia of the day: this can be written simply as

(VECTYPE) {0}

Yes, it works even with unsigned long. :) It is a C99 compound literal.

Paolo

#4 lore raw

Re: [Qemu-devel] [PATCHv5 02/10] add a zero splat vector to qemu-common.h

Am 26.03.2013 um 11:14 schrieb Paolo Bonzini <[email protected]>:

Quoted 33 lines

Il 26/03/2013 10:58, Peter Lieven ha scritto:

Quoted 19 lines

---
include/qemu-common.h | 3 +++

1 file changed, 3 insertions(+)
include/qemu-common.h +1 -0
index d7ad3a7..9022646 100644
@@ -453,6 +453,7 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <altivec.h>
 #define VECTYPE        vector unsigned char
 #define SPLAT(p)       vec_splat(vec_ld(0, p), 0)
+#define ZERO_SPLAT     vec_splat(vec_ld(0, 0), 0)
 #define ALL_EQ(v1, v2) vec_all_eq(v1, v2)
 /* altivec.h may redefine the bool macro as vector type.
 * Reset it to POSIX semantics. */
@@ -462,10 +463,12 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #include <emmintrin.h>

include/qemu-common.h void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size); · line 473

Quoted 10 lines
include/qemu-common.h +2 -0
 #define VECTYPE        __m128i
 #define SPLAT(p)       _mm_set1_epi8(*(p))
+#define ZERO_SPLAT     _mm_setzero_si128()
 #define ALL_EQ(v1, v2) (_mm_movemask_epi8(_mm_cmpeq_epi8(v1, v2)) == 0xFFFF)
 #else
 #define VECTYPE        unsigned long
 #define SPLAT(p)       (*(p) * (~0UL / 255))
+#define ZERO_SPLAT     0x0UL
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif

C trivia of the day: this can be written simply as

Quoted 1 line

(VECTYPE) {0}

you are the first who mentions this ;-)

respin?

Peter

#5 lore raw

Re: [Qemu-devel] [PATCHv5 02/10] add a zero splat vector to qemu-common.h

Il 26/03/2013 11:17, Peter Lieven ha scritto:

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

C trivia of the day: this can be written simply as

(VECTYPE) {0}

you are the first who mentions this ;-)

respin?

Yes, please.

Paolo

#6 lore raw

Re: [Qemu-devel] [PATCHv5 02/10] add a zero splat vector to qemu-common.h

Am 26.03.2013 um 11:17 schrieb Paolo Bonzini <[email protected]>:

Quoted 10 lines

Il 26/03/2013 11:17, Peter Lieven ha scritto:

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

C trivia of the day: this can be written simply as

(VECTYPE) {0}

you are the first who mentions this ;-)

respin?

Yes, please.

Ok, I wait if someone finds something else.

Peter

Quoted 2 lines

Paolo

#7 lore raw

PATCH v5 3/10 cutils: add a function to find non-zero content in a buffer

In reply to Peter Lieven

this adds buffer_find_nonzero_offset() which is a SSE2/Altivec optimized function that searches for non-zero content in a buffer.

the function starts full unrolling only after the first few chunks have been checked one by one. analyzing real memory page data has revealed that non-zero pages are non-zero within the first 256-512 bits in most cases. as this function is also heavily used to check for zero memory pages this tweak has been made to avoid the high setup costs of the fully unrolled check for non-zero pages.

due to the optimizations used in the function there are restrictions on buffer address and search length. the function can_use_buffer_find_nonzero_content() can be used to check if the function can be used safely.

---
 include/qemu-common.h |   13 ++++++++++++
 util/cutils.c         |   55 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
include/qemu-common.h +13 -0
index 9022646..7c7c244 100644
@@ -472,4 +472,17 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif
 
+#define BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR 8
+static inline bool
+can_use_buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    if (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
+               * sizeof(VECTYPE)) == 0
+        && ((uintptr_t) buf) % sizeof(VECTYPE) == 0) {
+        return true;
+    }
+    return false;
+}
+size_t buffer_find_nonzero_offset(const void *buf, size_t len);
+
 #endif
util/cutils.c +55 -0

61 lines not on this page · raw message

--
1.7.9.5

#8 lore raw

Re: [Qemu-devel] [PATCHv5 03/10] cutils: add a function to find non-zero content in a buffer

Quoted 65 lines · Peter Lieven

this adds buffer_find_nonzero_offset() which is a SSE2/Altivec optimized function that searches for non-zero content in a buffer.

the function starts full unrolling only after the first few chunks have been checked one by one. analyzing real memory page data has revealed that non-zero pages are non-zero within the first 256-512 bits in most cases. as this function is also heavily used to check for zero memory pages this tweak has been made to avoid the high setup costs of the fully unrolled check for non-zero pages.

due to the optimizations used in the function there are restrictions on buffer address and search length. the function can_use_buffer_find_nonzero_content() can be used to check if the function can be used safely.

---
 include/qemu-common.h |   13 ++++++++++++
 util/cutils.c         |   55 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
include/qemu-common.h +13 -0
index 9022646..7c7c244 100644
@@ -472,4 +472,17 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif
 
+#define BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR 8
+static inline bool
+can_use_buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    if (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
+               * sizeof(VECTYPE)) == 0
+        && ((uintptr_t) buf) % sizeof(VECTYPE) == 0) {
+        return true;
+    }
+    return false;
+}
+size_t buffer_find_nonzero_offset(const void *buf, size_t len);
+
 #endif
util/cutils.c +13 -0
index 1439da4..0314a18 100644
@@ -143,6 +143,61 @@ int qemu_fdatasync(int fd)
 }
 
 /*
+ * Searches for an area with non-zero content in a buffer
+ *
+ * Attention! The len must be a multiple of
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
+ * and addr must be a multiple of sizeof(VECTYPE) due to
+ * restriction of optimizations in this function.
+ *
+ * can_use_buffer_find_nonzero_offset() can be used to check
+ * these requirements.
+ *
+ * The return value is the offset of the non-zero area rounded
+ * down to a multiple of sizeof(VECTYPE) for the first
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR chunks and down to

util/cutils.c int qemu_fdatasync(int fd) · line 168

Quoted 10 lines
util/cutils.c +10 -0
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
+ * afterwards.
+ *
+ * If the buffer is all zero the return value is equal to len.
+ */
+
+size_t buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    VECTYPE *p = (VECTYPE *)buf;
+    VECTYPE zero = ZERO_SPLAT;

If you have to resplit it anyways, what about changing this to:

-    VECTYPE *p = (VECTYPE *)buf;
-    VECTYPE zero = ZERO_SPLAT;
+    const VECTYPE *p = buf;
+    const VECTYPE zero = ZERO_SPLAT;
     size_t i;
Quoted 1 line

From the "I hate casts" film?

Thanks, Juan.

#9 lore raw

Re: [Qemu-devel] [PATCHv5 03/10] cutils: add a function to find non-zero content in a buffer

Am 26.03.2013 um 11:38 schrieb Juan Quintela <[email protected]>:

Quoted 79 lines
Quoted 65 lines · Peter Lieven

this adds buffer_find_nonzero_offset() which is a SSE2/Altivec optimized function that searches for non-zero content in a buffer.

the function starts full unrolling only after the first few chunks have been checked one by one. analyzing real memory page data has revealed that non-zero pages are non-zero within the first 256-512 bits in most cases. as this function is also heavily used to check for zero memory pages this tweak has been made to avoid the high setup costs of the fully unrolled check for non-zero pages.

due to the optimizations used in the function there are restrictions on buffer address and search length. the function can_use_buffer_find_nonzero_content() can be used to check if the function can be used safely.

---

include/qemu-common.h |   13 ++++++++++++
util/cutils.c         |   55 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 68 insertions(+)
include/qemu-common.h +13 -0
index 9022646..7c7c244 100644
@@ -472,4 +472,17 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif
 
+#define BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR 8
+static inline bool
+can_use_buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    if (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
+               * sizeof(VECTYPE)) == 0
+        && ((uintptr_t) buf) % sizeof(VECTYPE) == 0) {
+        return true;
+    }
+    return false;
+}
+size_t buffer_find_nonzero_offset(const void *buf, size_t len);
+
 #endif
util/cutils.c +13 -0
index 1439da4..0314a18 100644
@@ -143,6 +143,61 @@ int qemu_fdatasync(int fd)
 }
 
 /*
+ * Searches for an area with non-zero content in a buffer
+ *
+ * Attention! The len must be a multiple of
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
+ * and addr must be a multiple of sizeof(VECTYPE) due to
+ * restriction of optimizations in this function.
+ *
+ * can_use_buffer_find_nonzero_offset() can be used to check
+ * these requirements.
+ *
+ * The return value is the offset of the non-zero area rounded
+ * down to a multiple of sizeof(VECTYPE) for the first
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR chunks and down to

util/cutils.c int qemu_fdatasync(int fd) · line 168

Quoted 10 lines
util/cutils.c +10 -0
+ * BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR * sizeof(VECTYPE)
+ * afterwards.
+ *
+ * If the buffer is all zero the return value is equal to len.
+ */
+
+size_t buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    VECTYPE *p = (VECTYPE *)buf;
+    VECTYPE zero = ZERO_SPLAT;

If you have to resplit it anyways, what about changing this to:

Review

Quoted 4 lines
(quoted diff) +2 -2
-    VECTYPE *p = (VECTYPE *)buf;
-    VECTYPE zero = ZERO_SPLAT;
+    const VECTYPE *p = buf;
+    const VECTYPE zero = ZERO_SPLAT;

i would change it to

const VECTYPE zero = (VECTYPE) {0};

and drop patch 2 again.

Peter

Quoted 5 lines
    size_t i;

From the "I hate casts" film?

Thanks, Juan.

#10 lore raw

Re: [Qemu-devel] [PATCHv5 03/10] cutils: add a function to find non-zero content in a buffer

In reply to Peter Lieven
Quoted 31 lines · Peter Lieven

this adds buffer_find_nonzero_offset() which is a SSE2/Altivec optimized function that searches for non-zero content in a buffer.

the function starts full unrolling only after the first few chunks have been checked one by one. analyzing real memory page data has revealed that non-zero pages are non-zero within the first 256-512 bits in most cases. as this function is also heavily used to check for zero memory pages this tweak has been made to avoid the high setup costs of the fully unrolled check for non-zero pages.

due to the optimizations used in the function there are restrictions on buffer address and search length. the function can_use_buffer_find_nonzero_content() can be used to check if the function can be used safely.

---
 include/qemu-common.h |   13 ++++++++++++
 util/cutils.c         |   55 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
include/qemu-common.h +1 -0
index 9022646..7c7c244 100644
@@ -472,4 +472,17 @@ void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size);
 #define ALL_EQ(v1, v2) ((v1) == (v2))
 #endif
 
+#define BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR 8

include/qemu-common.h void hexdump(const char *buf, FILE *fp, const char *prefix, size_t size); · line 485

Quoted 10 lines
include/qemu-common.h +10 -0
+static inline bool
+can_use_buffer_find_nonzero_offset(const void *buf, size_t len)
+{
+    if (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
+               * sizeof(VECTYPE)) == 0
+        && ((uintptr_t) buf) % sizeof(VECTYPE) == 0) {
+        return true;
+    }
+    return false;
+}

This can be spelled as:

return (len % (BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
* sizeof(VECTYPE)) == 0
&& ((uintptr_t) buf) % sizeof(VECTYPE) == 0);;

But I don't care too much.

#11 lore raw

PATCH v5 4/10 buffer_is_zero: use vector optimizations if possible

In reply to Peter Lieven

performance gain on SSE2 is approx. 20-25%. altivec is not tested. performance for unsigned long arithmetic is unchanged.

---
 util/cutils.c |    5 +++++
 1 file changed, 5 insertions(+)
util/cutils.c +5 -0
index 0314a18..daf032a 100644
@@ -215,6 +215,11 @@ bool buffer_is_zero(const void *buf, size_t len)
     long d0, d1, d2, d3;
     const long * const data = buf;
 
+    /* use vector optimized zero check if possible */
+    if (can_use_buffer_find_nonzero_offset(buf, len)) {
+        return buffer_find_nonzero_offset(buf, len) == len;
+    }
+
     assert(len % (4 * sizeof(long)) == 0);
     len /= sizeof(long);
 
Signature · 1 line
1.7.9.5
#12 lore raw

PATCH v5 5/10 bitops: unroll while loop in find_next_bit()

In reply to Peter Lieven

this patch adopts the loop unrolling idea of bitmap_is_zero() to speed up the skipping of large areas with zeros in find_next_bit().

this routine is extensively used to find dirty pages in live migration.

testing only the find_next_bit performance on a zeroed bitfield the loop onrolling decreased executing time by approx. 50% on x86_64.

---
 util/bitops.c |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
util/bitops.c +17 -1
index e72237a..227c38b 100644
@@ -42,7 +42,23 @@ unsigned long find_next_bit(const unsigned long *addr, unsigned long size,
         size -= BITS_PER_LONG;
         result += BITS_PER_LONG;
     }
-    while (size & ~(BITS_PER_LONG-1)) {
+    while (size >= 4*BITS_PER_LONG) {
+        unsigned long d1, d2, d3;
+        tmp = *p;
+        d1 = *(p+1);
+        d2 = *(p+2);
+        d3 = *(p+3);
+        if (tmp) {
+            goto found_middle;
+        }
+        if (d1 | d2 | d3) {
+            break;
+        }
+        p += 4;
+        result += 4*BITS_PER_LONG;
+        size -= 4*BITS_PER_LONG;
+    }
+    while (size >= BITS_PER_LONG) {
         if ((tmp = *(p++))) {
             goto found_middle;
         }
Signature · 1 line
1.7.9.5
#13 lore raw

PATCH v5 6/10 migration: search for zero instead of dup pages

In reply to Peter Lieven

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

---
 arch_init.c |   21 ++++++---------------
 1 file changed, 6 insertions(+), 15 deletions(-)
arch_init.c +6 -15
index 35974c2..dd5deff 100644
@@ -146,19 +146,10 @@ int qemu_read_default_config_files(bool userconfig)
     return 0;
 }
 
-static int is_dup_page(uint8_t *page)
+static inline bool is_zero_page(uint8_t *p)
 {
-    VECTYPE *p = (VECTYPE *)page;
-    VECTYPE val = SPLAT(page);
-    int i;
-
-    for (i = 0; i < TARGET_PAGE_SIZE / sizeof(VECTYPE); i++) {
-        if (!ALL_EQ(val, p[i])) {
-            return 0;
-        }
-    }
-
-    return 1;
+    return buffer_find_nonzero_offset(p, TARGET_PAGE_SIZE) ==
+        TARGET_PAGE_SIZE;
 }
 
 /* struct contains XBZRLE cache and a static page
@@ -445,12 +436,12 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
 
             /* In doubt sent page as normal */
             bytes_sent = -1;
-            if (is_dup_page(p)) {
+            if (is_zero_page(p)) {
                 acct_info.dup_pages++;
                 bytes_sent = save_block_hdr(f, block, offset, cont,
                                             RAM_SAVE_FLAG_COMPRESS);
-                qemu_put_byte(f, *p);
-                bytes_sent += 1;
+                qemu_put_byte(f, 0);
+                bytes_sent++;
             } else if (migrate_use_xbzrle()) {
                 current_addr = block->offset + offset;
                 bytes_sent = save_xbzrle_page(f, p, current_addr, block,
Signature · 1 line
1.7.9.5
#14 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Am 26.03.2013 um 10:58 hat Peter Lieven geschrieben:

Quoted 9 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

Quoted 7 lines

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

The problem seems to be that the RAM of a guest is in fact _not_ zeroed during initialisation. It hits my test case reliably because I'm running with MALLOC_PERTURB_. Now I'm wondering if in practice this happens only under such test conditions, or if real guests could be affected as well and we should make sure to get zeroed memory for RAM.

Kevin

#15 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Il 05/04/2013 21:23, Kevin Wolf ha scritto:

Quoted 16 lines
Quoted 9 lines
Quoted 9 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

Quoted 7 lines
Quoted 7 lines

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Quoted 8 lines

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

The problem seems to be that the RAM of a guest is in fact _not_ zeroed during initialisation. It hits my test case reliably because I'm running with MALLOC_PERTURB_. Now I'm wondering if in practice this happens only under such test conditions, or if real guests could be affected as well and we should make sure to get zeroed memory for RAM.

I think we should MADV_DONTNEED it.

Paolo

#16 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Am 05.04.2013 um 22:00 schrieb Paolo Bonzini <[email protected]>:

Quoted 20 lines

Il 05/04/2013 21:23, Kevin Wolf ha scritto:

Quoted 19 lines
Quoted 9 lines
Quoted 9 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

Quoted 7 lines
Quoted 7 lines

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

Quoted 7 lines
Quoted 5 lines

The problem seems to be that the RAM of a guest is in fact _not_ zeroed during initialisation. It hits my test case reliably because I'm running with MALLOC_PERTURB_. Now I'm wondering if in practice this happens only under such test conditions, or if real guests could be affected as well and we should make sure to get zeroed memory for RAM.

I think we should MADV_DONTNEED it.

This does not guarantee that the memory is unmapped afaik. Sadly, I think we have to revert

migration: do not sent zero pages in bulk stage

The memory assigned by posix_memalign is most likely zero as all GFP_USER pages
are zeroed out by the kernel on alloc (at least under Linux), but if the page is reused in the same process it is not necessarily zero anymore.

What I was trying to achieve with this patch is that the memset when receiving a zero_page at the target was allocating memory and the MADV_DONTNEED was not immediately dropping the page. This lead to memory pressure and swapping etc. on overcommitted systems.

What I would propose as a solution for this is after reverting the "do not sent zero pages" patch is sth like this when receiving a compressed page:

if (ch != 0 || !is_zero_page(host)) {
    memset(host, ch, TARGET_PAGE_SIZE);
}

Regarding Kevins observation of the speed regression in iotest 007 this is simply if MALLOC_PERTURB_ is used there are simply no zero pages, but only dup pages in memory. On a real system the observation is that pages are either zero or not dup at all.

Peter

Quoted 2 lines

Paolo

#17 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

In reply to Paolo Bonzini

Am 05.04.2013 um 22:00 schrieb Paolo Bonzini <[email protected]>:

Quoted 20 lines

Il 05/04/2013 21:23, Kevin Wolf ha scritto:

Quoted 19 lines
Quoted 9 lines
Quoted 9 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

Quoted 7 lines
Quoted 7 lines

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

Quoted 7 lines
Quoted 5 lines

The problem seems to be that the RAM of a guest is in fact _not_ zeroed during initialisation. It hits my test case reliably because I'm running with MALLOC_PERTURB_. Now I'm wondering if in practice this happens only under such test conditions, or if real guests could be affected as well and we should make sure to get zeroed memory for RAM.

I think we should MADV_DONTNEED it.

i have to correct myself, on Linux it seems that MADV_DONTNEED guarantees that subsequent reads will return a zero filled page.

If I am right with that we could MADV_DONTNEED the memory on startup and keep not sending zero pages in bulk stage if we are running under Linux. Am I right with that?

Peter

#18 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Il 06/04/2013 00:06, Peter Lieven ha scritto:

Quoted 6 lines
Quoted 1 line
Quoted 1 line

I think we should MADV_DONTNEED it.

i have to correct myself, on Linux it seems that MADV_DONTNEED guarantees that subsequent reads will return a zero filled page.

If I am right with that we could MADV_DONTNEED the memory on startup and keep not sending zero pages in bulk stage if we are running under Linux. Am I right with that?

Actually we can use mmap+munmap to allocate a well-aligned, always zero block for the RAM.

Paolo

#19 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Am 08.04.2013 um 10:38 schrieb Paolo Bonzini <[email protected]>:

Quoted 10 lines

Il 06/04/2013 00:06, Peter Lieven ha scritto:

Quoted 6 lines
Quoted 1 line
Quoted 1 line

I think we should MADV_DONTNEED it.

i have to correct myself, on Linux it seems that MADV_DONTNEED guarantees that subsequent reads will return a zero filled page.

If I am right with that we could MADV_DONTNEED the memory on startup and keep not sending zero pages in bulk stage if we are running under Linux. Am I right with that?

Actually we can use mmap+munmap to allocate a well-aligned, always zero block for the RAM.

like this?

exec.c +21 -23
index 786987a..dfd1ad3 100644
@@ -1048,27 +1048,31 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
         new_block->host = host;
         new_block->flags |= RAM_PREALLOC_MASK;
     } else {
-        if (mem_path) {
+        if (xen_enabled()) {
+            xen_ram_alloc(new_block->offset, size, mr);
+            memory_try_enable_merging(new_block->host, size);
+        } else {
+            if (mem_path) {
 #if defined (__linux__) && !defined(TARGET_S390X)
-            new_block->host = file_ram_alloc(new_block, size, mem_path);
-            if (!new_block->host) {
-                new_block->host = qemu_vmalloc(size);
-                memory_try_enable_merging(new_block->host, size);
-            }
+                new_block->host = file_ram_alloc(new_block, size, mem_path);
 #else
-            fprintf(stderr, "-mem-path option unsupported\n");
-            exit(1);
+                fprintf(stderr, "-mem-path option unsupported\n");
+                exit(1);
 #endif
-        } else {
-            if (xen_enabled()) {
-                xen_ram_alloc(new_block->offset, size, mr);
-            } else if (kvm_enabled()) {
+            } 
+            if (!new_block->host && kvm_enabled()) {
                 /* some s390/kvm configurations have special constraints */
                 new_block->host = kvm_vmalloc(size);
-            } else {
-                new_block->host = qemu_vmalloc(size);
             }
-            memory_try_enable_merging(new_block->host, size);
+            if (!new_block->host) {
+                new_block->host = mmap(0, size, PROT_READ | PROT_WRITE, 
+                                       MAP_ANONYMOUS |  MAP_PRIVATE, -1, 0);
+                if (new_block->host == MAP_FAILED) {
+                    fprintf(stderr,"qemu_ram_alloc: can't mmap RAM pages %d\n",errno);
+                    exit(1);
+                }
+                memory_try_enable_merging(new_block->host, size);
+            }
         }
     }
     new_block->length = size;
@@ -1142,25 +1146,19 @@ void qemu_ram_free(ram_addr_t addr)
                 ;
             } else if (mem_path) {
 #if defined (__linux__) && !defined(TARGET_S390X)
+                munmap(block->host, block->length);
                 if (block->fd) {
-                    munmap(block->host, block->length);
                     close(block->fd);
-                } else {
-                    qemu_vfree(block->host);
                 }
 #else
                 abort();
 #endif
             } else {
-#if defined(TARGET_S390X) && defined(CONFIG_KVM)
-                munmap(block->host, block->length);
-#else
                 if (xen_enabled()) {
                     xen_invalidate_map_cache_entry(block->host);
                 } else {
-                    qemu_vfree(block->host);
+                    munmap(block->host, block->length);
                 }
-#endif
             }
             g_free(block);
             break;
kvm-all.c +1 -1
index 9b433d3..90be8e3 100644
@@ -1742,7 +1742,7 @@ void *kvm_vmalloc(ram_addr_t size)
         return mem;
     }
 #endif
-    return qemu_vmalloc(size);
+    return NULL;
 }
 
 void kvm_setup_guest_memory(void *start, size_t size)
#20 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Il 08/04/2013 11:25, Peter Lieven ha scritto:

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

Actually we can use mmap+munmap to allocate a well-aligned, always zero block for the RAM.

like this?

This doesn't align it. I'll send a patch shortly.

Paolo

#21 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

In reply to Kevin Wolf

Am 05.04.2013 um 21:23 schrieb Kevin Wolf <[email protected]>:

Quoted 14 lines

Am 26.03.2013 um 10:58 hat Peter Lieven geschrieben:

Quoted 13 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Quoted 7 lines
Quoted 3 lines

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

Can you try if the following solves your issue:

exec.c +1 -0
index 786987a..54baa4a 100644
@@ -1071,6 +1071,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
             memory_try_enable_merging(new_block->host, size);
         }
     }
+    qemu_madvise(new_block->host, size, QEMU_MADV_DONTNEED);
     new_block->length = size;
 
     /* Keep the list sorted from biggest to smallest block.  */

BR,
Peter

#22 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Am 08.04.2013 um 10:33 schrieb Peter Lieven <[email protected]>:

Quoted 25 lines

Am 05.04.2013 um 21:23 schrieb Kevin Wolf <[email protected]>:

Quoted 21 lines

Am 26.03.2013 um 10:58 hat Peter Lieven geschrieben:

Quoted 9 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

Quoted 7 lines

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

actually this might be better?!:

exec.c +5 -0
index 786987a..334a46e 100644
@@ -1071,6 +1071,11 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
             memory_try_enable_merging(new_block->host, size);
         }
     }
+    if (mem_prealloc) {
+        memset(new_block->host, size, 0x00);
+    } else {
+        qemu_madvise(new_block->host, size, QEMU_MADV_DONTNEED);
+    }
     new_block->length = size;
 
     /* Keep the list sorted from biggest to smallest block.  */
#23 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

In reply to Peter Lieven

Am 08.04.2013 um 10:33 hat Peter Lieven geschrieben:

Quoted 32 lines

Am 05.04.2013 um 21:23 schrieb Kevin Wolf <[email protected]>:

Quoted 14 lines

Am 26.03.2013 um 10:58 hat Peter Lieven geschrieben:

Quoted 13 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Quoted 7 lines
Quoted 3 lines

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

Can you try if the following solves your issue:

exec.c +0 -0
index 786987a..54baa4a 100644

exec.c ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host, · line 1077

Quoted 7 lines
exec.c +1 -0
@@ -1071,6 +1071,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
             memory_try_enable_merging(new_block->host, size);
         }
     }
+    qemu_madvise(new_block->host, size, QEMU_MADV_DONTNEED);
     new_block->length = size;
 
     /* Keep the list sorted from biggest to smallest block.  */

It does. But perhaps Paolo's suggestion of using mmap() to allocate the memory would be better. I'm not sure how MADV_DONTNEED behaves on non-Linux.

Kevin

#24 lore raw

Re: [Qemu-devel] [PATCHv5 06/10] migration: search for zero instead of dup pages

Am 08.04.2013 um 10:49 schrieb Kevin Wolf <[email protected]>:

Quoted 39 lines

Am 08.04.2013 um 10:33 hat Peter Lieven geschrieben:

Quoted 38 lines

Am 05.04.2013 um 21:23 schrieb Kevin Wolf <[email protected]>:

Quoted 14 lines

Am 26.03.2013 um 10:58 hat Peter Lieven geschrieben:

Quoted 13 lines

virtually all dup pages are zero pages. remove the special is_dup_page() function and use the optimized buffer_find_nonzero_offset() function instead.

here buffer_find_nonzero_offset() is used directly to avoid the unnecssary additional checks in buffer_is_zero().

raw performace gain checking 1 GByte zeroed memory over is_dup_page() is approx. 10-12% with SSE2 and 8-10% with unsigned long arithmedtic.

Quoted 7 lines
Quoted 3 lines

Okay, so I bisected again and this is the second patch that is involved in the slowness of qemu-iotests case 007.

Can you try if the following solves your issue:

exec.c +1 -0
index 786987a..54baa4a 100644
@@ -1071,6 +1071,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
            memory_try_enable_merging(new_block->host, size);
        }
    }
+    qemu_madvise(new_block->host, size, QEMU_MADV_DONTNEED);
    new_block->length = size;
 
Quoted 5 lines

exec.c ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host, · line 1077

Quoted 1 line
exec.c +0 -0
    /* Keep the list sorted from biggest to smallest block.  */

It does. But perhaps Paolo's suggestion of using mmap() to allocate the memory would be better. I'm not sure how MADV_DONTNEED behaves on non-Linux.

its not guaranteed to zero memory.

Peter

Quoted 2 lines

Kevin

#25 lore raw

PATCH v5 7/10 migration: add an indicator for bulk state of ram migration

In reply to Peter Lieven

the first round of ram transfer is special since all pages are dirty and thus all memory pages are transferred to the target. this patch adds a boolean variable to track this stage.

---
 arch_init.c |    3 +++
 1 file changed, 3 insertions(+)
arch_init.c +3 -0
index dd5deff..1291bd2 100644
@@ -319,6 +319,7 @@ static ram_addr_t last_offset;
 static unsigned long *migration_bitmap;
 static uint64_t migration_dirty_pages;
 static uint32_t last_version;
+static bool ram_bulk_stage;
 
 static inline
 ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
@@ -426,6 +427,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
             if (!block) {
                 block = QTAILQ_FIRST(&ram_list.blocks);
                 complete_round = true;
+                ram_bulk_stage = false;
             }
         } else {
             uint8_t *p;
@@ -529,6 +531,7 @@ static void reset_ram_globals(void)
     last_sent_block = NULL;
     last_offset = 0;
     last_version = ram_list.version;
+    ram_bulk_stage = true;
 }
 
 #define MAX_WAIT 50 /* ms, half buffered_file limit */
Signature · 1 line
1.7.9.5
#26 lore raw

PATCH v5 8/10 migration: do not sent zero pages in bulk stage

In reply to Peter Lieven

during bulk stage of ram migration if a page is a zero page do not send it at all.
the memory at the destination reads as zero anyway.

even if there is an madvise with QEMU_MADV_DONTNEED at the target upon receipt of a zero page I have observed that the target starts swapping if the memory is overcommitted. it seems that the pages are dropped asynchronously.

this patch also updates QMP to return the number of skipped pages in MigrationStats.

---
 arch_init.c                   |   24 ++++++++++++++++++++----
 hmp.c                         |    2 ++
 include/migration/migration.h |    2 ++
 migration.c                   |    3 ++-
 qapi-schema.json              |    8 +++++---
 qmp-commands.hx               |    3 ++-
 6 files changed, 33 insertions(+), 9 deletions(-)
arch_init.c +20 -4
index 1291bd2..3a0d02e 100644
@@ -183,6 +183,7 @@ int64_t xbzrle_cache_resize(int64_t new_size)
 /* accounting for migration statistics */
 typedef struct AccountingInfo {
     uint64_t dup_pages;
+    uint64_t skipped_pages;
     uint64_t norm_pages;
     uint64_t iterations;
     uint64_t xbzrle_bytes;
@@ -208,6 +209,16 @@ uint64_t dup_mig_pages_transferred(void)
     return acct_info.dup_pages;
 }
 
+uint64_t skipped_mig_bytes_transferred(void)
+{
+    return acct_info.skipped_pages * TARGET_PAGE_SIZE;
+}
+
+uint64_t skipped_mig_pages_transferred(void)
+{
+    return acct_info.skipped_pages;
+}
+
 uint64_t norm_mig_bytes_transferred(void)
 {
     return acct_info.norm_pages * TARGET_PAGE_SIZE;
@@ -440,10 +451,15 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
             bytes_sent = -1;
             if (is_zero_page(p)) {
                 acct_info.dup_pages++;
-                bytes_sent = save_block_hdr(f, block, offset, cont,
-                                            RAM_SAVE_FLAG_COMPRESS);
-                qemu_put_byte(f, 0);
-                bytes_sent++;
+                if (!ram_bulk_stage) {
+                    bytes_sent = save_block_hdr(f, block, offset, cont,
+                                                RAM_SAVE_FLAG_COMPRESS);
+                    qemu_put_byte(f, 0);
+                    bytes_sent++;
+                } else {
+                    acct_info.skipped_pages++;
+                    bytes_sent = 0;
+                }
             } else if (migrate_use_xbzrle()) {
                 current_addr = block->offset + offset;
                 bytes_sent = save_xbzrle_page(f, p, current_addr, block,
hmp.c +2 -0
index b0a861c..e3e833e 100644
@@ -173,6 +173,8 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
                        info->ram->total >> 10);
         monitor_printf(mon, "duplicate: %" PRIu64 " pages\n",
                        info->ram->duplicate);
+        monitor_printf(mon, "skipped: %" PRIu64 " pages\n",
+                       info->ram->skipped);
         monitor_printf(mon, "normal: %" PRIu64 " pages\n",
                        info->ram->normal);
         monitor_printf(mon, "normal bytes: %" PRIu64 " kbytes\n",
include/migration/migration.h +2 -0
index bb617fd..e2acec6 100644
@@ -96,6 +96,8 @@ extern SaveVMHandlers savevm_ram_handlers;
 
 uint64_t dup_mig_bytes_transferred(void);
 uint64_t dup_mig_pages_transferred(void);
+uint64_t skipped_mig_bytes_transferred(void);
+uint64_t skipped_mig_pages_transferred(void);
 uint64_t norm_mig_bytes_transferred(void);
 uint64_t norm_mig_pages_transferred(void);
 uint64_t xbzrle_mig_bytes_transferred(void);
migration.c +2 -1
index 185d112..7fb2147 100644
@@ -197,11 +197,11 @@ MigrationInfo *qmp_query_migrate(Error **errp)
         info->ram->remaining = ram_bytes_remaining();
         info->ram->total = ram_bytes_total();
         info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
         info->ram->normal = norm_mig_pages_transferred();
         info->ram->normal_bytes = norm_mig_bytes_transferred();
         info->ram->dirty_pages_rate = s->dirty_pages_rate;
 
-
         if (blk_mig_active()) {
             info->has_disk = true;
             info->disk = g_malloc0(sizeof(*info->disk));
@@ -227,6 +227,7 @@ MigrationInfo *qmp_query_migrate(Error **errp)
         info->ram->remaining = 0;
         info->ram->total = ram_bytes_total();
         info->ram->duplicate = dup_mig_pages_transferred();
+        info->ram->skipped = skipped_mig_pages_transferred();
         info->ram->normal = norm_mig_pages_transferred();
         info->ram->normal_bytes = norm_mig_bytes_transferred();
         break;
qapi-schema.json +5 -3
index 088f4e1..d6a8812 100644
@@ -496,7 +496,9 @@
 #
 # @total: total amount of bytes involved in the migration process
 #
-# @duplicate: number of duplicate pages (since 1.2)
+# @duplicate: number of duplicate (zero) pages (since 1.2)
+#
+# @skipped: number of skipped zero pages (since 1.5)
 #
 # @normal : number of normal pages (since 1.2)
 #
@@ -509,8 +511,8 @@
 ##
 { 'type': 'MigrationStats',
   'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
-           'duplicate': 'int', 'normal': 'int', 'normal-bytes': 'int',
-           'dirty-pages-rate' : 'int' } }
+           'duplicate': 'int', 'skipped': 'int', 'normal': 'int',
+           'normal-bytes': 'int', 'dirty-pages-rate' : 'int' } }
 
 ##
 # @XBZRLECacheStats
qmp-commands.hx +2 -1
index b370060..fed74c6 100644
@@ -2442,7 +2442,8 @@ The main json-object contains the following:
          - "transferred": amount transferred (json-int)
          - "remaining": amount remaining (json-int)
          - "total": total (json-int)
-         - "duplicate": number of duplicated pages (json-int)
+         - "duplicate": number of duplicated (zero) pages (json-int)
+         - "skipped": number of skipped zero pages (json-int)
          - "normal" : number of normal pages transferred (json-int)
          - "normal-bytes" : number of normal bytes transferred (json-int)
 - "disk": only present if "status" is "active" and it is a block migration,
Signature · 1 line
1.7.9.5
#27 lore raw

PATCH v5 9/10 migration: do not search dirty pages in bulk stage

In reply to Peter Lieven

avoid searching for dirty pages just increment the page offset. all pages are dirty anyway.

---
 arch_init.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch_init.c +7 -1
index 3a0d02e..a522735 100644
@@ -340,7 +340,13 @@ ram_addr_t migration_bitmap_find_and_reset_dirty(MemoryRegion *mr,
     unsigned long nr = base + (start >> TARGET_PAGE_BITS);
     unsigned long size = base + (int128_get64(mr->size) >> TARGET_PAGE_BITS);
 
-    unsigned long next = find_next_bit(migration_bitmap, size, nr);
+    unsigned long next;
+
+    if (ram_bulk_stage && nr > base) {
+        next = nr + 1;
+    } else {
+        next = find_next_bit(migration_bitmap, size, nr);
+    }
 
     if (next < size) {
         clear_bit(next, migration_bitmap);
Signature · 1 line
1.7.9.5
#28 lore raw

PATCH v5 10/10 migration: use XBZRLE only after bulk stage

In reply to Peter Lieven

at the beginning of migration all pages are marked dirty and in the first round a bulk migration of all pages is performed.

currently all these pages are copied to the page cache regardless of whether they are frequently updated or not. this doesn't make sense since most of these pages are never transferred again.

this patch changes the XBZRLE transfer to only be used after the bulk stage has been completed. that means a page is added to the page cache the second time it is transferred and XBZRLE can benefit from the third time of transfer.

since the page cache is likely smaller than the number of pages it's also likely that in the second round the page is missing in the cache due to collisions in the bulk phase.

on the other hand a lot of unnecessary mallocs, memdups and frees are saved.

the following results have been taken earlier while executing the test program from docs/xbzrle.txt. (+) with the patch and (-) without. (thanks to Eric Blake for reformatting and comments)

+ total time: 22185 milliseconds
- total time: 22410 milliseconds

Shaved 0.3 seconds, better than 1%!

+ downtime: 29 milliseconds
- downtime: 21 milliseconds

Not sure why downtime seemed worse, but probably not the end of the world.

+ transferred ram: 706034 kbytes
- transferred ram: 721318 kbytes

Fewer bytes sent - good.

+ remaining ram: 0 kbytes
- remaining ram: 0 kbytes
+ total ram: 1057216 kbytes
- total ram: 1057216 kbytes
+ duplicate: 108556 pages
- duplicate: 105553 pages
+ normal: 175146 pages
- normal: 179589 pages
+ normal bytes: 700584 kbytes
- normal bytes: 718356 kbytes

Fewer normal bytes...

+ cache size: 67108864 bytes
- cache size: 67108864 bytes
+ xbzrle transferred: 3127 kbytes
- xbzrle transferred: 630 kbytes

...and more compressed pages sent - good.

+ xbzrle pages: 117811 pages
- xbzrle pages: 21527 pages
+ xbzrle cache miss: 18750
- xbzrle cache miss: 179589

And very good improvement on the cache miss rate.

+ xbzrle overflow : 0
- xbzrle overflow : 0

---
 arch_init.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch_init.c +1 -1
index a522735..e1af898 100644
@@ -466,7 +466,7 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
                     acct_info.skipped_pages++;
                     bytes_sent = 0;
                 }
-            } else if (migrate_use_xbzrle()) {
+            } else if (!ram_bulk_stage && migrate_use_xbzrle()) {
                 current_addr = block->offset + offset;
                 bytes_sent = save_xbzrle_page(f, p, current_addr, block,
                                               offset, cont, last_stage);
Signature · 1 line
1.7.9.5
Reviewed-by on the v5 series
#29 lore raw
In reply to Peter Lieven
Quoted 62 lines · Peter Lieven

this is v5 of my patch series with various optimizations in zero buffer checking and migration tweaks.

thanks especially to Eric Blake, Orit Wassermann and Paolo Bonzini for reviewing.

v5:
- move zero splat vector to a different patch
- fix indentation of can_user_buffer_find_nonzero_offset()
- do not unroll the first loop in buffer_find_nonzero_offset() to optimize it for zero page checking
- use an older unrolled version of find_next_bit() without SIMD instruction as there is no evidence that the vectorized version is better if not even worse and the code is easier to understand.
- added a word in the commit message of patch 8 about the skipped pages field in QMP MigrationStats.
- fixed the order of key-value pairs of MigrationStats in qapi-schema.json
- updated info about the performance benefit of is_zero_page() to the latest benchmark results in the commit message.

v4:
- do not inline buffer_find_nonzero_offset()
- inline can_usebuffer_find_nonzero_offset() correctly
- readd asserts in buffer_find_nonzero_offset() as profiling shows they do not hurt.
- change last occurences of scalar 8 by BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
- avoid deferencing p already in patch 5 where we know that the page (p) is zero
- explicitly set bytes_sent = 0 if we skip a zero page. bytes_sent was 0 before, but it was not obvious.
- add accounting information for skipped zero pages
- fix errors reported by checkpatch.pl

v3:
- remove asserts, inline functions and add a check function if buffer_find_nonzero_offset() can be used.
- use above check function in buffer_is_zero() and find_next_bit().
- use buffer_is_nonzero_offset() directly to find zero pages. we know that all requirements are met for memory pages.
- fix C89 violation in buffer_is_zero().
- avoid derefencing p in ram_save_block() if we already know the page is zero.
- fix initialization of last_offset in reset_ram_globals().
- avoid skipping pages with offset == 0 in bulk stage in migration_bitmap_find_and_reset_dirty().
- compared to v1 check for zero pages also after bulk ram migration as there are guests (e.g. Windows) which zero out large amount of memory while running.

v2:
- fix description, add trivial zero check and add asserts to buffer_find_nonzero_offset.
- add a constant for the unroll factor of buffer_find_nonzero_offset
- replace is_dup_page() by buffer_is_zero()
- added test results to xbzrle patch
- optimize descriptions

Quoted 11 lines

Peter Lieven (10):
move vector definitions to qemu-common.h
add a zero splat vector to qemu-common.h
cutils: add a function to find non-zero content in a buffer
buffer_is_zero: use vector optimizations if possible
bitops: unroll while loop in find_next_bit()
migration: search for zero instead of dup pages
migration: add an indicator for bulk state of ram migration
migration: do not sent zero pages in bulk stage
migration: do not search dirty pages in bulk stage
migration: use XBZRLE only after bulk stage

Really nice series.
I did some minor review changes, but they are more nits that anynthing else.

Thanks, Juan.

#30 lore raw

Am 26.03.2013 um 11:46 schrieb Juan Quintela <[email protected]>:

Quoted 75 lines
Quoted 62 lines · Peter Lieven

this is v5 of my patch series with various optimizations in zero buffer checking and migration tweaks.

thanks especially to Eric Blake, Orit Wassermann and Paolo Bonzini for reviewing.

v5:
- move zero splat vector to a different patch
- fix indentation of can_user_buffer_find_nonzero_offset()
- do not unroll the first loop in buffer_find_nonzero_offset() to optimize it for zero page checking
- use an older unrolled version of find_next_bit() without SIMD instruction as there is no evidence that the vectorized version is better if not even worse and the code is easier to understand.
- added a word in the commit message of patch 8 about the skipped pages field in QMP MigrationStats.
- fixed the order of key-value pairs of MigrationStats in qapi-schema.json
- updated info about the performance benefit of is_zero_page() to the latest benchmark results in the commit message.

v4:
- do not inline buffer_find_nonzero_offset()
- inline can_usebuffer_find_nonzero_offset() correctly
- readd asserts in buffer_find_nonzero_offset() as profiling shows they do not hurt.
- change last occurences of scalar 8 by BUFFER_FIND_NONZERO_OFFSET_UNROLL_FACTOR
- avoid deferencing p already in patch 5 where we know that the page (p) is zero
- explicitly set bytes_sent = 0 if we skip a zero page. bytes_sent was 0 before, but it was not obvious.
- add accounting information for skipped zero pages
- fix errors reported by checkpatch.pl

v3:
- remove asserts, inline functions and add a check function if buffer_find_nonzero_offset() can be used.
- use above check function in buffer_is_zero() and find_next_bit().
- use buffer_is_nonzero_offset() directly to find zero pages. we know that all requirements are met for memory pages.
- fix C89 violation in buffer_is_zero().
- avoid derefencing p in ram_save_block() if we already know the page is zero.
- fix initialization of last_offset in reset_ram_globals().
- avoid skipping pages with offset == 0 in bulk stage in migration_bitmap_find_and_reset_dirty().
- compared to v1 check for zero pages also after bulk ram migration as there are guests (e.g. Windows) which zero out large amount of memory while running.

v2:
- fix description, add trivial zero check and add asserts to buffer_find_nonzero_offset.
- add a constant for the unroll factor of buffer_find_nonzero_offset
- replace is_dup_page() by buffer_is_zero()
- added test results to xbzrle patch
- optimize descriptions

Quoted 11 lines

Peter Lieven (10):
move vector definitions to qemu-common.h
add a zero splat vector to qemu-common.h
cutils: add a function to find non-zero content in a buffer
buffer_is_zero: use vector optimizations if possible
bitops: unroll while loop in find_next_bit()
migration: search for zero instead of dup pages
migration: add an indicator for bulk state of ram migration
migration: do not sent zero pages in bulk stage
migration: do not search dirty pages in bulk stage
migration: use XBZRLE only after bulk stage

Quoted 1 line

Really nice series.

Thank you.

Quoted 2 lines

I did some minor review changes, but they are more nits that anynthing else.

Ok, where is your repository?

Peter

Quoted 4 lines

Thanks, Juan.