Skip to contentSkip to conversation

support dirtyrate measurement with dirty bitmap

PATCH Superseded [email protected] 1 participant 3 messages Jul 16, 2021

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover support dirtyrate measurement with dirty bitmap

From: Hyman Huang(黄勇) <[email protected]>

v5:
- let recording dirty pages after memory_global_dirty_log_sync make dirtyrate result more accurate

v4:
- drop the first commit:
"KVM: introduce kvm_get_manual_dirty_log_protect"

- clear dirty log unconditionally so that the first commit can be dropped.

- rename global var DirtyRateDirtyPages to total_dirty_pages

- stat the dirty pages along with the existing loop in cpu_physical_memory_set_dirty_lebitmap

- stat the increased dirty pages like the way of dirty-ring

- add BQL when fetch dirty log and clear dirty log

- do not clear dirty log after measuring.

v4 implementation clear log unconditionally so that the kvm_get_manual_dirty_log_protect can be dropped and so do the first commit.

other main modification is add the BQL when fetch and clear dirty log.

last modification is do not clear dirty log after measuring. if dirty tracking be stopped after measuring, clear dirty log make no sense.if dirty tracking is running after mesauring, clear dirty log can be handled by the caller who is interested in it.

v3:
- do not touch cpu_physical_memory_sync_dirty_bitmap

- rename global var DirtyRateIncreasedPages to DirtyRateDirtyPages

- stat dirty pages in cpu_physical_memory_set_dirty_lebitmap, which is on the execution path of memory_global_dirty_log_sync and can be used for dirty rate measuring.

the v3 implemention runs well, we could get accurate dirtyrate as v1 and simplify the implementation heavyly. it do not touch the any ram_list.dirty_memory[*], so do has no conflict with migraion at all.
if migration is running at the same time with dirtyrate measuring, measuring may reset protection of pages after memory_global_dirty_log_sync before migration iteration, but it has no side affect because kvm_log_clear_one_slot can guarantee that same dirty_bmap in a slot shouldn't be cleared twice.

v2:
- drop the DIRTY_MEMORY_DIRTY_RATE dirty bits

- reuse the DIRTY_MEMORY_MIGRATION dirty bits to stat the dirty pages.

- introduce global var DirtyRateIncreasedPages to stat the increased dirty pages

- simplify the logic of calculation. skip the 1'round of log sync unconditionally

changes of this version are based on Peter's advice, like the version 1, it is posted for the sake of RFC. ideally, this patshset may be merged after the commit:
"migration/dirtyrate: implement dirty-ring dirtyrate calculation"

v1:
the dirtyrate measurement implemented by page-sampling originally, it is not accurate in some scenarios, so we have introduced dirty-ring based dirtyrate measurement(maybe it will be merged soon), it fix the accuracy of page-sampling, and more importantly, it is at the granualrity of vcpu.

dirty-ring method can be used when dirty-ring enable, as supplementary, we introduce dirty-bitmap method to calculating dirtyrate when dirty log enable, so that we can also get the accurate dirtyrate if needed in the absence of dirty-ring.

three things has done to implement the measurement:
- introduce a fresh new dirty bits named DIRTY_MEMORY_DIRTY_RATE, which is used to store dirty bitmap after fetching it from kvm. why we do not reuse the existing DIRTY_MEMORY_MIGRATION dirty bits is we do not want to interfere with migration of and let implementation clear, this is also the reason why dirty_memory be split.

DIRTY_MEMORY_DIRTY_RATE dirty bits will be filled when memory_global_dirty_log_sync executed if GLOBAL_DIRTY_DIRTY_RATE bit be set in the global_dirty_tracking flag.

- introduce kvm_get_manual_dirty_log_protect function so that we can probe the protect caps of kvm when calculating.

- implement dirtyrate measurement with dirty bitmap with following step:
1. start the dirty log.

2. probe the protect cap, if KVM_DIRTY_LOG_INITIALLY_SET enable, skip skip the 1'R and do the reset page protection manually, since kvm file bitmap with 1 bits if this cap is enabled.

3. clear the DIRTY_MEMORY_DIRTY_RATE dirty bits, prepare to store the dirty bitmap.

4. start memory_global_dirty_log_sync and fetch dirty bitmap from kvm

5. reap the DIRTY_MEMORY_DIRTY_RATE dirty bits and do the calculation.

this patchset rebases on the commit "migration/dirtyrate: implement dirty-ring dirtyrate calculation", since the above feature has not been merged, so we post this patch for the sake of RFC. ideally, this patshset may be merged after it.

Please, review, thanks !

Best Regards !

Hyman Huang(黄勇) (2):
memory: introduce total_dirty_pages to stat dirty pages migration/dirtyrate: implement dirty-bitmap dirtyrate calculation

 hmp-commands.hx         |   9 ++--
 include/exec/ram_addr.h |   4 ++
 migration/dirtyrate.c   | 122 ++++++++++++++++++++++++++++++++++++++++++++----
 qapi/migration.json     |   6 ++-
 4 files changed, 126 insertions(+), 15 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v5 1/2 memory: introduce total_dirty_pages to stat dirty pages

From: Hyman Huang(黄勇) <[email protected]>

introduce global var total_dirty_pages to stat dirty pages along with memory_global_dirty_log_sync.

---
 include/exec/ram_addr.h | 4 ++++
 migration/dirtyrate.c   | 7 +++++++
 2 files changed, 11 insertions(+)
include/exec/ram_addr.h +4 -0
index 45c9132..e98d665 100644
@@ -26,6 +26,8 @@
 #include "exec/ramlist.h"
 #include "exec/ramblock.h"
 
+extern uint64_t total_dirty_pages;
+
 /**
  * clear_bmap_size: calculate clear bitmap size
  *
@@ -366,6 +368,7 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
             for (k = 0; k < nr; k++) {
                 if (bitmap[k]) {
                     unsigned long temp = leul_to_cpu(bitmap[k]);
+                    total_dirty_pages += ctpopl(temp);
 
                     qatomic_or(&blocks[DIRTY_MEMORY_VGA][idx][offset], temp);
 
@@ -403,6 +406,7 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
         for (i = 0; i < len; i++) {
             if (bitmap[i] != 0) {
                 c = leul_to_cpu(bitmap[i]);
+                total_dirty_pages += ctpopl(c);
                 do {
                     j = ctzl(c);
                     c &= ~(1ul << j);
migration/dirtyrate.c +7 -0
index f92c4b4..17b3d2c 100644
@@ -28,6 +28,13 @@
 #include "sysemu/runstate.h"
 #include "exec/memory.h"
 
+/*
+ * total_dirty_pages is procted by BQL and is used
+ * to stat dirty pages during the period of two
+ * memory_global_dirty_log_sync
+ */
+uint64_t total_dirty_pages;
+
 typedef struct DirtyPageRecord {
     uint64_t start_pages;
     uint64_t end_pages;
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v5 2/2 migration/dirtyrate: implement dirty-bitmap dirtyrate calculation

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

introduce dirty-bitmap mode as the third method of calc-dirty-rate. implement dirty-bitmap dirtyrate calculation, which can be used to measuring dirtyrate in the absence of dirty-ring.

introduce "dirty_bitmap:-b" option in hmp calc_dirty_rate to indicate dirty bitmap method should be used for calculation.

---
 hmp-commands.hx       |   9 ++--
 migration/dirtyrate.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++----
 qapi/migration.json   |   6 ++-
 3 files changed, 115 insertions(+), 15 deletions(-)
hmp-commands.hx +5 -4
index f7fc9d7..605973c 100644
@@ -1738,9 +1738,10 @@ ERST
 
     {
         .name       = "calc_dirty_rate",
-        .args_type  = "dirty_ring:-r,second:l,sample_pages_per_GB:l?",
-        .params     = "[-r] second [sample_pages_per_GB]",
-        .help       = "start a round of guest dirty rate measurement (using -d to"
-                      "\n\t\t\t specify dirty ring as the method of calculation)",
+        .args_type  = "dirty_ring:-r,dirty_bitmap:-b,second:l,sample_pages_per_GB:l?",
+        .params     = "[-r] [-b] second [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement (using -r to"
+                      "\n\t\t\t specify dirty ring as the method of calculation and"
+                      "\n\t\t\t -b to specify dirty bitmap as method of calculation)",
         .cmd        = hmp_calc_dirty_rate,
     },
migration/dirtyrate.c +106 -9

164 lines not on this page · raw message

qapi/migration.json +4 -2
index de35528..0b00976 100644
@@ -1747,13 +1747,15 @@
 #
 # @page-sampling: calculate dirtyrate by sampling pages.
 #
-# @dirty-ring: calculate dirtyrate by via dirty ring.
+# @dirty-ring: calculate dirtyrate by dirty ring.
+#
+# @dirty-bitmap: calculate dirtyrate by dirty bitmap.
 #
 # Since: 6.1
 #
 ##
 { 'enum': 'DirtyRateMeasureMode',
-  'data': ['page-sampling', 'dirty-ring'] }
+  'data': ['page-sampling', 'dirty-ring', 'dirty-bitmap'] }
 
 ##
 # @DirtyRateInfo:
Signature · 1 line
1.8.3.1