Skip to contentSkip to conversation

support dirtyrate at the granualrity of vcpu

PATCH Superseded [email protected] 5 participants 23 messages Jun 7, 2021 – Jun 11, 2021

v3 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v3 cover support dirtyrate at the granualrity of vcpu

From: Hyman Huang(黄勇) <[email protected]>

v3:
- pick up "migration/dirtyrate: make sample page count configurable" to make patchset apply master correctly

v2:
- rebase to "migration/dirtyrate: make sample page count configurable"

- rename "vcpu" to "per_vcpu" to show the per-vcpu method

- squash patch 5/6 into a single one, squash patch 1/2 also

- pick up "hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds"

- make global_dirty_log a bitmask to make sure both migration and dirty could not intefer with each other

- add memory free callback to prevent memory leaking

the most different of v2 fron v1 is that we make the global_dirty_log a bitmask. the reason is dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem.

make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate.

all references to global_dirty_log should be untouched because any bit set there should justify that global dirty logging is enabled.

Please review, thanks !

v1:

Since the Dirty Ring on QEMU part has been merged recently, how to use this feature is under consideration.

In the scene of migration, it is valuable to provide a more accurante interface to track dirty memory than existing one, so that the upper layer application can make a wise decision, or whatever. More importantly,
dirtyrate info at the granualrity of vcpu could provide a possibility to make migration convergent by imposing restriction on vcpu. With Dirty Ring, we can calculate dirtyrate efficiently and cheaply.

The old interface implemented by sampling pages, it consumes cpu resource, and the larger guest memory size become, the more cpu resource it consumes, namely, hard to scale. New interface has no such drawback.

Please review, thanks !

Best Regards !

Hyman Huang(黄勇) (6):
migration/dirtyrate: make sample page count configurable
KVM: introduce dirty_pages and kvm_dirty_ring_enabled migration/dirtyrate: add per-vcpu option for calc-dirty-rate migration/dirtyrate: adjust struct DirtyRateStat
memory: make global_dirty_log a bitmask migration/dirtyrate: implement dirty-ring dirtyrate calculation

Peter Xu (1):
hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds

 accel/kvm/kvm-all.c    |   7 +
 hmp-commands-info.hx   |  13 ++
 hmp-commands.hx        |  15 ++
 include/exec/memory.h  |  13 +-
 include/hw/core/cpu.h  |   1 +
 include/monitor/hmp.h  |   2 +
 include/sysemu/kvm.h   |   1 +
 migration/dirtyrate.c  | 347 ++++++++++++++++++++++++++++++++++++++---
 migration/dirtyrate.h  |  27 +++-
 migration/ram.c        |   8 +-
 migration/trace-events |   5 +
 qapi/migration.json    |  38 ++++-
 softmmu/memory.c       |  36 +++--
 13 files changed, 468 insertions(+), 45 deletions(-)
Signature · 1 line
2.18.2
#1 lore raw

PATCH v3 1/7 migration/dirtyrate: make sample page count configurable

From: Hyman Huang(黄勇) <[email protected]>

introduce optional sample-pages argument in calc-dirty-rate, making sample page count per GB configurable so that more accurate dirtyrate can be calculated.

---
 migration/dirtyrate.c | 31 +++++++++++++++++++++++++++----
 migration/dirtyrate.h |  8 +++++++-
 qapi/migration.json   | 13 ++++++++++---
 3 files changed, 44 insertions(+), 8 deletions(-)
migration/dirtyrate.c +27 -4
index ccb98147e8..2ee3890721 100644
@@ -48,6 +48,12 @@ static bool is_sample_period_valid(int64_t sec)
     return true;
 }
 
+static bool is_sample_pages_valid(int64_t pages)
+{
+    return pages >= MIN_SAMPLE_PAGE_COUNT &&
+           pages <= MAX_SAMPLE_PAGE_COUNT;
+}
+
 static int dirtyrate_set_state(int *state, int old_state, int new_state)
 {
     assert(new_state < DIRTY_RATE_STATUS__MAX);
@@ -72,13 +78,15 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
     info->status = CalculatingState;
     info->start_time = DirtyStat.start_time;
     info->calc_time = DirtyStat.calc_time;
+    info->sample_pages = DirtyStat.sample_pages;
 
     trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState));
 
     return info;
 }
 
-static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time)
+static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time,
+                                uint64_t sample_pages)
 {
     DirtyStat.total_dirty_samples = 0;
     DirtyStat.total_sample_count = 0;
@@ -86,6 +94,7 @@ static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time)
     DirtyStat.dirty_rate = -1;
     DirtyStat.start_time = start_time;
     DirtyStat.calc_time = calc_time;
+    DirtyStat.sample_pages = sample_pages;
 }
 
 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info)
@@ -361,6 +370,7 @@ void *get_dirtyrate_thread(void *arg)
     int ret;
     int64_t start_time;
     int64_t calc_time;
+    uint64_t sample_pages;
 
     ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,
                               DIRTY_RATE_STATUS_MEASURING);
@@ -371,7 +381,8 @@ void *get_dirtyrate_thread(void *arg)
 
     start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000;
     calc_time = config.sample_period_seconds;
-    init_dirtyrate_stat(start_time, calc_time);
+    sample_pages = config.sample_pages_per_gigabytes;
+    init_dirtyrate_stat(start_time, calc_time, sample_pages);
 
     calculate_dirtyrate(config);
 
@@ -383,7 +394,8 @@ void *get_dirtyrate_thread(void *arg)
     return NULL;
 }
 
-void qmp_calc_dirty_rate(int64_t calc_time, Error **errp)
+void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,
+                         int64_t sample_pages, Error **errp)
 {
     static struct DirtyRateConfig config;
     QemuThread thread;
@@ -404,6 +416,17 @@ void qmp_calc_dirty_rate(int64_t calc_time, Error **errp)
         return;
     }
 
+    if (has_sample_pages) {
+        if (!is_sample_pages_valid(sample_pages)) {
+            error_setg(errp, "sample-pages is out of range[%d, %d].",
+                            MIN_SAMPLE_PAGE_COUNT,
+                            MAX_SAMPLE_PAGE_COUNT);
+            return;
+        }
+    } else {
+        sample_pages = DIRTYRATE_DEFAULT_SAMPLE_PAGES;
+    }
+
     /*
      * Init calculation state as unstarted.
      */
@@ -415,7 +438,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, Error **errp)
     }
 
     config.sample_period_seconds = calc_time;
-    config.sample_pages_per_gigabytes = DIRTYRATE_DEFAULT_SAMPLE_PAGES;
+    config.sample_pages_per_gigabytes = sample_pages;
     qemu_thread_create(&thread, "get_dirtyrate", get_dirtyrate_thread,
                        (void *)&config, QEMU_THREAD_DETACHED);
 }
migration/dirtyrate.h +7 -1
index 6ec429534d..e1fd29089e 100644
@@ -15,7 +15,6 @@
 
 /*
  * Sample 512 pages per GB as default.
- * TODO: Make it configurable.
  */
 #define DIRTYRATE_DEFAULT_SAMPLE_PAGES            512
 
@@ -35,6 +34,12 @@
 #define MIN_FETCH_DIRTYRATE_TIME_SEC              1
 #define MAX_FETCH_DIRTYRATE_TIME_SEC              60
 
+/*
+ * Take 1/16 pages in 1G as the maxmum sample page count
+ */
+#define MIN_SAMPLE_PAGE_COUNT                     128
+#define MAX_SAMPLE_PAGE_COUNT                     16384
+
 struct DirtyRateConfig {
     uint64_t sample_pages_per_gigabytes; /* sample pages per GB */
     int64_t sample_period_seconds; /* time duration between two sampling */
@@ -63,6 +68,7 @@ struct DirtyRateStat {
     int64_t dirty_rate; /* dirty rate in MB/s */
     int64_t start_time; /* calculation start time in units of second */
     int64_t calc_time; /* time duration of two sampling in units of second */
+    uint64_t sample_pages; /* sample pages per GB */
 };
 
 void *get_dirtyrate_thread(void *arg);
qapi/migration.json +10 -3
index 7a5bdf9a0d..770ae54c17 100644
@@ -1740,6 +1740,9 @@
 #
 # @calc-time: time in units of second for sample dirty pages
 #
+# @sample-pages: page count per GB for sample dirty pages
+#                the default value is 512 (since 6.1)
+#
 # Since: 5.2
 #
 ##
@@ -1747,7 +1750,8 @@
   'data': {'*dirty-rate': 'int64',
            'status': 'DirtyRateStatus',
            'start-time': 'int64',
-           'calc-time': 'int64'} }
+           'calc-time': 'int64',
+           'sample-pages': 'uint64'} }
 
 ##
 # @calc-dirty-rate:
@@ -1756,13 +1760,16 @@
 #
 # @calc-time: time in units of second for sample dirty pages
 #
+# @sample-pages: page count per GB for sample dirty pages
+#                the default value is 512 (since 6.1)
+#
 # Since: 5.2
 #
 # Example:
-#   {"command": "calc-dirty-rate", "data": {"calc-time": 1} }
+#   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
 #
 ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }
 
 ##
 # @query-dirty-rate:
Signature · 1 line
2.18.2
#2 lore raw

Re: [PATCH v3 1/7] migration/dirtyrate: make sample page count configurable

Quoted 11 lines · On Mon, Jun 07, 2021 at 09:11:34AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

introduce optional sample-pages argument in calc-dirty-rate, making sample page count per GB configurable so that more accurate dirtyrate can be calculated.

---

+++ b/qapi/migration.json
# Example:

qapi/migration.json line 1772

Quoted 6 lines
qapi/migration.json +2 -2
-#   {"command": "calc-dirty-rate", "data": {"calc-time": 1} }
+#   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
 #
 ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }

Long line. Please wrap at 80 columns.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [PATCH v3 1/7] migration/dirtyrate: make sample page count configurable

Quoted 10 lines · * Eric Blake ([email protected]) wrote:
Quoted 8 lines · On Mon, Jun 07, 2021 at 09:11:34AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

introduce optional sample-pages argument in calc-dirty-rate, making sample page count per GB configurable so that more accurate dirtyrate can be calculated.

---

Quoted 10 lines

qapi/migration.json line 1772

Quoted 8 lines

+++ b/qapi/migration.json
# Example:

qapi/migration.json +2 -2
-#   {"command": "calc-dirty-rate", "data": {"calc-time": 1} }
+#   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
 #
 ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }

Long line. Please wrap at 80 columns.

I can fix up that.

Quoted 4 lines
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#4 lore raw

PATCH v3 2/7 hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds

In reply to [email protected]

From: Peter Xu <[email protected]>

These two commands are missing when adding the QMP sister commands. Add them, so developers can play with them easier.

---
 hmp-commands-info.hx  | 13 ++++++++++++
 hmp-commands.hx       | 14 +++++++++++++
 include/monitor/hmp.h |  2 ++
 migration/dirtyrate.c | 47 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+)
hmp-commands-info.hx +13 -0
index b2347a6aea..fb59c27200 100644
@@ -867,3 +867,16 @@ SRST
   ``info replay``
     Display the record/replay information: mode and the current icount.
 ERST
+
+    {
+        .name       = "dirty_rate",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show dirty rate information",
+        .cmd        = hmp_info_dirty_rate,
+    },
+
+SRST
+  ``info dirty_rate``
+    Display the vcpu dirty rate information.
+ERST
hmp-commands.hx +14 -0
index 2d21fe5ad4..84dcc3aae6 100644
@@ -1727,3 +1727,17 @@ ERST
         .flags      = "p",
     },
 
+SRST
+``calc_dirty_rate`` *second*
+  Start a round of dirty rate measurement with the period specified in *second*.
+  The result of the dirty rate measurement may be observed with ``info
+  dirty_rate`` command.
+ERST
+
+    {
+        .name       = "calc_dirty_rate",
+        .args_type  = "second:l,sample_pages_per_GB:l?",
+        .params     = "second [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement",
+        .cmd        = hmp_calc_dirty_rate,
+    },
include/monitor/hmp.h +2 -0
index 605d57287a..3baa1058e2 100644
@@ -129,5 +129,7 @@ void hmp_info_replay(Monitor *mon, const QDict *qdict);
 void hmp_replay_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_delete_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_seek(Monitor *mon, const QDict *qdict);
+void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict);
+void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict);
 
 #endif
migration/dirtyrate.c +47 -0

56 lines not on this page · raw message

Signature · 1 line
2.18.2
Reviewed-by on v3 2/7
#5 lore raw

Re: [PATCH v3 2/7] hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds

Quoted 7 lines · * [email protected] ([email protected]) wrote:

From: Peter Xu <[email protected]>

These two commands are missing when adding the QMP sister commands. Add them, so developers can play with them easier.

Quoted 127 lines
---
 hmp-commands-info.hx  | 13 ++++++++++++
 hmp-commands.hx       | 14 +++++++++++++
 include/monitor/hmp.h |  2 ++
 migration/dirtyrate.c | 47 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+)
hmp-commands-info.hx +13 -0
index b2347a6aea..fb59c27200 100644
@@ -867,3 +867,16 @@ SRST
   ``info replay``
     Display the record/replay information: mode and the current icount.
 ERST
+
+    {
+        .name       = "dirty_rate",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show dirty rate information",
+        .cmd        = hmp_info_dirty_rate,
+    },
+
+SRST
+  ``info dirty_rate``
+    Display the vcpu dirty rate information.
+ERST
hmp-commands.hx +14 -0
index 2d21fe5ad4..84dcc3aae6 100644
@@ -1727,3 +1727,17 @@ ERST
         .flags      = "p",
     },
 
+SRST
+``calc_dirty_rate`` *second*
+  Start a round of dirty rate measurement with the period specified in *second*.
+  The result of the dirty rate measurement may be observed with ``info
+  dirty_rate`` command.
+ERST
+
+    {
+        .name       = "calc_dirty_rate",
+        .args_type  = "second:l,sample_pages_per_GB:l?",
+        .params     = "second [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement",
+        .cmd        = hmp_calc_dirty_rate,
+    },
include/monitor/hmp.h +2 -0
index 605d57287a..3baa1058e2 100644
@@ -129,5 +129,7 @@ void hmp_info_replay(Monitor *mon, const QDict *qdict);
 void hmp_replay_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_delete_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_seek(Monitor *mon, const QDict *qdict);
+void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict);
+void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict);
 
 #endif
migration/dirtyrate.c +47 -0
index 2ee3890721..320c56ba2c 100644
@@ -20,6 +20,9 @@
 #include "ram.h"
 #include "trace.h"
 #include "dirtyrate.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
+#include "qapi/qmp/qdict.h"
 
 static int CalculatingState = DIRTY_RATE_STATUS_UNSTARTED;
 static struct DirtyRateStat DirtyStat;
@@ -447,3 +450,47 @@ struct DirtyRateInfo *qmp_query_dirty_rate(Error **errp)
 {
     return query_dirty_rate_info();
 }
+
+void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict)
+{
+    DirtyRateInfo *info = query_dirty_rate_info();
+
+    monitor_printf(mon, "Status: %s\n",
+                   DirtyRateStatus_str(info->status));
+    monitor_printf(mon, "Start Time: %"PRIi64" (ms)\n",
+                   info->start_time);
+    monitor_printf(mon, "Sample Pages: %"PRIu64" (per GB)\n",
+                   info->sample_pages);
+    monitor_printf(mon, "Period: %"PRIi64" (sec)\n",
+                   info->calc_time);
+    monitor_printf(mon, "Dirty rate: ");
+    if (info->has_dirty_rate) {
+        monitor_printf(mon, "%"PRIi64" (MB/s)\n", info->dirty_rate);
+    } else {
+        monitor_printf(mon, "(not ready)\n");
+    }
+    g_free(info);
+}
+
+void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict)
+{
+    int64_t sec = qdict_get_try_int(qdict, "second", 0);
+    int64_t sample_pages = qdict_get_try_int(qdict, "sample_pages_per_GB", -1);
+    bool has_sample_pages = (sample_pages != -1);
+    Error *err = NULL;
+
+    if (!sec) {
+        monitor_printf(mon, "Incorrect period length specified!\n");
+        return;
+    }
+
+    qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, &err);
+    if (err) {
+        hmp_handle_error(mon, err);
+        return;
+    }
+
+    monitor_printf(mon, "Starting dirty rate measurement with period %"PRIi64
+                   " seconds\n", sec);
+    monitor_printf(mon, "[Please use 'info dirty_rate' to check results]\n");
+}
Signature · 1 line
2.18.2
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#6 lore raw

Re: [PATCH v3 2/7] hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds

In reply to [email protected]
Quoted 7 lines · * [email protected] ([email protected]) wrote:

From: Peter Xu <[email protected]>

These two commands are missing when adding the QMP sister commands. Add them, so developers can play with them easier.

I've queued 1 and 2 (with a line wrap on 1); we can take the rest after Peter is happy with the other stuff.

Dave

Quoted 127 lines
---
 hmp-commands-info.hx  | 13 ++++++++++++
 hmp-commands.hx       | 14 +++++++++++++
 include/monitor/hmp.h |  2 ++
 migration/dirtyrate.c | 47 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+)
hmp-commands-info.hx +13 -0
index b2347a6aea..fb59c27200 100644
@@ -867,3 +867,16 @@ SRST
   ``info replay``
     Display the record/replay information: mode and the current icount.
 ERST
+
+    {
+        .name       = "dirty_rate",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show dirty rate information",
+        .cmd        = hmp_info_dirty_rate,
+    },
+
+SRST
+  ``info dirty_rate``
+    Display the vcpu dirty rate information.
+ERST
hmp-commands.hx +14 -0
index 2d21fe5ad4..84dcc3aae6 100644
@@ -1727,3 +1727,17 @@ ERST
         .flags      = "p",
     },
 
+SRST
+``calc_dirty_rate`` *second*
+  Start a round of dirty rate measurement with the period specified in *second*.
+  The result of the dirty rate measurement may be observed with ``info
+  dirty_rate`` command.
+ERST
+
+    {
+        .name       = "calc_dirty_rate",
+        .args_type  = "second:l,sample_pages_per_GB:l?",
+        .params     = "second [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement",
+        .cmd        = hmp_calc_dirty_rate,
+    },
include/monitor/hmp.h +2 -0
index 605d57287a..3baa1058e2 100644
@@ -129,5 +129,7 @@ void hmp_info_replay(Monitor *mon, const QDict *qdict);
 void hmp_replay_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_delete_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_seek(Monitor *mon, const QDict *qdict);
+void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict);
+void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict);
 
 #endif
migration/dirtyrate.c +47 -0
index 2ee3890721..320c56ba2c 100644
@@ -20,6 +20,9 @@
 #include "ram.h"
 #include "trace.h"
 #include "dirtyrate.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
+#include "qapi/qmp/qdict.h"
 
 static int CalculatingState = DIRTY_RATE_STATUS_UNSTARTED;
 static struct DirtyRateStat DirtyStat;
@@ -447,3 +450,47 @@ struct DirtyRateInfo *qmp_query_dirty_rate(Error **errp)
 {
     return query_dirty_rate_info();
 }
+
+void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict)
+{
+    DirtyRateInfo *info = query_dirty_rate_info();
+
+    monitor_printf(mon, "Status: %s\n",
+                   DirtyRateStatus_str(info->status));
+    monitor_printf(mon, "Start Time: %"PRIi64" (ms)\n",
+                   info->start_time);
+    monitor_printf(mon, "Sample Pages: %"PRIu64" (per GB)\n",
+                   info->sample_pages);
+    monitor_printf(mon, "Period: %"PRIi64" (sec)\n",
+                   info->calc_time);
+    monitor_printf(mon, "Dirty rate: ");
+    if (info->has_dirty_rate) {
+        monitor_printf(mon, "%"PRIi64" (MB/s)\n", info->dirty_rate);
+    } else {
+        monitor_printf(mon, "(not ready)\n");
+    }
+    g_free(info);
+}
+
+void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict)
+{
+    int64_t sec = qdict_get_try_int(qdict, "second", 0);
+    int64_t sample_pages = qdict_get_try_int(qdict, "sample_pages_per_GB", -1);
+    bool has_sample_pages = (sample_pages != -1);
+    Error *err = NULL;
+
+    if (!sec) {
+        monitor_printf(mon, "Incorrect period length specified!\n");
+        return;
+    }
+
+    qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, &err);
+    if (err) {
+        hmp_handle_error(mon, err);
+        return;
+    }
+
+    monitor_printf(mon, "Starting dirty rate measurement with period %"PRIi64
+                   " seconds\n", sec);
+    monitor_printf(mon, "[Please use 'info dirty_rate' to check results]\n");
+}
Signature · 1 line
2.18.2
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#7 lore raw

PATCH v3 3/7 KVM: introduce dirty_pages and kvm_dirty_ring_enabled

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

dirty_pages is used to calculate dirtyrate via dirty ring, when enabled, kvm-reaper will increase the dirty pages after gfns being dirtied.

kvm_dirty_ring_enabled shows if kvm-reaper is working. dirtyrate thread could use it to check if measurement can base on dirty ring feature.

---
 accel/kvm/kvm-all.c   | 7 +++++++
 include/hw/core/cpu.h | 1 +
 include/sysemu/kvm.h  | 1 +
 3 files changed, 9 insertions(+)
accel/kvm/kvm-all.c +7 -0
index c7ec538850..bc012f0bee 100644
@@ -469,6 +469,7 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
     cpu->kvm_fd = ret;
     cpu->kvm_state = s;
     cpu->vcpu_dirty = true;
+    cpu->dirty_pages = 0;
 
     mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
     if (mmap_size < 0) {
@@ -743,6 +744,7 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu)
         count++;
     }
     cpu->kvm_fetch_index = fetch;
+    cpu->dirty_pages += count;
 
     return count;
 }
@@ -2293,6 +2295,11 @@ bool kvm_vcpu_id_is_valid(int vcpu_id)
     return vcpu_id >= 0 && vcpu_id < kvm_max_vcpu_id(s);
 }
 
+bool kvm_dirty_ring_enabled(void)
+{
+    return kvm_state->kvm_dirty_ring_size ? true : false;
+}
+
 static int kvm_init(MachineState *ms)
 {
     MachineClass *mc = MACHINE_GET_CLASS(ms);
include/hw/core/cpu.h +1 -0
index 4e0ea68efc..80fcb1d563 100644
@@ -374,6 +374,7 @@ struct CPUState {
     struct kvm_run *kvm_run;
     struct kvm_dirty_gfn *kvm_dirty_gfns;
     uint32_t kvm_fetch_index;
+    uint64_t dirty_pages;
 
     /* Used for events with 'vcpu' and *without* the 'disabled' properties */
     DECLARE_BITMAP(trace_dstate_delayed, CPU_TRACE_DSTATE_MAX_EVENTS);
include/sysemu/kvm.h +1 -0
index a1ab1ee12d..7b22aeb6ae 100644
@@ -547,4 +547,5 @@ bool kvm_cpu_check_are_resettable(void);
 
 bool kvm_arch_cpu_check_are_resettable(void);
 
+bool kvm_dirty_ring_enabled(void);
 #endif
Signature · 1 line
2.18.2
#8 lore raw

PATCH v3 4/7 migration/dirtyrate: add per-vcpu option for calc-dirty-rate

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

calculate dirtyrate for each vcpu if vcpu is true, add the dirtyrate of each vcpu to the return value also.

---
 migration/dirtyrate.c | 45 ++++++++++++++++++++++++++++++++++++++-----
 migration/dirtyrate.h |  1 +
 qapi/migration.json   | 29 ++++++++++++++++++++++++++--
 3 files changed, 68 insertions(+), 7 deletions(-)
migration/dirtyrate.c +40 -5
index 320c56ba2c..5947c688f6 100644
@@ -397,8 +397,11 @@ void *get_dirtyrate_thread(void *arg)
     return NULL;
 }
 
-void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,
-                         int64_t sample_pages, Error **errp)
+void qmp_calc_dirty_rate(int64_t calc_time,
+                         bool per_vcpu,
+                         bool has_sample_pages,
+                         int64_t sample_pages,
+                         Error **errp)
 {
     static struct DirtyRateConfig config;
     QemuThread thread;
@@ -419,6 +422,12 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,
         return;
     }
 
+    if (has_sample_pages && per_vcpu) {
+        error_setg(errp, "per-vcpu and sample-pages are mutually exclusive, "
+                         "only one of then can be specified!\n");
+        return;
+    }
+
     if (has_sample_pages) {
         if (!is_sample_pages_valid(sample_pages)) {
             error_setg(errp, "sample-pages is out of range[%d, %d].",
@@ -430,6 +439,15 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,
         sample_pages = DIRTYRATE_DEFAULT_SAMPLE_PAGES;
     }
 
+    /*
+     * Vcpu method only works when kvm dirty ring is enabled.
+     */
+    if (per_vcpu && !kvm_dirty_ring_enabled()) {
+        error_setg(errp, "dirty ring is disabled or conflict with migration"
+                         "use sample method or remeasure later.");
+        return;
+    }
+
     /*
      * Init calculation state as unstarted.
      */
@@ -442,6 +460,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,
 
     config.sample_period_seconds = calc_time;
     config.sample_pages_per_gigabytes = sample_pages;
+    config.per_vcpu = per_vcpu;
     qemu_thread_create(&thread, "get_dirtyrate", get_dirtyrate_thread,
                        (void *)&config, QEMU_THREAD_DETACHED);
 }
@@ -459,13 +478,22 @@ void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict)
                    DirtyRateStatus_str(info->status));
     monitor_printf(mon, "Start Time: %"PRIi64" (ms)\n",
                    info->start_time);
-    monitor_printf(mon, "Sample Pages: %"PRIu64" (per GB)\n",
-                   info->sample_pages);
     monitor_printf(mon, "Period: %"PRIi64" (sec)\n",
                    info->calc_time);
+    if (info->has_sample_pages) {
+        monitor_printf(mon, "Sample Pages: %"PRIu64" (per GB)\n",
+                       info->sample_pages);
+    }
     monitor_printf(mon, "Dirty rate: ");
     if (info->has_dirty_rate) {
         monitor_printf(mon, "%"PRIi64" (MB/s)\n", info->dirty_rate);
+        if (info->per_vcpu && info->has_vcpu_dirty_rate) {
+            DirtyRateVcpuList *rate, *head = info->vcpu_dirty_rate;
+            for (rate = head; rate != NULL; rate = rate->next) {
+                monitor_printf(mon, "vcpu[%"PRIi64"], Dirty rate: %"PRIi64"\n",
+                               rate->value->id, rate->value->dirty_rate);
+            }
+        }
     } else {
         monitor_printf(mon, "(not ready)\n");
     }
@@ -477,6 +505,7 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict)
     int64_t sec = qdict_get_try_int(qdict, "second", 0);
     int64_t sample_pages = qdict_get_try_int(qdict, "sample_pages_per_GB", -1);
     bool has_sample_pages = (sample_pages != -1);
+    bool per_vcpu = qdict_get_try_bool(qdict, "per_vcpu", false);
     Error *err = NULL;
 
     if (!sec) {
@@ -484,7 +513,13 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict)
         return;
     }
 
-    qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, &err);
+    if (has_sample_pages && per_vcpu) {
+        monitor_printf(mon, "per_vcpu and sample_pages are mutually exclusive, "
+                       "only one of then can be specified!\n");
+        return;
+    }
+
+    qmp_calc_dirty_rate(sec, per_vcpu, has_sample_pages, sample_pages, &err);
     if (err) {
         hmp_handle_error(mon, err);
         return;
migration/dirtyrate.h +1 -0
index e1fd29089e..ec82716b40 100644
@@ -43,6 +43,7 @@
 struct DirtyRateConfig {
     uint64_t sample_pages_per_gigabytes; /* sample pages per GB */
     int64_t sample_period_seconds; /* time duration between two sampling */
+    bool per_vcpu; /* calculate dirtyrate for each vcpu using dirty ring */
 };
 
 /*
qapi/migration.json +27 -2
index 770ae54c17..7eef988182 100644
@@ -1708,6 +1708,21 @@
 { 'event': 'UNPLUG_PRIMARY',
   'data': { 'device-id': 'str' } }
 
+##
+# @DirtyRateVcpu:
+#
+# Dirty rate of vcpu.
+#
+# @id: vcpu index.
+#
+# @dirty-rate: dirty rate.
+#
+# Since: 6.1
+#
+##
+{ 'struct': 'DirtyRateVcpu',
+  'data': { 'id': 'int', 'dirty-rate': 'int64' } }
+
 ##
 # @DirtyRateStatus:
 #
@@ -1743,6 +1758,10 @@
 # @sample-pages: page count per GB for sample dirty pages
 #                the default value is 512 (since 6.1)
 #
+# @per-vcpu: calculate dirtyrate for each vcpu (Since 6.1)
+#
+# @vcpu-dirty-rate: dirtyrate for each vcpu (Since 6.1)
+#
 # Since: 5.2
 #
 ##
@@ -1751,7 +1770,9 @@
            'status': 'DirtyRateStatus',
            'start-time': 'int64',
            'calc-time': 'int64',
-           'sample-pages': 'uint64'} }
+           '*sample-pages': 'uint64',
+           'per-vcpu': 'bool',
+           '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } }
 
 ##
 # @calc-dirty-rate:
@@ -1760,6 +1781,10 @@
 #
 # @calc-time: time in units of second for sample dirty pages
 #
+# @per-vcpu: calculate vcpu dirty rate if true, the default value is
+#            false, note that the per-vcpu and sample-pages are mutually
+#            exclusive (since 6.1)
+#
 # @sample-pages: page count per GB for sample dirty pages
 #                the default value is 512 (since 6.1)
 #
@@ -1769,7 +1794,7 @@
 #   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
 #
 ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', 'per-vcpu': 'bool', '*sample-pages': 'int'} }
 
 ##
 # @query-dirty-rate:
Signature · 1 line
2.18.2
#9 lore raw

Re: [PATCH v3 4/7] migration/dirtyrate: add per-vcpu option for calc-dirty-rate

qapi/migration.json line 1775

Quoted 8 lines · On Mon, Jun 07, 2021 at 09:12:36AM +0800, [email protected] wrote:
qapi/migration.json +3 -1
@@ -1751,7 +1770,9 @@
            'status': 'DirtyRateStatus',
            'start-time': 'int64',
            'calc-time': 'int64',
-           'sample-pages': 'uint64'} }
+           '*sample-pages': 'uint64',
+           'per-vcpu': 'bool',
+           '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } }

Ideally we shouldn't touch existing exported fields? I know it's not a problem since it's just introduced and together with the series.. The other alternative is to keep it as is but just ignore it in the result (or set it to zero for per-vcpu sampling)? No strong opinion.

Quoted 15 lines
qapi/migration.json +4 -0
 ##
 # @calc-dirty-rate:
@@ -1760,6 +1781,10 @@
 #
 # @calc-time: time in units of second for sample dirty pages
 #
+# @per-vcpu: calculate vcpu dirty rate if true, the default value is
+#            false, note that the per-vcpu and sample-pages are mutually
+#            exclusive (since 6.1)
+#
 # @sample-pages: page count per GB for sample dirty pages
 #                the default value is 512 (since 6.1)
 #

qapi/migration.json line 1797

Quoted 5 lines
qapi/migration.json +1 -1
@@ -1769,7 +1794,7 @@
 #   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
 #
 ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', 'per-vcpu': 'bool', '*sample-pages': 'int'} }

I still think exporting this new feature should happen as/in the last patch.

Also, wondering whether the name "per-vcpu" would let people think of "samping pages per-vcpu only", not showing that it's a completely new mechanism inside. How about make it a new parameter "*mode" (and I think even with per-vcpu it should be optional as "*per-vcpu") default to "page-sampling" and add a new "dirty-ring"?

Actually I'm also wondering whether dirty log could be anything useful here in the future as a 3rd mode (then the "*mode" idea should be more useful if so), basically for old kernels where dirty ring is not there, we can timely call memory_global_dirty_log_sync() and calculate dirty pages there just like what we do with dirty rings (without calling migration_bitmap_sync(), so we don't need to deliver dirty bits from kvmslots to ramblocks, just pick them up from kvm and do the accounting for pure dirty rate measurement). That's a wild idea though, so just raise it up. Would that be anything useful?

Signature · 1 line
Peter Xu
#10 lore raw

Re: [PATCH v3 4/7] migration/dirtyrate: add per-vcpu option for calc-dirty-rate

Quoted 14 lines · Peter Xu

qapi/migration.json line 1775

Quoted 8 lines · On Mon, Jun 07, 2021 at 09:12:36AM +0800, [email protected] wrote:
qapi/migration.json +3 -1
@@ -1751,7 +1770,9 @@
             'status': 'DirtyRateStatus',
             'start-time': 'int64',
             'calc-time': 'int64',
-           'sample-pages': 'uint64'} }
+           '*sample-pages': 'uint64',
+           'per-vcpu': 'bool',
+           '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } }

Ideally we shouldn't touch existing exported fields? I know it's not a problem since it's just introduced and together with the series.. The other alternative is to keep it as is but just ignore it in the result (or set it to zero for per-vcpu sampling)? No strong opinion.

i'll change 'sample-pages' to '*sample-pages' in the first commit "migration/dirtyrate: make sample page count configurable" so that the following patch won't touch existing exported field.

Quoted 15 lines
Quoted 15 lines
qapi/migration.json +4 -0
  ##
  # @calc-dirty-rate:
@@ -1760,6 +1781,10 @@
  #
  # @calc-time: time in units of second for sample dirty pages
  #
+# @per-vcpu: calculate vcpu dirty rate if true, the default value is
+#            false, note that the per-vcpu and sample-pages are mutually
+#            exclusive (since 6.1)
+#
  # @sample-pages: page count per GB for sample dirty pages
  #                the default value is 512 (since 6.1)
  #
Quoted 8 lines

qapi/migration.json line 1797

Quoted 6 lines
qapi/migration.json +1 -1
@@ -1769,7 +1794,7 @@
  #   {"command": "calc-dirty-rate", "data": {"calc-time": 1, 'sample-pages': 512} }
  #
  ##
-{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*sample-pages': 'int'} }
+{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', 'per-vcpu': 'bool', '*sample-pages': 'int'} }

I still think exporting this new feature should happen as/in the last patch.

indeed, the last version ignore this advice, i'll introduce the "per-vcpu" in the last patch, before it implemented.

Quoted 6 lines

Also, wondering whether the name "per-vcpu" would let people think of "samping pages per-vcpu only", not showing that it's a completely new mechanism inside. How about make it a new parameter "*mode" (and I think even with per-vcpu it should be optional as "*per-vcpu") default to "page-sampling" and add a new "dirty-ring"?

i prefer the "*mode" parameter to show the different dirty rate method.

Quoted 9 lines

Actually I'm also wondering whether dirty log could be anything useful here in the future as a 3rd mode (then the "*mode" idea should be more useful if so), basically for old kernels where dirty ring is not there, we can timely call memory_global_dirty_log_sync() and calculate dirty pages there just like what we do with dirty rings (without calling migration_bitmap_sync(), so we don't need to deliver dirty bits from kvmslots to ramblocks, just pick them up from kvm and do the accounting for pure dirty rate measurement). That's a wild idea though, so just raise it up. Would that be anything useful?

uh, actually this idea is what i'm working on to stat the memory heat(trying to reduce the transferred memory in migration), theoretically it can be used for dirty rate measurement also, maybe i could try it after this patchset work having done.

Quoted 1 line
Signature · 3 lines
Best regard

Hyman Huang(黄勇)
#11 lore raw

Re: [PATCH v3 4/7] migration/dirtyrate: add per-vcpu option for calc-dirty-rate

On Tue, Jun 08, 2021 at 12:16:03AM +0800, Hyman Huang wrote:

[...]

Quoted 8 lines
Quoted 8 lines

Actually I'm also wondering whether dirty log could be anything useful here in the future as a 3rd mode (then the "*mode" idea should be more useful if so), basically for old kernels where dirty ring is not there, we can timely call memory_global_dirty_log_sync() and calculate dirty pages there just like what we do with dirty rings (without calling migration_bitmap_sync(), so we don't need to deliver dirty bits from kvmslots to ramblocks, just pick them up from kvm and do the accounting for pure dirty rate measurement). That's a wild idea though, so just raise it up. Would that be anything useful?

Quoted 4 lines

uh, actually this idea is what i'm working on to stat the memory heat(trying to reduce the transferred memory in migration), theoretically it can be used for dirty rate measurement also, maybe i could try it after this patchset work having done.

Yes, definitely no need to do that in this series; it just proves that "*mode" parameter is better in case a 3rd one could come. Thanks,

Signature · 1 line
Peter Xu
#12 lore raw

PATCH v3 5/7 migration/dirtyrate: adjust struct DirtyRateStat

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

use union to store stat data of two mutual exclusive methods.

---
 migration/dirtyrate.c | 40 +++++++++++++++++++++-------------------
 migration/dirtyrate.h | 18 +++++++++++++++---
 2 files changed, 36 insertions(+), 22 deletions(-)
migration/dirtyrate.c +21 -19
index 5947c688f6..055145c24c 100644
@@ -88,33 +88,39 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
     return info;
 }
 
-static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time,
-                                uint64_t sample_pages)
+static void init_dirtyrate_stat(int64_t start_time,
+                                struct DirtyRateConfig config)
 {
-    DirtyStat.total_dirty_samples = 0;
-    DirtyStat.total_sample_count = 0;
-    DirtyStat.total_block_mem_MB = 0;
     DirtyStat.dirty_rate = -1;
     DirtyStat.start_time = start_time;
-    DirtyStat.calc_time = calc_time;
-    DirtyStat.sample_pages = sample_pages;
+    DirtyStat.calc_time = config.sample_period_seconds;
+    DirtyStat.sample_pages = config.sample_pages_per_gigabytes;
+
+    if (config.per_vcpu) {
+        DirtyStat.method.vcpu.nvcpu = -1;
+        DirtyStat.method.vcpu.rates = NULL;
+    } else {
+        DirtyStat.method.vm.total_dirty_samples = 0;
+        DirtyStat.method.vm.total_sample_count = 0;
+        DirtyStat.method.vm.total_block_mem_MB = 0;
+    }
 }
 
 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info)
 {
-    DirtyStat.total_dirty_samples += info->sample_dirty_count;
-    DirtyStat.total_sample_count += info->sample_pages_count;
+    DirtyStat.method.vm.total_dirty_samples += info->sample_dirty_count;
+    DirtyStat.method.vm.total_sample_count += info->sample_pages_count;
     /* size of total pages in MB */
-    DirtyStat.total_block_mem_MB += (info->ramblock_pages *
+    DirtyStat.method.vm.total_block_mem_MB += (info->ramblock_pages *
                                      TARGET_PAGE_SIZE) >> 20;
 }
 
 static void update_dirtyrate(uint64_t msec)
 {
     uint64_t dirtyrate;
-    uint64_t total_dirty_samples = DirtyStat.total_dirty_samples;
-    uint64_t total_sample_count = DirtyStat.total_sample_count;
-    uint64_t total_block_mem_MB = DirtyStat.total_block_mem_MB;
+    uint64_t total_dirty_samples = DirtyStat.method.vm.total_dirty_samples;
+    uint64_t total_sample_count = DirtyStat.method.vm.total_sample_count;
+    uint64_t total_block_mem_MB = DirtyStat.method.vm.total_block_mem_MB;
 
     dirtyrate = total_dirty_samples * total_block_mem_MB *
                 1000 / (total_sample_count * msec);
@@ -327,7 +333,7 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
         update_dirtyrate_stat(block_dinfo);
     }
 
-    if (DirtyStat.total_sample_count == 0) {
+    if (DirtyStat.method.vm.total_sample_count == 0) {
         return false;
     }
 
@@ -372,8 +378,6 @@ void *get_dirtyrate_thread(void *arg)
     struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;
     int ret;
     int64_t start_time;
-    int64_t calc_time;
-    uint64_t sample_pages;
 
     ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,
                               DIRTY_RATE_STATUS_MEASURING);
@@ -383,9 +387,7 @@ void *get_dirtyrate_thread(void *arg)
     }
 
     start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000;
-    calc_time = config.sample_period_seconds;
-    sample_pages = config.sample_pages_per_gigabytes;
-    init_dirtyrate_stat(start_time, calc_time, sample_pages);
+    init_dirtyrate_stat(start_time, config);
 
     calculate_dirtyrate(config);
 
migration/dirtyrate.h +15 -3
index ec82716b40..af32e33d5d 100644
@@ -59,17 +59,29 @@ struct RamblockDirtyInfo {
     uint32_t *hash_result; /* array of hash result for sampled pages */
 };
 
+typedef struct SampleVMStat {
+    uint64_t total_dirty_samples; /* total dirty sampled page */
+    uint64_t total_sample_count; /* total sampled pages */
+    uint64_t total_block_mem_MB; /* size of total sampled pages in MB */
+} SampleVMStat;
+
+typedef struct VcpuStat {
+    int nvcpu; /* number of vcpu */
+    DirtyRateVcpu *rates; /* array of dirty rate for each vcpu */
+} VcpuStat;
+
 /*
  * Store calculation statistics for each measure.
  */
 struct DirtyRateStat {
-    uint64_t total_dirty_samples; /* total dirty sampled page */
-    uint64_t total_sample_count; /* total sampled pages */
-    uint64_t total_block_mem_MB; /* size of total sampled pages in MB */
     int64_t dirty_rate; /* dirty rate in MB/s */
     int64_t start_time; /* calculation start time in units of second */
     int64_t calc_time; /* time duration of two sampling in units of second */
     uint64_t sample_pages; /* sample pages per GB */
+    union {
+        SampleVMStat vm;
+        VcpuStat vcpu;
+    } method;
 };
 
 void *get_dirtyrate_thread(void *arg);
Signature · 1 line
2.18.2
#13 lore raw

PATCH v3 6/7 memory: make global_dirty_log a bitmask

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem.

make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate.

all references to global_dirty_log should be untouched because any bit set there should justify that global dirty logging is enabled.

---
 include/exec/memory.h | 13 ++++++++++---
 migration/ram.c       |  8 ++++----
 softmmu/memory.c      | 36 +++++++++++++++++++++++++++---------
 3 files changed, 41 insertions(+), 16 deletions(-)
include/exec/memory.h +10 -3
index c158fd7084..94c7088299 100644
@@ -55,7 +55,10 @@ static inline void fuzz_dma_read_cb(size_t addr,
 }
 #endif
 
-extern bool global_dirty_log;
+#define GLOBAL_DIRTY_MIGRATION  (1U<<0)
+#define GLOBAL_DIRTY_DIRTY_RATE (1U<<1)
+
+extern int global_dirty_log;
 
 typedef struct MemoryRegionOps MemoryRegionOps;
 
@@ -2099,13 +2102,17 @@ void memory_listener_unregister(MemoryListener *listener);
 
 /**
  * memory_global_dirty_log_start: begin dirty logging for all regions
+ *
+ * @flags: purpose of start dirty log, migration or dirty rate
  */
-void memory_global_dirty_log_start(void);
+void memory_global_dirty_log_start(int flags);
 
 /**
  * memory_global_dirty_log_stop: end dirty logging for all regions
+ *
+ * @flags: purpose of stop dirty log, migration or dirty rate
  */
-void memory_global_dirty_log_stop(void);
+void memory_global_dirty_log_stop(int flags);
 
 void mtree_info(bool flatview, bool dispatch_tree, bool owner, bool disabled);
 
migration/ram.c +4 -4
index 60ea913c54..9ce31af9d1 100644
@@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque)
         /* caller have hold iothread lock or is in a bh, so there is
          * no writing race against the migration bitmap
          */
-        memory_global_dirty_log_stop();
+        memory_global_dirty_log_stop(GLOBAL_DIRTY_MIGRATION);
     }
 
     RAMBLOCK_FOREACH_NOT_IGNORED(block) {
@@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs)
         ram_list_init_bitmaps();
         /* We don't use dirty log with background snapshots */
         if (!migrate_background_snapshot()) {
-            memory_global_dirty_log_start();
+            memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION);
             migration_bitmap_sync_precopy(rs);
         }
     }
@@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void)
             /* Discard this dirty bitmap record */
             bitmap_zero(block->bmap, block->max_length >> TARGET_PAGE_BITS);
         }
-        memory_global_dirty_log_start();
+        memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION);
     }
     ram_state->migration_dirty_pages = 0;
     qemu_mutex_unlock_ramlist();
@@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void)
 {
     RAMBlock *block;
 
-    memory_global_dirty_log_stop();
+    memory_global_dirty_log_stop(GLOBAL_DIRTY_MIGRATION);
     RAMBLOCK_FOREACH_NOT_IGNORED(block) {
         g_free(block->bmap);
         block->bmap = NULL;
softmmu/memory.c +27 -9
index c19b0be6b1..b93baba82d 100644
@@ -39,7 +39,7 @@
 static unsigned memory_region_transaction_depth;
 static bool memory_region_update_pending;
 static bool ioeventfd_update_pending;
-bool global_dirty_log;
+int global_dirty_log;
 
 static QTAILQ_HEAD(, MemoryListener) memory_listeners
     = QTAILQ_HEAD_INITIALIZER(memory_listeners);
@@ -2659,14 +2659,20 @@ void memory_global_after_dirty_log_sync(void)
 
 static VMChangeStateEntry *vmstate_change;
 
-void memory_global_dirty_log_start(void)
+void memory_global_dirty_log_start(int flags)
 {
     if (vmstate_change) {
         qemu_del_vm_change_state_handler(vmstate_change);
         vmstate_change = NULL;
     }
 
-    global_dirty_log = true;
+    if (flags & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log |= GLOBAL_DIRTY_MIGRATION;
+    }
+
+    if (flags & GLOBAL_DIRTY_DIRTY_RATE) {
+        global_dirty_log |= GLOBAL_DIRTY_DIRTY_RATE;
+    }
 
     MEMORY_LISTENER_CALL_GLOBAL(log_global_start, Forward);
 
@@ -2676,9 +2682,15 @@ void memory_global_dirty_log_start(void)
     memory_region_transaction_commit();
 }
 
-static void memory_global_dirty_log_do_stop(void)
+static void memory_global_dirty_log_do_stop(int flags)
 {
-    global_dirty_log = false;
+    if (flags & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log &= ~GLOBAL_DIRTY_MIGRATION;
+    }
+
+    if (flags & GLOBAL_DIRTY_DIRTY_RATE) {
+        global_dirty_log &= ~GLOBAL_DIRTY_DIRTY_RATE;
+    }
 
     /* Refresh DIRTY_MEMORY_MIGRATION bit.  */
     memory_region_transaction_begin();
@@ -2691,8 +2703,10 @@ static void memory_global_dirty_log_do_stop(void)
 static void memory_vm_change_state_handler(void *opaque, bool running,
                                            RunState state)
 {
+    int *flags = opaque;
     if (running) {
-        memory_global_dirty_log_do_stop();
+        memory_global_dirty_log_do_stop(*flags);
+        g_free(opaque);
 
         if (vmstate_change) {
             qemu_del_vm_change_state_handler(vmstate_change);
@@ -2701,18 +2715,22 @@ static void memory_vm_change_state_handler(void *opaque, bool running,
     }
 }
 
-void memory_global_dirty_log_stop(void)
+void memory_global_dirty_log_stop(int flags)
 {
+    int *opaque = NULL;
     if (!runstate_is_running()) {
         if (vmstate_change) {
             return;
         }
+
+        opaque = g_malloc0(sizeof(opaque));
+        *opaque = flags;
         vmstate_change = qemu_add_vm_change_state_handler(
-                                memory_vm_change_state_handler, NULL);
+                                memory_vm_change_state_handler, opaque);
         return;
     }
 
-    memory_global_dirty_log_do_stop();
+    memory_global_dirty_log_do_stop(flags);
 }
 
 static void listener_add_address_space(MemoryListener *listener,
Signature · 1 line
2.18.2
#14 lore raw

Re: [PATCH v3 6/7] memory: make global_dirty_log a bitmask

Quoted 26 lines · On Mon, Jun 07, 2021 at 09:13:12AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem.

make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate.

all references to global_dirty_log should be untouched because any bit set there should justify that global dirty logging is enabled.

---
 include/exec/memory.h | 13 ++++++++++---
 migration/ram.c       |  8 ++++----
 softmmu/memory.c      | 36 +++++++++++++++++++++++++++---------
 3 files changed, 41 insertions(+), 16 deletions(-)
include/exec/memory.h +0 -0
index c158fd7084..94c7088299 100644

include/exec/memory.h static inline void fuzz_dma_read_cb(size_t addr, · line 59

Quoted 6 lines
include/exec/memory.h +2 -1
@@ -55,7 +55,10 @@ static inline void fuzz_dma_read_cb(size_t addr,
 }
 #endif
 
-extern bool global_dirty_log;
+#define GLOBAL_DIRTY_MIGRATION  (1U<<0)
+#define GLOBAL_DIRTY_DIRTY_RATE (1U<<1)

Add some comment for each dirty log reason would be nice.

softmmu/memory.c line 42

Quoted 10 lines
softmmu/memory.c +1 -1
index c19b0be6b1..b93baba82d 100644
@@ -39,7 +39,7 @@
 static unsigned memory_region_transaction_depth;
 static bool memory_region_update_pending;
 static bool ioeventfd_update_pending;
-bool global_dirty_log;
+int global_dirty_log;

Maybe unsigned int would make more sense for a bitmask?

Quoted 13 lines
softmmu/memory.c +1 -1
 static QTAILQ_HEAD(, MemoryListener) memory_listeners
     = QTAILQ_HEAD_INITIALIZER(memory_listeners);
@@ -2659,14 +2659,20 @@ void memory_global_after_dirty_log_sync(void)
 
 static VMChangeStateEntry *vmstate_change;
 
-void memory_global_dirty_log_start(void)
+void memory_global_dirty_log_start(int flags)
 {
     if (vmstate_change) {
         qemu_del_vm_change_state_handler(vmstate_change);
         vmstate_change = NULL;

softmmu/memory.c void memory_global_after_dirty_log_sync(void) · line 2675

Quoted 10 lines
softmmu/memory.c +7 -1
     }
 
-    global_dirty_log = true;
+    if (flags & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log |= GLOBAL_DIRTY_MIGRATION;
+    }
+
+    if (flags & GLOBAL_DIRTY_DIRTY_RATE) {
+        global_dirty_log |= GLOBAL_DIRTY_DIRTY_RATE;
+    }

Instead of separate "if"s, perhaps something like this?

  #define  GLOBAL_DIRTY_MASK  (0x3)

  assert(!(flags & (~GLOBAL_DIRTY_MASK)));
  assert(!(global_dirty_log ^ flags));
  global_dirty_log |= flags;
Quoted 8 lines
softmmu/memory.c +0 -1
     MEMORY_LISTENER_CALL_GLOBAL(log_global_start, Forward);
 
@@ -2676,9 +2682,15 @@ void memory_global_dirty_log_start(void)
     memory_region_transaction_commit();
 }
 
-static void memory_global_dirty_log_do_stop(void)

softmmu/memory.c void memory_global_dirty_log_start(void) · line 2693

Quoted 10 lines
softmmu/memory.c +8 -1
+static void memory_global_dirty_log_do_stop(int flags)
 {
-    global_dirty_log = false;
+    if (flags & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log &= ~GLOBAL_DIRTY_MIGRATION;
+    }
+
+    if (flags & GLOBAL_DIRTY_DIRTY_RATE) {
+        global_dirty_log &= ~GLOBAL_DIRTY_DIRTY_RATE;
+    }

Same here? Thanks,

Signature · 1 line
Peter Xu
#15 lore raw

PATCH v3 7/7 migration/dirtyrate: implement dirty-ring dirtyrate calculation

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

use dirty ring feature to implement dirtyrate calculation. to enable it, set vcpu option as true in calc-dirty-rate.

add per_vcpu as mandatory option in calc_dirty_rate, to calculate dirty rate for vcpu, and use hmp cmd:
(qemu) calc_dirty_rate 1 on

---
 hmp-commands.hx        |   7 +-
 migration/dirtyrate.c  | 226 ++++++++++++++++++++++++++++++++++++++---
 migration/trace-events |   5 +
 3 files changed, 220 insertions(+), 18 deletions(-)
hmp-commands.hx +4 -3
index 84dcc3aae6..cc24ab2ab1 100644
@@ -1736,8 +1736,9 @@ ERST
 
     {
         .name       = "calc_dirty_rate",
-        .args_type  = "second:l,sample_pages_per_GB:l?",
-        .params     = "second [sample_pages_per_GB]",
-        .help       = "start a round of guest dirty rate measurement",
+        .args_type  = "second:l,per_vcpu:b,sample_pages_per_GB:l?",
+        .params     = "second on|off [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement, "
+                      "calculate for vcpu use on|off",
         .cmd        = hmp_calc_dirty_rate,
     },
migration/dirtyrate.c +211 -15

300 lines not on this page · raw message

migration/trace-events +5 -0
index 860c4f4025..4c5a658665 100644
@@ -330,6 +330,11 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, uint32_t crc) "ramblock n
 calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32
 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock name: %s, ramblock size: %" PRIu64
 find_page_matched(const char *idstr) "ramblock %s addr or size changed"
+dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64
+dirtyrate_do_calculate_vcpu(int idx, uint64_t pages, int64_t seconds) "vcpu[%d]: dirty %"PRIu64 " pages in %"PRIi64 " seconds"
+dirtyrate_powerdown_callback(void) ""
+dirtyrate_dirty_log_start(void) ""
+dirtyrate_dirty_log_stop(void) ""
 
 # block.c
 migration_block_init_shared(const char *blk_device_name) "Start migration for %s with shared base image"
Signature · 1 line
2.18.2
#16 lore raw

Re: [PATCH v3 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation

Quoted 21 lines · On Mon, Jun 07, 2021 at 09:15:20AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

use dirty ring feature to implement dirtyrate calculation. to enable it, set vcpu option as true in calc-dirty-rate.

add per_vcpu as mandatory option in calc_dirty_rate, to calculate dirty rate for vcpu, and use hmp cmd:
(qemu) calc_dirty_rate 1 on

---
 hmp-commands.hx        |   7 +-
 migration/dirtyrate.c  | 226 ++++++++++++++++++++++++++++++++++++++---
 migration/trace-events |   5 +
 3 files changed, 220 insertions(+), 18 deletions(-)
hmp-commands.hx +0 -0
index 84dcc3aae6..cc24ab2ab1 100644

hmp-commands.hx ERST · line 1739

Quoted 7 lines
hmp-commands.hx +1 -3
@@ -1736,8 +1736,9 @@ ERST
 
     {
         .name       = "calc_dirty_rate",
-        .args_type  = "second:l,sample_pages_per_GB:l?",
-        .params     = "second [sample_pages_per_GB]",
-        .help       = "start a round of guest dirty rate measurement",
+        .args_type  = "second:l,per_vcpu:b,sample_pages_per_GB:l?",

How about "dirty-ring:-r"? Then it's: "(qemu) calc_dirty_rate -r 10". It can still be a bool in HMP even if it's a "*mode" in qmp. We can further make "-l" for dirty logging (if we want that at last) and make two flags exclusive.

Quoted 24 lines
hmp-commands.hx +3 -0
+        .params     = "second on|off [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement, "
+                      "calculate for vcpu use on|off",
         .cmd        = hmp_calc_dirty_rate,
     },
migration/dirtyrate.c +4 -0
index 055145c24c..e432118f49 100644
@@ -16,6 +16,9 @@
 #include "cpu.h"
 #include "exec/ramblock.h"
 #include "qemu/rcu_queue.h"
+#include "qemu/main-loop.h"
+#include "sysemu/kvm.h"
+#include "sysemu/runstate.h"
 #include "qapi/qapi-commands-migration.h"
 #include "ram.h"
 #include "trace.h"
@@ -23,9 +26,38 @@
 #include "monitor/hmp.h"
 #include "monitor/monitor.h"
 #include "qapi/qmp/qdict.h"
+#include "exec/memory.h"

migration/dirtyrate.c line 39

Quoted 10 lines
migration/dirtyrate.c +10 -0
+
+typedef enum {
+    CALC_NONE = 0,
+    CALC_DIRTY_RING,
+    CALC_SAMPLE_PAGES,
+} CalcMethod;
+
+typedef struct DirtyPageRecord {
+    int64_t start_pages;
+    int64_t end_pages;

Why not uint64_t? Note that we can also detect overflows using end_pages < start_pages when needed, but imho we don't even need to worry about it - since even if overflowed, "end - start" will still generate the right value..

migration/dirtyrate.c line 46

Quoted 7 lines
migration/dirtyrate.c +4 -0
+} DirtyPageRecord;
+
+static DirtyPageRecord *dirty_pages;
 
 static int CalculatingState = DIRTY_RATE_STATUS_UNSTARTED;
 static struct DirtyRateStat DirtyStat;
+static CalcMethod last_method = CALC_NONE;

How about simply name it "dirty_rate_method" as it's "current" not "last"?

migration/dirtyrate.c line 56

Quoted 10 lines
migration/dirtyrate.c +10 -0
+bool register_powerdown_callback = false;
+
+static void dirtyrate_powerdown_req(Notifier *n, void *opaque)
+{
+    if (last_method == CALC_DIRTY_RING) {
+        g_free(DirtyStat.method.vcpu.rates);
+        DirtyStat.method.vcpu.rates = NULL;
+    }
+    trace_dirtyrate_powerdown_callback();
+}

In the cover letter, you did mention this as "add memory free callback to prevent memory leaking" but I didn't really follow..

If VM quits, QEMU quits, things got freed anyways (by OS)?

Quoted 16 lines
migration/dirtyrate.c +5 -0
+
+static Notifier dirtyrate_powerdown_notifier = {
+    .notify = dirtyrate_powerdown_req
+};
 
 static int64_t set_sample_page_period(int64_t msec, int64_t initial_time)
 {
@@ -72,6 +104,7 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
 {
     int64_t dirty_rate = DirtyStat.dirty_rate;
     struct DirtyRateInfo *info = g_malloc0(sizeof(DirtyRateInfo));
+    DirtyRateVcpuList *head = NULL, **tail = &head;
 
     if (qatomic_read(&CalculatingState) == DIRTY_RATE_STATUS_MEASURED) {
         info->has_dirty_rate = true;

migration/dirtyrate.c static struct DirtyRateInfo *query_dirty_rate_info(void) · line 122

Quoted 10 lines
migration/dirtyrate.c +6 -1
@@ -81,7 +114,22 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
     info->status = CalculatingState;
     info->start_time = DirtyStat.start_time;
     info->calc_time = DirtyStat.calc_time;
-    info->sample_pages = DirtyStat.sample_pages;
+
+    if (last_method == CALC_DIRTY_RING) {
+        int i = 0;
+        info->per_vcpu = true;
+        info->has_vcpu_dirty_rate = true;
+        for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {

I would also suggest not use "vcpu" as name of field, maybe also "dirty_ring"?

And I think we can omit the "method" too and compilers should know it (same to the other places)? Then it can be written as DirtyState.dirty_ring.nvcpu.

Quoted 44 lines
migration/dirtyrate.c +28 -9
+            DirtyRateVcpu *rate = g_malloc0(sizeof(DirtyRateVcpu));
+            rate->id = DirtyStat.method.vcpu.rates[i].id;
+            rate->dirty_rate = DirtyStat.method.vcpu.rates[i].dirty_rate;
+            QAPI_LIST_APPEND(tail, rate);
+        }
+        info->vcpu_dirty_rate = head;
+    } else {
+        info->has_sample_pages = true;
+        info->sample_pages = DirtyStat.sample_pages;
+    }
 
     trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState));
 
@@ -94,15 +142,37 @@ static void init_dirtyrate_stat(int64_t start_time,
     DirtyStat.dirty_rate = -1;
     DirtyStat.start_time = start_time;
     DirtyStat.calc_time = config.sample_period_seconds;
-    DirtyStat.sample_pages = config.sample_pages_per_gigabytes;
-
-    if (config.per_vcpu) {
-        DirtyStat.method.vcpu.nvcpu = -1;
-        DirtyStat.method.vcpu.rates = NULL;
-    } else {
-        DirtyStat.method.vm.total_dirty_samples = 0;
-        DirtyStat.method.vm.total_sample_count = 0;
-        DirtyStat.method.vm.total_block_mem_MB = 0;
+    DirtyStat.sample_pages =
+        config.per_vcpu ? -1 : config.sample_pages_per_gigabytes;
+
+    if (unlikely(!register_powerdown_callback)) {
+        qemu_register_powerdown_notifier(&dirtyrate_powerdown_notifier);
+        register_powerdown_callback = true;
+    }
+
+    switch (last_method) {
+    case CALC_NONE:
+    case CALC_SAMPLE_PAGES:
+        if (config.per_vcpu) {
+            DirtyStat.method.vcpu.nvcpu = -1;
+            DirtyStat.method.vcpu.rates = NULL;
+        } else {
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;

migration/dirtyrate.c static void init_dirtyrate_stat(int64_t start_time, · line 172

Quoted 10 lines
migration/dirtyrate.c +10 -0
+        }
+        break;
+    case CALC_DIRTY_RING:
+        if (!config.per_vcpu) {
+            g_free(DirtyStat.method.vcpu.rates);
+            DirtyStat.method.vcpu.rates = NULL;
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;
+        }

I'm a bit confused; why it's CALC_DIRTY_RING but per_vcpu not set? Why we need to care about "last_method" at all?..

migration/dirtyrate.c static void init_dirtyrate_stat(int64_t start_time, · line 174

Quoted 2 lines
migration/dirtyrate.c +2 -0
+        break;
+    default:

We can abort() here.

Quoted 23 lines
migration/dirtyrate.c +7 -2
+        break;
     }
 }
 
@@ -316,7 +386,7 @@ find_block_matched(RAMBlock *block, int count,
 }
 
 static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
-                                  int block_count)
+                                   int block_count)
 {
     struct RamblockDirtyInfo *block_dinfo = NULL;
     RAMBlock *block = NULL;
@@ -340,14 +410,125 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
     return true;
 }
 
-static void calculate_dirtyrate(struct DirtyRateConfig config)
+static void record_dirtypages(CPUState *cpu, bool start)
+{
+    if (start) {
+        dirty_pages[cpu->cpu_index].start_pages = cpu->dirty_pages;
+    } else {

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 427

Quoted 10 lines
migration/dirtyrate.c +10 -0
+        dirty_pages[cpu->cpu_index].end_pages = cpu->dirty_pages;
+    }
+}
+
+static void dirtyrate_global_dirty_log_start(void)
+{
+    /* dirty logging is enabled already */
+    if (global_dirty_log) {
+        return;
+    }

If it's a bitmask already, then we'd want to drop this..

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 432

Quoted 5 lines
migration/dirtyrate.c +5 -0
+
+    qemu_mutex_lock_iothread();
+    memory_global_dirty_log_start(GLOBAL_DIRTY_DIRTY_RATE);
+    qemu_mutex_unlock_iothread();
+    trace_dirtyrate_dirty_log_start();

How about moving this trace into memory_global_dirty_log_start() of the other patch, dumps the bitmask?

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 442

Quoted 10 lines
migration/dirtyrate.c +10 -0
+}
+
+static void dirtyrate_global_dirty_log_stop(void)
+{
+    /* migration is in process, do not stop dirty logging,
+     * just clear the GLOBAL_DIRTY_DIRTY_RATE bit */
+    if (global_dirty_log & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log &= ~(GLOBAL_DIRTY_DIRTY_RATE);
+        return;
+    }

IIUC we don't need this either..

memory_global_dirty_log_start|stop() will make sure all things work already, we should only use these apis and stop caring about migration at all.

Or did I miss something?

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 447

Quoted 5 lines
migration/dirtyrate.c +5 -0
+
+    qemu_mutex_lock_iothread();
+    memory_global_dirty_log_stop(GLOBAL_DIRTY_DIRTY_RATE);
+    qemu_mutex_unlock_iothread();
+    trace_dirtyrate_dirty_log_stop();

Same question here; maybe better to move into memory_global_dirty_log_stop()? Can make it trace_global_dirty_changed(bitmask) and call at start/stop.

Quoted 6 lines
migration/dirtyrate.c +6 -0
+}
+
+static int64_t do_calculate_dirtyrate_vcpu(int idx)
+{
+    uint64_t memory_size_MB;
+    int64_t time_s;

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 463

Quoted 10 lines
migration/dirtyrate.c +10 -0
+    uint64_t start_pages = dirty_pages[idx].start_pages;
+    uint64_t end_pages = dirty_pages[idx].end_pages;
+    uint64_t dirty_pages = 0;
+
+    /* uint64_t over the INT64_MAX */
+    if (unlikely(end_pages < start_pages)) {
+        dirty_pages = INT64_MAX - start_pages + end_pages + 1;
+    } else {
+        dirty_pages = end_pages - start_pages;
+    }

As mentioned above, IMHO this would be enough:

           dirty_pages = end_pages - start_pages;

even if rare overflowed happened.

Quoted 26 lines
migration/dirtyrate.c +26 -0
+
+    memory_size_MB = (dirty_pages * TARGET_PAGE_SIZE) >> 20;
+    time_s = DirtyStat.calc_time;
+
+    trace_dirtyrate_do_calculate_vcpu(idx, dirty_pages, time_s);
+
+    return memory_size_MB / time_s;
+}
+
+static void calculate_dirtyrate_vcpu(struct DirtyRateConfig config)
+{
+    CPUState *cpu;
+    int64_t msec = 0;
+    int64_t start_time;
+    uint64_t dirtyrate = 0;
+    uint64_t dirtyrate_sum = 0;
+    int nvcpu = 0;
+    int i = 0;
+
+    CPU_FOREACH(cpu) {
+        nvcpu++;
+    }
+
+    dirty_pages = g_malloc0(sizeof(*dirty_pages) * nvcpu);
+
+    dirtyrate_global_dirty_log_start();

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 499

Quoted 10 lines
migration/dirtyrate.c +10 -0
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, true);
+    }
+
+    DirtyStat.method.vcpu.nvcpu = nvcpu;
+    if (last_method != CALC_DIRTY_RING) {
+        DirtyStat.method.vcpu.rates =
+            g_malloc0(sizeof(DirtyRateVcpu) * nvcpu);
+    }

I don't see a strong need to optimize malloc() for continuous dirty rate measurements. Can we simply malloc() for every measurement we need?

If we really want this, it would be nice to make it a follow up patch, but we'd better justify why it helps...

Btw, I think it's better the malloc()s happen before measuring starts, e.g.:

cpu_foreach { nvcpu++ }
rates = malloc(...)
dirty_pages = malloc(...)

global_dirty_log_start(DIRTY_RATE)
cpu_foreach { record_dirtypages() }
...

Quoted 23 lines
migration/dirtyrate.c +23 -0
+
+    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    DirtyStat.start_time = start_time / 1000;
+
+    msec = config.sample_period_seconds * 1000;
+    msec = set_sample_page_period(msec, start_time);
+    DirtyStat.calc_time = msec / 1000;
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, false);
+    }
+
+    dirtyrate_global_dirty_log_stop();
+
+    for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {
+        dirtyrate = do_calculate_dirtyrate_vcpu(i);
+        DirtyStat.method.vcpu.rates[i].id = i;
+        DirtyStat.method.vcpu.rates[i].dirty_rate = dirtyrate;
+        dirtyrate_sum += dirtyrate;
+    }
+
+    DirtyStat.dirty_rate = dirtyrate_sum / DirtyStat.method.vcpu.nvcpu;
+    g_free(dirty_pages);

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 350

Quoted 10 lines
migration/dirtyrate.c +3 -1
+}
+
+static void calculate_dirtyrate_sample_vm(struct DirtyRateConfig config)
 {
     struct RamblockDirtyInfo *block_dinfo = NULL;
     int block_count = 0;
     int64_t msec = 0;
     int64_t initial_time;
 
-    rcu_register_thread();

Better to make this a separate patch.

Thanks,

Quoted 67 lines
migration/dirtyrate.c +17 -2
     rcu_read_lock();
     initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     if (!record_ramblock_hash_info(&block_dinfo, config, &block_count)) {
@@ -364,13 +545,24 @@ static void calculate_dirtyrate(struct DirtyRateConfig config)
     if (!compare_page_hash_info(block_dinfo, block_count)) {
         goto out;
     }
-
     update_dirtyrate(msec);
 
 out:
     rcu_read_unlock();
     free_ramblock_dirty_info(block_dinfo, block_count);
-    rcu_unregister_thread();
+}
+
+static void calculate_dirtyrate(struct DirtyRateConfig config)
+{
+    if (config.per_vcpu) {
+        calculate_dirtyrate_vcpu(config);
+        last_method = CALC_DIRTY_RING;
+    } else {
+        calculate_dirtyrate_sample_vm(config);
+        last_method = CALC_SAMPLE_PAGES;
+    }
+
+    trace_dirtyrate_calculate(DirtyStat.dirty_rate);
 }
 
 void *get_dirtyrate_thread(void *arg)
@@ -379,6 +571,8 @@ void *get_dirtyrate_thread(void *arg)
     int ret;
     int64_t start_time;
 
+    rcu_register_thread();
+
     ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,
                               DIRTY_RATE_STATUS_MEASURING);
     if (ret == -1) {
@@ -396,6 +590,8 @@ void *get_dirtyrate_thread(void *arg)
     if (ret == -1) {
         error_report("change dirtyrate state failed.");
     }
+
+    rcu_unregister_thread();
     return NULL;
 }
 
migration/trace-events +5 -0
index 860c4f4025..4c5a658665 100644
@@ -330,6 +330,11 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, uint32_t crc) "ramblock n
 calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32
 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock name: %s, ramblock size: %" PRIu64
 find_page_matched(const char *idstr) "ramblock %s addr or size changed"
+dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64
+dirtyrate_do_calculate_vcpu(int idx, uint64_t pages, int64_t seconds) "vcpu[%d]: dirty %"PRIu64 " pages in %"PRIi64 " seconds"
+dirtyrate_powerdown_callback(void) ""
+dirtyrate_dirty_log_start(void) ""
+dirtyrate_dirty_log_stop(void) ""
 
 # block.c
 migration_block_init_shared(const char *blk_device_name) "Start migration for %s with shared base image"
Signature · 1 line
2.18.2
Signature · 1 line
Peter Xu
#17 lore raw

Re: [PATCH v3 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation

Quoted 64 lines · Peter Xu
Quoted 21 lines · On Mon, Jun 07, 2021 at 09:15:20AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

use dirty ring feature to implement dirtyrate calculation. to enable it, set vcpu option as true in calc-dirty-rate.

add per_vcpu as mandatory option in calc_dirty_rate, to calculate dirty rate for vcpu, and use hmp cmd:
(qemu) calc_dirty_rate 1 on

---
  hmp-commands.hx        |   7 +-
  migration/dirtyrate.c  | 226 ++++++++++++++++++++++++++++++++++++++---
  migration/trace-events |   5 +
  3 files changed, 220 insertions(+), 18 deletions(-)
hmp-commands.hx +0 -0
index 84dcc3aae6..cc24ab2ab1 100644

hmp-commands.hx ERST · line 1739

Quoted 7 lines
hmp-commands.hx +1 -3
@@ -1736,8 +1736,9 @@ ERST
 
      {
          .name       = "calc_dirty_rate",
-        .args_type  = "second:l,sample_pages_per_GB:l?",
-        .params     = "second [sample_pages_per_GB]",
-        .help       = "start a round of guest dirty rate measurement",
+        .args_type  = "second:l,per_vcpu:b,sample_pages_per_GB:l?",

How about "dirty-ring:-r"? Then it's: "(qemu) calc_dirty_rate -r 10". It can still be a bool in HMP even if it's a "*mode" in qmp. We can further make "-l" for dirty logging (if we want that at last) and make two flags exclusive.

Quoted 30 lines
hmp-commands.hx +3 -0
+        .params     = "second on|off [sample_pages_per_GB]",
+        .help       = "start a round of guest dirty rate measurement, "
+                      "calculate for vcpu use on|off",
          .cmd        = hmp_calc_dirty_rate,
      },
migration/dirtyrate.c +10 -0
index 055145c24c..e432118f49 100644
@@ -16,6 +16,9 @@
  #include "cpu.h"
  #include "exec/ramblock.h"
  #include "qemu/rcu_queue.h"
+#include "qemu/main-loop.h"
+#include "sysemu/kvm.h"
+#include "sysemu/runstate.h"
  #include "qapi/qapi-commands-migration.h"
  #include "ram.h"
  #include "trace.h"
@@ -23,9 +26,38 @@
  #include "monitor/hmp.h"
  #include "monitor/monitor.h"
  #include "qapi/qmp/qdict.h"
+#include "exec/memory.h"
+
+typedef enum {
+    CALC_NONE = 0,
+    CALC_DIRTY_RING,
+    CALC_SAMPLE_PAGES,
+} CalcMethod;
Quoted 8 lines

migration/dirtyrate.c line 39

Quoted 4 lines
migration/dirtyrate.c +4 -0
+
+typedef struct DirtyPageRecord {
+    int64_t start_pages;
+    int64_t end_pages;

Why not uint64_t? Note that we can also detect overflows using end_pages < start_pages when needed, but imho we don't even need to worry about it - since even if overflowed, "end - start" will still generate the right value..

yeah, it's more efficient. i'll change the type next version.

Quoted 10 lines

migration/dirtyrate.c line 46

Quoted 8 lines
migration/dirtyrate.c +4 -0
+} DirtyPageRecord;
+
+static DirtyPageRecord *dirty_pages;
 
  static int CalculatingState = DIRTY_RATE_STATUS_UNSTARTED;
  static struct DirtyRateStat DirtyStat;
+static CalcMethod last_method = CALC_NONE;

How about simply name it "dirty_rate_method" as it's "current" not "last"?

yeah !

Quoted 15 lines

migration/dirtyrate.c line 56

Quoted 11 lines
migration/dirtyrate.c +10 -0
+bool register_powerdown_callback = false;
+
+static void dirtyrate_powerdown_req(Notifier *n, void *opaque)
+{
+    if (last_method == CALC_DIRTY_RING) {
+        g_free(DirtyStat.method.vcpu.rates);
+        DirtyStat.method.vcpu.rates = NULL;
+    }
+    trace_dirtyrate_powerdown_callback();
+}

In the cover letter, you did mention this as "add memory free callback to prevent memory leaking" but I didn't really follow..

Quoted 1 line

If VM quits, QEMU quits, things got freed anyways (by OS)?

ok, i add this callback just ensure the qemu do free the struct logically. but it seems that this can't be done in many scenarios like qemu process has been killed, and the callback can't be triggered. so letting the OS handle the free work is a wise decision.

Quoted 82 lines
Quoted 17 lines
migration/dirtyrate.c +5 -0
+
+static Notifier dirtyrate_powerdown_notifier = {
+    .notify = dirtyrate_powerdown_req
+};
 
  static int64_t set_sample_page_period(int64_t msec, int64_t initial_time)
  {
@@ -72,6 +104,7 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
  {
      int64_t dirty_rate = DirtyStat.dirty_rate;
      struct DirtyRateInfo *info = g_malloc0(sizeof(DirtyRateInfo));
+    DirtyRateVcpuList *head = NULL, **tail = &head;
 
      if (qatomic_read(&CalculatingState) == DIRTY_RATE_STATUS_MEASURED) {
          info->has_dirty_rate = true;

migration/dirtyrate.c static struct DirtyRateInfo *query_dirty_rate_info(void) · line 122

Quoted 10 lines
migration/dirtyrate.c +6 -1
@@ -81,7 +114,22 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)
      info->status = CalculatingState;
      info->start_time = DirtyStat.start_time;
      info->calc_time = DirtyStat.calc_time;
-    info->sample_pages = DirtyStat.sample_pages;
+
+    if (last_method == CALC_DIRTY_RING) {
+        int i = 0;
+        info->per_vcpu = true;
+        info->has_vcpu_dirty_rate = true;
+        for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {

I would also suggest not use "vcpu" as name of field, maybe also "dirty_ring"?

And I think we can omit the "method" too and compilers should know it (same to the other places)? Then it can be written as DirtyState.dirty_ring.nvcpu.

Quoted 49 lines
migration/dirtyrate.c +33 -9
+            DirtyRateVcpu *rate = g_malloc0(sizeof(DirtyRateVcpu));
+            rate->id = DirtyStat.method.vcpu.rates[i].id;
+            rate->dirty_rate = DirtyStat.method.vcpu.rates[i].dirty_rate;
+            QAPI_LIST_APPEND(tail, rate);
+        }
+        info->vcpu_dirty_rate = head;
+    } else {
+        info->has_sample_pages = true;
+        info->sample_pages = DirtyStat.sample_pages;
+    }
 
      trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState));
 
@@ -94,15 +142,37 @@ static void init_dirtyrate_stat(int64_t start_time,
      DirtyStat.dirty_rate = -1;
      DirtyStat.start_time = start_time;
      DirtyStat.calc_time = config.sample_period_seconds;
-    DirtyStat.sample_pages = config.sample_pages_per_gigabytes;
-
-    if (config.per_vcpu) {
-        DirtyStat.method.vcpu.nvcpu = -1;
-        DirtyStat.method.vcpu.rates = NULL;
-    } else {
-        DirtyStat.method.vm.total_dirty_samples = 0;
-        DirtyStat.method.vm.total_sample_count = 0;
-        DirtyStat.method.vm.total_block_mem_MB = 0;
+    DirtyStat.sample_pages =
+        config.per_vcpu ? -1 : config.sample_pages_per_gigabytes;
+
+    if (unlikely(!register_powerdown_callback)) {
+        qemu_register_powerdown_notifier(&dirtyrate_powerdown_notifier);
+        register_powerdown_callback = true;
+    }
+
+    switch (last_method) {
+    case CALC_NONE:
+    case CALC_SAMPLE_PAGES:
+        if (config.per_vcpu) {
+            DirtyStat.method.vcpu.nvcpu = -1;
+            DirtyStat.method.vcpu.rates = NULL;
+        } else {
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;
+        }
+        break;
+    case CALC_DIRTY_RING:
+        if (!config.per_vcpu) {
+            g_free(DirtyStat.method.vcpu.rates);
Quoted 8 lines

migration/dirtyrate.c static void init_dirtyrate_stat(int64_t start_time, · line 163

Quoted 5 lines
migration/dirtyrate.c +5 -0
+            DirtyStat.method.vcpu.rates = NULL;
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;
+        }

I'm a bit confused; why it's CALC_DIRTY_RING but per_vcpu not set? Why we need to care about "last_method" at all?..

this two method share the union, the sample method use the local variable of SampleVMStat type, the dirty ring method should alloc rates of DirtyRateVcpu type every time qmp calc_dirty_rate called in case of add vcpu, once rates has been store dirty rate value, it can't be freed until the next time of executing calc_dirty_rate, because info dirty rate may access the rates struct, so the rates struct can only be freed before calc_dirty_rate with sample.

Quoted 42 lines

migration/dirtyrate.c static void init_dirtyrate_stat(int64_t start_time, · line 174

Quoted 3 lines
migration/dirtyrate.c +2 -0
+        break;
+    default:

We can abort() here.

Quoted 23 lines
migration/dirtyrate.c +7 -2
+        break;
      }
  }
 
@@ -316,7 +386,7 @@ find_block_matched(RAMBlock *block, int count,
  }
 
  static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
-                                  int block_count)
+                                   int block_count)
  {
      struct RamblockDirtyInfo *block_dinfo = NULL;
      RAMBlock *block = NULL;
@@ -340,14 +410,125 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info,
      return true;
  }
 
-static void calculate_dirtyrate(struct DirtyRateConfig config)
+static void record_dirtypages(CPUState *cpu, bool start)
+{
+    if (start) {
+        dirty_pages[cpu->cpu_index].start_pages = cpu->dirty_pages;
+    } else {

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 427

Quoted 10 lines
migration/dirtyrate.c +10 -0
+        dirty_pages[cpu->cpu_index].end_pages = cpu->dirty_pages;
+    }
+}
+
+static void dirtyrate_global_dirty_log_start(void)
+{
+    /* dirty logging is enabled already */
+    if (global_dirty_log) {
+        return;
+    }

If it's a bitmask already, then we'd want to drop this..

Quoted 8 lines

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 432

Quoted 5 lines
migration/dirtyrate.c +5 -0
+
+    qemu_mutex_lock_iothread();
+    memory_global_dirty_log_start(GLOBAL_DIRTY_DIRTY_RATE);
+    qemu_mutex_unlock_iothread();
+    trace_dirtyrate_dirty_log_start();

How about moving this trace into memory_global_dirty_log_start() of the other patch, dumps the bitmask?

it's a good idea.

Quoted 47 lines

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 442

Quoted 11 lines
migration/dirtyrate.c +10 -0
+}
+
+static void dirtyrate_global_dirty_log_stop(void)
+{
+    /* migration is in process, do not stop dirty logging,
+     * just clear the GLOBAL_DIRTY_DIRTY_RATE bit */
+    if (global_dirty_log & GLOBAL_DIRTY_MIGRATION) {
+        global_dirty_log &= ~(GLOBAL_DIRTY_DIRTY_RATE);
+        return;
+    }

IIUC we don't need this either..

memory_global_dirty_log_start|stop() will make sure all things work already, we should only use these apis and stop caring about migration at all.

Or did I miss something?

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 447

Quoted 5 lines
migration/dirtyrate.c +5 -0
+
+    qemu_mutex_lock_iothread();
+    memory_global_dirty_log_stop(GLOBAL_DIRTY_DIRTY_RATE);
+    qemu_mutex_unlock_iothread();
+    trace_dirtyrate_dirty_log_stop();

Same question here; maybe better to move into memory_global_dirty_log_stop()? Can make it trace_global_dirty_changed(bitmask) and call at start/stop.

Quoted 6 lines
migration/dirtyrate.c +6 -0
+}
+
+static int64_t do_calculate_dirtyrate_vcpu(int idx)
+{
+    uint64_t memory_size_MB;
+    int64_t time_s;

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 463

Quoted 10 lines
migration/dirtyrate.c +10 -0
+    uint64_t start_pages = dirty_pages[idx].start_pages;
+    uint64_t end_pages = dirty_pages[idx].end_pages;
+    uint64_t dirty_pages = 0;
+
+    /* uint64_t over the INT64_MAX */
+    if (unlikely(end_pages < start_pages)) {
+        dirty_pages = INT64_MAX - start_pages + end_pages + 1;
+    } else {
+        dirty_pages = end_pages - start_pages;
+    }

As mentioned above, IMHO this would be enough:

Quoted 3 lines
            dirty_pages = end_pages - start_pages;

even if rare overflowed happened.

yeah, i'll drop the old algorithm

Quoted 46 lines
Quoted 27 lines
migration/dirtyrate.c +26 -0
+
+    memory_size_MB = (dirty_pages * TARGET_PAGE_SIZE) >> 20;
+    time_s = DirtyStat.calc_time;
+
+    trace_dirtyrate_do_calculate_vcpu(idx, dirty_pages, time_s);
+
+    return memory_size_MB / time_s;
+}
+
+static void calculate_dirtyrate_vcpu(struct DirtyRateConfig config)
+{
+    CPUState *cpu;
+    int64_t msec = 0;
+    int64_t start_time;
+    uint64_t dirtyrate = 0;
+    uint64_t dirtyrate_sum = 0;
+    int nvcpu = 0;
+    int i = 0;
+
+    CPU_FOREACH(cpu) {
+        nvcpu++;
+    }
+
+    dirty_pages = g_malloc0(sizeof(*dirty_pages) * nvcpu);
+
+    dirtyrate_global_dirty_log_start();

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 499

Quoted 10 lines
migration/dirtyrate.c +10 -0
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, true);
+    }
+
+    DirtyStat.method.vcpu.nvcpu = nvcpu;
+    if (last_method != CALC_DIRTY_RING) {
+        DirtyStat.method.vcpu.rates =
+            g_malloc0(sizeof(DirtyRateVcpu) * nvcpu);
+    }

I don't see a strong need to optimize malloc() for continuous dirty rate measurements. Can we simply malloc() for every measurement we need?

If we really want this, it would be nice to make it a follow up patch, but we'd better justify why it helps...

Btw, I think it's better the malloc()s happen before measuring starts, e.g.:

Quoted 8 lines
   cpu_foreach { nvcpu++ }
   rates = malloc(...)
   dirty_pages = malloc(...)

   global_dirty_log_start(DIRTY_RATE)
   cpu_foreach { record_dirtypages() }
   ...

ok, adjust it next version.

Quoted 106 lines
Quoted 23 lines
migration/dirtyrate.c +23 -0
+
+    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    DirtyStat.start_time = start_time / 1000;
+
+    msec = config.sample_period_seconds * 1000;
+    msec = set_sample_page_period(msec, start_time);
+    DirtyStat.calc_time = msec / 1000;
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, false);
+    }
+
+    dirtyrate_global_dirty_log_stop();
+
+    for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {
+        dirtyrate = do_calculate_dirtyrate_vcpu(i);
+        DirtyStat.method.vcpu.rates[i].id = i;
+        DirtyStat.method.vcpu.rates[i].dirty_rate = dirtyrate;
+        dirtyrate_sum += dirtyrate;
+    }
+
+    DirtyStat.dirty_rate = dirtyrate_sum / DirtyStat.method.vcpu.nvcpu;
+    g_free(dirty_pages);

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 350

Quoted 10 lines
migration/dirtyrate.c +3 -1
+}
+
+static void calculate_dirtyrate_sample_vm(struct DirtyRateConfig config)
  {
      struct RamblockDirtyInfo *block_dinfo = NULL;
      int block_count = 0;
      int64_t msec = 0;
      int64_t initial_time;
 
-    rcu_register_thread();

Better to make this a separate patch.

Thanks,

Quoted 67 lines
migration/dirtyrate.c +17 -2
      rcu_read_lock();
      initial_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
      if (!record_ramblock_hash_info(&block_dinfo, config, &block_count)) {
@@ -364,13 +545,24 @@ static void calculate_dirtyrate(struct DirtyRateConfig config)
      if (!compare_page_hash_info(block_dinfo, block_count)) {
          goto out;
      }
-
      update_dirtyrate(msec);
 
  out:
      rcu_read_unlock();
      free_ramblock_dirty_info(block_dinfo, block_count);
-    rcu_unregister_thread();
+}
+
+static void calculate_dirtyrate(struct DirtyRateConfig config)
+{
+    if (config.per_vcpu) {
+        calculate_dirtyrate_vcpu(config);
+        last_method = CALC_DIRTY_RING;
+    } else {
+        calculate_dirtyrate_sample_vm(config);
+        last_method = CALC_SAMPLE_PAGES;
+    }
+
+    trace_dirtyrate_calculate(DirtyStat.dirty_rate);
  }
 
  void *get_dirtyrate_thread(void *arg)
@@ -379,6 +571,8 @@ void *get_dirtyrate_thread(void *arg)
      int ret;
      int64_t start_time;
 
+    rcu_register_thread();
+
      ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,
                                DIRTY_RATE_STATUS_MEASURING);
      if (ret == -1) {
@@ -396,6 +590,8 @@ void *get_dirtyrate_thread(void *arg)
      if (ret == -1) {
          error_report("change dirtyrate state failed.");
      }
+
+    rcu_unregister_thread();
      return NULL;
  }
 
migration/trace-events +5 -0
index 860c4f4025..4c5a658665 100644
@@ -330,6 +330,11 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, uint32_t crc) "ramblock n
  calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32
  skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock name: %s, ramblock size: %" PRIu64
  find_page_matched(const char *idstr) "ramblock %s addr or size changed"
+dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64
+dirtyrate_do_calculate_vcpu(int idx, uint64_t pages, int64_t seconds) "vcpu[%d]: dirty %"PRIu64 " pages in %"PRIi64 " seconds"
+dirtyrate_powerdown_callback(void) ""
+dirtyrate_dirty_log_start(void) ""
+dirtyrate_dirty_log_stop(void) ""
 
  # block.c
  migration_block_init_shared(const char *blk_device_name) "Start migration for %s with shared base image"
Signature · 1 line
2.18.2
Signature · 3 lines
Best regard

Hyman Huang(黄勇)
#18 lore raw

Re: [PATCH v3 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation

Quoted 23 lines · Hyman Huang
Quoted 21 lines
Quoted 10 lines
migration/dirtyrate.c +10 -0
+    switch (last_method) {
+    case CALC_NONE:
+    case CALC_SAMPLE_PAGES:
+        if (config.per_vcpu) {
+            DirtyStat.method.vcpu.nvcpu = -1;
+            DirtyStat.method.vcpu.rates = NULL;
+        } else {
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;

migration/dirtyrate.c static void init_dirtyrate_stat(int64_t start_time, · line 172

Quoted 10 lines
migration/dirtyrate.c +10 -0
+        }
+        break;
+    case CALC_DIRTY_RING:
+        if (!config.per_vcpu) {
+            g_free(DirtyStat.method.vcpu.rates);
+            DirtyStat.method.vcpu.rates = NULL;
+            DirtyStat.method.vm.total_dirty_samples = 0;
+            DirtyStat.method.vm.total_sample_count = 0;
+            DirtyStat.method.vm.total_block_mem_MB = 0;
+        }
Quoted 2 lines

I'm a bit confused; why it's CALC_DIRTY_RING but per_vcpu not set? Why we need to care about "last_method" at all?..

Quoted 7 lines

this two method share the union, the sample method use the local variable of SampleVMStat type, the dirty ring method should alloc rates of DirtyRateVcpu type every time qmp calc_dirty_rate called in case of add vcpu, once rates has been store dirty rate value, it can't be freed until the next time of executing calc_dirty_rate, because info dirty rate may access the rates struct, so the rates struct can only be freed before calc_dirty_rate with sample.

OK, then please consider separate the cleanup and init, otherwise as mode grows, it'll be a N*N matrix, afaict..

So we can always clean everything first depending on the old mode, then update last_method (or whatever its new name is...) to the new mode, and init the stat for the new mode. Then it'll always be 2*N.

Another note is that I also suggest you do all these in the main thread, not the sampling thread. Because if you do that in sampling thread it means main thread can "query" at any time, then we'd better need a mutex to protect the whole section when you update either last_method or DirtyStat. In main thread it should be serialized already as there's either BQL or a single QMP monitor thread.

Oh wait... not sure whether "only one QMP iothread" will still keep true forever.. So maybe always introduce a mutex and protect all those fields; who knows when QMP will start concurrent execution of these "info" commands (it looks doable.. :).

Signature · 1 line
Peter Xu
#19 lore raw

Re: [PATCH v3 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation

In reply to [email protected]
Quoted 39 lines · On Mon, Jun 07, 2021 at 09:15:20AM +0800, [email protected] wrote:
migration/dirtyrate.c +39 -0
+static void calculate_dirtyrate_vcpu(struct DirtyRateConfig config)
+{
+    CPUState *cpu;
+    int64_t msec = 0;
+    int64_t start_time;
+    uint64_t dirtyrate = 0;
+    uint64_t dirtyrate_sum = 0;
+    int nvcpu = 0;
+    int i = 0;
+
+    CPU_FOREACH(cpu) {
+        nvcpu++;
+    }
+
+    dirty_pages = g_malloc0(sizeof(*dirty_pages) * nvcpu);
+
+    dirtyrate_global_dirty_log_start();
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, true);
+    }
+
+    DirtyStat.method.vcpu.nvcpu = nvcpu;
+    if (last_method != CALC_DIRTY_RING) {
+        DirtyStat.method.vcpu.rates =
+            g_malloc0(sizeof(DirtyRateVcpu) * nvcpu);
+    }
+
+    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    DirtyStat.start_time = start_time / 1000;
+
+    msec = config.sample_period_seconds * 1000;
+    msec = set_sample_page_period(msec, start_time);
+    DirtyStat.calc_time = msec / 1000;
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, false);
+    }
+

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 521

Quoted 10 lines
migration/dirtyrate.c +10 -0
+    dirtyrate_global_dirty_log_stop();
+
+    for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {
+        dirtyrate = do_calculate_dirtyrate_vcpu(i);
+        DirtyStat.method.vcpu.rates[i].id = i;
+        DirtyStat.method.vcpu.rates[i].dirty_rate = dirtyrate;
+        dirtyrate_sum += dirtyrate;
+    }
+
+    DirtyStat.dirty_rate = dirtyrate_sum / DirtyStat.method.vcpu.nvcpu;

Why you'd like to divide with nvcpu? Isn't dirtyrate_sum exactly what we want? As I don't think we care about average per-vcpu dirty rate, but total here.

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 523

Quoted 2 lines
migration/dirtyrate.c +2 -0
+    g_free(dirty_pages);
+}

I did a run with 4G mem VM, alloc 1G and dirty it with 500MB/s, then

- With old way: I got 95MB/s
- With new way: I got 128MB/s

The new way has the output with:

Dirty rate: 128 (MB/s)
vcpu[0], Dirty rate: 0
vcpu[1], Dirty rate: 1
vcpu[2], Dirty rate: 0
vcpu[3], Dirty rate: 511

I think if without the division, it'll be 512MB/s, which is matching the dirty workload I initiated.

Signature · 1 line
Peter Xu
#20 lore raw

Re: [PATCH v3 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation

Quoted 45 lines · Peter Xu
Quoted 44 lines · On Mon, Jun 07, 2021 at 09:15:20AM +0800, [email protected] wrote:
migration/dirtyrate.c +44 -0
+static void calculate_dirtyrate_vcpu(struct DirtyRateConfig config)
+{
+    CPUState *cpu;
+    int64_t msec = 0;
+    int64_t start_time;
+    uint64_t dirtyrate = 0;
+    uint64_t dirtyrate_sum = 0;
+    int nvcpu = 0;
+    int i = 0;
+
+    CPU_FOREACH(cpu) {
+        nvcpu++;
+    }
+
+    dirty_pages = g_malloc0(sizeof(*dirty_pages) * nvcpu);
+
+    dirtyrate_global_dirty_log_start();
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, true);
+    }
+
+    DirtyStat.method.vcpu.nvcpu = nvcpu;
+    if (last_method != CALC_DIRTY_RING) {
+        DirtyStat.method.vcpu.rates =
+            g_malloc0(sizeof(DirtyRateVcpu) * nvcpu);
+    }
+
+    start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+    DirtyStat.start_time = start_time / 1000;
+
+    msec = config.sample_period_seconds * 1000;
+    msec = set_sample_page_period(msec, start_time);
+    DirtyStat.calc_time = msec / 1000;
+
+    CPU_FOREACH(cpu) {
+        record_dirtypages(cpu, false);
+    }
+
+    dirtyrate_global_dirty_log_stop();
+
+    for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) {
+        dirtyrate = do_calculate_dirtyrate_vcpu(i);
+        DirtyStat.method.vcpu.rates[i].id = i;
Quoted 9 lines

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 521

Quoted 5 lines
migration/dirtyrate.c +5 -0
+        DirtyStat.method.vcpu.rates[i].dirty_rate = dirtyrate;
+        dirtyrate_sum += dirtyrate;
+    }
+
+    DirtyStat.dirty_rate = dirtyrate_sum / DirtyStat.method.vcpu.nvcpu;

Why you'd like to divide with nvcpu? Isn't dirtyrate_sum exactly what we want? As I don't think we care about average per-vcpu dirty rate, but total here.

the initial idea of mine is that the qmp output dirty rate represent the average dirty rate, my mistake.indeed, the vm dirty rate should not be the average of vcpu's, i'll fix it the next version.

Quoted 19 lines

migration/dirtyrate.c static bool compare_page_hash_info(struct RamblockDirtyInfo *info, · line 523

Quoted 2 lines
migration/dirtyrate.c +2 -0
+    g_free(dirty_pages);
+}

I did a run with 4G mem VM, alloc 1G and dirty it with 500MB/s, then

- With old way: I got 95MB/s
- With new way: I got 128MB/s

The new way has the output with:

Dirty rate: 128 (MB/s)
vcpu[0], Dirty rate: 0
vcpu[1], Dirty rate: 1
vcpu[2], Dirty rate: 0
vcpu[3], Dirty rate: 511

I think if without the division, it'll be 512MB/s, which is matching the dirty workload I initiated.

Signature · 3 lines
Best regard

Hyman Huang(黄勇)
author
#21 lore raw
In reply to [email protected]

cc Chuan Zheng <[email protected]>

Quoted 83 lines · 在 2021/6/7 9:11, [email protected] 写道:

From: Hyman Huang(黄勇) <[email protected]>

v3:
- pick up "migration/dirtyrate: make sample page count configurable" to make patchset apply master correctly

v2:
- rebase to "migration/dirtyrate: make sample page count configurable"

- rename "vcpu" to "per_vcpu" to show the per-vcpu method

- squash patch 5/6 into a single one, squash patch 1/2 also

- pick up "hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds"

- make global_dirty_log a bitmask to make sure both migration and dirty could not intefer with each other

- add memory free callback to prevent memory leaking

the most different of v2 fron v1 is that we make the global_dirty_log a bitmask. the reason is dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem.

make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate.

all references to global_dirty_log should be untouched because any bit set there should justify that global dirty logging is enabled.

Please review, thanks !

v1:

Since the Dirty Ring on QEMU part has been merged recently, how to use this feature is under consideration.

In the scene of migration, it is valuable to provide a more accurante interface to track dirty memory than existing one, so that the upper layer application can make a wise decision, or whatever. More importantly,
dirtyrate info at the granualrity of vcpu could provide a possibility to make migration convergent by imposing restriction on vcpu. With Dirty Ring, we can calculate dirtyrate efficiently and cheaply.

The old interface implemented by sampling pages, it consumes cpu resource, and the larger guest memory size become, the more cpu resource it consumes, namely, hard to scale. New interface has no such drawback.

Please review, thanks !

Best Regards !

Hyman Huang(黄勇) (6):
migration/dirtyrate: make sample page count configurable
KVM: introduce dirty_pages and kvm_dirty_ring_enabled migration/dirtyrate: add per-vcpu option for calc-dirty-rate migration/dirtyrate: adjust struct DirtyRateStat
memory: make global_dirty_log a bitmask migration/dirtyrate: implement dirty-ring dirtyrate calculation

Peter Xu (1):
hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds

  accel/kvm/kvm-all.c    |   7 +
  hmp-commands-info.hx   |  13 ++
  hmp-commands.hx        |  15 ++
  include/exec/memory.h  |  13 +-
  include/hw/core/cpu.h  |   1 +
  include/monitor/hmp.h  |   2 +
  include/sysemu/kvm.h   |   1 +
  migration/dirtyrate.c  | 347 ++++++++++++++++++++++++++++++++++++++---
  migration/dirtyrate.h  |  27 +++-
  migration/ram.c        |   8 +-
  migration/trace-events |   5 +
  qapi/migration.json    |  38 ++++-
  softmmu/memory.c       |  36 +++--
  13 files changed, 468 insertions(+), 45 deletions(-)
#22 lore raw
In reply to [email protected]
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Type: series
Message-id: [email protected]
Subject: [PATCH v3 0/7] support dirtyrate at the granualrity of vcpu

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
db25537 migration/dirtyrate: implement dirty-ring dirtyrate calculation 720f923 memory: make global_dirty_log a bitmask
a3323fd migration/dirtyrate: adjust struct DirtyRateStat 04b16bd migration/dirtyrate: add per-vcpu option for calc-dirty-rate 0ec6909 KVM: introduce dirty_pages and kvm_dirty_ring_enabled 8828757 hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds dd4342c migration/dirtyrate: make sample page count configurable

=== OUTPUT BEGIN ===
1/7 Checking commit dd4342c3ec2f (migration/dirtyrate: make sample page count configurable) 2/7 Checking commit 8828757e0b0e (hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds) 3/7 Checking commit 0ec6909b55f6 (KVM: introduce dirty_pages and kvm_dirty_ring_enabled) 4/7 Checking commit 04b16bd9b236 (migration/dirtyrate: add per-vcpu option for calc-dirty-rate)
ERROR: Error messages should not contain newlines
#42: FILE: migration/dirtyrate.c:427:
+ "only one of then can be specified!\n");

total: 1 errors, 0 warnings, 158 lines checked

Patch 4/7 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

5/7 Checking commit a3323fddecd9 (migration/dirtyrate: adjust struct DirtyRateStat) 6/7 Checking commit 720f923fec0d (memory: make global_dirty_log a bitmask)
ERROR: spaces required around that '<<' (ctx:VxV)
#36: FILE: include/exec/memory.h:58:
+#define GLOBAL_DIRTY_MIGRATION (1U<<0)

                                    ^

ERROR: spaces required around that '<<' (ctx:VxV)
#37: FILE: include/exec/memory.h:59:
+#define GLOBAL_DIRTY_DIRTY_RATE (1U<<1)

                                    ^

total: 2 errors, 0 warnings, 145 lines checked

Patch 6/7 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

7/7 Checking commit db255379e746 (migration/dirtyrate: implement dirty-ring dirtyrate calculation)
ERROR: do not initialise globals to 0 or NULL
#75: FILE: migration/dirtyrate.c:47:
+bool register_powerdown_callback = false;

WARNING: Block comments use a leading /* on a separate line #208: FILE: migration/dirtyrate.c:437:
+ /* migration is in process, do not stop dirty logging,

WARNING: Block comments use a trailing */ on a separate line #209: FILE: migration/dirtyrate.c:438:
+ * just clear the GLOBAL_DIRTY_DIRTY_RATE bit */

total: 1 errors, 2 warnings, 322 lines checked

Patch 7/7 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]