Skip to contentSkip to conversation

support dirty restraint on vCPU

PATCH Superseded [email protected] 3 participants 13 messages Mar 31, 2022 – Apr 27, 2022 Arm/Amlogic Meson SoC support

v22 of 25 Superseded Latest is v25 →

lore
Share & tools
lore
#0 lore raw

v22 cover support dirty restraint on vCPU

From: Hyman Huang(黄勇) <[email protected]>

This is v22 of dirtylimit series.
The following is the history of the patchset, since v22 kind of different from the original version, i made abstracts of changelog:

RFC and v1: https://lore.kernel.org/qemu-devel/[email protected]/
v2: https://lore.kernel.org/qemu-devel/[email protected]/
v1->v2 changelog:
- rename some function and variables. refactor the original algo of dirtylimit. Thanks for the comments given by Juan Quintela.
v3: https://lore.kernel.org/qemu-devel/[email protected]/
v4: https://lore.kernel.org/qemu-devel/[email protected]/
v5: https://lore.kernel.org/qemu-devel/[email protected]/
v6: https://lore.kernel.org/qemu-devel/[email protected]/
v7: https://lore.kernel.org/qemu-devel/[email protected]/
v2->v7 changelog:
- refactor the docs, annotation and fix bugs of the original algo of dirtylimit. Thanks for the review given by Markus Armbruster.
v8: https://lore.kernel.org/qemu-devel/[email protected]/
v9: https://lore.kernel.org/qemu-devel/[email protected]/
v10: https://lore.kernel.org/qemu-devel/[email protected]/
v7->v10 changelog:
- introduce a simpler but more efficient algo of dirtylimit inspired by Peter Xu.
- keep polishing the annotation suggested by Markus Armbruster.
v11: https://lore.kernel.org/qemu-devel/[email protected]/
v12: https://lore.kernel.org/qemu-devel/[email protected]/
v13: https://lore.kernel.org/qemu-devel/[email protected]/
v10->v13 changelog:
- handle the hotplug/unplug scenario.
- refactor the new algo, split the commit and make the code more clean.
v14: https://lore.kernel.org/qemu-devel/[email protected]/
v13->v14 changelog:
- sent by accident.
v15: https://lore.kernel.org/qemu-devel/[email protected]/
v16: https://lore.kernel.org/qemu-devel/[email protected]/
v17: https://lore.kernel.org/qemu-devel/[email protected]/
v14->v17 changelog:
- do some code clean and fix test bug reported by Dr. David Alan Gilbert.
v18: https://lore.kernel.org/qemu-devel/[email protected]/
v19: https://lore.kernel.org/qemu-devel/[email protected]/
v20: https://lore.kernel.org/qemu-devel/[email protected]/
v21: https://lore.kernel.org/qemu-devel/[email protected]/
v17->v21 changelog:
- add qtest, fix bug and do code clean.
v21->v22 changelog:
- move the vcpu dirty limit test into migration-test and do some modification suggested by Peter.

Please review.

Yong.

Abstract
========

This patchset introduce a mechanism to impose dirty restraint on vCPU, aiming to keep the vCPU running in a certain dirtyrate given by user. dirty restraint on vCPU maybe an alternative method to implement convergence logic for live migration, which could improve guest memory performance during migration compared with traditional method in theory.

For the current live migration implementation, the convergence logic throttles all vCPUs of the VM, which has some side effects. -'read processes' on vCPU will be unnecessarily penalized
- throttle increase percentage step by step, which seems struggling to find the optimal throttle percentage when dirtyrate is high.
- hard to predict the remaining time of migration if the throttling percentage reachs 99%

to a certain extent, the dirty restraint machnism can fix these effects by throttling at vCPU granularity during migration.

the implementation is rather straightforward, we calculate vCPU dirtyrate via the Dirty Ring mechanism periodically as the commit 0e21bf246 "implement dirty-ring dirtyrate calculation" does, for vCPU that be specified to impose dirty restraint, we throttle it periodically as the auto-converge does, once after throttling, we compare the quota dirtyrate with current dirtyrate, if current dirtyrate is not under the quota, increase the throttling percentage until current dirtyrate is under the quota.

this patchset is the basis of implmenting a new auto-converge method for live migration, we introduce two qmp commands for impose/cancel the dirty restraint on specified vCPU, so it also can be an independent api to supply the upper app such as libvirt, which can use it to implement the convergence logic during live migration, supplemented with the qmp 'calc-dirty-rate' command or whatever.

we post this patchset for RFC and any corrections and suggetions about the implementation, api, throttleing algorithm or whatever are very appreciated!

Please review, thanks !

Best Regards !

Hyman Huang (8):
accel/kvm/kvm-all: Refactor per-vcpu dirty ring reaping
cpus: Introduce cpu_list_generation_id migration/dirtyrate: Refactor dirty page rate calculation softmmu/dirtylimit: Implement vCPU dirtyrate calculation periodically accel/kvm/kvm-all: Introduce kvm_dirty_ring_size function softmmu/dirtylimit: Implement virtual CPU throttle softmmu/dirtylimit: Implement dirty page rate limit
tests: Add dirty page rate limit test

 accel/kvm/kvm-all.c             |  46 ++-
 accel/stubs/kvm-stub.c          |   6 +
 cpus-common.c                   |   8 +
 hmp-commands-info.hx            |  13 +
 hmp-commands.hx                 |  32 +++
 include/exec/cpu-common.h       |   1 +
 include/exec/memory.h           |   5 +-
 include/hw/core/cpu.h           |   6 +
 include/monitor/hmp.h           |   3 +
 include/sysemu/dirtylimit.h     |  37 +++
 include/sysemu/dirtyrate.h      |  28 ++
 include/sysemu/kvm.h            |   2 +
 migration/dirtyrate.c           | 227 +++++++++------
 migration/dirtyrate.h           |   7 +-
 qapi/migration.json             |  80 ++++++
 softmmu/dirtylimit.c            | 602 ++++++++++++++++++++++++++++++++++++++++
 softmmu/meson.build             |   1 +
 softmmu/trace-events            |   7 +
 tests/qtest/migration-helpers.c |  22 ++
 tests/qtest/migration-helpers.h |   2 +
 tests/qtest/migration-test.c    | 255 +++++++++++++++++
 tests/qtest/qmp-cmd-test.c      |   2 +
 22 files changed, 1293 insertions(+), 99 deletions(-)
 create mode 100644 include/sysemu/dirtylimit.h
 create mode 100644 include/sysemu/dirtyrate.h
 create mode 100644 softmmu/dirtylimit.c
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v22 1/8 accel/kvm/kvm-all: Refactor per-vcpu dirty ring reaping

From: Hyman Huang(黄勇) <[email protected]>

Add a non-required argument 'CPUState' to kvm_dirty_ring_reap so that it can cover single vcpu dirty-ring-reaping scenario.

---
 accel/kvm/kvm-all.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)
accel/kvm/kvm-all.c +13 -10
index 5f1377c..c30870ae 100644
@@ -756,17 +756,20 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu)
 }
 
 /* Must be with slots_lock held */
-static uint64_t kvm_dirty_ring_reap_locked(KVMState *s)
+static uint64_t kvm_dirty_ring_reap_locked(KVMState *s, CPUState* cpu)
 {
     int ret;
-    CPUState *cpu;
     uint64_t total = 0;
     int64_t stamp;
 
     stamp = get_clock();
 
-    CPU_FOREACH(cpu) {
-        total += kvm_dirty_ring_reap_one(s, cpu);
+    if (cpu) {
+        total = kvm_dirty_ring_reap_one(s, cpu);
+    } else {
+        CPU_FOREACH(cpu) {
+            total += kvm_dirty_ring_reap_one(s, cpu);
+        }
     }
 
     if (total) {
@@ -787,7 +790,7 @@ static uint64_t kvm_dirty_ring_reap_locked(KVMState *s)
  * Currently for simplicity, we must hold BQL before calling this.  We can
  * consider to drop the BQL if we're clear with all the race conditions.
  */
-static uint64_t kvm_dirty_ring_reap(KVMState *s)
+static uint64_t kvm_dirty_ring_reap(KVMState *s, CPUState *cpu)
 {
     uint64_t total;
 
@@ -807,7 +810,7 @@ static uint64_t kvm_dirty_ring_reap(KVMState *s)
      *     reset below.
      */
     kvm_slots_lock();
-    total = kvm_dirty_ring_reap_locked(s);
+    total = kvm_dirty_ring_reap_locked(s, cpu);
     kvm_slots_unlock();
 
     return total;
@@ -854,7 +857,7 @@ static void kvm_dirty_ring_flush(void)
      * vcpus out in a synchronous way.
      */
     kvm_cpu_synchronize_kick_all();
-    kvm_dirty_ring_reap(kvm_state);
+    kvm_dirty_ring_reap(kvm_state, NULL);
     trace_kvm_dirty_ring_flush(1);
 }
 
@@ -1398,7 +1401,7 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
                  * Not easy.  Let's cross the fingers until it's fixed.
                  */
                 if (kvm_state->kvm_dirty_ring_size) {
-                    kvm_dirty_ring_reap_locked(kvm_state);
+                    kvm_dirty_ring_reap_locked(kvm_state, NULL);
                 } else {
                     kvm_slot_get_dirty_log(kvm_state, mem);
                 }
@@ -1470,7 +1473,7 @@ static void *kvm_dirty_ring_reaper_thread(void *data)
         r->reaper_state = KVM_DIRTY_RING_REAPER_REAPING;
 
         qemu_mutex_lock_iothread();
-        kvm_dirty_ring_reap(s);
+        kvm_dirty_ring_reap(s, NULL);
         qemu_mutex_unlock_iothread();
 
         r->reaper_iteration++;
@@ -2957,7 +2960,7 @@ int kvm_cpu_exec(CPUState *cpu)
              */
             trace_kvm_dirty_ring_full(cpu->cpu_index);
             qemu_mutex_lock_iothread();
-            kvm_dirty_ring_reap(kvm_state);
+            kvm_dirty_ring_reap(kvm_state, NULL);
             qemu_mutex_unlock_iothread();
             ret = 0;
             break;
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH v22 3/8 migration/dirtyrate: Refactor dirty page rate calculation

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

abstract out dirty log change logic into function
global_dirty_log_change.

abstract out dirty page rate calculation logic via dirty-ring into function vcpu_calculate_dirtyrate.

abstract out mathematical dirty page rate calculation into do_calculate_dirtyrate, decouple it from DirtyStat.

rename set_sample_page_period to dirty_stat_wait, which is well-understood and will be reused in dirtylimit.

handle cpu hotplug/unplug scenario during measurement of dirty page rate.

export util functions outside migration.

---
 include/sysemu/dirtyrate.h |  28 ++++++
 migration/dirtyrate.c      | 227 ++++++++++++++++++++++++++++-----------------
 migration/dirtyrate.h      |   7 +-
 3 files changed, 174 insertions(+), 88 deletions(-)
 create mode 100644 include/sysemu/dirtyrate.h
include/sysemu/dirtyrate.h new file +28 -0
new file mode 100644
index 0000000..4d3b9a4
@@ -0,0 +1,28 @@
+/*
+ * dirty page rate helper functions
+ *
+ * Copyright (c) 2022 CHINA TELECOM CO.,LTD.
+ *
+ * Authors:
+ *  Hyman Huang(黄勇) <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_DIRTYRATE_H
+#define QEMU_DIRTYRATE_H
+
+typedef struct VcpuStat {
+    int nvcpu; /* number of vcpu */
+    DirtyRateVcpu *rates; /* array of dirty rate for each vcpu */
+} VcpuStat;
+
+int64_t vcpu_calculate_dirtyrate(int64_t calc_time_ms,
+                                 VcpuStat *stat,
+                                 unsigned int flag,
+                                 bool one_shot);
+
+void global_dirty_log_change(unsigned int flag,
+                             bool start);
+#endif
migration/dirtyrate.c +144 -83

290 lines not on this page · raw message

migration/dirtyrate.h +2 -5
index 69d4c5b..594a5c0 100644
@@ -13,6 +13,8 @@
 #ifndef QEMU_MIGRATION_DIRTYRATE_H
 #define QEMU_MIGRATION_DIRTYRATE_H
 
+#include "sysemu/dirtyrate.h"
+
 /*
  * Sample 512 pages per GB as default.
  */
@@ -65,11 +67,6 @@ typedef struct SampleVMStat {
     uint64_t total_block_mem_MB; /* size of total sampled pages in MB */
 } SampleVMStat;
 
-typedef struct VcpuStat {
-    int nvcpu; /* number of vcpu */
-    DirtyRateVcpu *rates; /* array of dirty rate for each vcpu */
-} VcpuStat;
-
 /*
  * Store calculation statistics for each measure.
  */
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v22 2/8 cpus: Introduce cpu_list_generation_id

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Introduce cpu_list_generation_id to track cpu list generation so that cpu hotplug/unplug can be detected during measurement of dirty page rate.

cpu_list_generation_id could be used to detect changes of cpu list, which is prepared for dirty page rate measurement.

---
 cpus-common.c             | 8 ++++++++
 include/exec/cpu-common.h | 1 +
 2 files changed, 9 insertions(+)
cpus-common.c +8 -0
index db459b4..793364d 100644
@@ -73,6 +73,12 @@ static int cpu_get_free_index(void)
 }
 
 CPUTailQ cpus = QTAILQ_HEAD_INITIALIZER(cpus);
+static unsigned int cpu_list_generation_id;
+
+unsigned int cpu_list_generation_id_get(void)
+{
+    return cpu_list_generation_id;
+}
 
 void cpu_list_add(CPUState *cpu)
 {
@@ -84,6 +90,7 @@ void cpu_list_add(CPUState *cpu)
         assert(!cpu_index_auto_assigned);
     }
     QTAILQ_INSERT_TAIL_RCU(&cpus, cpu, node);
+    cpu_list_generation_id++;
 }
 
 void cpu_list_remove(CPUState *cpu)
@@ -96,6 +103,7 @@ void cpu_list_remove(CPUState *cpu)
 
     QTAILQ_REMOVE_RCU(&cpus, cpu, node);
     cpu->cpu_index = UNASSIGNED_CPU_INDEX;
+    cpu_list_generation_id++;
 }
 
 CPUState *qemu_get_cpu(int index)
include/exec/cpu-common.h +1 -0
index 50a7d29..7cba376 100644
@@ -32,6 +32,7 @@ extern intptr_t qemu_host_page_mask;
 void qemu_init_cpu_list(void);
 void cpu_list_lock(void);
 void cpu_list_unlock(void);
+unsigned int cpu_list_generation_id_get(void);
 
 void tcg_flush_softmmu_tlb(CPUState *cs);
 
Signature · 1 line
1.8.3.1
#7 lore raw

PATCH v22 7/8 softmmu/dirtylimit: Implement dirty page rate limit

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Implement dirtyrate calculation periodically basing on dirty-ring and throttle virtual CPU until it reachs the quota dirty page rate given by user.

Introduce qmp commands "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit", "query-vcpu-dirty-limit" to enable, disable, query dirty page limit for virtual CPU.

Meanwhile, introduce corresponding hmp commands "set_vcpu_dirty_limit", "cancel_vcpu_dirty_limit", "info vcpu_dirty_limit" so the feature can be more usable.

"query-vcpu-dirty-limit" success depends on enabling dirty page rate limit, so just add it to the list of skipped command to ensure qmp-cmd-test run successfully.

---
 hmp-commands-info.hx       |  13 +++
 hmp-commands.hx            |  32 ++++++++
 include/monitor/hmp.h      |   3 +
 qapi/migration.json        |  80 +++++++++++++++++++
 softmmu/dirtylimit.c       | 195 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qtest/qmp-cmd-test.c |   2 +
 6 files changed, 325 insertions(+)
hmp-commands-info.hx +13 -0
index adfa085..016717d 100644
@@ -865,6 +865,19 @@ SRST
     Display the vcpu dirty rate information.
 ERST
 
+    {
+        .name       = "vcpu_dirty_limit",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show dirty page limit information of all vCPU",
+        .cmd        = hmp_info_vcpu_dirty_limit,
+    },
+
+SRST
+  ``info vcpu_dirty_limit``
+    Display the vcpu dirty page limit information.
+ERST
+
 #if defined(TARGET_I386)
     {
         .name       = "sgx",
hmp-commands.hx +32 -0
index 8476277..82ab75f 100644
@@ -1746,3 +1746,35 @@ ERST
                       "\n\t\t\t -b to specify dirty bitmap as method of calculation)",
         .cmd        = hmp_calc_dirty_rate,
     },
+
+SRST
+``set_vcpu_dirty_limit``
+  Set dirty page rate limit on virtual CPU, the information about all the
+  virtual CPU dirty limit status can be observed with ``info vcpu_dirty_limit``
+  command.
+ERST
+
+    {
+        .name       = "set_vcpu_dirty_limit",
+        .args_type  = "dirty_rate:l,cpu_index:l?",
+        .params     = "dirty_rate [cpu_index]",
+        .help       = "set dirty page rate limit, use cpu_index to set limit"
+                      "\n\t\t\t\t\t on a specified virtual cpu",
+        .cmd        = hmp_set_vcpu_dirty_limit,
+    },
+
+SRST
+``cancel_vcpu_dirty_limit``
+  Cancel dirty page rate limit on virtual CPU, the information about all the
+  virtual CPU dirty limit status can be observed with ``info vcpu_dirty_limit``
+  command.
+ERST
+
+    {
+        .name       = "cancel_vcpu_dirty_limit",
+        .args_type  = "cpu_index:l?",
+        .params     = "[cpu_index]",
+        .help       = "cancel dirty page rate limit, use cpu_index to cancel"
+                      "\n\t\t\t\t\t limit on a specified virtual cpu",
+        .cmd        = hmp_cancel_vcpu_dirty_limit,
+    },
include/monitor/hmp.h +3 -0
index 96d0148..478820e 100644
@@ -131,6 +131,9 @@ void hmp_replay_delete_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_seek(Monitor *mon, const QDict *qdict);
 void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict);
 void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict);
+void hmp_set_vcpu_dirty_limit(Monitor *mon, const QDict *qdict);
+void hmp_cancel_vcpu_dirty_limit(Monitor *mon, const QDict *qdict);
+void hmp_info_vcpu_dirty_limit(Monitor *mon, const QDict *qdict);
 void hmp_human_readable_text_helper(Monitor *mon,
                                     HumanReadableText *(*qmp_handler)(Error **));
 
qapi/migration.json +80 -0

86 lines not on this page · raw message

softmmu/dirtylimit.c +195 -0

206 lines not on this page · raw message

tests/qtest/qmp-cmd-test.c +2 -0
index 7f103ea..4b216a0 100644
@@ -110,6 +110,8 @@ static bool query_is_ignored(const char *cmd)
         "query-sev-capabilities",
         "query-sgx",
         "query-sgx-capabilities",
+        /* Success depends on enabling dirty page rate limit */
+        "query-vcpu-dirty-limit",
         NULL
     };
     int i;
Signature · 1 line
1.8.3.1
#5 lore raw

PATCH v22 5/8 accel/kvm/kvm-all: Introduce kvm_dirty_ring_size function

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Introduce kvm_dirty_ring_size util function to help calculate dirty ring ful time.

---
 accel/kvm/kvm-all.c    | 5 +++++
 accel/stubs/kvm-stub.c | 6 ++++++
 include/sysemu/kvm.h   | 2 ++
 3 files changed, 13 insertions(+)
accel/kvm/kvm-all.c +5 -0
index c30870ae..95deab9 100644
@@ -2313,6 +2313,11 @@ bool kvm_dirty_ring_enabled(void)
     return kvm_state->kvm_dirty_ring_size ? true : false;
 }
 
+uint32_t kvm_dirty_ring_size(void)
+{
+    return kvm_state->kvm_dirty_ring_size;
+}
+
 static int kvm_init(MachineState *ms)
 {
     MachineClass *mc = MACHINE_GET_CLASS(ms);
accel/stubs/kvm-stub.c +6 -0
index 3345882..c5aafaa 100644
@@ -148,3 +148,9 @@ bool kvm_dirty_ring_enabled(void)
 {
     return false;
 }
+
+uint32_t kvm_dirty_ring_size(void)
+{
+    return 0;
+}
+#endif
include/sysemu/kvm.h +2 -0
index a783c78..efd6dee 100644
@@ -582,4 +582,6 @@ bool kvm_cpu_check_are_resettable(void);
 bool kvm_arch_cpu_check_are_resettable(void);
 
 bool kvm_dirty_ring_enabled(void);
+
+uint32_t kvm_dirty_ring_size(void);
 #endif
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH v22 4/8 softmmu/dirtylimit: Implement vCPU dirtyrate calculation periodically

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Introduce the third method GLOBAL_DIRTY_LIMIT of dirty tracking for calculate dirtyrate periodly for dirty page rate limit.

Add dirtylimit.c to implement dirtyrate calculation periodly, which will be used for dirty page rate limit.

Add dirtylimit.h to export util functions for dirty page rate limit implementation.

---
 include/exec/memory.h       |   5 +-
 include/sysemu/dirtylimit.h |  22 +++++++++
 softmmu/dirtylimit.c        | 116 ++++++++++++++++++++++++++++++++++++++++++++
 softmmu/meson.build         |   1 +
 4 files changed, 143 insertions(+), 1 deletion(-)
 create mode 100644 include/sysemu/dirtylimit.h
 create mode 100644 softmmu/dirtylimit.c
include/exec/memory.h +4 -1
index 4d5997e..88ca510 100644
@@ -69,7 +69,10 @@ static inline void fuzz_dma_read_cb(size_t addr,
 /* Dirty tracking enabled because measuring dirty rate */
 #define GLOBAL_DIRTY_DIRTY_RATE (1U << 1)
 
-#define GLOBAL_DIRTY_MASK  (0x3)
+/* Dirty tracking enabled because dirty limit */
+#define GLOBAL_DIRTY_LIMIT      (1U << 2)
+
+#define GLOBAL_DIRTY_MASK  (0x7)
 
 extern unsigned int global_dirty_tracking;
 
include/sysemu/dirtylimit.h new file +22 -0
new file mode 100644
index 0000000..da459f0
@@ -0,0 +1,22 @@
+/*
+ * Dirty page rate limit common functions
+ *
+ * Copyright (c) 2022 CHINA TELECOM CO.,LTD.
+ *
+ * Authors:
+ *  Hyman Huang(黄勇) <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#ifndef QEMU_DIRTYRLIMIT_H
+#define QEMU_DIRTYRLIMIT_H
+
+#define DIRTYLIMIT_CALC_TIME_MS         1000    /* 1000ms */
+
+int64_t vcpu_dirty_rate_get(int cpu_index);
+void vcpu_dirty_rate_stat_start(void);
+void vcpu_dirty_rate_stat_stop(void);
+void vcpu_dirty_rate_stat_initialize(void);
+void vcpu_dirty_rate_stat_finalize(void);
+#endif
softmmu/dirtylimit.c new file +116 -0

116 lines not on this page · raw message

softmmu/meson.build +1 -0
index 8138248..3272af1 100644
@@ -4,6 +4,7 @@ specific_ss.add(when: 'CONFIG_SOFTMMU', if_true: [files(
   'memory.c',
   'physmem.c',
   'qtest.c',
+  'dirtylimit.c',
 )])
 
 specific_ss.add(when: ['CONFIG_SOFTMMU', 'CONFIG_TCG'], if_true: [files(
Signature · 1 line
1.8.3.1
#6 lore raw

PATCH v22 6/8 softmmu/dirtylimit: Implement virtual CPU throttle

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Setup a negative feedback system when vCPU thread handling KVM_EXIT_DIRTY_RING_FULL exit by introducing throttle_us_per_full field in struct CPUState. Sleep throttle_us_per_full microseconds to throttle vCPU if dirtylimit is in service.

---
 accel/kvm/kvm-all.c         |  20 ++-
 include/hw/core/cpu.h       |   6 +
 include/sysemu/dirtylimit.h |  15 +++
 softmmu/dirtylimit.c        | 291 ++++++++++++++++++++++++++++++++++++++++++++
 softmmu/trace-events        |   7 ++
 5 files changed, 338 insertions(+), 1 deletion(-)
accel/kvm/kvm-all.c +19 -1
index 95deab9..6cbeca0 100644
@@ -45,6 +45,7 @@
 #include "qemu/guest-random.h"
 #include "sysemu/hw_accel.h"
 #include "kvm-cpus.h"
+#include "sysemu/dirtylimit.h"
 
 #include "hw/boards.h"
 
@@ -476,6 +477,7 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
     cpu->kvm_state = s;
     cpu->vcpu_dirty = true;
     cpu->dirty_pages = 0;
+    cpu->throttle_us_per_full = 0;
 
     mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
     if (mmap_size < 0) {
@@ -1469,6 +1471,11 @@ static void *kvm_dirty_ring_reaper_thread(void *data)
          */
         sleep(1);
 
+        /* keep sleeping so that dirtylimit not be interfered by reaper */
+        if (dirtylimit_in_service()) {
+            continue;
+        }
+
         trace_kvm_dirty_ring_reaper("wakeup");
         r->reaper_state = KVM_DIRTY_RING_REAPER_REAPING;
 
@@ -2965,8 +2972,19 @@ int kvm_cpu_exec(CPUState *cpu)
              */
             trace_kvm_dirty_ring_full(cpu->cpu_index);
             qemu_mutex_lock_iothread();
-            kvm_dirty_ring_reap(kvm_state, NULL);
+            /*
+             * We throttle vCPU by making it sleep once it exit from kernel
+             * due to dirty ring full. In the dirtylimit scenario, reaping
+             * all vCPUs after a single vCPU dirty ring get full result in
+             * the miss of sleep, so just reap the ring-fulled vCPU.
+             */
+            if (dirtylimit_in_service()) {
+                kvm_dirty_ring_reap(kvm_state, cpu);
+            } else {
+                kvm_dirty_ring_reap(kvm_state, NULL);
+            }
             qemu_mutex_unlock_iothread();
+            dirtylimit_vcpu_execute(cpu);
             ret = 0;
             break;
         case KVM_EXIT_SYSTEM_EVENT:
include/hw/core/cpu.h +6 -0
index 1369736..2380001 100644
@@ -418,6 +418,12 @@ struct CPUState {
      */
     bool throttle_thread_scheduled;
 
+    /*
+     * Sleep throttle_us_per_full microseconds once dirty ring is full
+     * if dirty page rate limit is enabled.
+     */
+    int64_t throttle_us_per_full;
+
     bool ignore_memory_transaction_failures;
 
     /* Used for user-only emulation of prctl(PR_SET_UNALIGN). */
include/sysemu/dirtylimit.h +15 -0
index da459f0..8d2c1f3 100644
@@ -19,4 +19,19 @@ void vcpu_dirty_rate_stat_start(void);
 void vcpu_dirty_rate_stat_stop(void);
 void vcpu_dirty_rate_stat_initialize(void);
 void vcpu_dirty_rate_stat_finalize(void);
+
+void dirtylimit_state_lock(void);
+void dirtylimit_state_unlock(void);
+void dirtylimit_state_initialize(void);
+void dirtylimit_state_finalize(void);
+bool dirtylimit_in_service(void);
+bool dirtylimit_vcpu_index_valid(int cpu_index);
+void dirtylimit_process(void);
+void dirtylimit_change(bool start);
+void dirtylimit_set_vcpu(int cpu_index,
+                         uint64_t quota,
+                         bool enable);
+void dirtylimit_set_all(uint64_t quota,
+                        bool enable);
+void dirtylimit_vcpu_execute(CPUState *cpu);
 #endif
softmmu/dirtylimit.c +291 -0

321 lines not on this page · raw message

softmmu/trace-events +7 -0
index 9c88887..22606dc 100644
@@ -31,3 +31,10 @@ runstate_set(int current_state, const char *current_state_str, int new_state, co
 system_wakeup_request(int reason) "reason=%d"
 qemu_system_shutdown_request(int reason) "reason=%d"
 qemu_system_powerdown_request(void) ""
+
+#dirtylimit.c
+dirtylimit_state_initialize(int max_cpus) "dirtylimit state initialize: max cpus %d"
+dirtylimit_state_finalize(void)
+dirtylimit_throttle_pct(int cpu_index, uint64_t pct, int64_t time_us) "CPU[%d] throttle percent: %" PRIu64 ", throttle adjust time %"PRIi64 " us"
+dirtylimit_set_vcpu(int cpu_index, uint64_t quota) "CPU[%d] set dirty page rate limit %"PRIu64
+dirtylimit_vcpu_execute(int cpu_index, int64_t sleep_time_us) "CPU[%d] sleep %"PRIi64 " us"
Signature · 1 line
1.8.3.1
#8 lore raw

PATCH v22 8/8 tests: Add dirty page rate limit test

In reply to [email protected]

From: Hyman Huang(黄勇) <[email protected]>

Add dirty page rate limit test if kernel support dirty ring, create a standalone file to implement the test case.

The following qmp commands are covered by this test case:
"calc-dirty-rate", "query-dirty-rate", "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit" and "query-vcpu-dirty-limit".

---
 tests/qtest/migration-helpers.c |  22 ++++
 tests/qtest/migration-helpers.h |   2 +
 tests/qtest/migration-test.c    | 255 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 279 insertions(+)
tests/qtest/migration-helpers.c +22 -0
index 4ee2601..1e594f9 100644
@@ -76,6 +76,28 @@ QDict *wait_command(QTestState *who, const char *command, ...)
 }
 
 /*
+ * Execute the qmp command only
+ */
+QDict *qmp_command(QTestState *who, const char *command, ...)
+{
+    va_list ap;
+    QDict *resp, *ret;
+
+    va_start(ap, command);
+    resp = qtest_vqmp(who, command, ap);
+    va_end(ap);
+
+    g_assert(!qdict_haskey(resp, "error"));
+    g_assert(qdict_haskey(resp, "return"));
+
+    ret = qdict_get_qdict(resp, "return");
+    qobject_ref(ret);
+    qobject_unref(resp);
+
+    return ret;
+}
+
+/*
  * Send QMP command "migrate".
  * Arguments are built from @fmt... (formatted like
  * qobject_from_jsonf_nofail()) with "uri": @uri spliced in.
tests/qtest/migration-helpers.h +2 -0
index 555adaf..04939fa 100644
@@ -22,6 +22,8 @@ QDict *wait_command_fd(QTestState *who, int fd, const char *command, ...);
 G_GNUC_PRINTF(2, 3)
 QDict *wait_command(QTestState *who, const char *command, ...);
 
+QDict *qmp_command(QTestState *who, const char *command, ...);
+
 G_GNUC_PRINTF(3, 4)
 void migrate_qmp(QTestState *who, const char *uri, const char *fmt, ...);
 
tests/qtest/migration-test.c +255 -0

279 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Acked-by on v22 8/8
#9 lore raw

Re: [PATCH v22 8/8] tests: Add dirty page rate limit test

Quoted 10 lines · On Fri, Apr 01, 2022 at 01:49:24AM +0800, [email protected] wrote:

From: Hyman Huang(黄勇) <[email protected]>

Add dirty page rate limit test if kernel support dirty ring, create a standalone file to implement the test case.

The following qmp commands are covered by this test case:
"calc-dirty-rate", "query-dirty-rate", "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit" and "query-vcpu-dirty-limit".

Thanks,

Signature · 1 line
Peter Xu
author
#10 lore raw
In reply to [email protected]

Ping.
Hi, David and Peter, how do you think this patchset?
Is it suitable for queueing ? or is there still something need to be
done ?

Yong

Quoted 136 lines · 在 2022/4/1 1:49, [email protected] 写道:

From: Hyman Huang(黄勇) <[email protected]>

This is v22 of dirtylimit series.
The following is the history of the patchset, since v22 kind of different from the original version, i made abstracts of changelog:

RFC and v1: https://lore.kernel.org/qemu-devel/[email protected]/
v2: https://lore.kernel.org/qemu-devel/[email protected]/
v1->v2 changelog:
- rename some function and variables. refactor the original algo of dirtylimit. Thanks for the comments given by Juan Quintela.
v3: https://lore.kernel.org/qemu-devel/[email protected]/
v4: https://lore.kernel.org/qemu-devel/[email protected]/
v5: https://lore.kernel.org/qemu-devel/[email protected]/
v6: https://lore.kernel.org/qemu-devel/[email protected]/
v7: https://lore.kernel.org/qemu-devel/[email protected]/
v2->v7 changelog:
- refactor the docs, annotation and fix bugs of the original algo of dirtylimit. Thanks for the review given by Markus Armbruster.
v8: https://lore.kernel.org/qemu-devel/[email protected]/
v9: https://lore.kernel.org/qemu-devel/[email protected]/
v10: https://lore.kernel.org/qemu-devel/[email protected]/
v7->v10 changelog:
- introduce a simpler but more efficient algo of dirtylimit inspired by Peter Xu.
- keep polishing the annotation suggested by Markus Armbruster.
v11: https://lore.kernel.org/qemu-devel/[email protected]/
v12: https://lore.kernel.org/qemu-devel/[email protected]/
v13: https://lore.kernel.org/qemu-devel/[email protected]/
v10->v13 changelog:
- handle the hotplug/unplug scenario.
- refactor the new algo, split the commit and make the code more clean.
v14: https://lore.kernel.org/qemu-devel/[email protected]/
v13->v14 changelog:
- sent by accident.
v15: https://lore.kernel.org/qemu-devel/[email protected]/
v16: https://lore.kernel.org/qemu-devel/[email protected]/
v17: https://lore.kernel.org/qemu-devel/[email protected]/
v14->v17 changelog:
- do some code clean and fix test bug reported by Dr. David Alan Gilbert.
v18: https://lore.kernel.org/qemu-devel/[email protected]/
v19: https://lore.kernel.org/qemu-devel/[email protected]/
v20: https://lore.kernel.org/qemu-devel/[email protected]/
v21: https://lore.kernel.org/qemu-devel/[email protected]/
v17->v21 changelog:
- add qtest, fix bug and do code clean.
v21->v22 changelog:
- move the vcpu dirty limit test into migration-test and do some modification suggested by Peter.

Please review.

Yong.

Abstract
========

This patchset introduce a mechanism to impose dirty restraint on vCPU, aiming to keep the vCPU running in a certain dirtyrate given by user. dirty restraint on vCPU maybe an alternative method to implement convergence logic for live migration, which could improve guest memory performance during migration compared with traditional method in theory.

For the current live migration implementation, the convergence logic throttles all vCPUs of the VM, which has some side effects. -'read processes' on vCPU will be unnecessarily penalized
- throttle increase percentage step by step, which seems struggling to find the optimal throttle percentage when dirtyrate is high.
- hard to predict the remaining time of migration if the throttling percentage reachs 99%

to a certain extent, the dirty restraint machnism can fix these effects by throttling at vCPU granularity during migration.

the implementation is rather straightforward, we calculate vCPU dirtyrate via the Dirty Ring mechanism periodically as the commit 0e21bf246 "implement dirty-ring dirtyrate calculation" does, for vCPU that be specified to impose dirty restraint, we throttle it periodically as the auto-converge does, once after throttling, we compare the quota dirtyrate with current dirtyrate, if current dirtyrate is not under the quota, increase the throttling percentage until current dirtyrate is under the quota.

this patchset is the basis of implmenting a new auto-converge method for live migration, we introduce two qmp commands for impose/cancel the dirty restraint on specified vCPU, so it also can be an independent api to supply the upper app such as libvirt, which can use it to implement the convergence logic during live migration, supplemented with the qmp 'calc-dirty-rate' command or whatever.

we post this patchset for RFC and any corrections and suggetions about the implementation, api, throttleing algorithm or whatever are very appreciated!

Please review, thanks !

Best Regards !

Hyman Huang (8):
accel/kvm/kvm-all: Refactor per-vcpu dirty ring reaping
cpus: Introduce cpu_list_generation_id migration/dirtyrate: Refactor dirty page rate calculation softmmu/dirtylimit: Implement vCPU dirtyrate calculation periodically accel/kvm/kvm-all: Introduce kvm_dirty_ring_size function softmmu/dirtylimit: Implement virtual CPU throttle softmmu/dirtylimit: Implement dirty page rate limit
tests: Add dirty page rate limit test

  accel/kvm/kvm-all.c             |  46 ++-
  accel/stubs/kvm-stub.c          |   6 +
  cpus-common.c                   |   8 +
  hmp-commands-info.hx            |  13 +
  hmp-commands.hx                 |  32 +++
  include/exec/cpu-common.h       |   1 +
  include/exec/memory.h           |   5 +-
  include/hw/core/cpu.h           |   6 +
  include/monitor/hmp.h           |   3 +
  include/sysemu/dirtylimit.h     |  37 +++
  include/sysemu/dirtyrate.h      |  28 ++
  include/sysemu/kvm.h            |   2 +
  migration/dirtyrate.c           | 227 +++++++++------
  migration/dirtyrate.h           |   7 +-
  qapi/migration.json             |  80 ++++++
  softmmu/dirtylimit.c            | 602 ++++++++++++++++++++++++++++++++++++++++
  softmmu/meson.build             |   1 +
  softmmu/trace-events            |   7 +
  tests/qtest/migration-helpers.c |  22 ++
  tests/qtest/migration-helpers.h |   2 +
  tests/qtest/migration-test.c    | 255 +++++++++++++++++
  tests/qtest/qmp-cmd-test.c      |   2 +
  22 files changed, 1293 insertions(+), 99 deletions(-)

create mode 100644 include/sysemu/dirtylimit.h
create mode 100644 include/sysemu/dirtyrate.h
create mode 100644 softmmu/dirtylimit.c

#11 lore raw

Hi, Yong,

Quoted 3 lines · Hyman

Ping.
Hi, David and Peter, how do you think this patchset?
Is it suitable for queueing ? or is there still something need to be done ?

It keeps looking good to me in general, let's see whether the maintainers have any comments. Thanks,

Signature · 1 line
Peter Xu
#12 lore raw
Quoted 9 lines · Peter Xu

Hi, Yong,

Quoted 3 lines · Hyman

Ping.
Hi, David and Peter, how do you think this patchset?
Is it suitable for queueing ? or is there still something need to be done ?

It keeps looking good to me in general, let's see whether the maintainers have any comments. Thanks,

Whose review is still needed? Or is this ready?