support dirtyrate at the granualrity of vcpu
v7 cover support dirtyrate at the granualrity of vcpu
From: Hyman Huang(黄勇) <[email protected]>
v7
- fix the code style problem, sorry about that
v6:
- pick up commit "KVM: introduce dirty_pages and kvm_dirty_ring_enabled" which has been dropped in verison 5
v5:
- rename global_dirty_log to global_dirty_tracking on Peter's advice
- make global_dirty_tracking a bitmask:
1. add assert statement to ensure starting dirty tracking repeatly not allowed.
2. add assert statement to ensure dirty tracking cannot be stopped without having been started.
- protecting dirty rate stat info:
1. drop the mutext for protecting dirty rate introduced in version 4
2. change the code block in query_dirty_rate_info so that requirements of "safe racing" to the dirty rate stat can be meet
- make the helper function "record_dirtypages" inline and change the global var dirty_pages to local var
- free DirtyRateVcpuList in case of memory leak
please review, thanks a lot.
v4:
- make global_dirty_log a bitmask:
1. add comments about dirty log bitmask
2. use assert statement to check validity of flags
3. add trace to log bitmask changes
- introduce mode option to show what method calculation should be used, also, export mode option in the as last commmit
- split cleanup and init of dirty rate stat and move it in the main thread
- change the fields of DirtyPageRecord to uint64_t type so that we can calculation the increased dirty pages with the formula as Peter's advice: dirty pages = end_pages - start_pages
- introduce mutex to protect dirty rate stat info
- adjust order of registering thread
- drop the memory free callback
this version modify some code on Peter's advice, reference to:
https://lore.kernel.org/qemu-devel/YL5nNYXmrqMlXF3v@t490s/
thanks again.
v3:
- pick up "migration/dirtyrate: make sample page count configurable" to make patchset apply master correctly
v2:
- rebase to "migration/dirtyrate: make sample page count configurable"
- rename "vcpu" to "per_vcpu" to show the per-vcpu method
- squash patch 5/6 into a single one, squash patch 1/2 also
- pick up "hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds"
- make global_dirty_log a bitmask to make sure both migration and dirty could not intefer with each other
- add memory free callback to prevent memory leaking
the most different of v2 fron v1 is that we make the global_dirty_log a bitmask. the reason is dirty rate measurement may start or stop dirty logging during calculation. this conflict with migration because stop dirty log make migration leave dirty pages out then that'll be a problem.
make global_dirty_log a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty log aims for, migration or dirty rate.
all references to global_dirty_log should be untouched because any bit set there should justify that global dirty logging is enabled.
Please review, thanks !
v1:
Since the Dirty Ring on QEMU part has been merged recently, how to use this feature is under consideration.
In the scene of migration, it is valuable to provide a more accurante interface to track dirty memory than existing one, so that the upper layer application can make a wise decision, or whatever. More importantly,
dirtyrate info at the granualrity of vcpu could provide a possibility to make migration convergent by imposing restriction on vcpu. With Dirty Ring, we can calculate dirtyrate efficiently and cheaply.
The old interface implemented by sampling pages, it consumes cpu resource, and the larger guest memory size become, the more cpu resource it consumes, namely, hard to scale. New interface has no such drawback.
Please review, thanks !
Best Regards !
Hyman Huang(黄勇) (7):
KVM: introduce dirty_pages and kvm_dirty_ring_enabled
memory: rename global_dirty_log to global_dirty_tracking
memory: make global_dirty_tracking a bitmask migration/dirtyrate: introduce struct and adjust DirtyRateStat migration/dirtyrate: adjust order of registering thread migration/dirtyrate: move init step of calculation to main thread migration/dirtyrate: implement dirty-ring dirtyrate calculation
accel/kvm/kvm-all.c | 7 ++ hmp-commands.hx | 7 +- include/exec/memory.h | 18 +++- include/exec/ram_addr.h | 4 +- include/hw/core/cpu.h | 1 + include/sysemu/kvm.h | 1 + migration/dirtyrate.c | 267 ++++++++++++++++++++++++++++++++++++++++++------ migration/dirtyrate.h | 19 +++- migration/ram.c | 8 +- migration/trace-events | 2 + qapi/migration.json | 46 ++++++++- softmmu/memory.c | 36 +++++-- softmmu/trace-events | 1 + 13 files changed, 355 insertions(+), 62 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v7 1/7 KVM: introduce dirty_pages and kvm_dirty_ring_enabled
From: Hyman Huang(黄勇) <[email protected]>
dirty_pages is used to calculate dirtyrate via dirty ring, when enabled, kvm-reaper will increase the dirty pages after gfns being dirtied.
kvm_dirty_ring_enabled shows if kvm-reaper is working. dirtyrate thread could use it to check if measurement can base on dirty ring feature.
--- accel/kvm/kvm-all.c | 7 +++++++ include/hw/core/cpu.h | 1 + include/sysemu/kvm.h | 1 + 3 files changed, 9 insertions(+)
accel/kvm/kvm-all.c +7 -0
@@ -469,6 +469,7 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp) cpu->kvm_fd = ret; cpu->kvm_state = s; cpu->vcpu_dirty = true; + cpu->dirty_pages =0 ; mmap_size =kvm_ioctl (s,KVM_GET_VCPU_MMAP_SIZE ,0 );if (mmap_size <0 ) {
@@ -743,6 +744,7 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu) count++; } cpu->kvm_fetch_index = fetch; + cpu->dirty_pages += count;return count; }
@@ -2293,6 +2295,11 @@ bool kvm_vcpu_id_is_valid(int vcpu_id)return vcpu_id >=0 && vcpu_id <kvm_max_vcpu_id (s); } +bool kvm_dirty_ring_enabled (void ) +{ +return kvm_state->kvm_dirty_ring_size ? true : false; +} +static int kvm_init (MachineState *ms) {MachineClass *mc =MACHINE_GET_CLASS (ms);
include/hw/core/cpu.h +1 -0
@@ -374,6 +374,7 @@ struct CPUState {struct kvm_run *kvm_run;struct kvm_dirty_gfn *kvm_dirty_gfns;uint32_t kvm_fetch_index; +uint64_t dirty_pages;/* Used for events with 'vcpu' and *without* the 'disabled' properties */ DECLARE_BITMAP (trace_dstate_delayed,CPU_TRACE_DSTATE_MAX_EVENTS );
include/sysemu/kvm.h +1 -0
@@ -547,4 +547,5 @@ bool kvm_cpu_check_are_resettable(void);bool kvm_arch_cpu_check_are_resettable (void ); +bool kvm_dirty_ring_enabled (void );#endif
Signature · 1 line
1.8.3.1
PATCH v7 2/7 memory: rename global_dirty_log to global_dirty_tracking
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
since dirty ring has been introduced, there are two methods to track dirty pages of vm. it seems that "logging" has a hint on the method, so rename the global_dirty_log to global_dirty_tracking would make description more accurate.
--- include/exec/memory.h | 2 +- include/exec/ram_addr.h | 4 ++-- softmmu/memory.c | 10 +++++----- 3 files changed, 8 insertions(+), 8 deletions(-)
include/exec/memory.h +1 -1
@@ -55,7 +55,7 @@ static inline void fuzz_dma_read_cb(size_t addr, }#endif -extern bool global_dirty_log; +extern bool global_dirty_tracking;typedef struct MemoryRegionOps MemoryRegionOps ;
include/exec/ram_addr.h +2 -2
@@ -372,7 +372,7 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,qatomic_or (&blocks[DIRTY_MEMORY_VGA ][idx][offset], temp); -if (global_dirty_log) { +if (global_dirty_tracking) {qatomic_or ( &blocks[DIRTY_MEMORY_MIGRATION ][idx][offset], temp);
@@ -395,7 +395,7 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap, }else {uint8_t clients =tcg_enabled () ?DIRTY_CLIENTS_ALL :DIRTY_CLIENTS_NOCODE ; -if (!global_dirty_log) { +if (!global_dirty_tracking) { clients &= ~(1 <<DIRTY_MEMORY_MIGRATION ); }
softmmu/memory.c +5 -5
@@ -39,7 +39,7 @@static unsigned memory_region_transaction_depth;static bool memory_region_update_pending;static bool ioeventfd_update_pending; -bool global_dirty_log; +bool global_dirty_tracking; static QTAILQ_HEAD(, MemoryListener) memory_listeners =QTAILQ_HEAD_INITIALIZER (memory_listeners);
@@ -1813,7 +1813,7 @@ uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)uint8_t mask = mr->dirty_log_mask;RAMBlock *rb = mr->ram_block; -if (global_dirty_log && ((rb &&qemu_ram_is_migratable (rb)) || +if (global_dirty_tracking && ((rb &&qemu_ram_is_migratable (rb)) ||memory_region_is_iommu (mr))) { mask |= (1 <<DIRTY_MEMORY_MIGRATION ); }
@@ -2666,7 +2666,7 @@ void memory_global_dirty_log_start(void) vmstate_change =NULL ; } - global_dirty_log = true; + global_dirty_tracking = true;MEMORY_LISTENER_CALL_GLOBAL (log_global_start, Forward);
@@ -2678,7 +2678,7 @@ void memory_global_dirty_log_start(void)static void memory_global_dirty_log_do_stop (void ) { - global_dirty_log = false; + global_dirty_tracking = false;/* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin ();
@@ -2724,7 +2724,7 @@ static void listener_add_address_space(MemoryListener *listener,if (listener->begin) { listener->begin (listener); } -if (global_dirty_log) { +if (global_dirty_tracking) {if (listener->log_global_start) { listener->log_global_start (listener); }
Signature · 1 line
1.8.3.1
PATCH v7 3/7 memory: make global_dirty_tracking a bitmask
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
dirty rate measurement may start or stop dirty tracking during calculation. this conflict with migration because stop dirty tracking make migration leave dirty pages out then that'll be a problem.
make global_dirty_tracking a bitmask can let both migration and dirty rate measurement work fine. introduce GLOBAL_DIRTY_MIGRATION and GLOBAL_DIRTY_DIRTY_RATE to distinguish what current dirty tracking aims for, migration or dirty rate.
--- include/exec/memory.h | 18 +++++++++++++++--- migration/ram.c | 8 ++++---- softmmu/memory.c | 32 +++++++++++++++++++++++--------- softmmu/trace-events | 1 + 4 files changed, 43 insertions(+), 16 deletions(-)
include/exec/memory.h +15 -3
@@ -55,7 +55,15 @@ static inline void fuzz_dma_read_cb(size_t addr, }#endif -extern bool global_dirty_tracking; +/* Possible bits for global_dirty_log */ + +/* Dirty tracking enabled because migration is running */ +#define GLOBAL_DIRTY_MIGRATION (1U << 0) + +/* Dirty tracking enabled because measuring dirty rate */ +#define GLOBAL_DIRTY_DIRTY_RATE (1U << 1) + +extern unsigned int global_dirty_tracking;typedef struct MemoryRegionOps MemoryRegionOps ;
@@ -2099,13 +2107,17 @@ void memory_listener_unregister(MemoryListener *listener);/** * memory_global_dirty_log_start: begin dirty logging for all regions +* +* @flags: purpose of starting dirty log, migration or dirty rate */ -void memory_global_dirty_log_start (void ); +void memory_global_dirty_log_start (unsigned int flags);/** * memory_global_dirty_log_stop: end dirty logging for all regions +* +* @flags: purpose of stopping dirty log, migration or dirty rate */ -void memory_global_dirty_log_stop (void ); +void memory_global_dirty_log_stop (unsigned int flags);void mtree_info (bool flatview,bool dispatch_tree,bool owner,bool disabled);
migration/ram.c +4 -4
@@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque)/* caller have hold iothread lock or is in a bh, so there is * no writing race against the migration bitmap */ -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION ); }RAMBLOCK_FOREACH_NOT_IGNORED (block) {
@@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs)ram_list_init_bitmaps ();/* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot ()) { -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION );migration_bitmap_sync_precopy (rs); } }
@@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void)/* Discard this dirty bitmap record */ bitmap_zero (block->bmap, block->max_length >>TARGET_PAGE_BITS ); } -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION ); } ram_state->migration_dirty_pages =0 ;qemu_mutex_unlock_ramlist ();
@@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void) {RAMBlock *block; -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION );RAMBLOCK_FOREACH_NOT_IGNORED (block) {g_free (block->bmap); block->bmap =NULL ;
softmmu/memory.c +23 -9
@@ -39,7 +39,7 @@static unsigned memory_region_transaction_depth;static bool memory_region_update_pending;static bool ioeventfd_update_pending; -bool global_dirty_tracking; +unsigned int global_dirty_tracking; static QTAILQ_HEAD(, MemoryListener) memory_listeners =QTAILQ_HEAD_INITIALIZER (memory_listeners);
@@ -2659,14 +2659,19 @@ void memory_global_after_dirty_log_sync(void)static VMChangeStateEntry *vmstate_change; -void memory_global_dirty_log_start (void ) +void memory_global_dirty_log_start (unsigned int flags) {if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change); vmstate_change =NULL ; } - global_dirty_tracking = true; +#define GLOBAL_DIRTY_MASK (0x3) +assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert (!(global_dirty_tracking & flags)); + global_dirty_tracking |= flags; + +trace_global_dirty_changed (global_dirty_tracking);MEMORY_LISTENER_CALL_GLOBAL (log_global_start, Forward);
@@ -2676,9 +2681,13 @@ void memory_global_dirty_log_start(void)memory_region_transaction_commit (); } -static void memory_global_dirty_log_do_stop (void ) +static void memory_global_dirty_log_do_stop (unsigned int flags) { - global_dirty_tracking = false; +assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert ((global_dirty_tracking & flags) == flags); + global_dirty_tracking &= ~flags; + +trace_global_dirty_changed (global_dirty_tracking);/* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin ();
@@ -2691,8 +2700,10 @@ static void memory_global_dirty_log_do_stop(void)static void memory_vm_change_state_handler (void *opaque,bool running,RunState state) { +unsigned int *flags = opaque;if (running) { -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (*flags); +g_free (opaque);if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change);
@@ -2701,18 +2712,21 @@ static void memory_vm_change_state_handler(void *opaque, bool running, } } -void memory_global_dirty_log_stop (void ) +void memory_global_dirty_log_stop (unsigned int flags) { +unsigned int *opaque =NULL ;if (!runstate_is_running ()) {if (vmstate_change) {return ; } + opaque =g_malloc0 (sizeof (opaque)); + *opaque = flags; vmstate_change =qemu_add_vm_change_state_handler ( - memory_vm_change_state_handler,NULL ); + memory_vm_change_state_handler, opaque);return ; } -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (flags); } static void listener_add_address_space(MemoryListener *listener,
softmmu/trace-events +1 -0
@@ -18,6 +18,7 @@ memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t flatview_new(void *view, void *root) "%p (root %p)" flatview_destroy(void *view, void *root) "%p (root %p)" flatview_destroy_rcu(void *view, void *root) "%p (root %p)" +global_dirty_changed(unsigned int bitmask) "bitmask 0x%"PRIx32 # vl.c vm_state_notify(int running, int reason, const char *reason_str) "running %d reason %d (%s)"
Signature · 1 line
1.8.3.1
Re: [PATCH v7 3/7] memory: make global_dirty_tracking a bitmask
Quoted 118 lines · On Thu, Jun 17, 2021 at 10:12:04PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
dirty rate measurement may start or stop dirty tracking during calculation. this conflict with migration because stop dirty tracking make migration leave dirty pages out then that'll be a problem.
make
global_dirty_trackinga bitmask can let both migration and dirty rate measurement work fine. introduceGLOBAL_DIRTY_MIGRATIONandGLOBAL_DIRTY_DIRTY_RATEto distinguish what current dirty tracking aims for, migration or dirty rate.Signed-off-by Hyman Huang(黄勇) <[email protected]>--- include/exec/memory.h | 18 +++++++++++++++--- migration/ram.c | 8 ++++---- softmmu/memory.c | 32 +++++++++++++++++++++++--------- softmmu/trace-events | 1 + 4 files changed, 43 insertions(+), 16 deletions(-)include/exec/memory.h +15
-3@@ -55,7 +55,15 @@ static inline void fuzz_dma_read_cb(size_t addr, }#endif -extern bool global_dirty_tracking; +/* Possible bits for global_dirty_log */ + +/* Dirty tracking enabled because migration is running */ +#define GLOBAL_DIRTY_MIGRATION (1U << 0) + +/* Dirty tracking enabled because measuring dirty rate */ +#define GLOBAL_DIRTY_DIRTY_RATE (1U << 1) + +extern unsigned int global_dirty_tracking;typedef struct MemoryRegionOps MemoryRegionOps ;@@ -2099,13 +2107,17 @@ void memory_listener_unregister(MemoryListener *listener);/** * memory_global_dirty_log_start: begin dirty logging for all regions +* +* @flags: purpose of starting dirty log, migration or dirty rate */ -void memory_global_dirty_log_start (void ); +void memory_global_dirty_log_start (unsigned int flags);/** * memory_global_dirty_log_stop: end dirty logging for all regions +* +* @flags: purpose of stopping dirty log, migration or dirty rate */ -void memory_global_dirty_log_stop (void ); +void memory_global_dirty_log_stop (unsigned int flags);void mtree_info (bool flatview,bool dispatch_tree,bool owner,bool disabled);migration/ram.c +4
-4@@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque)/* caller have hold iothread lock or is in a bh, so there is * no writing race against the migration bitmap */ -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION ); }RAMBLOCK_FOREACH_NOT_IGNORED (block) {@@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs)ram_list_init_bitmaps ();/* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot ()) { -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION );migration_bitmap_sync_precopy (rs); } }@@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void)/* Discard this dirty bitmap record */ bitmap_zero (block->bmap, block->max_length >>TARGET_PAGE_BITS ); } -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION ); } ram_state->migration_dirty_pages =0 ;qemu_mutex_unlock_ramlist ();@@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void) {RAMBlock *block; -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION );RAMBLOCK_FOREACH_NOT_IGNORED (block) {g_free (block->bmap); block->bmap =NULL ;softmmu/memory.c +1
-1@@ -39,7 +39,7 @@static unsigned memory_region_transaction_depth;static bool memory_region_update_pending;static bool ioeventfd_update_pending; -bool global_dirty_tracking; +unsigned int global_dirty_tracking; static QTAILQ_HEAD(, MemoryListener) memory_listeners =QTAILQ_HEAD_INITIALIZER (memory_listeners);@@ -2659,14 +2659,19 @@ void memory_global_after_dirty_log_sync(void)static VMChangeStateEntry *vmstate_change;
softmmu/memory.c void memory_global_after_dirty_log_sync(void) · line 2669
Quoted 10 lines
softmmu/memory.c +2
-2-void memory_global_dirty_log_start (void ) +void memory_global_dirty_log_start (unsigned int flags) {if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change); vmstate_change =NULL ; } - global_dirty_tracking = true; +#define GLOBAL_DIRTY_MASK (0x3)
I should raised this earlier... but I think better move this macro to the defines of the bits.
Quoted 40 lines
softmmu/memory.c +14
-3+assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert (!(global_dirty_tracking & flags)); + global_dirty_tracking |= flags; + +trace_global_dirty_changed (global_dirty_tracking);MEMORY_LISTENER_CALL_GLOBAL (log_global_start, Forward);@@ -2676,9 +2681,13 @@ void memory_global_dirty_log_start(void)memory_region_transaction_commit (); } -static void memory_global_dirty_log_do_stop (void ) +static void memory_global_dirty_log_do_stop (unsigned int flags) { - global_dirty_tracking = false; +assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert ((global_dirty_tracking & flags) == flags); + global_dirty_tracking &= ~flags; + +trace_global_dirty_changed (global_dirty_tracking);/* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin ();@@ -2691,8 +2700,10 @@ static void memory_global_dirty_log_do_stop(void)static void memory_vm_change_state_handler (void *opaque,bool running,RunState state) { +unsigned int *flags = opaque;if (running) { -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (*flags); +g_free (opaque);if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change);@@ -2701,18 +2712,21 @@ static void memory_vm_change_state_handler(void *opaque, bool running, } }
softmmu/memory.c static void memory_vm_change_state_handler(void *opaque, bool running, · line 2723
Quoted 10 lines
softmmu/memory.c +4
-1-void memory_global_dirty_log_stop (void ) +void memory_global_dirty_log_stop (unsigned int flags) { +unsigned int *opaque =NULL ;if (!runstate_is_running ()) {if (vmstate_change) {return ; } + opaque =g_malloc0 (sizeof (opaque)); + *opaque = flags;
Here the flags can be directly casted into a "void *" so we could avoid malloc/free. You call..
I also still think it's easier to squash previous renaming patch into this one.
Thanks,
Quoted 9 lines
softmmu/memory.c +2
-2vmstate_change =qemu_add_vm_change_state_handler ( - memory_vm_change_state_handler,NULL ); + memory_vm_change_state_handler, opaque);return ; } -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (flags); }
Signature · 1 line
Peter Xu
Re: [PATCH v7 3/7] memory: make global_dirty_tracking a bitmask
Quoted 127 lines · Peter Xu
Quoted 126 lines · On Thu, Jun 17, 2021 at 10:12:04PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
dirty rate measurement may start or stop dirty tracking during calculation. this conflict with migration because stop dirty tracking make migration leave dirty pages out then that'll be a problem.
make
global_dirty_trackinga bitmask can let both migration and dirty rate measurement work fine. introduceGLOBAL_DIRTY_MIGRATIONandGLOBAL_DIRTY_DIRTY_RATEto distinguish what current dirty tracking aims for, migration or dirty rate.Signed-off-by Hyman Huang(黄勇) <[email protected]>--- include/exec/memory.h | 18 +++++++++++++++--- migration/ram.c | 8 ++++---- softmmu/memory.c | 32 +++++++++++++++++++++++--------- softmmu/trace-events | 1 + 4 files changed, 43 insertions(+), 16 deletions(-)include/exec/memory.h +15
-3@@ -55,7 +55,15 @@ static inline void fuzz_dma_read_cb(size_t addr, }#endif -extern bool global_dirty_tracking; +/* Possible bits for global_dirty_log */ + +/* Dirty tracking enabled because migration is running */ +#define GLOBAL_DIRTY_MIGRATION (1U << 0) + +/* Dirty tracking enabled because measuring dirty rate */ +#define GLOBAL_DIRTY_DIRTY_RATE (1U << 1) + +extern unsigned int global_dirty_tracking;typedef struct MemoryRegionOps MemoryRegionOps ;@@ -2099,13 +2107,17 @@ void memory_listener_unregister(MemoryListener *listener);/** * memory_global_dirty_log_start: begin dirty logging for all regions +* +* @flags: purpose of starting dirty log, migration or dirty rate */ -void memory_global_dirty_log_start (void ); +void memory_global_dirty_log_start (unsigned int flags);/** * memory_global_dirty_log_stop: end dirty logging for all regions +* +* @flags: purpose of stopping dirty log, migration or dirty rate */ -void memory_global_dirty_log_stop (void ); +void memory_global_dirty_log_stop (unsigned int flags);void mtree_info (bool flatview,bool dispatch_tree,bool owner,bool disabled);migration/ram.c +4
-4@@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque)/* caller have hold iothread lock or is in a bh, so there is * no writing race against the migration bitmap */ -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION ); }RAMBLOCK_FOREACH_NOT_IGNORED (block) {@@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs)ram_list_init_bitmaps ();/* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot ()) { -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION );migration_bitmap_sync_precopy (rs); } }@@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void)/* Discard this dirty bitmap record */ bitmap_zero (block->bmap, block->max_length >>TARGET_PAGE_BITS ); } -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION ); } ram_state->migration_dirty_pages =0 ;qemu_mutex_unlock_ramlist ();@@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void) {RAMBlock *block; -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION );RAMBLOCK_FOREACH_NOT_IGNORED (block) {g_free (block->bmap); block->bmap =NULL ;softmmu/memory.c +2
-2@@ -39,7 +39,7 @@static unsigned memory_region_transaction_depth;static bool memory_region_update_pending;static bool ioeventfd_update_pending; -bool global_dirty_tracking; +unsigned int global_dirty_tracking; static QTAILQ_HEAD(, MemoryListener) memory_listeners =QTAILQ_HEAD_INITIALIZER (memory_listeners);@@ -2659,14 +2659,19 @@ void memory_global_after_dirty_log_sync(void)static VMChangeStateEntry *vmstate_change; -void memory_global_dirty_log_start (void ) +void memory_global_dirty_log_start (unsigned int flags) {if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change); vmstate_change =NULL ; }
Quoted 6 lines
softmmu/memory.c void memory_global_after_dirty_log_sync(void) · line 2669
Quoted 2 lines
softmmu/memory.c +1
-1- global_dirty_tracking = true; +#define GLOBAL_DIRTY_MASK (0x3)I should raised this earlier... but I think better move this macro to the defines of the bits.
OK
Quoted 54 lines
Quoted 40 lines
softmmu/memory.c +14
-3+assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert (!(global_dirty_tracking & flags)); + global_dirty_tracking |= flags; + +trace_global_dirty_changed (global_dirty_tracking);MEMORY_LISTENER_CALL_GLOBAL (log_global_start, Forward);@@ -2676,9 +2681,13 @@ void memory_global_dirty_log_start(void)memory_region_transaction_commit (); } -static void memory_global_dirty_log_do_stop (void ) +static void memory_global_dirty_log_do_stop (unsigned int flags) { - global_dirty_tracking = false; +assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert ((global_dirty_tracking & flags) == flags); + global_dirty_tracking &= ~flags; + +trace_global_dirty_changed (global_dirty_tracking);/* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin ();@@ -2691,8 +2700,10 @@ static void memory_global_dirty_log_do_stop(void)static void memory_vm_change_state_handler (void *opaque,bool running,RunState state) { +unsigned int *flags = opaque;if (running) { -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (*flags); +g_free (opaque);if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change);@@ -2701,18 +2712,21 @@ static void memory_vm_change_state_handler(void *opaque, bool running, } }softmmu/memory.c static void memory_vm_change_state_handler(void *opaque, bool running, · line 2723
Quoted 10 lines
softmmu/memory.c +4
-1-void memory_global_dirty_log_stop (void ) +void memory_global_dirty_log_stop (unsigned int flags) { +unsigned int *opaque =NULL ;if (!runstate_is_running ()) {if (vmstate_change) {return ; } + opaque =g_malloc0 (sizeof (opaque)); + *opaque = flags;Here the flags can be directly casted into a "void *" so we could avoid malloc/free. You call..
Quoted 2 lines
I also still think it's easier to squash previous renaming patch into this one.
OK
Quoted 12 lines
Thanks,
Quoted 9 lines
softmmu/memory.c +2
-2vmstate_change =qemu_add_vm_change_state_handler ( - memory_vm_change_state_handler,NULL ); + memory_vm_change_state_handler, opaque);return ; } -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (flags); }
Signature · 3 lines
Best regard Hyman Huang(黄勇)
Re: [PATCH v7 3/7] memory: make global_dirty_tracking a bitmask
In reply to Peter XuQuoted 173 lines · Peter Xu
Quoted 118 lines · On Thu, Jun 17, 2021 at 10:12:04PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
dirty rate measurement may start or stop dirty tracking during calculation. this conflict with migration because stop dirty tracking make migration leave dirty pages out then that'll be a problem.
make
global_dirty_trackinga bitmask can let both migration and dirty rate measurement work fine. introduceGLOBAL_DIRTY_MIGRATIONandGLOBAL_DIRTY_DIRTY_RATEto distinguish what current dirty tracking aims for, migration or dirty rate.Signed-off-by Hyman Huang(黄勇) <[email protected]>--- include/exec/memory.h | 18 +++++++++++++++--- migration/ram.c | 8 ++++---- softmmu/memory.c | 32 +++++++++++++++++++++++--------- softmmu/trace-events | 1 + 4 files changed, 43 insertions(+), 16 deletions(-)include/exec/memory.h +15
-3@@ -55,7 +55,15 @@ static inline void fuzz_dma_read_cb(size_t addr, }#endif -extern bool global_dirty_tracking; +/* Possible bits for global_dirty_log */ + +/* Dirty tracking enabled because migration is running */ +#define GLOBAL_DIRTY_MIGRATION (1U << 0) + +/* Dirty tracking enabled because measuring dirty rate */ +#define GLOBAL_DIRTY_DIRTY_RATE (1U << 1) + +extern unsigned int global_dirty_tracking;typedef struct MemoryRegionOps MemoryRegionOps ;@@ -2099,13 +2107,17 @@ void memory_listener_unregister(MemoryListener *listener);/** * memory_global_dirty_log_start: begin dirty logging for all regions +* +* @flags: purpose of starting dirty log, migration or dirty rate */ -void memory_global_dirty_log_start (void ); +void memory_global_dirty_log_start (unsigned int flags);/** * memory_global_dirty_log_stop: end dirty logging for all regions +* +* @flags: purpose of stopping dirty log, migration or dirty rate */ -void memory_global_dirty_log_stop (void ); +void memory_global_dirty_log_stop (unsigned int flags);void mtree_info (bool flatview,bool dispatch_tree,bool owner,bool disabled);migration/ram.c +4
-4@@ -2190,7 +2190,7 @@ static void ram_save_cleanup(void *opaque)/* caller have hold iothread lock or is in a bh, so there is * no writing race against the migration bitmap */ -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION ); }RAMBLOCK_FOREACH_NOT_IGNORED (block) {@@ -2652,7 +2652,7 @@ static void ram_init_bitmaps(RAMState *rs)ram_list_init_bitmaps ();/* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot ()) { -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION );migration_bitmap_sync_precopy (rs); } }@@ -3393,7 +3393,7 @@ void colo_incoming_start_dirty_log(void)/* Discard this dirty bitmap record */ bitmap_zero (block->bmap, block->max_length >>TARGET_PAGE_BITS ); } -memory_global_dirty_log_start (); +memory_global_dirty_log_start (GLOBAL_DIRTY_MIGRATION ); } ram_state->migration_dirty_pages =0 ;qemu_mutex_unlock_ramlist ();@@ -3405,7 +3405,7 @@ void colo_release_ram_cache(void) {RAMBlock *block; -memory_global_dirty_log_stop (); +memory_global_dirty_log_stop (GLOBAL_DIRTY_MIGRATION );RAMBLOCK_FOREACH_NOT_IGNORED (block) {g_free (block->bmap); block->bmap =NULL ;softmmu/memory.c +1
-1@@ -39,7 +39,7 @@static unsigned memory_region_transaction_depth;static bool memory_region_update_pending;static bool ioeventfd_update_pending; -bool global_dirty_tracking; +unsigned int global_dirty_tracking; static QTAILQ_HEAD(, MemoryListener) memory_listeners =QTAILQ_HEAD_INITIALIZER (memory_listeners);@@ -2659,14 +2659,19 @@ void memory_global_after_dirty_log_sync(void)static VMChangeStateEntry *vmstate_change;softmmu/memory.c void memory_global_after_dirty_log_sync(void) · line 2669
Quoted 10 lines
softmmu/memory.c +2
-2-void memory_global_dirty_log_start (void ) +void memory_global_dirty_log_start (unsigned int flags) {if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change); vmstate_change =NULL ; } - global_dirty_tracking = true; +#define GLOBAL_DIRTY_MASK (0x3)I should raised this earlier... but I think better move this macro to the defines of the bits.
Quoted 40 lines
softmmu/memory.c +14
-3+assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert (!(global_dirty_tracking & flags)); + global_dirty_tracking |= flags; + +trace_global_dirty_changed (global_dirty_tracking);MEMORY_LISTENER_CALL_GLOBAL (log_global_start, Forward);@@ -2676,9 +2681,13 @@ void memory_global_dirty_log_start(void)memory_region_transaction_commit (); } -static void memory_global_dirty_log_do_stop (void ) +static void memory_global_dirty_log_do_stop (unsigned int flags) { - global_dirty_tracking = false; +assert (flags && !(flags & (~GLOBAL_DIRTY_MASK ))); +assert ((global_dirty_tracking & flags) == flags); + global_dirty_tracking &= ~flags; + +trace_global_dirty_changed (global_dirty_tracking);/* Refresh DIRTY_MEMORY_MIGRATION bit. */ memory_region_transaction_begin ();@@ -2691,8 +2700,10 @@ static void memory_global_dirty_log_do_stop(void)static void memory_vm_change_state_handler (void *opaque,bool running,RunState state) { +unsigned int *flags = opaque;if (running) { -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (*flags); +g_free (opaque);if (vmstate_change) {qemu_del_vm_change_state_handler (vmstate_change);@@ -2701,18 +2712,21 @@ static void memory_vm_change_state_handler(void *opaque, bool running, } }
Quoted 14 lines
softmmu/memory.c static void memory_vm_change_state_handler(void *opaque, bool running, · line 2723
Quoted 10 lines
softmmu/memory.c +4
-1-void memory_global_dirty_log_stop (void ) +void memory_global_dirty_log_stop (unsigned int flags) { +unsigned int *opaque =NULL ;if (!runstate_is_running ()) {if (vmstate_change) {return ; } + opaque =g_malloc0 (sizeof (opaque)); + *opaque = flags;Here the flags can be directly casted into a "void *" so we could avoid malloc/free. You call..
yeah, get it! think it's kind of werid but havn't figured out.
Quoted 14 lines
I also still think it's easier to squash previous renaming patch into this one.
Thanks,
Quoted 9 lines
softmmu/memory.c +2
-2vmstate_change =qemu_add_vm_change_state_handler ( - memory_vm_change_state_handler,NULL ); + memory_vm_change_state_handler, opaque);return ; } -memory_global_dirty_log_do_stop (); +memory_global_dirty_log_do_stop (flags); }
Signature · 3 lines
Best regard Hyman Huang(黄勇)
PATCH v7 4/7 migration/dirtyrate: introduce struct and adjust DirtyRateStat
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
introduce "DirtyRateMeasureMode" to specify what method should be used to calculate dirty rate, introduce "DirtyRateVcpu" to store dirty rate fore each vcpu.
use union to store stat data of specific mode
--- migration/dirtyrate.c | 48 ++++++++++++++++++++++++++++-------------------- migration/dirtyrate.h | 19 ++++++++++++++++--- qapi/migration.json | 30 ++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 23 deletions(-)
migration/dirtyrate.c +28 -20
@@ -88,33 +88,44 @@ static struct DirtyRateInfo *query_dirty_rate_info(void)return info; } -static void init_dirtyrate_stat (int64_t start_time,int64_t calc_time, -uint64_t sample_pages) +static void init_dirtyrate_stat (int64_t start_time, +struct DirtyRateConfig config) { - DirtyStat.total_dirty_samples =0 ; - DirtyStat.total_sample_count =0 ; - DirtyStat.total_block_mem_MB =0 ; DirtyStat.dirty_rate =-1 ; DirtyStat.start_time = start_time; - DirtyStat.calc_time = calc_time; - DirtyStat.sample_pages = sample_pages; + DirtyStat.calc_time = config.sample_period_seconds; + DirtyStat.sample_pages = config.sample_pages_per_gigabytes; + +switch (config.mode) { +case DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING : + DirtyStat.page_sampling.total_dirty_samples =0 ; + DirtyStat.page_sampling.total_sample_count =0 ; + DirtyStat.page_sampling.total_block_mem_MB =0 ; +break ; +case DIRTY_RATE_MEASURE_MODE_DIRTY_RING : + DirtyStat.dirty_ring.nvcpu =-1 ; + DirtyStat.dirty_ring.rates =NULL ; +break ; +default : +break ; + } }static void update_dirtyrate_stat (struct RamblockDirtyInfo *info) { - DirtyStat.total_dirty_samples += info->sample_dirty_count; - DirtyStat.total_sample_count += info->sample_pages_count; + DirtyStat.page_sampling.total_dirty_samples += info->sample_dirty_count; + DirtyStat.page_sampling.total_sample_count += info->sample_pages_count;/* size of total pages in MB */ - DirtyStat.total_block_mem_MB += (info->ramblock_pages * -TARGET_PAGE_SIZE ) >>20 ; + DirtyStat.page_sampling.total_block_mem_MB += (info->ramblock_pages * +TARGET_PAGE_SIZE ) >>20 ; }static void update_dirtyrate (uint64_t msec) {uint64_t dirtyrate; -uint64_t total_dirty_samples = DirtyStat.total_dirty_samples; -uint64_t total_sample_count = DirtyStat.total_sample_count; -uint64_t total_block_mem_MB = DirtyStat.total_block_mem_MB; +uint64_t total_dirty_samples = DirtyStat.page_sampling.total_dirty_samples; +uint64_t total_sample_count = DirtyStat.page_sampling.total_sample_count; +uint64_t total_block_mem_MB = DirtyStat.page_sampling.total_block_mem_MB; dirtyrate = total_dirty_samples * total_block_mem_MB *1000 / (total_sample_count * msec);
@@ -327,7 +338,7 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info,update_dirtyrate_stat (block_dinfo); } -if (DirtyStat.total_sample_count ==0 ) { +if (DirtyStat.page_sampling.total_sample_count ==0 ) {return false; }
@@ -372,8 +383,6 @@ void *get_dirtyrate_thread(void *arg)struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;int ret;int64_t start_time; -int64_t calc_time; -uint64_t sample_pages; ret =dirtyrate_set_state (&CalculatingState,DIRTY_RATE_STATUS_UNSTARTED ,DIRTY_RATE_STATUS_MEASURING );
@@ -383,9 +392,7 @@ void *get_dirtyrate_thread(void *arg) } start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; - calc_time = config.sample_period_seconds; - sample_pages = config.sample_pages_per_gigabytes; -init_dirtyrate_stat (start_time, calc_time, sample_pages); +init_dirtyrate_stat (start_time, config);calculate_dirtyrate (config);
@@ -442,6 +449,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, config.sample_period_seconds = calc_time; config.sample_pages_per_gigabytes = sample_pages; + config.mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ;qemu_thread_create (&thread,"get_dirtyrate" , get_dirtyrate_thread, (void *)&config,QEMU_THREAD_DETACHED ); }
migration/dirtyrate.h +16 -3
@@ -43,6 +43,7 @@struct DirtyRateConfig {uint64_t sample_pages_per_gigabytes;/* sample pages per GB */ int64_t sample_period_seconds;/* time duration between two sampling */ +DirtyRateMeasureMode mode;/* mode of dirtyrate measurement */ };/*
@@ -58,17 +59,29 @@ struct RamblockDirtyInfo {uint32_t *hash_result;/* array of hash result for sampled pages */ }; +typedef struct SampleVMStat { +uint64_t total_dirty_samples;/* total dirty sampled page */ +uint64_t total_sample_count;/* total sampled pages */ +uint64_t total_block_mem_MB;/* size of total sampled pages in MB */ +}SampleVMStat ; + +typedef struct VcpuStat { +int nvcpu;/* number of vcpu */ +DirtyRateVcpu *rates;/* array of dirty rate for each vcpu */ +}VcpuStat ; +/* * Store calculation statistics for each measure. */ struct DirtyRateStat { -uint64_t total_dirty_samples;/* total dirty sampled page */ -uint64_t total_sample_count;/* total sampled pages */ -uint64_t total_block_mem_MB;/* size of total sampled pages in MB */ int64_t dirty_rate;/* dirty rate in MB/s */ int64_t start_time;/* calculation start time in units of second */ int64_t calc_time;/* time duration of two sampling in units of second */ uint64_t sample_pages;/* sample pages per GB */ +union { +SampleVMStat page_sampling; +VcpuStat dirty_ring; + }; };void *get_dirtyrate_thread (void *arg);
qapi/migration.json +30 -0
@@ -1709,6 +1709,21 @@ 'data': { 'device-id': 'str' } } ## +# @DirtyRateVcpu: +# +# Dirty rate of vcpu. +# +# @id: vcpu index. +# +# @dirty-rate: dirty rate. +# +# Since: 6.1 +# +## +{ 'struct': 'DirtyRateVcpu', + 'data': { 'id': 'int', 'dirty-rate': 'int64' } } + +## # @DirtyRateStatus: # # An enumeration of dirtyrate status.
@@ -1726,6 +1741,21 @@ 'data': [ 'unstarted', 'measuring', 'measured'] } ## +# @DirtyRateMeasureMode: +# +# An enumeration of mode of measuring dirtyrate. +# +# @page-sampling: calculate dirtyrate by sampling pages. +# +# @dirty-ring: calculate dirtyrate by via dirty ring. +# +# Since: 6.1 +# +## +{ 'enum': 'DirtyRateMeasureMode', + 'data': [ 'none', 'page-sampling', 'dirty-ring'] } + +## # @DirtyRateInfo: # # Information about current dirty page rate of vm.
Signature · 1 line
1.8.3.1
Re: [PATCH v7 4/7] migration/dirtyrate: introduce struct and adjust DirtyRateStat
Quoted 4 lines · On Thu, Jun 17, 2021 at 10:12:05PM +0800, [email protected] wrote:
##
qapi/migration.json +3
-0+# @DirtyRateMeasureMode: +# +# An enumeration of mode of measuring dirtyrate.
qapi/migration.json line 1756
Quoted 10 lines
qapi/migration.json +10
-0+# +# @page-sampling: calculate dirtyrate by sampling pages. +# +# @dirty-ring: calculate dirtyrate by via dirty ring. +# +# Since: 6.1 +# +## +{ 'enum': 'DirtyRateMeasureMode', + 'data': [ 'none', 'page-sampling', 'dirty-ring'] }
I still don't get why we need "none" mode. Could you explain?
Signature · 1 line
Peter Xu
Re: [PATCH v7 4/7] migration/dirtyrate: introduce struct and adjust DirtyRateStat
Quoted 18 lines · Peter Xu
Quoted 4 lines · On Thu, Jun 17, 2021 at 10:12:05PM +0800, [email protected] wrote:
##
qapi/migration.json +3
-0+# @DirtyRateMeasureMode: +# +# An enumeration of mode of measuring dirtyrate.qapi/migration.json line 1756
Quoted 10 lines
qapi/migration.json +10
-0+# +# @page-sampling: calculate dirtyrate by sampling pages. +# +# @dirty-ring: calculate dirtyrate by via dirty ring. +# +# Since: 6.1 +# +## +{ 'enum': 'DirtyRateMeasureMode', + 'data': [ 'none', 'page-sampling', 'dirty-ring'] }I still don't get why we need "none" mode. Could you explain?
sorry, forget about this modification
Signature · 3 lines
Best regard Hyman Huang(黄勇)
PATCH v7 5/7 migration/dirtyrate: adjust order of registering thread
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
registering get_dirtyrate thread in advance so that both page-sampling and dirty-ring mode can be covered.
--- migration/dirtyrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
migration/dirtyrate.c +3 -2
@@ -352,7 +352,6 @@ static void calculate_dirtyrate(struct DirtyRateConfig config)int64_t msec =0 ;int64_t initial_time; -rcu_register_thread ();rcu_read_lock (); initial_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );if (!record_ramblock_hash_info (&block_dinfo, config, &block_count)) {
@@ -375,7 +374,6 @@ static void calculate_dirtyrate(struct DirtyRateConfig config) out:rcu_read_unlock ();free_ramblock_dirty_info (block_dinfo, block_count); -rcu_unregister_thread (); }void *get_dirtyrate_thread (void *arg)
@@ -383,6 +381,7 @@ void *get_dirtyrate_thread(void *arg)struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;int ret;int64_t start_time; +rcu_register_thread (); ret =dirtyrate_set_state (&CalculatingState,DIRTY_RATE_STATUS_UNSTARTED ,DIRTY_RATE_STATUS_MEASURING );
@@ -401,6 +400,8 @@ void *get_dirtyrate_thread(void *arg)if (ret ==-1 ) {error_report ("change dirtyrate state failed." ); } + +rcu_unregister_thread ();return NULL ; }
Signature · 1 line
1.8.3.1
PATCH v7 6/7 migration/dirtyrate: move init step of calculation to main thread
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
since main thread may "query dirty rate" at any time, it's better to move init step into main thead so that synchronization overhead between "main" and "get_dirtyrate" can be reduced.
--- migration/dirtyrate.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
migration/dirtyrate.c +19 -4
@@ -26,6 +26,7 @@static int CalculatingState =DIRTY_RATE_STATUS_UNSTARTED ;static struct DirtyRateStat DirtyStat; +static DirtyRateMeasureMode dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_NONE ;static int64_t set_sample_page_period (int64_t msec,int64_t initial_time) {
@@ -111,6 +112,11 @@ static void init_dirtyrate_stat(int64_t start_time, } } +static void cleanup_dirtyrate_stat (struct DirtyRateConfig config) +{ +/* TODO */ +} +static void update_dirtyrate_stat (struct RamblockDirtyInfo *info) { DirtyStat.page_sampling.total_dirty_samples += info->sample_dirty_count;
@@ -380,7 +386,6 @@ void *get_dirtyrate_thread(void *arg) {struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;int ret; -int64_t start_time;rcu_register_thread (); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,
@@ -390,9 +395,6 @@ void *get_dirtyrate_thread(void *arg)return NULL ; } - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; -init_dirtyrate_stat (start_time, config); -calculate_dirtyrate (config); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_MEASURING,
@@ -411,6 +413,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,static struct DirtyRateConfig config;QemuThread thread;int ret; +int64_t start_time;/* * If the dirty rate is already being measured, don't attempt to start.
@@ -451,6 +454,18 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, config.sample_period_seconds = calc_time; config.sample_pages_per_gigabytes = sample_pages; config.mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ; + +cleanup_dirtyrate_stat (config); + +/* +* update dirty rate mode so that we can figure out what mode has +* been used in last calculation +**/ + dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ; + + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; +init_dirtyrate_stat (start_time, config); +qemu_thread_create (&thread,"get_dirtyrate" , get_dirtyrate_thread, (void *)&config,QEMU_THREAD_DETACHED ); }
Signature · 1 line
1.8.3.1
Re: [PATCH v7 6/7] migration/dirtyrate: move init step of calculation to main thread
Quoted 62 lines · On Thu, Jun 17, 2021 at 10:12:07PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
since main thread may "query dirty rate" at any time, it's better to move init step into main thead so that synchronization overhead between "main" and "
get_dirtyrate" can be reduced.Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/dirtyrate.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)migration/dirtyrate.c +7
-4@@ -26,6 +26,7 @@static int CalculatingState =DIRTY_RATE_STATUS_UNSTARTED ;static struct DirtyRateStat DirtyStat; +static DirtyRateMeasureMode dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_NONE ;static int64_t set_sample_page_period (int64_t msec,int64_t initial_time) {@@ -111,6 +112,11 @@ static void init_dirtyrate_stat(int64_t start_time, } } +static void cleanup_dirtyrate_stat (struct DirtyRateConfig config) +{ +/* TODO */ +} +static void update_dirtyrate_stat (struct RamblockDirtyInfo *info) { DirtyStat.page_sampling.total_dirty_samples += info->sample_dirty_count;@@ -380,7 +386,6 @@ void *get_dirtyrate_thread(void *arg) {struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;int ret; -int64_t start_time;rcu_register_thread (); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,@@ -390,9 +395,6 @@ void *get_dirtyrate_thread(void *arg)return NULL ; } - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; -init_dirtyrate_stat (start_time, config); -calculate_dirtyrate (config); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_MEASURING,@@ -411,6 +413,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,static struct DirtyRateConfig config;QemuThread thread;int ret; +int64_t start_time;/* * If the dirty rate is already being measured, don't attempt to start.
migration/dirtyrate.c void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, · line 458
Quoted 5 lines
migration/dirtyrate.c +2
-0@@ -451,6 +454,18 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, config.sample_period_seconds = calc_time; config.sample_pages_per_gigabytes = sample_pages; config.mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ; + +cleanup_dirtyrate_stat (config);
This line should ideally be moved into the next patch, as sampling itself doesn't need it.
migration/dirtyrate.c void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, · line 464
Quoted 6 lines
migration/dirtyrate.c +6
-0+ +/* +* update dirty rate mode so that we can figure out what mode has +* been used in last calculation +**/ + dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ;
This line is odd. Would page sampling broken if without this line? We need to make sure each commit keeps the old way working..
Thanks,
Quoted 10 lines
migration/dirtyrate.c +4
-0+ + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; +init_dirtyrate_stat (start_time, config); +qemu_thread_create (&thread,"get_dirtyrate" , get_dirtyrate_thread, (void *)&config,QEMU_THREAD_DETACHED ); }Signature · 1 line
1.8.3.1
Signature · 1 line
Peter Xu
Re: [PATCH v7 6/7] migration/dirtyrate: move init step of calculation to main thread
Quoted 78 lines · Peter Xu
Quoted 62 lines · On Thu, Jun 17, 2021 at 10:12:07PM +0800, [email protected] wrote:
From: Hyman Huang(黄勇) <[email protected]>
since main thread may "query dirty rate" at any time, it's better to move init step into main thead so that synchronization overhead between "main" and "
get_dirtyrate" can be reduced.Signed-off-by Hyman Huang(黄勇) <[email protected]>--- migration/dirtyrate.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)migration/dirtyrate.c +7
-4@@ -26,6 +26,7 @@static int CalculatingState =DIRTY_RATE_STATUS_UNSTARTED ;static struct DirtyRateStat DirtyStat; +static DirtyRateMeasureMode dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_NONE ;static int64_t set_sample_page_period (int64_t msec,int64_t initial_time) {@@ -111,6 +112,11 @@ static void init_dirtyrate_stat(int64_t start_time, } } +static void cleanup_dirtyrate_stat (struct DirtyRateConfig config) +{ +/* TODO */ +} +static void update_dirtyrate_stat (struct RamblockDirtyInfo *info) { DirtyStat.page_sampling.total_dirty_samples += info->sample_dirty_count;@@ -380,7 +386,6 @@ void *get_dirtyrate_thread(void *arg) {struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg;int ret; -int64_t start_time;rcu_register_thread (); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_UNSTARTED,@@ -390,9 +395,6 @@ void *get_dirtyrate_thread(void *arg)return NULL ; } - start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; -init_dirtyrate_stat (start_time, config); -calculate_dirtyrate (config); ret = dirtyrate_set_state(&CalculatingState, DIRTY_RATE_STATUS_MEASURING,@@ -411,6 +413,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages,static struct DirtyRateConfig config;QemuThread thread;int ret; +int64_t start_time;/* * If the dirty rate is already being measured, don't attempt to start. migration/dirtyrate.c void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, · line 458
Quoted 5 lines
migration/dirtyrate.c +2
-0@@ -451,6 +454,18 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, config.sample_period_seconds = calc_time; config.sample_pages_per_gigabytes = sample_pages; config.mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ; + +cleanup_dirtyrate_stat (config);This line should ideally be moved into the next patch, as sampling itself doesn't need it. >
migration/dirtyrate.c void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, · line 464
Quoted 6 lines
migration/dirtyrate.c +6
-0+ +/* +* update dirty rate mode so that we can figure out what mode has +* been used in last calculation +**/ + dirtyrate_mode =DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING ;
Quoted 3 lines
This line is odd. Would page sampling broken if without this line? We need to make sure each commit keeps the old way working..
yes, i'll drop this to make sure each commit keeps the old way working
Quoted 13 lines
Thanks,
Quoted 10 lines
migration/dirtyrate.c +4
-0+ + start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) /1000 ; +init_dirtyrate_stat (start_time, config); +qemu_thread_create (&thread,"get_dirtyrate" , get_dirtyrate_thread, (void *)&config,QEMU_THREAD_DETACHED ); }Signature · 1 line
1.8.3.1
Signature · 3 lines
Best regard Hyman Huang(黄勇)
PATCH v7 7/7 migration/dirtyrate: implement dirty-ring dirtyrate calculation
In reply to [email protected]From: Hyman Huang(黄勇) <[email protected]>
use dirty ring feature to implement dirtyrate calculation.
introduce mode option in qmp calc_dirty_rate to specify what method should be used when calculating dirtyrate, either page-sampling or dirty-ring should be passed.
introduce "dirty_ring:-r" option in hmp calc_dirty_rate to indicate dirty ring method should be used for calculation.
--- hmp-commands.hx | 7 +- migration/dirtyrate.c | 199 ++++++++++++++++++++++++++++++++++++++++++++++--- migration/trace-events | 2 + qapi/migration.json | 16 +++- 4 files changed, 207 insertions(+), 17 deletions(-)
hmp-commands.hx +4 -3
@@ -1738,8 +1738,9 @@ ERST { .name = "calc_dirty_rate", - .args_type = "second:l,sample_pages_per_GB:l?", - .params = "second [sample_pages_per_GB]", - .help = "start a round of guest dirty rate measurement", + .args_type = "dirty_ring:-r,second:l,sample_pages_per_GB:l?", + .params = "[-r] second [sample_pages_per_GB]", + .help = "start a round of guest dirty rate measurement (using -d to" + "\n\t\t\t specify dirty ring as the method of calculation)", .cmd = hmp_calc_dirty_rate, },
migration/dirtyrate.c +187 -12
296 lines not on this page · raw message
migration/trace-events +2 -0
@@ -330,6 +330,8 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, uint32_t crc) "ramblock n calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock name: %s, ramblock size: %" PRIu64 find_page_matched(const char *idstr) "ramblock %s addr or size changed" +dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64 " MB/s" +dirtyrate_do_calculate_vcpu(int idx, uint64_t rate) "vcpu[%d]: %"PRIu64 " MB/s" # block.c migration_block_init_shared(const char *blk_device_name) "Start migration for %s with shared base image"
qapi/migration.json +14 -2
@@ -1773,6 +1773,12 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mode containing method of calculate dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# +# @vcpu-dirty-rate: dirtyrate for each vcpu if dirty-ring +# mode specified (Since 6.1) +# # Since: 5.2 # ##
@@ -1781,7 +1787,9 @@ 'status': 'DirtyRateStatus', 'start-time': 'int64', 'calc-time': 'int64', - 'sample-pages': 'uint64'} } + 'sample-pages': 'uint64', + 'mode': 'DirtyRateMeasureMode', + '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } } ## # @calc-dirty-rate:
@@ -1793,6 +1801,9 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mechanism of calculating dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# # Since: 5.2 # # Example:
@@ -1801,7 +1812,8 @@ # ## { 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', - '*sample-pages': 'int'} } + '*sample-pages': 'int', + '*mode': 'DirtyRateMeasureMode'} } ## # @query-dirty-rate:
Signature · 1 line
1.8.3.1
Re: [PATCH v7 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation
migration/dirtyrate.c void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict) · line 657
Quoted 13 lines · On Thu, Jun 17, 2021 at 10:12:08PM +0800, [email protected] wrote:
migration/dirtyrate.c +6
-0@@ -487,12 +646,23 @@ void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict) info->sample_pages);monitor_printf (mon,"Period: %" PRIi64" (sec)\n" , info->calc_time); +monitor_printf (mon,"Mode: %s\n" , +DirtyRateMeasureMode_str (info->mode));monitor_printf (mon,"Dirty rate: " );if (info->has_dirty_rate) {monitor_printf (mon,"%" PRIi64" (MB/s)\n" , info->dirty_rate); +if (info->has_vcpu_dirty_rate) { +DirtyRateVcpuList *rate, *head = info->vcpu_dirty_rate; +for (rate = head; rate !=NULL ; rate = rate->next) { + monitor_printf(mon, "vcpu[%"PRIi64"], Dirty rate: %"PRIi64"\n",
Maybe ending it with " (MB/s)" to match the dirty rate line?
Otherwise this patch looks good to me, thanks. It's just that I still have a few trivial comments out there, so this patch still may need some adjustment.
Quoted 10 lines
migration/dirtyrate.c +5
-0+ rate->value->id, rate->value->dirty_rate); + } + } }else {monitor_printf (mon,"(not ready)\n" ); } + +qapi_free_DirtyRateVcpuList (info->vcpu_dirty_rate);g_free (info); }
Signature · 1 line
Peter Xu