KVM: arm64: Properly restore PMU state during live-migration
PATCH 1/1 KVM: arm64: Properly restore PMU state during live-migration
Currently if a guest is live-migrated while it is actively using perf counters, then after live-migrate it will notice that all counters would suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.
Usually on live-migration guest state is restored using KVM_SET_ONE_REG ioctl interface, which simply restores the value of PMU registers values but does not re-program the perf events so that the guest can seamlessly use these counters even after live-migration like it was doing before live-migration.
Instead there are two completely different code path between guest accessing PMU registers and VMM restoring counters on live-migration.
In case of KVM_SET_ONE_REG:
kvm_arm_set_reg()
...... kvm_arm_sys_reg_set_reg()
........... reg_from_user()
but in case when guest tries to access these counters:
handle_exit()
..... kvm_handle_sys_reg() ..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()
The drawback of using the KVM_SET_ONE_REG interface is that the host pmu events which were registered for the source instance and not present for the destination instance. Thus passively restoring PMCR_EL0 using KVM_SET_ONE_REG interface would not create the necessary host pmu events which are crucial for seamless guest experience across live migration.
In ordet to fix the situation, on first vcpu load we should restore PMCR_EL0 in the same exact way like the guest was trying to access these counters. And then we will also recreate the relevant host pmu events.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)void kvm_set_pmu_events (u32 set,struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_host (struct kvm_vcpu *vcpu);#else
arch/arm64/kvm/arm.c +1 -0
@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled (&vcpu->arch))kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);
arch/arm64/kvm/pmu-emul.c +8 -2
@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); } -if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_C || !kvm_arm_pmu_v3_restored (vcpu))kvm_pmu_set_counter_value (vcpu,ARMV8_PMU_CYCLE_IDX ,0 ); -if (val &ARMV8_PMU_PMCR_P ) { +/* +* All the counters needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_P || !kvm_arm_pmu_v3_restored (vcpu)) {for_each_set_bit (i, &mask,32 )kvm_pmu_set_counter_value (vcpu, i,0 ); }
arch/arm64/kvm/pmu.c +15 -0
@@ -161,6 +161,21 @@ static void kvm_vcpu_pmu_disable_el0(unsigned long events) } } +/* +* Restore PMU events on first vcpu load. +*/ +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu) +{ +if (kvm_arm_pmu_v3_restored (vcpu)) +return ; + +u64 val =__vcpu_sys_reg (vcpu,PMCR_EL0 ); + +kvm_pmu_handle_pmcr (vcpu, val); + + vcpu->arch.pmu.restored = true; +} +/* * On VHE ensure that only guest events have EL0 counting enabled. * This is called from both vcpu_{load,put} and the sysreg handling.
include/kvm/arm_pmu.h +3 -0
@@ -32,10 +32,12 @@ struct kvm_pmu {struct kvm_pmc pmc[ARMV8_PMU_MAX_COUNTERS ];DECLARE_BITMAP (chained,ARMV8_PMU_MAX_COUNTER_PAIRS );bool created; +bool restored;bool irq_level;struct irq_work overflow_work; }; +#define kvm_arm_pmu_v3_restored (v) ((v)->arch.pmu.restored)#define kvm_arm_pmu_irq_initialized (v) ((v)->arch.pmu.irq_num >= VGIC_NR_SGIS)u64 kvm_pmu_get_counter_value (struct kvm_vcpu *vcpu,u64 select_idx);void kvm_pmu_set_counter_value (struct kvm_vcpu *vcpu,u64 select_idx,u64 val);
@@ -67,6 +69,7 @@ struct kvm_pmu { };#define kvm_arm_pmu_irq_initialized (v) (false) +#define kvm_arm_pmu_v3_restored (v) (false)static inline u64 kvm_pmu_get_counter_value (struct kvm_vcpu *vcpu,u64 select_idx) {
Signature · 12 lines
2.31.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879
kernel test robot
build botThank you for the patch! Perhaps something to improve:
Hi Jinank,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kvmarm/next]
[also build test WARNING on arm/for-next soc/for-next arm64/for-next/core v5.13-rc4 next-20210603] [cannot apply to xlnx/master]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Jinank-Jain/KVM-arm64-Properly-restore-PMU-state-during-live-migration/20210603-190755 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next config: arm64-allyesconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/08cb3fc07cb1a68f15eb4a3e68e886c40732e699
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Jinank-Jain/KVM-arm64-Properly-restore-PMU-state-during-live-migration/20210603-190755 git checkout 08cb3fc07cb1a68f15eb4a3e68e886c40732e699
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
arch/arm64/kvm/pmu.c: In function 'kvm_vcpu_pmu_restore':
>> arch/arm64/kvm/pmu.c:172:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
172 | u64 val = __vcpu_sys_reg(vcpu, PMCR_EL0);
| ^~~vim +172 arch/arm64/kvm/pmu.c
163
164 /*
165 * Restore PMU events on first vcpu load.
166 */
167 void kvm_vcpu_pmu_restore(struct kvm_vcpu *vcpu)
168 {
169 if (kvm_arm_pmu_v3_restored(vcpu))
170 return;
171
> 172 u64 val = __vcpu_sys_reg(vcpu, PMCR_EL0);
173
174 kvm_pmu_handle_pmcr(vcpu, val);
175
176 vcpu->arch.pmu.restored = true;
177 }
178
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Marc Zyngier
maintainerHi Jinank,
Quoted 31 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using perf counters, then after live-migrate it will notice that all counters would suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.
Usually on live-migration guest state is restored using
KVM_SET_ONE_REGioctl interface, which simply restores the value of PMU registers values but does not re-program the perf events so that the guest can seamlessly use these counters even after live-migration like it was doing before live-migration.Instead there are two completely different code path between guest accessing PMU registers and VMM restoring counters on live-migration.
In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()
Quoted 3 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the host pmu events which were registered for the source instance and not present for the destination instance.
I can't parse this sentence. Do you mean "are not present"?
Quoted 42 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host pmu events which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access these counters. And then we will also recreate the relevant host pmu events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)void kvm_set_pmu_events (u32 set,struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_host (struct kvm_vcpu *vcpu);#else arch/arm64/kvm/arm.c +0
-0
arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 411
Quoted 4 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);
If this only needs to be run once per vcpu, why not trigger it from kvm_arm_pmu_v3_enable(), which is also called once per vcpu?
This can done on the back of a request, saving most of the overhead and not requiring any extra field. Essentially, something like the (untested) patch below.
Quoted 8 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled (&vcpu->arch))kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);arch/arm64/kvm/pmu-emul.c +0
-0
arch/arm64/kvm/pmu-emul.c void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) · line 580
Quoted 8 lines
arch/arm64/kvm/pmu-emul.c +4
-1@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); } -if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_C || !kvm_arm_pmu_v3_restored (vcpu))
Why? There is no architectural guarantee that a counter resets to 0 without writing PMCR_EL0.C. And if you want the guest to continue counting where it left off, resetting the counter is at best counter-productive.
So I must be missing something...
arch/arm64/kvm/pmu-emul.c void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) · line 586
Quoted 7 lines
arch/arm64/kvm/pmu-emul.c +4
-1kvm_pmu_set_counter_value (vcpu,ARMV8_PMU_CYCLE_IDX ,0 ); -if (val &ARMV8_PMU_PMCR_P ) { +/* +* All the counters needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_P || !kvm_arm_pmu_v3_restored (vcpu)) {
Same thing here.
arch/arm64/kvm/pmu-emul.c void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) · line 589
Quoted 3 lines
arch/arm64/kvm/pmu-emul.c +0
-0for_each_set_bit (i, &mask,32 )kvm_pmu_set_counter_value (vcpu, i,0 ); }
The rest of the changes should be unnecessary with the patch below.
Thanks,
M.
From 1be188ab71867632a8c17384be6e55f47f42aa8b Mon Sep 17 00:00:00 2001
From: Marc Zyngier <[email protected]>
Date: Thu, 3 Jun 2021 16:50:02 +0100
Subject: [PATCH] KVM: arm64: Restore PMU configuration on first run
Restoring a guest with an active virtual PMU results in no perf counters being instanciated on the host side. Not quite what you'd expect from a restore.
In order to fix this, force a writeback of PMCR_EL0 on the first run of a vcpu (using a new request so that it happens once the vcpu has been loaded). This will in turn create all the host-side counters that were missing.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 4 ++++ arch/arm64/kvm/pmu-emul.c | 3 +++ 3 files changed, 8 insertions(+)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -46,6 +46,7 @@#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2)#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/kvm/arm.c +4 -0
@@ -689,6 +689,10 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)vgic_v4_load (vcpu);preempt_enable (); } + +if (kvm_check_request (KVM_REQ_RELOAD_PMU , vcpu)) +kvm_pmu_handle_pmcr (vcpu, +__vcpu_sys_reg (vcpu,PMCR_EL0 )); } }
arch/arm64/kvm/pmu-emul.c +3 -0
@@ -850,6 +850,9 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)return -EINVAL ; } +/* One-off reload of the PMU on first run */ +kvm_make_request (KVM_REQ_RELOAD_PMU , vcpu); +return 0 ; }
Signature · 5 lines
2.30.2 -- Without deviation from the norm, progress is not possible.
Quoted 125 lines · Marc Zyngier
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi Jinank,
Quoted 35 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using perf
counters, then after live-migrate it will notice that all counters would
suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.Usually on live-migration guest state is restored using
KVM_SET_ONE_REG
ioctl interface, which simply restores the value of PMU registers values but does not re-program the perf events so that the guest can seamlessly
use these counters even after live-migration like it was doing before
live-migration.Instead there are two completely different code path between guest accessing PMU registers and VMM restoring counters on live-migration.
In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()Quoted 5 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the host pmu
events which were registered for the source instance and not present for
the destination instance.I can't parse this sentence. Do you mean "are not present"?
Quoted 47 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host pmu events
which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access these counters. And then we will also recreate the relevant host pmu
events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_host (struct kvm_vcpu *vcpu);#else
arch/arm64/kvm/arm.c +0
-0arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, · line 412
Quoted 5 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);If this only needs to be run once per vcpu, why not trigger it from
kvm_arm_pmu_v3_enable(), which is also called once per vcpu?This can done on the back of a request, saving most of the overhead and not requiring any extra field. Essentially, something like the (untested) patch below.
Quoted 12 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled (&vcpu->arch))kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu);arch/arm64/kvm/pmu-emul.c +0
-0@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); }
Quoted 10 lines
arch/arm64/kvm/pmu-emul.c void kvm_pmu_handle_pmcr(struct kvm_vcpu · line 581
Quoted 5 lines
arch/arm64/kvm/pmu-emul.c +4
-1-if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_C || !kvm_arm_pmu_v3_restored (vcpu))Why? There is no architectural guarantee that a counter resets to 0 without writing PMCR_EL0.C. And if you want the guest to continue counting where it left off, resetting the counter is at best counter-productive.
Without this we would not be resetting PMU which is required for creating host perf events. With the patch that you suggested we are restoring PMCR_EL0 properly but still missing recreation of host perf events. And without host perf events, guest would still zeros after live migration. In my opinion we have two ways to fix it. We can fix it inside the kernel or let userspace/VMM set those bits before restarting the guest on the destination machine. What do you think?
Quoted 98 lines
So I must be missing something...
Quoted 10 lines
arch/arm64/kvm/pmu-emul.c +2
-1kvm_pmu_set_counter_value (vcpu,ARMV8_PMU_CYCLE_IDX ,0 ); -if (val &ARMV8_PMU_PMCR_P ) { +/* +* All the counters needs to reset in case of first vcpu load.
arch/arm64/kvm/pmu-emul.c +2
-0+*/ +if (val &ARMV8_PMU_PMCR_P || !kvm_arm_pmu_v3_restored (vcpu)){
Same thing here.
Quoted 3 lines
for_each_set_bit(i, &mask, 32) kvm_pmu_set_counter_value(vcpu, i, 0); }The rest of the changes should be unnecessary with the patch below.
Thanks,
M.From
1be188ab71867632a8c17384be6e55f47f42aa8bMon Sep 17 00:00:00 2001
From: Marc Zyngier <[email protected]>
Date: Thu, 3 Jun 2021 16:50:02 +0100
Subject: [PATCH] KVM: arm64: Restore PMU configuration on first runRestoring a guest with an active virtual PMU results in no perf counters being instanciated on the host side. Not quite what you'd expect from a restore.
In order to fix this, force a writeback of
PMCR_EL0on the first run of a vcpu (using a new request so that it happens once the vcpu has been loaded). This will in turn create all the host-side counters that were missing.Reported-by Jinank Jain <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 4 ++++ arch/arm64/kvm/pmu-emul.c | 3 +++ 3 files changed, 8 insertions(+)arch/arm64/include/asm/kvm_host.h +1
-0@@ -46,6 +46,7 @@#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2)#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
KVM_DIRTY_LOG_INITIALLY_SET)arch/arm64/kvm/arm.c +4
-0@@ -689,6 +689,10 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)vgic_v4_load (vcpu);preempt_enable (); } + +if (kvm_check_request (KVM_REQ_RELOAD_PMU , vcpu)) +kvm_pmu_handle_pmcr (vcpu, +__vcpu_sys_reg (vcpu,PMCR_EL0 )); }}
arch/arm64/kvm/pmu-emul.c +3
-1@@ -850,6 +850,9 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)return -EINVAL ; } +/* One-off reload of the PMU on first run */ +kvm_make_request (KVM_REQ_RELOAD_PMU , vcpu); +return 0 ; } --2.30.2
Quoted 2 lines
--
Without deviation from the norm, progress is not possible.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Marc Zyngier
maintainerQuoted 133 lines · Jinank
Quoted 120 lines · Marc Zyngier
Hi Jinank,
Quoted 35 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using perf
counters, then after live-migrate it will notice that all counters would
suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.Usually on live-migration guest state is restored using
KVM_SET_ONE_REG
ioctl interface, which simply restores the value of PMU registers values but does not re-program the perf events so that the guest can seamlessly
use these counters even after live-migration like it was doing before
live-migration.Instead there are two completely different code path between guest accessing PMU registers and VMM restoring counters on live-migration.
In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()Quoted 5 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the host pmu
events which were registered for the source instance and not present for
the destination instance.I can't parse this sentence. Do you mean "are not present"?
Quoted 47 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host pmu events
which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access these counters. And then we will also recreate the relevant host pmu
events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_host (struct kvm_vcpu *vcpu);#else
arch/arm64/kvm/arm.c +0
-0arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, · line 412
Quoted 5 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);If this only needs to be run once per vcpu, why not trigger it from
kvm_arm_pmu_v3_enable(), which is also called once per vcpu?This can done on the back of a request, saving most of the overhead and not requiring any extra field. Essentially, something like the (untested) patch below.
Quoted 12 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled (&vcpu->arch))kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu);arch/arm64/kvm/pmu-emul.c +0
-0@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); }Quoted 10 lines
arch/arm64/kvm/pmu-emul.c void kvm_pmu_handle_pmcr(struct kvm_vcpu · line 581
Quoted 5 lines
arch/arm64/kvm/pmu-emul.c +4
-1-if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load. +*/ +if (val &ARMV8_PMU_PMCR_C || !kvm_arm_pmu_v3_restored (vcpu))Why? There is no architectural guarantee that a counter resets to 0 without writing PMCR_EL0.C. And if you want the guest to continue counting where it left off, resetting the counter is at best counter-productive.
Quoted 4 lines
Without this we would not be resetting PMU which is required for creating host perf events. With the patch that you suggested we are restoring
PMCR_EL0properly but still missing recreation of host perf events.
How? The request that gets set on the first vcpu run will call kvm_pmu_handle_pmcr() -> kvm_pmu_enable_counter_mask() -> kvm_pmu_create_perf_event(). What are we missing?
Quoted 4 lines
And without host perf events, guest would still zeros after live migration. In my opinion we have two ways to fix it. We can fix it inside the kernel or let userspace/VMM set those bits before restarting the guest on the destination machine. What do you think?
I think either you're missing my point above, or I'm completely missing yours. And I still don't understand why you want to zero the counters that you have just restored. How does that help?
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Marc.
Quoted 163 lines · Marc Zyngier
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 149 lines · Jinank
Quoted 134 lines · Marc Zyngier
Hi Jinank,
Quoted 40 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using
perf
counters, then after live-migrate it will notice that all counters
would
suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.Usually on live-migration guest state is restored using
KVM_SET_ONE_REG
ioctl interface, which simply restores the value of PMU registers
values but does not re-program the perf events so that the guest
can seamlessly
use these counters even after live-migration like it was doing before
live-migration.Instead there are two completely different code path between guest
accessing PMU registers and VMM restoring counters on live-migration.In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()Quoted 5 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the host pmu
events which were registered for the source instance and not present for
the destination instance.I can't parse this sentence. Do you mean "are not present"?
Quoted 52 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host pmu
events
which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access
these counters. And then we will also recreate the relevant host
pmu
events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void
kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); #elsearch/arm64/kvm/arm.c +0
-0arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu · line 413
Quoted 6 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);If this only needs to be run once per vcpu, why not trigger it from
kvm_arm_pmu_v3_enable(), which is also called once per vcpu?This can done on the back of a request, saving most of the overhead
and not requiring any extra field. Essentially, something like the
(untested) patch below.Quoted 13 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled(&vcpu->arch)) kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu);arch/arm64/kvm/pmu- renamed from arch/arm64/kvm/pmu-emul.c +0
-0
emul.c
indexfd167d4f4215..12a40f4b5f0d100644arch/arm64/kvm/pmu-emul.c +0
-0@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); }Quoted 13 lines
Quoted 7 lines
arch/arm64/kvm/pmu-emul.c +2
-1-if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load.
arch/arm64/kvm/pmu-emul.c +2
-0+*/ + if (val & ARMV8_PMU_PMCR_C ||!kvm_arm_pmu_v3_restored(vcpu))
Why? There is no architectural guarantee that a counter resets to 0
without writing PMCR_EL0.C. And if you want the guest to continue counting where it left off, resetting the counter is at best counter-productive.Quoted 5 lines
Without this we would not be resetting PMU which is required for creating host perf events. With the patch that you suggested we are restoring
PMCR_EL0properly but still missing recreation of host perf
events.
Quoted 4 lines
How? The request that gets set on the first vcpu run will call
kvm_pmu_handle_pmcr()->kvm_pmu_enable_counter_mask()->kvm_pmu_create_perf_event(). What are we missing?
I found out what I was missing. I was working with an older kernel which was missing this upstream patch:
https://lore.kernel.org/lkml/[email protected]/
Quoted 5 lines
Quoted 4 lines
And without host perf events, guest would still zeros after live migration. In my opinion we have two ways to fix it. We can fix it inside the kernel or let userspace/VMM set those bits before restarting the guest on the destination machine. What do you think?
Quoted 8 lines
I think either you're missing my point above, or I'm completely missing yours. And I still don't understand why you want to zero the counters that you have just restored. How does that help?
M.--
Without deviation from the norm, progress is not possible.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Marc Zyngier
maintainerQuoted 172 lines · Jinank
Hi Marc.
Quoted 163 lines · Marc Zyngier
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 149 lines · Jinank
Quoted 134 lines · Marc Zyngier
Hi Jinank,
Quoted 40 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using
perf
counters, then after live-migrate it will notice that all counters
would
suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.Usually on live-migration guest state is restored using
KVM_SET_ONE_REG
ioctl interface, which simply restores the value of PMU registers
values but does not re-program the perf events so that the guest
can seamlessly
use these counters even after live-migration like it was doing before
live-migration.Instead there are two completely different code path between guest
accessing PMU registers and VMM restoring counters on live-migration.In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()Quoted 5 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the host pmu
events which were registered for the source instance and not present for
the destination instance.I can't parse this sentence. Do you mean "are not present"?
Quoted 52 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host pmu
events
which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access
these counters. And then we will also recreate the relevant host
pmu
events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void
kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); #elsearch/arm64/kvm/arm.c +0
-0arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu · line 413
Quoted 6 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);If this only needs to be run once per vcpu, why not trigger it from
kvm_arm_pmu_v3_enable(), which is also called once per vcpu?This can done on the back of a request, saving most of the overhead
and not requiring any extra field. Essentially, something like the
(untested) patch below.Quoted 13 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled(&vcpu->arch)) kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu);arch/arm64/kvm/pmu- renamed from arch/arm64/kvm/pmu-emul.c +0
-0
emul.c
indexfd167d4f4215..12a40f4b5f0d100644arch/arm64/kvm/pmu-emul.c +0
-0@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)kvm_pmu_disable_counter_mask (vcpu, mask); }Quoted 13 lines
Quoted 7 lines
arch/arm64/kvm/pmu-emul.c +2
-1-if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load.
arch/arm64/kvm/pmu-emul.c +2
-0+*/ + if (val & ARMV8_PMU_PMCR_C ||!kvm_arm_pmu_v3_restored(vcpu))
Why? There is no architectural guarantee that a counter resets to 0
without writing PMCR_EL0.C. And if you want the guest to continue counting where it left off, resetting the counter is at best counter-productive.Quoted 5 lines
Without this we would not be resetting PMU which is required for creating host perf events. With the patch that you suggested we are restoring
PMCR_EL0properly but still missing recreation of host perf
events.Quoted 4 lines
How? The request that gets set on the first vcpu run will call
kvm_pmu_handle_pmcr()->kvm_pmu_enable_counter_mask()->kvm_pmu_create_perf_event(). What are we missing?
Quoted 4 lines
I found out what I was missing. I was working with an older kernel which was missing this upstream patch:
:-(
Please test whatever you send with an upstream kernel. Actually, please *develop* on an upstream kernel. This will avoid this kind of discussion where we talk past each other, and make it plain that your production kernel is lacking all sorts of fixes.
Now, can you please state whether or not this patch fixes it for you *on an upstream kernel*? I have no interest in results from a production kernel.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 208 lines · Marc Zyngier
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 188 lines · Jinank
Hi Marc.
Quoted 180 lines · Marc Zyngier
CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you can confirm the sender
and know the content is safe.Quoted 161 lines · Jinank
Quoted 144 lines · Marc Zyngier
Hi Jinank,
Quoted 43 lines · Jinank Jain
Currently if a guest is live-migrated while it is actively using
perf
counters, then after live-migrate it will notice that all counters
would
suddenly start reporting 0s. This is due to the fact we are not
re-creating the relevant perf events inside the kernel.Usually on live-migration guest state is restored using
KVM_SET_ONE_REG
ioctl interface, which simply restores the value of PMU registers
values but does not re-program the perf events so that the guest
can seamlessly
use these counters even after live-migration like it was doing
before
live-migration.Instead there are two completely different code path between
guest
accessing PMU registers and VMM restoring counters on live-migration.In case of
KVM_SET_ONE_REG:
kvm_arm_set_reg()
......kvm_arm_sys_reg_set_reg()
...........reg_from_user()but in case when guest tries to access these counters:
handle_exit()
.....kvm_handle_sys_reg()..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()Quoted 7 lines
The drawback of using the
KVM_SET_ONE_REGinterface is that the
host pmu
events which were registered for the source instance and not
present for
the destination instance.I can't parse this sentence. Do you mean "are not present"?
Quoted 53 lines
Thus passively restoring
PMCR_EL0usingKVM_SET_ONE_REGinterface would not create the necessary host
pmu
events
which are crucial for seamless guest experience across live migration.In ordet to fix the situation, on first vcpu load we should restore
PMCR_EL0in the same exact way like the guest was trying to access
these counters. And then we will also recreate the relevant host
pmu
events.Signed-off-by Jinank Jain <[email protected]>Cc Alexander Graf (AWS) <[email protected]>Cc Marc Zyngier <[email protected]>Cc James Morse <[email protected]>Cc Alexandru Elisei <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/pmu-emul.c | 10 ++++++++-- arch/arm64/kvm/pmu.c | 15 +++++++++++++++ include/kvm/arm_pmu.h | 3 +++ 5 files changed, 28 insertions(+), 2 deletions(-)arch/arm64/include/asm/kvm_host.h +1
-0@@ -745,6 +745,7 @@ static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);void kvm_clr_pmu_events (u32 clr); +void kvm_vcpu_pmu_restore (struct kvm_vcpu *vcpu);void kvm_vcpu_pmu_restore_guest (struct kvm_vcpu *vcpu);void
kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); #elsearch/arm64/kvm/arm.c +0
-0arch/arm64/kvm/arm.c void kvm_arch_vcpu_load(struct kvm_vcpu · line 413
Quoted 6 lines
arch/arm64/kvm/arm.c +1
-0@@ -408,6 +408,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu); +kvm_vcpu_pmu_restore (vcpu);If this only needs to be run once per vcpu, why not trigger it
fromkvm_arm_pmu_v3_enable(), which is also called once per vcpu?This can done on the back of a request, saving most of the overhead
and not requiring any extra field. Essentially, something like
the
(untested) patch below.Quoted 15 lines
arch/arm64/kvm/arm.c +0
-0kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled(&vcpu->arch)) kvm_make_request(KVM_REQ_RECORD_STEAL, vcpu);arch/arm64/kvm/pmu- renamed from arch/arm64/kvm/pmu-emul.c +0
-0
emul.c
indexfd167d4f4215..12a40f4b5f0d100644arch/arm64/kvm/pmu-emul.c +0
-0@@ -574,10 +574,16 @@ void kvm_pmu_handle_pmcr(struct
kvm_vcpu
*vcpu, u64 val)kvm_pmu_disable_counter_mask(vcpu, mask); }Quoted 15 lines
Quoted 7 lines
arch/arm64/kvm/pmu-emul.c +2
-1-if (val &ARMV8_PMU_PMCR_C ) +/* +* Cycle counter needs to reset in case of first vcpu load.
arch/arm64/kvm/pmu-emul.c +2
-0+*/ + if (val & ARMV8_PMU_PMCR_C ||!kvm_arm_pmu_v3_restored(vcpu))
Why? There is no architectural guarantee that a counter resets to
0
without writing PMCR_EL0.C. And if you want the guest to continue
counting where it left off, resetting the counter is at best counter-productive.Quoted 8 lines
Without this we would not be resetting PMU which is required for
creating host perf events. With the patch that you suggested we are
restoringPMCR_EL0properly but still missing recreation of host
perf
events.Quoted 4 lines
How? The request that gets set on the first vcpu run will call
kvm_pmu_handle_pmcr()->kvm_pmu_enable_counter_mask()->kvm_pmu_create_perf_event(). What are we missing?Quoted 4 lines
I found out what I was missing. I was working with an older kernel which was missing this upstream patch:
:-(
Please test whatever you send with an upstream kernel. Actually, please *develop* on an upstream kernel. This will avoid this kind of discussion where we talk past each other, and make it plain that your production kernel is lacking all sorts of fixes.
Quoted 6 lines
Now, can you please state whether or not this patch fixes it for you *on an upstream kernel*? I have no interest in results from a production kernel.
M.
Really sorry for the noise and I can confirm that your suggested patch fixes the problem for the upstream kernel i.e., if I live migrate a guest which is actively using perf events then the guest can continue using them even after live migration without interruption.
Quoted 2 lines
--
Without deviation from the norm, progress is not possible.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
PATCH v2 1/1 KVM: arm64: Properly restore PMU state during live-migration
In reply to Jinank JainCurrently if a guest is live-migrated while it is actively using perf counters, then after live-migrate it will notice that all counters would suddenly start reporting 0s. This is due to the fact we are not re-creating the relevant perf events inside the kernel.
Usually on live-migration guest state is restored using KVM_SET_ONE_REG ioctl interface, which simply restores the value of PMU registers values but does not re-program the perf events so that the guest can seamlessly use these counters even after live-migration like it was doing before live-migration.
Instead there are two completely different code path between guest accessing PMU registers and VMM restoring counters on live-migration.
In case of KVM_SET_ONE_REG:
kvm_arm_set_reg()
...... kvm_arm_sys_reg_set_reg()
........... reg_from_user()
but in case when guest tries to access these counters:
handle_exit()
..... kvm_handle_sys_reg() ..........perform_access() ...............access_pmu_evcntr() ...................kvm_pmu_set_counter_value() .......................kvm_pmu_create_perf_event()
The drawback of using the KVM_SET_ONE_REG interface is that the host pmu events which were registered for the source instance are not present for the destination instance. Thus, passively restoring PMCR_EL0 using KVM_SET_ONE_REG interface would not create the necessary host pmu events which are crucial for seamless guest experience across live migration.
In ordet to fix the situation, on first vcpu load we should restore PMCR_EL0 in the same exact way like the guest was trying to access these counters. And then we will also recreate the relevant host pmu events.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 4 ++++ arch/arm64/kvm/pmu-emul.c | 3 +++ 3 files changed, 8 insertions(+)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -46,6 +46,7 @@#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2)#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/kvm/arm.c +4 -0
@@ -689,6 +689,10 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)vgic_v4_load (vcpu);preempt_enable (); } + +if (kvm_check_request (KVM_REQ_RELOAD_PMU , vcpu)) +kvm_pmu_handle_pmcr (vcpu, +__vcpu_sys_reg (vcpu,PMCR_EL0 )); } }
arch/arm64/kvm/pmu-emul.c +3 -0
@@ -850,6 +850,9 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)return -EINVAL ; } +/* One-off reload of the PMU on first run */ +kvm_make_request (KVM_REQ_RELOAD_PMU , vcpu); +return 0 ; }
Signature · 12 lines
2.31.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879