KVM: arm64: Disabling disabled PMU counters wastes a lot of time
PATCH v2 1/1 KVM: arm64: Disabling disabled PMU counters wastes a lot of time
In a KVM guest on arm64, performance counters interrupts have an unnecessary overhead which slows down execution when using the "perf record" command and limits the "perf record" sampling period.
The problem is that when a guest VM disables counters by clearing the PMCR_EL0.E bit (bit 0), KVM will disable all counters defined in PMCR_EL0 even if they are not enabled in PMCNTENSET_EL0.
KVM disables a counter by calling into the perf framework, in particular by calling perf_event_create_kernel_counter() which is a time consuming operation. So, for example, with a Neoverse N1 CPU core which has 6 event counters and one cycle counter, KVM will always disable all 7 counters even if only one is enabled.
This typically happens when using the "perf record" command in a guest
VM: perf will disable all event counters with PMCNTENTSET_EL0 and only uses the cycle counter. And when using the "perf record" -F option with a high profiling frequency, the overhead of KVM disabling all counters instead of one on every counter interrupt becomes very noticeable.
The problem is fixed by having KVM disable only counters which are enabled in PMCNTENSET_EL0. If a counter is not enabled in PMCNTENSET_EL0 then KVM will not enable it when setting PMCR_EL0.E and it will remain disabled as long as it is not enabled in PMCNTENSET_EL0. So there is effectively no need to disable a counter when clearing PMCR_EL0.E if it is not enabled PMCNTENSET_EL0.
---
The patch is based on https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/pmu/reset-values
arch/arm64/kvm/pmu-emul.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/arm64/kvm/pmu-emul.c +5 -3
@@ -563,21 +563,23 @@ void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val)*/ void kvm_pmu_handle_pmcr (struct kvm_vcpu *vcpu,u64 val) { -unsigned long mask =kvm_pmu_valid_counter_mask (vcpu); +unsigned long mask;int i;if (val &ARMV8_PMU_PMCR_E ) {kvm_pmu_enable_counter_mask (vcpu,__vcpu_sys_reg (vcpu,PMCNTENSET_EL0 )); }else { -kvm_pmu_disable_counter_mask (vcpu, mask); +kvm_pmu_disable_counter_mask (vcpu, +__vcpu_sys_reg (vcpu,PMCNTENSET_EL0 )); }if (val &ARMV8_PMU_PMCR_C )kvm_pmu_set_counter_value (vcpu,ARMV8_PMU_CYCLE_IDX ,0 );if (val &ARMV8_PMU_PMCR_P ) { - mask &= ~BIT (ARMV8_PMU_CYCLE_IDX ); + mask =kvm_pmu_valid_counter_mask (vcpu) + &BIT (ARMV8_PMU_CYCLE_IDX );for_each_set_bit (i, &mask,32 )kvm_pmu_set_counter_value (vcpu, i,0 ); }
base-commit: 83f870a663592797c576846db3611e0a1664eda2
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 52 lines · Alexandre Chartre
In a KVM guest on arm64, performance counters interrupts have an unnecessary overhead which slows down execution when using the "perf record" command and limits the "perf record" sampling period.
The problem is that when a guest VM disables counters by clearing the PMCR_EL0.E bit (bit 0), KVM will disable all counters defined in
PMCR_EL0even if they are not enabled inPMCNTENSET_EL0.KVM disables a counter by calling into the perf framework, in particular by calling
perf_event_create_kernel_counter()which is a time consuming operation. So, for example, with a Neoverse N1 CPU core which has 6 event counters and one cycle counter, KVM will always disable all 7 counters even if only one is enabled.This typically happens when using the "perf record" command in a guest
VM: perf will disable all event counters withPMCNTENTSET_EL0and only uses the cycle counter. And when using the "perf record" -F option with a high profiling frequency, the overhead of KVM disabling all counters instead of one on every counter interrupt becomes very noticeable.The problem is fixed by having KVM disable only counters which are enabled in
PMCNTENSET_EL0. If a counter is not enabled inPMCNTENSET_EL0then KVM will not enable it when setting PMCR_EL0.E and it will remain disabled as long as it is not enabled inPMCNTENSET_EL0. So there is effectively no need to disable a counter when clearing PMCR_EL0.E if it is not enabledPMCNTENSET_EL0.Signed-off-by Alexandre Chartre <[email protected]>---
The patch is based on https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/pmu/reset-valuesarch/arm64/kvm/pmu-emul.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)arch/arm64/kvm/pmu-emul.c +2
-2@@ -563,21 +563,23 @@ void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val)*/ void kvm_pmu_handle_pmcr (struct kvm_vcpu *vcpu,u64 val) { -unsigned long mask =kvm_pmu_valid_counter_mask (vcpu); +unsigned long mask;int i;if (val &ARMV8_PMU_PMCR_E ) {kvm_pmu_enable_counter_mask (vcpu,__vcpu_sys_reg (vcpu,PMCNTENSET_EL0 )); }else { -kvm_pmu_disable_counter_mask (vcpu, mask); + kvm_pmu_disable_counter_mask(vcpu,
arch/arm64/kvm/pmu-emul.c void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val) · line 582
Quoted 10 lines
arch/arm64/kvm/pmu-emul.c +3
-1+ __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); }if (val &ARMV8_PMU_PMCR_C )kvm_pmu_set_counter_value (vcpu,ARMV8_PMU_CYCLE_IDX ,0 );if (val &ARMV8_PMU_PMCR_P ) { - mask &= ~BIT (ARMV8_PMU_CYCLE_IDX ); + mask =kvm_pmu_valid_counter_mask (vcpu) + &BIT (ARMV8_PMU_CYCLE_IDX );
This looks suspiciously opposite of what it replaces; however did we even need to do a bitwise operation here in the first place? Couldn't we skip the cycle counter by just limiting the for_each_set_bit iteration below to 31 bits?
Robin.
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +0
-0for_each_set_bit (i, &mask,32 )kvm_pmu_set_counter_value (vcpu, i,0 ); }base-commit:
83f870a663592797c576846db3611e0a1664eda2
Hi Robin,
Quoted 56 lines · Robin Murphy
Quoted 55 lines · Alexandre Chartre
In a KVM guest on arm64, performance counters interrupts have an unnecessary overhead which slows down execution when using the "perf record" command and limits the "perf record" sampling period.
The problem is that when a guest VM disables counters by clearing the PMCR_EL0.E bit (bit 0), KVM will disable all counters defined in
PMCR_EL0even if they are not enabled inPMCNTENSET_EL0.KVM disables a counter by calling into the perf framework, in particular by calling
perf_event_create_kernel_counter()which is a time consuming operation. So, for example, with a Neoverse N1 CPU core which has 6 event counters and one cycle counter, KVM will always disable all 7 counters even if only one is enabled.This typically happens when using the "perf record" command in a guest
VM: perf will disable all event counters withPMCNTENTSET_EL0and only uses the cycle counter. And when using the "perf record" -F option with a high profiling frequency, the overhead of KVM disabling all counters instead of one on every counter interrupt becomes very noticeable.The problem is fixed by having KVM disable only counters which are enabled in
PMCNTENSET_EL0. If a counter is not enabled inPMCNTENSET_EL0then KVM will not enable it when setting PMCR_EL0.E and it will remain disabled as long as it is not enabled inPMCNTENSET_EL0. So there is effectively no need to disable a counter when clearing PMCR_EL0.E if it is not enabledPMCNTENSET_EL0.Signed-off-by Alexandre Chartre <[email protected]>---
The patch is based on
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/pmu/reset-values
arch/arm64/kvm/pmu-emul.c| 8 +++++---1 file changed, 5 insertions(+), 3 deletions(-)arch/arm64/kvm/pmu-emul.c +3
-2@@ -563,21 +563,23 @@ void kvm_pmu_software_increment(struct kvm_vcpu *vcpu,u64 val) */ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) { - unsigned long mask = kvm_pmu_valid_counter_mask(vcpu); + unsigned long mask; int i; if (val & ARMV8_PMU_PMCR_E) { kvm_pmu_enable_counter_mask(vcpu, __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); } else { - kvm_pmu_disable_counter_mask(vcpu, mask); + kvm_pmu_disable_counter_mask(vcpu, + __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); }
Quoted 8 lines
arch/arm64/kvm/pmu-emul.c void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val) · line 582
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +2
-1if (val & ARMV8_PMU_PMCR_C) kvm_pmu_set_counter_value(vcpu, ARMV8_PMU_CYCLE_IDX, 0); if (val & ARMV8_PMU_PMCR_P) { - mask &= ~BIT(ARMV8_PMU_CYCLE_IDX); + mask = kvm_pmu_valid_counter_mask(vcpu) + & BIT(ARMV8_PMU_CYCLE_IDX);This looks suspiciously opposite of what it replaces;
It always sets the bit, which goes against the architecture and the code it was replacing, yes.
Quoted 3 lines
however did we even need to do a bitwise operation here in the first place? Couldn't we skip the cycle counter by just limiting the
for_each_set_bititeration below to 31 bits?
To quote myself [1]:
"Entertained the idea of restricting the number of bits in for_each_set_bit() to 31 since Linux (and the architecture, to some degree) treats the cycle count register as the 32nd event counter. Settled on this approach because I think it's clearer."
To expand on that, incorrectly resetting the cycle counter was introduced by a refactoring, so I preferred making it very clear that PMCR_EL0.P is not supposed to clear the cycle counter.
[1] https://lore.kernel.org/kvmarm/[email protected]/
Thanks,
Alex
Quoted 9 lines
Robin.
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +0
-0for_each_set_bit(i, &mask, 32) kvm_pmu_set_counter_value(vcpu, i, 0); }base-commit:
83f870a663592797c576846db3611e0a1664eda2
Quoted 68 lines · Alexandru Elisei
Hi Robin,
Quoted 56 lines · Robin Murphy
Quoted 55 lines · Alexandre Chartre
In a KVM guest on arm64, performance counters interrupts have an unnecessary overhead which slows down execution when using the "perf record" command and limits the "perf record" sampling period.
The problem is that when a guest VM disables counters by clearing the PMCR_EL0.E bit (bit 0), KVM will disable all counters defined in
PMCR_EL0even if they are not enabled inPMCNTENSET_EL0.KVM disables a counter by calling into the perf framework, in particular by calling
perf_event_create_kernel_counter()which is a time consuming operation. So, for example, with a Neoverse N1 CPU core which has 6 event counters and one cycle counter, KVM will always disable all 7 counters even if only one is enabled.This typically happens when using the "perf record" command in a guest
VM: perf will disable all event counters withPMCNTENTSET_EL0and only uses the cycle counter. And when using the "perf record" -F option with a high profiling frequency, the overhead of KVM disabling all counters instead of one on every counter interrupt becomes very noticeable.The problem is fixed by having KVM disable only counters which are enabled in
PMCNTENSET_EL0. If a counter is not enabled inPMCNTENSET_EL0then KVM will not enable it when setting PMCR_EL0.E and it will remain disabled as long as it is not enabled inPMCNTENSET_EL0. So there is effectively no need to disable a counter when clearing PMCR_EL0.E if it is not enabledPMCNTENSET_EL0.Signed-off-by Alexandre Chartre <[email protected]>---
The patch is based on
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/pmu/reset-valuesarch/arm64/kvm/pmu-emul.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)arch/arm64/kvm/pmu-emul.c +3
-2@@ -563,21 +563,23 @@ void kvm_pmu_software_increment(struct kvm_vcpu *vcpu,u64 val) */ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) { - unsigned long mask = kvm_pmu_valid_counter_mask(vcpu); + unsigned long mask; int i; if (val & ARMV8_PMU_PMCR_E) { kvm_pmu_enable_counter_mask(vcpu, __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); } else { - kvm_pmu_disable_counter_mask(vcpu, mask); + kvm_pmu_disable_counter_mask(vcpu, + __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); }Quoted 8 lines
arch/arm64/kvm/pmu-emul.c void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val) · line 582
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +2
-1if (val & ARMV8_PMU_PMCR_C) kvm_pmu_set_counter_value(vcpu, ARMV8_PMU_CYCLE_IDX, 0); if (val & ARMV8_PMU_PMCR_P) { - mask &= ~BIT(ARMV8_PMU_CYCLE_IDX); + mask = kvm_pmu_valid_counter_mask(vcpu) + & BIT(ARMV8_PMU_CYCLE_IDX);This looks suspiciously opposite of what it replaces;
Quoted 3 lines
It always sets the bit, which goes against the architecture and the code it was replacing, yes.
My bad, I screw up and I dropped the ~. I will resend.
Sorry,
alex.
Quoted 15 lines
Quoted 3 lines
however did we even need to do a bitwise operation here in the first place? Couldn't we skip the cycle counter by just limiting the
for_each_set_bititeration below to 31 bits?To quote myself [1]:
"Entertained the idea of restricting the number of bits in
for_each_set_bit()to 31 since Linux (and the architecture, to some degree) treats the cycle count register as the 32nd event counter. Settled on this approach because I think it's clearer."To expand on that, incorrectly resetting the cycle counter was introduced by a refactoring, so I preferred making it very clear that PMCR_EL0.P is not supposed to clear the cycle counter.
Quoted 15 lines
[1] https://lore.kernel.org/kvmarm/[email protected]/
Thanks,
Alex
Quoted 9 lines
Robin.
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +0
-0for_each_set_bit(i, &mask, 32) kvm_pmu_set_counter_value(vcpu, i, 0); }base-commit:
83f870a663592797c576846db3611e0a1664eda2
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 75 lines · Alexandru Elisei
Hi Robin,
Quoted 56 lines · Robin Murphy
Quoted 55 lines · Alexandre Chartre
In a KVM guest on arm64, performance counters interrupts have an unnecessary overhead which slows down execution when using the "perf record" command and limits the "perf record" sampling period.
The problem is that when a guest VM disables counters by clearing the PMCR_EL0.E bit (bit 0), KVM will disable all counters defined in
PMCR_EL0even if they are not enabled inPMCNTENSET_EL0.KVM disables a counter by calling into the perf framework, in particular by calling
perf_event_create_kernel_counter()which is a time consuming operation. So, for example, with a Neoverse N1 CPU core which has 6 event counters and one cycle counter, KVM will always disable all 7 counters even if only one is enabled.This typically happens when using the "perf record" command in a guest
VM: perf will disable all event counters withPMCNTENTSET_EL0and only uses the cycle counter. And when using the "perf record" -F option with a high profiling frequency, the overhead of KVM disabling all counters instead of one on every counter interrupt becomes very noticeable.The problem is fixed by having KVM disable only counters which are enabled in
PMCNTENSET_EL0. If a counter is not enabled inPMCNTENSET_EL0then KVM will not enable it when setting PMCR_EL0.E and it will remain disabled as long as it is not enabled inPMCNTENSET_EL0. So there is effectively no need to disable a counter when clearing PMCR_EL0.E if it is not enabledPMCNTENSET_EL0.Signed-off-by Alexandre Chartre <[email protected]>---
The patch is based on
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/pmu/reset-valuesarch/arm64/kvm/pmu-emul.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)arch/arm64/kvm/pmu-emul.c +3
-2@@ -563,21 +563,23 @@ void kvm_pmu_software_increment(struct kvm_vcpu *vcpu,u64 val) */ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val) { - unsigned long mask = kvm_pmu_valid_counter_mask(vcpu); + unsigned long mask; int i; if (val & ARMV8_PMU_PMCR_E) { kvm_pmu_enable_counter_mask(vcpu, __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); } else { - kvm_pmu_disable_counter_mask(vcpu, mask); + kvm_pmu_disable_counter_mask(vcpu, + __vcpu_sys_reg(vcpu, PMCNTENSET_EL0)); }Quoted 8 lines
arch/arm64/kvm/pmu-emul.c void kvm_pmu_software_increment(struct kvm_vcpu *vcpu, u64 val) · line 582
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +2
-1if (val & ARMV8_PMU_PMCR_C) kvm_pmu_set_counter_value(vcpu, ARMV8_PMU_CYCLE_IDX, 0); if (val & ARMV8_PMU_PMCR_P) { - mask &= ~BIT(ARMV8_PMU_CYCLE_IDX); + mask = kvm_pmu_valid_counter_mask(vcpu) + & BIT(ARMV8_PMU_CYCLE_IDX);This looks suspiciously opposite of what it replaces;
It always sets the bit, which goes against the architecture and the code it was replacing, yes.
Quoted 3 lines
however did we even need to do a bitwise operation here in the first place? Couldn't we skip the cycle counter by just limiting the
for_each_set_bititeration below to 31 bits?
Quoted 5 lines
To quote myself [1]:
"Entertained the idea of restricting the number of bits in
for_each_set_bit()to 31 since Linux (and the architecture, to some degree) treats the cycle count register as the 32nd event counter.
FWIW I wouldn't say there's any degree to it - we're iterating over the bits in a register where the cycle counter enable is unequivocally the 32nd bit.
Quoted 6 lines
Settled on this approach because I think it's
clearer."To expand on that, incorrectly resetting the cycle counter was introduced by a refactoring, so I preferred making it very clear that PMCR_EL0.P is not supposed to clear the cycle counter.
Fair enough, but if this has turned out to be a contentious hot path then masking the bit to zero and then deliberately iterating to see if it's set (find_next_bit() isn't exactly free) adds up to more overhead than a comment ;)
Robin.
Quoted 16 lines
[1] https://lore.kernel.org/kvmarm/[email protected]/
Thanks,
Alex
Quoted 9 lines
Robin.
Quoted 6 lines
arch/arm64/kvm/pmu-emul.c +0
-0for_each_set_bit(i, &mask, 32) kvm_pmu_set_counter_value(vcpu, i, 0); }base-commit:
83f870a663592797c576846db3611e0a1664eda2