arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path
PATCH v2 1/1 arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path
Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].
Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
---
Changes since v1:
- Rely on read_sanitised_ftr_reg instead of static var (will)
--- arch/arm64/include/asm/cpufeature.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +5 -1
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false; - mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }
Signature · 7 lines
2.35.3 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 4 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].
I am just curious, is this the only system register access (AA64MMFR1_EL1) causing such performance problems ?
Quoted 4 lines
Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround
Right, although cpu_has_hw_af() was added earlier via commit 47d7b15b88f9 ("arm64: cpufeature: introduce helper cpu_has_hw_af()"), but above commit did add this on regular page fault path via do_set_pte().
Quoted 2 lines
pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an
Right, accessing a system register on each page fault is not optimal.
Quoted 27 lines
explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>---
Changes since v1:
- Rely onread_sanitised_ftr_reginstead of static var (will)--- arch/arm64/include/asm/cpufeature.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/arm64/include/asm/cpufeature.h +0
-0@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false;
arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874
Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5
-1- mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }
LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 8 lines · Anshuman Khandual
Quoted 4 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].I am just curious, is this the only system register access (
AA64MMFR1_EL1) causing such performance problems ?
I haven't audited all the system registers. For AA64MMFR1_EL1 this is the only instance where the frequency of access affects performance in a meaningful way for my workload.
I have a real-world bug report about it, and by profiling vm exit events, I can also argue this is the only instance of any emulated msr read/write that happens frequently enough to change the order of magnitude of exit events measured by perf for my workload between, at least, 5.4 (it was introduced in v5.12, but I have data back to 5.4) and mainline.
Quoted 7 lines
Quoted 3 lines
Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaroundRight, although
cpu_has_hw_af()was added earlier via commit47d7b15b88f9("arm64: cpufeature: introduce helpercpu_has_hw_af()"), but above commit did add this on regular page fault path viado_set_pte().
Indeed. The only other usage of this function is in wp_page_copy, and, from what I can tell, it is in an unlikely() branch when COW is being performed on a page that was recently unmapped. It is not something frequent enough that I saw in profiling.
Quoted 5 lines
Quoted 2 lines
ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
}LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?
I think a full audit of the emulated system registers in kvm will be required to definitely answer it. But this instance is, by far, the hottest case in the codebase.
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 38 lines · Gabriel Krisman Bertazi
Quoted 8 lines · Anshuman Khandual
Quoted 4 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].I am just curious, is this the only system register access (
AA64MMFR1_EL1) causing such performance problems ?I haven't audited all the system registers. For
AA64MMFR1_EL1this is the only instance where the frequency of access affects performance in a meaningful way for my workload.I have a real-world bug report about it, and by profiling vm exit events, I can also argue this is the only instance of any emulated msr read/write that happens frequently enough to change the order of magnitude of exit events measured by perf for my workload between, at least, 5.4 (it was introduced in v5.12, but I have data back to 5.4) and mainline.
Quoted 7 lines
Quoted 3 lines
Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaroundRight, although
cpu_has_hw_af()was added earlier via commit47d7b15b88f9("arm64: cpufeature: introduce helpercpu_has_hw_af()"), but above commit did add this on regular page fault path viado_set_pte().Indeed. The only other usage of this function is in
wp_page_copy, and, from what I can tell, it is in anunlikely()branch when COW is being performed on a page that was recently unmapped. It is not something frequent enough that I saw in profiling.Quoted 3 lines
Quoted 2 lines
ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
}
Quoted 7 lines
Quoted 2 lines
LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?
I think a full audit of the emulated system registers in kvm will be required to definitely answer it. But this instance is, by far, the hottest case in the codebase.
Thanks for the additional details.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 15 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
Quoted 5 lines
This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
A small nit. This URL length causes a checkpatch.pl warning.
WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line)
Just wondering, if this URL can be replaced with a document identification number or something similar for a more cleaner commit message, not sure if such a thing is even feasible.
Quoted 17 lines
Signed-off-by Gabriel Krisman Bertazi <[email protected]>---
Changes since v1:
- Rely onread_sanitised_ftr_reginstead of static var (will)--- arch/arm64/include/asm/cpufeature.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/arm64/include/asm/cpufeature.h +0
-0@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false;
arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874
Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5
-1- mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }
FWIW
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 15 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
Quoted 7 lines
This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Hi,
Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerQuoted 21 lines · Gabriel Krisman Bertazi
Quoted 19 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
Quoted 8 lines
Quoted 3 lines
Signed-off-by Gabriel Krisman Bertazi <[email protected]>Hi,
Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?
Given this has been the case since v5.12 afaict and it's not a correctness issue, I was thinking we could queue this for 6.3?
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 7 lines · Will Deacon
Quoted 3 lines · Gabriel Krisman Bertazi
Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?
Given this has been the case since v5.12 afaict and it's not a correctness issue, I was thinking we could queue this for 6.3?
hey Will,
On a second thought, I'm fine with having it only in 6.3. :)
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Catalin Marinas
maintainerQuoted 7 lines · Gabriel Krisman Bertazi
arch/arm64/include/asm/cpufeature.h +0
-0@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false;
arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874
Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5
-1- mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }
You can probably make it slightly faster if you add a cpucap feature. We have ARM64_HW_DBM but not ARM64_HW_AF. Not sure you'd notice a difference in practice though.
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 9 lines · Catalin Marinas
Quoted 8 lines · Gabriel Krisman Bertazi
arch/arm64/include/asm/cpufeature.h +0
-0@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false;
Quoted 13 lines
arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874
Quoted 9 lines
arch/arm64/include/asm/cpufeature.h +5
-1- mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }You can probably make it slightly faster if you add a cpucap feature. We have
ARM64_HW_DBMbut notARM64_HW_AF. Not sure you'd notice a difference in practice though.
Hi Catalin,
I had a patch for that, but when I noticed the existence of ARM64_HW_DBM, I dropped it. it is mentioned quickly in the v1.
From my measurements, when comparing this version to a static variable caching the value (patch v1), the bsearch performance impact was not significant. I think it would be fine to keep this version.
but, of course, I can add the cpucap bit if it is preferred. Please, let me know.
Thanks,
Signature · 6 lines
Gabriel Krisman Bertazi _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 37 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.
Signed-off-by Gabriel Krisman Bertazi <[email protected]>---
Changes since v1:
- Rely onread_sanitised_ftr_reginstead of static var (will)--- arch/arm64/include/asm/cpufeature.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/arm64/include/asm/cpufeature.h +0
-0@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)if (!IS_ENABLED (CONFIG_ARM64_HW_AFDBM ))return false;
arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874
Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5
-1- mmfr1 =read_cpuid (ID_AA64MMFR1_EL1 ); +/* +* Use cached version to avoid emulated msr operation on KVM +* guests. +*/ + mmfr1 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR1_EL1 );return cpuid_feature_extract_unsigned_field (mmfr1,ID_AA64MMFR1_EL1_HAFDBS_SHIFT ); }
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 15 lines · Gabriel Krisman Bertazi
Accessing
AA64MMFR1_EL1is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].Commit
0388f9c74330("arm64: mm: Implementarch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.
Quoted 1 line
[...]
Applied to arm64 (for-next/misc), thanks!
[1/1] arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path
https://git.kernel.org/arm64/c/a89c6bcdac22
Signature · 7 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel