Skip to contentSkip to conversation

arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path

PATCH Applied Gabriel Krisman Bertazi 4 participants 12 messages Jan 9, 2023 – Jan 20, 2023 arm64

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

---
Changes since v1:
- Rely on read_sanitised_ftr_reg instead of static var (will)

---
 arch/arm64/include/asm/cpufeature.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +5 -1
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;
 
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }
Signature · 7 lines
2.35.3


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#1 lore raw
Quoted 4 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

I am just curious, is this the only system register access (AA64MMFR1_EL1) causing such performance problems ?

Quoted 4 lines

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround

Right, although cpu_has_hw_af() was added earlier via commit 47d7b15b88f9 ("arm64: cpufeature: introduce helper cpu_has_hw_af()"), but above commit did add this on regular page fault path via do_set_pte().

Quoted 2 lines

pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an

Right, accessing a system register on each page fault is not optimal.

Quoted 27 lines

explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

---
Changes since v1:
- Rely on read_sanitised_ftr_reg instead of static var (will)

---
 arch/arm64/include/asm/cpufeature.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +0 -0
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;

arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874

Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5 -1
 
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }

LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#2 lore raw
Quoted 8 lines · Anshuman Khandual
Quoted 4 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

I am just curious, is this the only system register access (AA64MMFR1_EL1) causing such performance problems ?

I haven't audited all the system registers. For AA64MMFR1_EL1 this is the only instance where the frequency of access affects performance in a meaningful way for my workload.

I have a real-world bug report about it, and by profiling vm exit events, I can also argue this is the only instance of any emulated msr read/write that happens frequently enough to change the order of magnitude of exit events measured by perf for my workload between, at least, 5.4 (it was introduced in v5.12, but I have data back to 5.4) and mainline.

Quoted 7 lines
Quoted 3 lines

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround

Right, although cpu_has_hw_af() was added earlier via commit 47d7b15b88f9 ("arm64: cpufeature: introduce helper cpu_has_hw_af()"), but above commit did add this on regular page fault path via do_set_pte().

Indeed. The only other usage of this function is in wp_page_copy, and, from what I can tell, it is in an unlikely() branch when COW is being performed on a page that was recently unmapped. It is not something frequent enough that I saw in profiling.

Quoted 5 lines
Quoted 2 lines

ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
}

LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?

I think a full audit of the emulated system registers in kvm will be required to definitely answer it. But this instance is, by far, the hottest case in the codebase.

Signature · 6 lines
Gabriel Krisman Bertazi

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#3 lore raw
Quoted 38 lines · Gabriel Krisman Bertazi
Quoted 8 lines · Anshuman Khandual
Quoted 4 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

I am just curious, is this the only system register access (AA64MMFR1_EL1) causing such performance problems ?

I haven't audited all the system registers. For AA64MMFR1_EL1 this is the only instance where the frequency of access affects performance in a meaningful way for my workload.

I have a real-world bug report about it, and by profiling vm exit events, I can also argue this is the only instance of any emulated msr read/write that happens frequently enough to change the order of magnitude of exit events measured by perf for my workload between, at least, 5.4 (it was introduced in v5.12, but I have data back to 5.4) and mainline.

Quoted 7 lines
Quoted 3 lines

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround

Right, although cpu_has_hw_af() was added earlier via commit 47d7b15b88f9 ("arm64: cpufeature: introduce helper cpu_has_hw_af()"), but above commit did add this on regular page fault path via do_set_pte().

Indeed. The only other usage of this function is in wp_page_copy, and, from what I can tell, it is in an unlikely() branch when COW is being performed on a page that was recently unmapped. It is not something frequent enough that I saw in profiling.

Quoted 3 lines
Quoted 2 lines

ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
}

Quoted 7 lines
Quoted 2 lines

LGTM but as mentioned earlier, are there not other similar instances or this is just more problematic being on direct page fault path ?

I think a full audit of the emulated system registers in kvm will be required to definitely answer it. But this instance is, by far, the hottest case in the codebase.

Thanks for the additional details.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Reviewed-by on v2 1/1
#4 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 15 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

Quoted 5 lines

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

A small nit. This URL length causes a checkpatch.pl warning.

WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line)

Just wondering, if this URL can be replaced with a document identification number or something similar for a more cleaner commit message, not sure if such a thing is even feasible.

Quoted 17 lines

---
Changes since v1:
- Rely on read_sanitised_ftr_reg instead of static var (will)

---
 arch/arm64/include/asm/cpufeature.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +0 -0
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;

arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874

Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5 -1
 
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }

FWIW

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#5 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 15 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

Quoted 7 lines

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

Hi,

Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?

Signature · 6 lines
Gabriel Krisman Bertazi

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#6 lore raw
Quoted 21 lines · Gabriel Krisman Bertazi
Quoted 19 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

Quoted 8 lines
Quoted 3 lines

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

Hi,

Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?

Given this has been the case since v5.12 afaict and it's not a correctness issue, I was thinking we could queue this for 6.3?

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#7 lore raw
Quoted 7 lines · Will Deacon
Quoted 3 lines · Gabriel Krisman Bertazi

Considering the performance impact on kvm guests, unless someone opposes, can we get this queued already for -rc5?

Given this has been the case since v5.12 afaict and it's not a correctness issue, I was thinking we could queue this for 6.3?

hey Will,

On a second thought, I'm fine with having it only in 6.3. :)

Signature · 6 lines
Gabriel Krisman Bertazi

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#8 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 7 lines · Gabriel Krisman Bertazi
arch/arm64/include/asm/cpufeature.h +0 -0
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;

arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874

Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5 -1
 
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }

You can probably make it slightly faster if you add a cpucap feature. We have ARM64_HW_DBM but not ARM64_HW_AF. Not sure you'd notice a difference in practice though.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#9 lore raw
Quoted 9 lines · Catalin Marinas
Quoted 8 lines · Gabriel Krisman Bertazi
arch/arm64/include/asm/cpufeature.h +0 -0
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;
 
Quoted 13 lines

arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874

Quoted 9 lines
arch/arm64/include/asm/cpufeature.h +5 -1
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }

You can probably make it slightly faster if you add a cpucap feature. We have ARM64_HW_DBM but not ARM64_HW_AF. Not sure you'd notice a difference in practice though.

Hi Catalin,

I had a patch for that, but when I noticed the existence of ARM64_HW_DBM, I dropped it. it is mentioned quickly in the v1.

From my measurements, when comparing this version to a static variable caching the value (patch v1), the bsearch performance impact was not significant. I think it would be fine to keep this version.

but, of course, I can add the cpucap bit if it is preferred. Please, let me know.

Thanks,

Signature · 6 lines
Gabriel Krisman Bertazi

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer Acked-by on v2 1/1
#10 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 37 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

This patch avoids the extra cost by using the sanitized cached value. It should be safe to do so, since this register mustn't change for a given cpu.

[1] https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Learn%20the%20Architecture/Armv8-A%20virtualization.pdf?revision=a765a7df-1a00-434d-b241-357bfda2dd31

---
Changes since v1:
- Rely on read_sanitised_ftr_reg instead of static var (will)

---
 arch/arm64/include/asm/cpufeature.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +0 -0
index 03d1c9d7af82..3d1a5677321e 100644
@@ -864,7 +864,11 @@ static inline bool cpu_has_hw_af(void)
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;

arch/arm64/include/asm/cpufeature.h static inline bool cpu_has_hw_af(void) · line 874

Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +5 -1
 
-	mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	/*
+	 * Use cached version to avoid emulated msr operation on KVM
+	 * guests.
+	 */
+	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer Applied
#11 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 15 lines · Gabriel Krisman Bertazi

Accessing AA64MMFR1_EL1 is expensive in KVM guests, since it is emulated in the hypervisor. In fact, ARM documentation mentions some feature registers are not supposed to be accessed frequently by the OS, and therefore should be emulated for guests [1].

Commit 0388f9c74330 ("arm64: mm: Implement arch_wants_old_prefaulted_pte()") introduced a read of this register in the page fault path. But, even when the feature of setting faultaround pages with the old flag is disabled for a given cpu, we are still paying the cost of checking the register on every pagefault. This results in an explosion of vmexit events in KVM guests, which directly impacts the performance of virtualized workloads. For instance, running kernbench yields a 15% increase in system time solely due to the increased vmexit cycles.

Quoted 1 line

[...]

Applied to arm64 (for-next/misc), thanks!

[1/1] arm64: Avoid repeated AA64MMFR1_EL1 register read on pagefault path

      https://git.kernel.org/arm64/c/a89c6bcdac22
Signature · 7 lines
Catalin


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel