Skip to contentSkip to conversation

HCPTR cp15 writes need isb?

DISCUSSION Vikram Sethi 2 participants 6 messages Jun 16, 2015 – Jun 17, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

What is your opinion on HCPTR cp15 writes being context changing and needing isb? I can submit a one line patch that adds the isb at the end of set_hcptr macro.

I did find some examples in ARMv7 ARM with an isb after cp15 write and an ARM blog [1] that states ISB "is used to ensure any previously executed context changing operations (including cp15 operations) will have completed by the time the ISB completed."

I also found this text from an older lkml thread [2] which seems to quote from an ARM document (although I cannot find the same text in ARMv7 ARM) "Any change to CP15 registers is guaranteed to be visible to subsequent instructions only after one of isb, exception, return from exception"

1 http://community.arm.com/groups/processors/blog/2011/10/19/memory-access-ordering-part-3--memory-access-ordering-in-the-arm-architecture
2 http://lkml.iu.edu/hypermail/linux/kernel/1105.1/00475.html

Thanks,
Vikram

Signature · 3 lines
Vikram Sethi
Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
#1 lore raw

On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:

Hi Vikram,

Quoted 8 lines

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

[...]

Can you look at the following patch (queued for 4.2)?

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html

Please let me know if this solves the issue you are seeing.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#2 lore raw
Quoted 13 lines · Marc Zyngier

On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:

Hi Vikram,

Quoted 8 lines

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

Quoted 7 lines

[...]

Can you look at the following patch (queued for 4.2)?

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html

Please let me know if this solves the issue you are seeing.

Don't we have the same issue the first time guest touches FP and traps i.e in switch_to_guest_vfp where we turn on floating point access in HCPTR and immediately access FPEXC in store_vfp_state without a isb?

My first attempt at a fix was similar to yours (add isb only in kvm_vcpu_return path after hcptr update) and while that helped the guest boot further, I still got hyp prefetch abort (hyp_pabt) later in the guest boot, until I also added isbs after the other HCPTR writes.

Quoted 4 lines

Thanks,

	M.
Signature · 3 lines
Vikram Sethi
Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
#3 lore raw
Quoted 18 lines · Vikram Sethi
Quoted 17 lines · Marc Zyngier

On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:

Hi Vikram,

Quoted 8 lines

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

[...]

Can you look at the following patch (queued for 4.2)?

Quoted 7 lines
Quoted 3 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html

Please let me know if this solves the issue you are seeing.

Don't we have the same issue the first time guest touches FP and traps i.e in switch_to_guest_vfp where we turn on floating point access in HCPTR and immediately access FPEXC in store_vfp_state without a isb?

Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?

arch/arm/kvm/interrupts.S +1 -0
index 3ac7aca..5b30047 100644
@@ -487,6 +487,7 @@ switch_to_guest_vfp:
 
 	@ NEON/VFP used.  Turn on VFP access.
 	set_hcptr vmexit, (HCPTR_TCP(10) | HCPTR_TCP(11))
+	isb	@ Force execution of HCPTR as we've just reenabled VFP access
 
 	@ Switch VFP/NEON hardware state to the guest's
 	add	r7, r0, #VCPU_VFP_HOST

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#4 lore raw
Quoted 21 lines · Marc Zyngier
Quoted 20 lines · Vikram Sethi
Quoted 19 lines · Marc Zyngier

On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:

Hi Vikram,

Quoted 8 lines

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

[...]

Can you look at the following patch (queued for 4.2)?

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Please let me know if this solves the issue you are seeing.

Don't we have the same issue the first time guest touches FP and traps i.e in switch_to_guest_vfp where we turn on floating point access in HCPTR and immediately access FPEXC in store_vfp_state without a isb?

Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?

Yes, the additional isb in switch_to_guest_vfp along with your original patch works for me.
When you refactor the original patch will it be cleaner to handle the isb in the set_hcptr macro whenever it is changed to not trap VFP access?

Quoted 17 lines
arch/arm/kvm/interrupts.S +1 -0
index 3ac7aca..5b30047 100644
@@ -487,6 +487,7 @@ switch_to_guest_vfp:
 
 	@ NEON/VFP used.  Turn on VFP access.
 	set_hcptr vmexit, (HCPTR_TCP(10) | HCPTR_TCP(11))
+	isb	@ Force execution of HCPTR as we've just reenabled VFP access
 
 	@ Switch VFP/NEON hardware state to the guest's
 	add	r7, r0, #VCPU_VFP_HOST

Thanks,

	M.
Signature · 3 lines
Vikram Sethi
Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
#5 lore raw
Quoted 27 lines · Vikram Sethi
Quoted 26 lines · Marc Zyngier
Quoted 25 lines · Vikram Sethi
Quoted 13 lines · Marc Zyngier

On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:

Hi Vikram,

Quoted 8 lines

Hi Marc, Christoffer, Catalin, Will,

I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.

Quoted 7 lines

[...]

Can you look at the following patch (queued for 4.2)?

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html

Please let me know if this solves the issue you are seeing.

Don't we have the same issue the first time guest touches FP and traps i.e in switch_to_guest_vfp where we turn on floating point access in HCPTR and immediately access FPEXC in store_vfp_state without a isb?

Quoted 6 lines
Quoted 2 lines

Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?

Yes, the additional isb in switch_to_guest_vfp along with your original patch works for me. When you refactor the original patch will it be cleaner to handle the
isb in the set_hcptr macro whenever it is changed to not trap VFP
access?

That's what I have done, but the result is a bit awkward, so I'm in two minds about it. I'll post it in a minute, please check that it still works for you (though I've checked that the generated code is the same).

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...