HCPTR cp15 writes need isb?
Subsystems (2)
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in set_hcptr macro in arch/arm/kvm/interrupts_head.S I am able to boot the Aarch32 guest, but without it, I see strange crashes to hyp_undef or hyp_pabt.
What is your opinion on HCPTR cp15 writes being context changing and needing isb? I can submit a one line patch that adds the isb at the end of set_hcptr macro.
I did find some examples in ARMv7 ARM with an isb after cp15 write and an ARM blog [1] that states ISB "is used to ensure any previously executed context changing operations (including cp15 operations) will have completed by the time the ISB completed."
I also found this text from an older lkml thread [2] which seems to quote from an ARM document (although I cannot find the same text in ARMv7 ARM) "Any change to CP15 registers is guaranteed to be visible to subsequent instructions only after one of isb, exception, return from exception"
1 http://community.arm.com/groups/processors/blog/2011/10/19/memory-access-ordering-part-3--memory-access-ordering-in-the-arm-architecture
2 http://lkml.iu.edu/hypermail/linux/kernel/1105.1/00475.html
Thanks,
Vikram
Signature · 3 lines
Vikram Sethi Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:
Hi Vikram,
Quoted 8 lines
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in
set_hcptrmacro inarch/arm/kvm/interrupts_head.SI am able to boot the Aarch32 guest, but without it, I see strange crashes tohyp_undeforhyp_pabt.
[...]
Can you look at the following patch (queued for 4.2)?
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html
Please let me know if this solves the issue you are seeing.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 13 lines · Marc Zyngier
On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:Hi Vikram,
Quoted 8 lines
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in
set_hcptrmacro inarch/arm/kvm/interrupts_head.SI am able to boot the Aarch32 guest, but without it, I see strange crashes tohyp_undeforhyp_pabt.
Quoted 7 lines
[...]
Can you look at the following patch (queued for 4.2)?
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html
Please let me know if this solves the issue you are seeing.
Don't we have the same issue the first time guest touches FP and traps i.e in switch_to_guest_vfp where we turn on floating point access in HCPTR and immediately access FPEXC in store_vfp_state without a isb?
My first attempt at a fix was similar to yours (add isb only in kvm_vcpu_return path after hcptr update) and while that helped the guest boot further, I still got hyp prefetch abort (hyp_pabt) later in the guest boot, until I also added isbs after the other HCPTR writes.
Quoted 4 lines
Thanks,
M.
Signature · 3 lines
Vikram Sethi Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 18 lines · Vikram Sethi
Quoted 17 lines · Marc Zyngier
On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:Hi Vikram,
Quoted 8 lines
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in
set_hcptrmacro inarch/arm/kvm/interrupts_head.SI am able to boot the Aarch32 guest, but without it, I see strange crashes tohyp_undeforhyp_pabt.[...]
Can you look at the following patch (queued for 4.2)?
Quoted 7 lines
Quoted 3 lines
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html
Please let me know if this solves the issue you are seeing.
Don't we have the same issue the first time guest touches FP and traps i.e in
switch_to_guest_vfpwhere we turn on floating point access in HCPTR and immediately access FPEXC instore_vfp_statewithout a isb?
Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?
arch/arm/kvm/interrupts.S +1 -0
@@ -487,6 +487,7 @@ switch_to_guest_vfp: @ NEON/VFP used. Turn on VFP access. set_hcptr vmexit, (HCPTR_TCP(10) | HCPTR_TCP(11)) + isb @ Force execution of HCPTR as we've just reenabled VFP access @ Switch VFP/NEON hardware state to the guest's add r7, r0, #VCPU_VFP_HOST
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 21 lines · Marc Zyngier
Quoted 20 lines · Vikram Sethi
Quoted 19 lines · Marc Zyngier
On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:Hi Vikram,
Quoted 8 lines
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in
set_hcptrmacro inarch/arm/kvm/interrupts_head.SI am able to boot the Aarch32 guest, but without it, I see strange crashes tohyp_undeforhyp_pabt.[...]
Can you look at the following patch (queued for 4.2)?
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Please let me know if this solves the issue you are seeing.
Don't we have the same issue the first time guest touches FP and traps i.e in
switch_to_guest_vfpwhere we turn on floating point access in HCPTR and immediately access FPEXC instore_vfp_statewithout a isb?Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?
Yes, the additional isb in switch_to_guest_vfp along with your original patch works for me.
When you refactor the original patch will it be cleaner to handle the isb in the set_hcptr macro whenever it is changed to not trap VFP access?
Quoted 17 lines
arch/arm/kvm/interrupts.S +1
-0@@ -487,6 +487,7 @@ switch_to_guest_vfp: @ NEON/VFP used. Turn on VFP access. set_hcptr vmexit, (HCPTR_TCP(10) | HCPTR_TCP(11)) + isb @ Force execution of HCPTR as we've just reenabled VFP access @ Switch VFP/NEON hardware state to the guest's add r7, r0, #VCPU_VFP_HOSTThanks,
M.
Signature · 3 lines
Vikram Sethi Qualcomm Technologies Inc, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 27 lines · Vikram Sethi
Quoted 26 lines · Marc Zyngier
Quoted 25 lines · Vikram Sethi
Quoted 13 lines · Marc Zyngier
On Tue, 16 Jun 2015 02:34:23 +0100
Vikram Sethi <[email protected]> wrote:Hi Vikram,
Quoted 8 lines
Hi Marc, Christoffer, Catalin, Will,
I'm seeing an issue with KVM HCPTR (cp15) writes on guest entry/exit on one of Qualcomm's CPU cores in AArch32 host and AArch32 guest mode. Our CPU architects believe that HCPTR cp15 writes are context changing and require an isb. With an isb in
set_hcptrmacro inarch/arm/kvm/interrupts_head.SI am able to boot the Aarch32 guest, but without it, I see strange crashes tohyp_undeforhyp_pabt.Quoted 7 lines
[...]
Can you look at the following patch (queued for 4.2)?
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/330955.html
Please let me know if this solves the issue you are seeing.
Don't we have the same issue the first time guest touches FP and traps i.e in
switch_to_guest_vfpwhere we turn on floating point access in HCPTR and immediately access FPEXC instore_vfp_statewithout a isb?
Quoted 6 lines
Quoted 2 lines
Good point, looks like my initial fix is incomplete. I'll repost a more complete fix but in the meantime, does adding the following work for you?
Yes, the additional isb in
switch_to_guest_vfpalong with your original patch works for me. When you refactor the original patch will it be cleaner to handle the
isb in theset_hcptrmacro whenever it is changed to not trap VFP
access?
That's what I have done, but the result is a bit awkward, so I'm in two minds about it. I'll post it in a minute, please check that it still works for you (though I've checked that the generated code is the same).
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...