Cortex-A77 erratum 1508412 workaround

Quoted 4 lines · Rob Herring

arch/arm64/include/asm/sysreg.h +0 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 

arch/arm64/include/asm/sysreg.h line 989

Quoted 7 lines

arch/arm64/include/asm/sysreg.h +7 -0

+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 5 lines · Catalin Marinas

Quoted 4 lines · Rob Herring

arch/arm64/include/asm/sysreg.h +0 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 

Quoted 12 lines

arch/arm64/include/asm/sysreg.h line 989

Quoted 7 lines

arch/arm64/include/asm/sysreg.h +7 -0

+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Disabling irqs around the PAR access would be simpler, I think (assuming this is needed).

Quoted 19 lines · Will Deacon

Quoted 17 lines · Catalin Marinas

arch/arm64/include/asm/sysreg.h line 989

Quoted 11 lines · Rob Herring

arch/arm64/include/asm/sysreg.h +7 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 
+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 2 lines

Disabling irqs around the PAR access would be simpler, I think (assuming this is needed).

This wouldn't work if it interrupts a guest.

Quoted 20 lines · Catalin Marinas

Quoted 19 lines · Will Deacon

Quoted 17 lines · Catalin Marinas

Review line 989

Quoted 11 lines · Rob Herring

(quoted diff) +7 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 
+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 5 lines

Quoted 3 lines

Disabling irqs around the PAR access would be simpler, I think (assuming
this is needed).

This wouldn't work if it interrupts a guest.

If we take an interrupt either side of the PAR_EL1 read and that we fully exit, the saving of PAR_EL1 on the way out solves the problem.

If we don't fully exit, but instead reenter the guest immediately (fixup_guest_exit() returns true), we'd need a DMB at that point, at least because of the GICv2 proxying code which performs device accesses on the guest's behalf.

Quoted 27 lines · Marc Zyngier

Quoted 20 lines · Catalin Marinas

Quoted 19 lines · Will Deacon

Quoted 17 lines · Catalin Marinas

Review line 989

Quoted 11 lines · Rob Herring

(quoted diff) +7 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 
+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 5 lines

Quoted 3 lines

Disabling irqs around the PAR access would be simpler, I think (assuming
this is needed).

This wouldn't work if it interrupts a guest.

Quoted 7 lines

If we take an interrupt either side of the PAR_EL1 read and that we fully exit, the saving of PAR_EL1 on the way out solves the problem.

If we don't fully exit, but instead reenter the guest immediately (fixup_guest_exit() returns true), we'd need a DMB at that point, at least because of the GICv2 proxying code which performs device accesses on the guest's behalf.

If you are ok with the diff below, I can fold it in:

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

index ca88ea416176..8770cf7ccd42 100644
@@ -420,7 +420,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
 	    kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 &&
 	    handle_tx2_tvm(vcpu))
-		return true;
+		goto guest;
 
 	/*

arch/arm64/kvm/hyp/include/hyp/switch.h +3 -3

 	 * We trap the first access to the FP/SIMD to save the host context

@@ -430,13 +430,13 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 	 * Similarly for trapped SVE accesses.
 	 */
 	if (__hyp_handle_fpsimd(vcpu))
-		return true;
+		goto guest;
 
 	if (__hyp_handle_ptrauth(vcpu))
-		return true;
+		goto guest;
 
 	if (!__populate_fault_info(vcpu))
-		return true;
+		goto guest;
 
 	if (static_branch_unlikely(&vgic_v2_cpuif_trap)) {

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

 		bool valid;

@@ -451,7 +451,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 			int ret = __vgic_v2_perform_cpuif_access(vcpu);
 
 			if (ret == 1)
-				return true;
+				goto guest;
 
 			/* Promote an illegal access to an SError.*/

arch/arm64/kvm/hyp/include/hyp/switch.h +6 -1

 			if (ret == -1)

@@ -467,12 +467,17 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 		int ret = __vgic_v3_perform_cpuif_access(vcpu);
 
 		if (ret == 1)
-			return true;
+			goto guest;
 	}
 
 exit:
 	/* Return to the host kernel and handle the exit */
 	return false;
+
+guest:
+	/* Re-enter the guest */
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));
+	return true;
 }
 

static inline bool __needs_ssbd_off(struct kvm_vcpu *vcpu)

Quoted 100 lines · Catalin Marinas

Quoted 27 lines · Marc Zyngier

Quoted 20 lines · Catalin Marinas

Quoted 19 lines · Will Deacon

Quoted 17 lines · Catalin Marinas

Review line 989

Quoted 11 lines · Rob Herring

(quoted diff) +7 -0

@@ -979,6 +980,14 @@
 		write_sysreg(__scs_new, sysreg);			\
 } while (0)
 
+#define read_sysreg_par() ({						\
+	u64 par;							\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par = read_sysreg(par_el1);					\
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));	\
+	par;								\
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 5 lines

Quoted 3 lines

Disabling irqs around the PAR access would be simpler, I think (assuming
this is needed).

This wouldn't work if it interrupts a guest.

Quoted 7 lines

If we take an interrupt either side of the PAR_EL1 read and that we fully exit, the saving of PAR_EL1 on the way out solves the problem.

If we don't fully exit, but instead reenter the guest immediately (fixup_guest_exit() returns true), we'd need a DMB at that point, at least because of the GICv2 proxying code which performs device accesses on the guest's behalf.

If you are ok with the diff below, I can fold it in:

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

index ca88ea416176..8770cf7ccd42 100644
@@ -420,7 +420,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
 	    kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 &&
 	    handle_tx2_tvm(vcpu))
-		return true;
+		goto guest;
 
 	/*

arch/arm64/kvm/hyp/include/hyp/switch.h +3 -3

 	 * We trap the first access to the FP/SIMD to save the host context

@@ -430,13 +430,13 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 	 * Similarly for trapped SVE accesses.
 	 */
 	if (__hyp_handle_fpsimd(vcpu))
-		return true;
+		goto guest;
 
 	if (__hyp_handle_ptrauth(vcpu))
-		return true;
+		goto guest;
 
 	if (!__populate_fault_info(vcpu))
-		return true;
+		goto guest;
 
 	if (static_branch_unlikely(&vgic_v2_cpuif_trap)) {

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

 		bool valid;

@@ -451,7 +451,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 			int ret = __vgic_v2_perform_cpuif_access(vcpu);
 
 			if (ret == 1)
-				return true;
+				goto guest;
 
 			/* Promote an illegal access to an SError.*/

arch/arm64/kvm/hyp/include/hyp/switch.h +6 -1

 			if (ret == -1)

@@ -467,12 +467,17 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
 		int ret = __vgic_v3_perform_cpuif_access(vcpu);
 
 		if (ret == 1)
-			return true;
+			goto guest;
 	}
 
 exit:
 	/* Return to the host kernel and handle the exit */
 	return false;
+
+guest:
+	/* Re-enter the guest */
+	asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));
+	return true;
 }
 

Quoted 1 line

static inline bool __needs_ssbd_off(struct kvm_vcpu *vcpu)

Looks good to me!

Quoted 101 lines · Marc Zyngier

Quoted 100 lines · Catalin Marinas

Quoted 27 lines · Marc Zyngier

Quoted 20 lines · Catalin Marinas

Quoted 19 lines · Will Deacon

Quoted 17 lines · Catalin Marinas

Review line 989

Quoted 11 lines · Rob Herring

(quoted diff) +7 -0

@@ -979,6 +980,14 @@
          write_sysreg(__scs_new, sysreg);                        \
 } while (0)
 
+#define read_sysreg_par() ({                                             \
+ u64 par;                                                        \
+ asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));    \
+ par = read_sysreg(par_el1);                                     \
+ asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));    \
+ par;                                                            \
+})

I was about to queue this up but one more point to clarify: can we get an interrupt at either side of the PAR_EL1 read and the handler do a device read, triggering the erratum? Do we need a DMB at exception entry/return?

Quoted 5 lines

Quoted 3 lines

Disabling irqs around the PAR access would be simpler, I think (assuming
this is needed).

This wouldn't work if it interrupts a guest.

Quoted 7 lines

If we take an interrupt either side of the PAR_EL1 read and that we fully exit, the saving of PAR_EL1 on the way out solves the problem.

If we don't fully exit, but instead reenter the guest immediately (fixup_guest_exit() returns true), we'd need a DMB at that point, at least because of the GICv2 proxying code which performs device accesses on the guest's behalf.

If you are ok with the diff below, I can fold it in:

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

index ca88ea416176..8770cf7ccd42 100644
@@ -420,7 +420,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
    if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
        kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 &&
        handle_tx2_tvm(vcpu))
-           return true;
+           goto guest;
 
    /*

* We trap the first access to the FP/SIMD to save the host context

arch/arm64/kvm/hyp/include/hyp/switch.h +3 -3

@@ -430,13 +430,13 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
     * Similarly for trapped SVE accesses.
     */
    if (__hyp_handle_fpsimd(vcpu))
-           return true;
+           goto guest;
 
    if (__hyp_handle_ptrauth(vcpu))
-           return true;
+           goto guest;
 
    if (!__populate_fault_info(vcpu))
-           return true;
+           goto guest;
 
    if (static_branch_unlikely(&vgic_v2_cpuif_trap)) {

            bool valid;

arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1

@@ -451,7 +451,7 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
                    int ret = __vgic_v2_perform_cpuif_access(vcpu);
 
                    if (ret == 1)
-                           return true;
+                           goto guest;
 
                    /* Promote an illegal access to an SError.*/

                    if (ret == -1)

arch/arm64/kvm/hyp/include/hyp/switch.h +6 -1

@@ -467,12 +467,17 @@ static inline bool fixup_guest_exit(struct
 kvm_vcpu *vcpu, u64 *exit_code)
            int ret = __vgic_v3_perform_cpuif_access(vcpu);
 
            if (ret == 1)
-                   return true;
+                   goto guest;
    }
 
 exit:
    /* Return to the host kernel and handle the exit */
    return false;
+
+guest:
+   /* Re-enter the guest */
+   asm(ALTERNATIVE("nop", "dmb sy", ARM64_WORKAROUND_1508412));
+   return true;
 }
 

Quoted 3 lines

Quoted 1 line

static inline bool __needs_ssbd_off(struct kvm_vcpu *vcpu)

Looks good to me!

Thanks Marc. Since it needs the local_irq_save() around the PAR_EL1 access in read_sysreg_par(), I'll wait for Rob to update the patches. Rob also asked the hardware guys for clarification on this scenario, so let's see what they reply.

This series implements the work-around for Cortex-A77 erratum 1508412. KVM guests which don't implement the workaround can still deadlock the system. This is also the case with the existing Cortex-A57 erratum 832075, so we add a warning message if an erratum can cause deadlock.

This series implements the work-around for Cortex-A77 erratum 1508412. KVM guests which don't implement the workaround can still deadlock the system. This is also the case with the existing Cortex-A57 erratum 832075, so we add a warning message if an erratum can cause deadlock.

Changes detailed in patches.

[...]