ARMv7: Add HYP mode switching support

Hello Andre,

arch/arm/cpu/armv7/nonsec_virt.S line 37

Quoted 11 lines

arch/arm/cpu/armv7/nonsec_virt.S new file +11 -0

+/* the vector table for secure state */
+_monitor_vectors:
+	.word 0	/* reset */
+	.word 0 /* undef */
+	adr pc, _secure_monitor
+	.word 0
+	.word 0
+	.word 0
+	.word 0
+	.word 0
+	.word 0	/* pad */

Could you explain why the last line is needed?

Quoted 8 lines · Masahiro Yamada

Hello Andre,

Quoted 4 lines

+/* the vector table for secure state */ +_monitor_vectors: +
.word 0 /* reset */ + .word 0 /* undef */ + adr pc,
_secure_monitor + .word 0 + .word 0 + .word 0 +
.word 0 + .word 0 + .word 0 /* pad */

Could you explain why the last line is needed?

I guess I cannot explain because it's probably not needed ;-) I copied this from somewhere else and accidentally this padding somehow sneaked in (from the actual addresses array, where the start vector was missing and thus the array ended at 7 * 4 bytes)

Do we need another version or is a follow-up patch sufficient? I'd like to avoid another review round for the sake of the reviewers.

The core specific part of the work is done in the assembly routine in nonsec_virt.S, introduced with the previous patch, but for the full glory we need to setup the GIC distributor interface once for the whole system, which is done in C here.
The routine is placed in arch/arm/cpu/armv7 to allow easy access from other ARMv7 boards.

We check the availability of the security extensions first.

Since we need a safe way to access the GIC, we use the PERIPHBASE registers on Cortex-A15 and A7 CPUs and do some sanity checks. Boards not implementing the CBAR can override this value via a configuration file variable.

Then we actually do the GIC enablement:
a) enable the GIC distributor, both for non-secure and secure state (GICD_CTLR[1:0] = 11b)
b) allow all interrupts to be handled from non-secure state (GICD_IGROUPRn = 0xFFFFFFFF)

The core specific GIC setup is then done in the assembly routine.

Signed-off-by Andre Przywara <[email protected]>

---
 arch/arm/cpu/armv7/Makefile  |   1 +
 arch/arm/cpu/armv7/virt-v7.c | 121 +++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h |   3 ++
 3 files changed, 125 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/virt-v7.c

arch/arm/cpu/armv7/Makefile +1 -0

index 11a8ad5..5813e87 100644
@@ -38,6 +38,7 @@ endif
 
 ifneq ($(CONFIG_ARMV7_NONSEC),)
 SOBJS	+= nonsec_virt.o
+COBJS	+= virt-v7.o
 endif
 
 SRCS	:= $(START:.o=.S) $(COBJS:.o=.c)

arch/arm/cpu/armv7/virt-v7.c new file +105 -0

new file mode 100644
index 0000000..689023f
@@ -0,0 +1,121 @@
+/*
+ * (C) Copyright 2013
+ * Andre Przywara, Linaro
+ *
+ * Routines to transition ARMv7 processors from secure into non-secure state
+ * needed to enable ARMv7 virtualization for current hypervisors
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <common.h>
+#include <asm/armv7.h>
+#include <asm/gic.h>
+#include <asm/io.h>
+
+unsigned long gic_dist_addr;
+
+static unsigned int read_id_pfr1(void)
+{
+	unsigned int reg;
+
+	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
+	return reg;
+}
+
+static unsigned long get_gicd_base_address(void)
+{
+#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
+	return CONFIG_ARM_GIC_BASE_ADDRESS + GIC_DIST_OFFSET;
+#else
+	unsigned midr;
+	unsigned periphbase;
+
+	/* check whether we are an Cortex-A15 or A7.
+	 * The actual HYP switch should work with all CPUs supporting
+	 * the virtualization extension, but we need the GIC address,
+	 * which we know only for sure for those two CPUs.
+	 */
+	asm("mrc p15, 0, %0, c0, c0, 0\n" : "=r"(midr));
+	switch (midr & MIDR_PRIMARY_PART_MASK) {
+	case MIDR_CORTEX_A9_R0P1:
+	case MIDR_CORTEX_A15_R0P0:
+	case MIDR_CORTEX_A7_R0P0:
+		break;
+	default:
+		printf("nonsec: could not determine GIC address.\n");
+		return -1;
+	}
+
+	/* get the GIC base address from the CBAR register */
+	asm("mrc p15, 4, %0, c15, c0, 0\n" : "=r" (periphbase));
+
+	/* the PERIPHBASE can be mapped above 4 GB (lower 8 bits used to
+	 * encode this). Bail out here since we cannot access this without
+	 * enabling paging.
+	 */
+	if ((periphbase & 0xff) != 0) {
+		printf("nonsec: PERIPHBASE is above 4 GB, no access.\n");
+		return -1;
+	}
+
+	return (periphbase & CBAR_MASK) + GIC_DIST_OFFSET;
+#endif
+}
+
+int armv7_switch_nonsec(void)
+{
+	unsigned int reg;
+	unsigned itlinesnr, i;
+
+	/* check whether the CPU supports the security extensions */
+	reg = read_id_pfr1();
+	if ((reg & 0xF0) == 0) {
+		printf("nonsec: Security extensions not implemented.\n");
+		return -1;
+	}
+
+	/* the SCR register will be set directly in the monitor mode handler,
+	 * according to the spec one should not tinker with it in secure state
+	 * in SVC mode. Do not try to read it once in non-secure state,
+	 * any access to it will trap.
+	 */
+
+	gic_dist_addr = get_gicd_base_address();
+	if (gic_dist_addr == -1)
+		return -1;
+
+	/* enable the GIC distributor */
+	writel(readl(gic_dist_addr + GICD_CTLR) | 0x03,

arch/arm/cpu/armv7/virt-v7.c new file +6 -0

+
+	/* call the non-sec switching code on this CPU */
+	_nonsec_init();
+
+	return 0;
+}

arch/arm/include/asm/armv7.h +3 -0

index 3e4b743..10ced11 100644
@@ -93,6 +93,9 @@ void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
 #ifdef CONFIG_ARMV7_NONSEC
+
+int armv7_switch_nonsec(void);
+
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
 #endif /* CONFIG_ARMV7_NONSEC */

Signature · 1 line

1.7.12.1

For the KVM and XEN hypervisors to be usable, we need to enter the kernel in HYP mode. Now that we already are in non-secure state, HYP mode switching is within short reach.

While doing the non-secure switch, we have to enable the HVC instruction and setup the HYP mode HVBAR (while still secure).

The actual switch is done by dropping back from a HYP mode handler without actually leaving HYP mode, so we introduce a new handler routine in our new secure exception vector table.

In the assembly switching routine we save and restore the banked LR and SP registers around the hypercall to do the actual HYP mode switch.

The C routine first checks whether we are in HYP mode already and also whether the virtualization extensions are available. It also checks whether the HYP mode switch was finally successful. The bootm command part only adds and adjusts some error reporting.

Signed-off-by Andre Przywara <[email protected]>

---
 arch/arm/cpu/armv7/Makefile      |  2 +-
 arch/arm/cpu/armv7/nonsec_virt.S | 43 +++++++++++++++++++++++++++++++++++-----
 arch/arm/cpu/armv7/virt-v7.c     | 37 ++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h     |  6 ++++--
 arch/arm/lib/bootm.c             |  7 ++++++-
 5 files changed, 86 insertions(+), 9 deletions(-)

arch/arm/cpu/armv7/Makefile +1 -1

index 5813e87..c20df3d 100644
@@ -36,7 +36,7 @@ ifneq ($(CONFIG_AM33XX)$(CONFIG_OMAP44XX)$(CONFIG_OMAP54XX)$(CONFIG_TEGRA)$(CONF
 SOBJS	+= lowlevel_init.o
 endif
 
-ifneq ($(CONFIG_ARMV7_NONSEC),)
+ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
 SOBJS	+= nonsec_virt.o
 COBJS	+= virt-v7.o
 endif

arch/arm/cpu/armv7/nonsec_virt.S +11 -3

index a88fa6b..fb1651d 100644
@@ -1,5 +1,5 @@
 /*
- * code for switching cores into non-secure state
+ * code for switching cores into non-secure state and into HYP mode
  *
  * Copyright (c) 2013	Andre Przywara <[email protected]>
  *

@@ -28,15 +28,16 @@
 #include <asm/armv7.h>
 
 .arch_extension sec
+.arch_extension virt
 
-/* the vector table for secure state */
+/* the vector table for secure state and HYP mode */
 _monitor_vectors:
 	.word 0	/* reset */
 	.word 0 /* undef */
 	adr pc, _secure_monitor
 	.word 0
 	.word 0
-	.word 0
+	adr pc, _hyp_trap
 	.word 0
 	.word 0
 	.word 0	/* pad */

@@ -54,10 +55,27 @@ _secure_monitor:
 	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
 	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
 
+#ifdef CONFIG_ARMV7_VIRT
+	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
+	and	r0, r0, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
+	cmp	r0, #(1 << CPUID_ARM_VIRT_SHIFT)
+	orreq	r1, r1, #0x100			@ allow HVC instruction
+#endif
+
 	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)

arch/arm/cpu/armv7/nonsec_virt.S +20 -2

+	mov pc, lr				@ do no switch modes, but
+						@ return to caller
+
 /*
  * Secondary CPUs start here and call the code for the core specific parts
  * of the non-secure and HYP mode transition. The GIC distributor specific

@@ -72,9 +90,13 @@ ENTRY(_smp_pen)
 	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
 
 	bl	_nonsec_init
+	mov	r12, r0				@ save GICC address
+#ifdef CONFIG_ARMV7_VIRT
+	bl	_switch_to_hyp
+#endif
 
-	ldr	r1, [r0, #GICC_IAR]		@ acknowledge IPI
-	str	r1, [r0, #GICC_EOIR]		@ signal end of interrupt
+	ldr	r1, [r12, #GICC_IAR]		@ acknowledge IPI
+	str	r1, [r12, #GICC_EOIR]		@ signal end of interrupt
 
 	adr	r0, _smp_pen			@ do not use this address again
 	b	smp_waitloop			@ wait for IPIs, board specific

@@ -161,3 +183,14 @@ ENTRY(_nonsec_init)
 
 	bx	lr
 ENDPROC(_nonsec_init)
+
+ENTRY(_switch_to_hyp)
+	mov	r0, lr
+	mov	r1, sp				@ save SVC copy of LR and SP
+	isb
+	hvc #0			 @ for older asm: .byte 0x70, 0x00, 0x40, 0xe1
+	mov	sp, r1
+	mov	lr, r0				@ restore SVC copy of LR and SP
+
+	bx	lr
+ENDPROC(_switch_to_hyp)

arch/arm/cpu/armv7/virt-v7.c +37 -0

index 50f0a3a..4dc8c45 100644
@@ -3,6 +3,7 @@
  * Andre Przywara, Linaro
  *
  * Routines to transition ARMv7 processors from secure into non-secure state
+ * and from non-secure SVC into HYP mode
  * needed to enable ARMv7 virtualization for current hypervisors
  *
  * See file CREDITS for list of people who contributed to this

@@ -31,6 +32,14 @@
 
 unsigned long gic_dist_addr;
 
+static unsigned int read_cpsr(void)
+{
+	unsigned int reg;
+
+	asm volatile ("mrs %0, cpsr\n" : "=r" (reg));
+	return reg;
+}
+
 static unsigned int read_id_pfr1(void)
 {
 	unsigned int reg;

@@ -90,6 +99,34 @@ void __weak smp_kick_all_cpus(void)
 	kick_secondary_cpus_gic(gic_dist_addr);
 }
 
+int armv7_switch_hyp(void)
+{
+	unsigned int reg;
+
+	/* check whether we are in HYP mode already */
+	if ((read_cpsr() & 0x1f) == 0x1a) {
+		debug("CPU already in HYP mode\n");
+		return 0;
+	}
+
+	/* check whether the CPU supports the virtualization extensions */
+	reg = read_id_pfr1();
+	if ((reg & CPUID_ARM_VIRT_MASK) != 1 << CPUID_ARM_VIRT_SHIFT) {
+		printf("HYP mode: Virtualization extensions not implemented.\n");
+		return -1;
+	}
+
+	/* call the HYP switching code on this CPU also */
+	_switch_to_hyp();
+
+	if ((read_cpsr() & 0x1F) != 0x1a) {
+		printf("HYP mode: switch not successful.\n");
+		return -1;
+	}
+
+	return 0;
+}
+
 int armv7_switch_nonsec(void)
 {
 	unsigned int reg;

arch/arm/include/asm/armv7.h +4 -2

index 06657fe..1a7b150 100644
@@ -92,14 +92,16 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
-#ifdef CONFIG_ARMV7_NONSEC
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 
 int armv7_switch_nonsec(void);
+int armv7_switch_hyp(void);
 
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
 void _smp_pen(void);
-#endif /* CONFIG_ARMV7_NONSEC */
+void _switch_to_hyp(void);
+#endif /* CONFIG_ARMV7_NONSEC || CONFIG_ARMV7_VIRT */
 
 #endif /* ! __ASSEMBLY__ */
 

arch/arm/lib/bootm.c +6 -1

index 6470eac..ff36319 100644
@@ -34,7 +34,7 @@
 #include <asm/bootm.h>
 #include <linux/compiler.h>
 
-#ifdef CONFIG_ARMV7_NONSEC
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 #include <asm/armv7.h>
 #endif
 

@@ -201,8 +201,13 @@ static void do_nonsec_virt_switch(void)
 {
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 	if (armv7_switch_nonsec() == 0)
+#ifdef CONFIG_ARMV7_VIRT
+		if (armv7_switch_hyp() == 0)
+			debug("entered HYP mode\n");
+#else
 		debug("entered non-secure state\n");
 #endif
+#endif
 }
 
 /* Subcommand: PREP */

Signature · 1 line

1.7.12.1

To enable hypervisors utilizing the ARMv7 virtualization extension on the Versatile Express board with the A15 core tile, we add the required configuration variable.
Also we define the board specific functions to do the SMP bringup:
smp_set_cpu_boot_addr() to set the start address for secondary cores and smp_waitloop() to wait for IPIs and jump to the start address.

This also serves as an example for what to do when adding support for new boards.

Signed-off-by Andre Przywara <[email protected]>

---
 board/armltd/vexpress/Makefile          |  7 +++++--
 board/armltd/vexpress/vexpress_common.c | 13 ++++++++++++
 board/armltd/vexpress/vexpress_smp.S    | 36 +++++++++++++++++++++++++++++++++
 include/configs/vexpress_ca15_tc2.h     |  4 ++++
 4 files changed, 58 insertions(+), 2 deletions(-)
 create mode 100644 board/armltd/vexpress/vexpress_smp.S

board/armltd/vexpress/Makefile +5 -2

index 6719f3d..282ef6d 100644
@@ -26,9 +26,12 @@ include $(TOPDIR)/config.mk
 LIB	= $(obj)lib$(BOARD).o
 
 COBJS	:= vexpress_common.o
+ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
+SOBJS	:= vexpress_smp.o
+endif
 
-SRCS	:= $(COBJS:.o=.c)
-OBJS	:= $(addprefix $(obj),$(COBJS))
+SRCS	:= $(COBJS:.o=.c) $(SOBJS:.o=.S)
+OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
 
 $(LIB):	$(obj).depend $(OBJS)
 	$(call cmd_link_o_target, $(OBJS))

board/armltd/vexpress/vexpress_common.c +13 -0

index 2c54869..66b810d 100644
@@ -272,3 +272,16 @@ ulong get_tbclk(void)
 {
 	return (ulong)CONFIG_SYS_HZ;
 }
+
+/* Setting the address at which secondary cores start from.
+ * Versatile Express uses one address for all cores, so ignore corenr
+ */
+void smp_set_core_boot_addr(unsigned long addr, int corenr)
+{
+	/* The SYSFLAGS register on VExpress needs to be cleared first
+	 * by writing to the next address, since any writes to the address
+	 * at offset 0 will only be ORed in
+	 */
+	writel(~0, CONFIG_SYSFLAGS_ADDR + 4);
+	writel(addr, CONFIG_SYSFLAGS_ADDR);
+}

board/armltd/vexpress/vexpress_smp.S new file +36 -0

new file mode 100644
index 0000000..41be2e7
@@ -0,0 +1,36 @@
+/*
+ * code for redirecting secondary cores to their start address
+ *
+ * Copyright (c) 2013	Andre Przywara <[email protected]>
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <config.h>
+#include <linux/linkage.h>
+
+/* void _smp_waitloop(unsigned previous_address); */
+ENTRY(smp_waitloop)
+	wfi
+	ldr	r1, =CONFIG_SYSFLAGS_ADDR	@ load start address
+	ldr	r1, [r1]
+	cmp	r0, r1			@ make sure we dont execute this code
+	beq	smp_waitloop		@ again (due to a spurious wakeup)
+	mov	pc, r1
+ENDPROC(smp_waitloop)

include/configs/vexpress_ca15_tc2.h +0 -0

index 4f425ac..14aa78e 100644

(for GIT URL and Changelog see below)

ARM CPUs with the virtualization extension have a new mode called HYP mode, which allows hypervisors to safely control and monitor guests. The current hypervisor implementations (KVM and Xen) require the kernel to be entered in that HYP mode.

This patch series introduces a configuration variable CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP mode. This is done automatically during execution of the bootm command.

The process of switching into HYP mode requires the CPU to be in secure state initially when entering u-boot, it will then setup some register and switch to non-secure state. This requires the GIC to be programmed properly first. Explanations about the details are in the commit messages of the respective patches.

The patches are structured like this:
1/8: prepare header file
2/8: add monitor handler (assembly)
3/8: add per CPU non-secure switch routine (assembly) 4/8: add system wide non-secure setup (C) 5/8: trigger non-secure switch during bootm command 6/8: add generic SMP functionality
7/8: add HYP mode switching
8/8: board specific code for ARM Versatile Express TC2

Since up to patch 6/8 this code works on non-virtualization capable CPUs also and there has been a request, there is now a second configuration variable CONFIG_ARMV7_NONSEC, which omits the final HYP mode switch and just goes into non-secure SVC state. You can specify either (or none) of them, the code cares about the dependency.

The code aims to be as generic as possible, though currently it has only been tested on the Versatile Express TC-2 board. The last patch thus enables the feature for that board and should serve as an example for supporting other boards.

For convenience there is a GIT tree which you can pull these patches from ("hypmode_v4" branch):
git://git.linaro.org/people/aprzywara/u-boot.git

Changes RFC..v1
* not a dedicated command anymore, code run by bootm & friends
* protecting code with #ifdefs to avoid unnecessary inclusion and accidental crashing (when accessing restricted registers)
* moving prototypes to header file to meet checkpatch recommendation
* adding comment as proposed by Christoffer

Changes v1..v2
mostly style and code layout changes
* restructure assembly code to live in a new file and not start.S
* split smp, nonsec_init and hyp_init to be separate functions
* used named constants from common header files
* split C function to be more readable
* extend comments to be more precise and elaborate
* add provision to override GIC base address (needed for Arndale?)
* add configuration variable to enable VExpress specific SMP code
* use writel/readl for MMIO GIC accesses
* remove superfluous isb instructions
* more minor fixes

Changes v2..v3
* fix clobbering of GICC address actually spoiling the stack
* do CNTFRQ setup in assembly per core (and not only once per SoC)
* moving the new code files into arch/arm/cpu/armv7
* add config variable for doing non-secure switch only
* use actual HYP and secure instructions mnemonics instead of the encoded byte sequence. This requires more recent compilers.
* make the identification of the CPU core more robust and saner
* use enum for error codes and rename them
* lots of smaller layout and style fixes

Changes v3..v4
* mask reserved bits in CBAR register
* move the VExpress board specific SMP code into the board directory
* embed error reporting in the respective functions and getting rid of the error code enum at all (by popular demand ;-)
* minor style fixes

Please review and comment!

Quoted 82 lines · Andre Przywara

(for GIT URL and Changelog see below)

ARM CPUs with the virtualization extension have a new mode called HYP mode, which allows hypervisors to safely control and monitor guests. The current hypervisor implementations (KVM and Xen) require the kernel to be entered in that HYP mode.

This patch series introduces a configuration variable CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP mode. This is done automatically during execution of the bootm command.

The process of switching into HYP mode requires the CPU to be in secure state initially when entering u-boot, it will then setup some register and switch to non-secure state. This requires the GIC to be programmed properly first. Explanations about the details are in the commit messages of the respective patches.

The patches are structured like this:
1/8: prepare header file
2/8: add monitor handler (assembly)
3/8: add per CPU non-secure switch routine (assembly) 4/8: add system wide non-secure setup (C) 5/8: trigger non-secure switch during bootm command 6/8: add generic SMP functionality
7/8: add HYP mode switching
8/8: board specific code for ARM Versatile Express TC2

Since up to patch 6/8 this code works on non-virtualization capable CPUs also and there has been a request, there is now a second configuration variable CONFIG_ARMV7_NONSEC, which omits the final HYP mode switch and just goes into non-secure SVC state. You can specify either (or none) of them, the code cares about the dependency.

The code aims to be as generic as possible, though currently it has only been tested on the Versatile Express TC-2 board. The last patch thus enables the feature for that board and should serve as an example for supporting other boards.

For convenience there is a GIT tree which you can pull these patches from ("hypmode_v4" branch):
git://git.linaro.org/people/aprzywara/u-boot.git

Changes RFC..v1
* not a dedicated command anymore, code run by bootm & friends
* protecting code with #ifdefs to avoid unnecessary inclusion and accidental crashing (when accessing restricted registers)
* moving prototypes to header file to meet checkpatch recommendation
* adding comment as proposed by Christoffer

Changes v1..v2
mostly style and code layout changes
* restructure assembly code to live in a new file and not start.S
* split smp, nonsec_init and hyp_init to be separate functions
* used named constants from common header files
* split C function to be more readable
* extend comments to be more precise and elaborate
* add provision to override GIC base address (needed for Arndale?)
* add configuration variable to enable VExpress specific SMP code
* use writel/readl for MMIO GIC accesses
* remove superfluous isb instructions
* more minor fixes

Changes v2..v3
* fix clobbering of GICC address actually spoiling the stack
* do CNTFRQ setup in assembly per core (and not only once per SoC)
* moving the new code files into arch/arm/cpu/armv7
* add config variable for doing non-secure switch only
* use actual HYP and secure instructions mnemonics instead of the encoded byte sequence. This requires more recent compilers.
* make the identification of the CPU core more robust and saner
* use enum for error codes and rename them
* lots of smaller layout and style fixes

Changes v3..v4
* mask reserved bits in CBAR register
* move the VExpress board specific SMP code into the board directory
* embed error reporting in the respective functions and getting rid of the error code enum at all (by popular demand ;-)
* minor style fixes

Quoted 2 lines

Please review and comment!

Only had those very few nits sent separately, which can be fixed later if we want. Otherwise looks good.

Reviewed-by Christoffer Dall <[email protected]>

Albert, Tom,
do you need more ACKs or Reviewed-bys?

On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:

[...]

Quoted 1 line

Quoted 2 lines

Albert, Tom, do you need more ACKs or Reviewed-bys?

Albert, Tom,

Can you let us know if you will accept a pull request for these patches? They look to be in pretty good shape?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Christoffer Dall

On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:

[...]

Quoted 3 lines

Albert, Tom, do you need more ACKs or Reviewed-bys?

Quoted 4 lines

Albert, Tom,

Can you let us know if you will accept a pull request for these patches? They look to be in pretty good shape?

It looks good to me, I just need Albert to pick them up. Thanks!

Quoted 14 lines · Tom Rini

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Quoted 11 lines · Christoffer Dall

On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:

[...]

Quoted 3 lines

Albert, Tom, do you need more ACKs or Reviewed-bys?

Albert, Tom,

Quoted 6 lines

Quoted 2 lines

Can you let us know if you will accept a pull request for these patches? They look to be in pretty good shape?

It looks good to me, I just need Albert to pick them up. Thanks!

Great! Do you need a pull request or do you just apply the patches from the mailing list?

(for GIT URL and Changelog see below)

ARM CPUs with the virtualization extension have a new mode called HYP mode, which allows hypervisors to safely control and monitor guests. The current hypervisor implementations (KVM and Xen) require the kernel to be entered in that HYP mode.

This patch series introduces a configuration variable CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP mode. This is done automatically during execution of the bootm command.

The process of switching into HYP mode requires the CPU to be in secure state initially when entering u-boot, it will then setup some register and switch to non-secure state. This requires the GIC to be programmed properly first. Explanations about the details are in the commit messages of the respective patches.

The patches are structured like this:
1/8: prepare header file
2/8: add monitor handler (assembly)
3/8: add per CPU non-secure switch routine (assembly) 4/8: add system wide non-secure setup (C) 5/8: trigger non-secure switch during bootm command 6/8: add generic SMP functionality
7/8: add HYP mode switching
8/8: board specific code for ARM Versatile Express TC2

Since up to patch 6/8 this code works on non-virtualization capable CPUs also and there has been a request, there is now a second configuration variable CONFIG_ARMV7_NONSEC, which omits the final HYP mode switch and just goes into non-secure SVC state. You can specify either (or none) of them, the code cares about the dependency.

The code aims to be as generic as possible, though currently it has only been tested on the Versatile Express TC-2 board. The last patch

thus enables the feature for that board and should serve as an example for supporting other boards.

For convenience there is a GIT tree which you can pull these patches from ("hypmode_v4" branch):
git://git.linaro.org/people/aprzywara/u-boot.git

Changes RFC..v1
* not a dedicated command anymore, code run by bootm & friends
* protecting code with #ifdefs to avoid unnecessary inclusion and accidental crashing (when accessing restricted registers)
* moving prototypes to header file to meet checkpatch recommendation
* adding comment as proposed by Christoffer

Changes v1..v2
mostly style and code layout changes
* restructure assembly code to live in a new file and not start.S
* split smp, nonsec_init and hyp_init to be separate functions
* used named constants from common header files
* split C function to be more readable
* extend comments to be more precise and elaborate
* add provision to override GIC base address (needed for Arndale?)
* add configuration variable to enable VExpress specific SMP code
* use writel/readl for MMIO GIC accesses
* remove superfluous isb instructions
* more minor fixes

Changes v2..v3
* fix clobbering of GICC address actually spoiling the stack
* do CNTFRQ setup in assembly per core (and not only once per SoC)
* moving the new code files into arch/arm/cpu/armv7
* add config variable for doing non-secure switch only
* use actual HYP and secure instructions mnemonics instead of the encoded byte sequence. This requires more recent compilers.
* make the identification of the CPU core more robust and saner
* use enum for error codes and rename them
* lots of smaller layout and style fixes

Changes v3..v4
* mask reserved bits in CBAR register
* move the VExpress board specific SMP code into the board directory
* embed error reporting in the respective functions and getting rid of the error code enum at all (by popular demand ;-)
* minor style fixes

Please review and comment!

Contributions and comments to support other boards are welcome.

Andre Przywara (8):
ARM: prepare armv7.h to be included from assembly source
ARM: add secure monitor handler to switch to non-secure state
ARM: add assembly routine to switch to non-secure state
ARM: add C function to switch to non-secure state
ARM: trigger non-secure state switch during bootm execution
ARM: add SMP support for non-secure switch
ARM: extend non-secure switch to also go into HYP mode
ARM: VExpress: enable ARMv7 virt support for VExpress A15

 arch/arm/cpu/armv7/Makefile             |   5 +
 arch/arm/cpu/armv7/nonsec_virt.S        | 196
++++++++++++++++++++++++++++++++
 arch/arm/cpu/armv7/virt-v7.c            | 172 ++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h            |  33 +++++-
 arch/arm/include/asm/gic.h              |  19 ++++
 arch/arm/lib/bootm.c                    |  18 +++
 board/armltd/vexpress/Makefile          |   7 +-
 board/armltd/vexpress/vexpress_common.c |  13 +++
 board/armltd/vexpress/vexpress_smp.S    |  36 ++++++
 include/common.h                        |   2 +
 include/configs/vexpress_ca15_tc2.h     |   4 +-
 11 files changed, 501 insertions(+), 4 deletions(-)
 create mode 100644 arch/arm/cpu/armv7/nonsec_virt.S
 create mode 100644 arch/arm/cpu/armv7/virt-v7.c
 create mode 100644 arch/arm/include/asm/gic.h
 create mode 100644 board/armltd/vexpress/vexpress_smp.S

--
1.7.12.1

_______________________________________________
kvmarm mailing list
kvmarm at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm

(for GIT URL and Changelog see below)