ARMv7: add PSCI support to u-boot

PSCI is an ARM standard that provides a generic interface that supervisory software can use to manage power in the following situations:
- Core idle management
- CPU hotplug
- big.LITTLE migration models
- System shutdown and reset

It basically allows the kernel to offload these tasks to the firmware, and rely on common kernel side code.

More importantly, it gives a way to ensure that CPUs enter the kernel at the appropriate exception level (ie HYP mode, to allow the use of the virtualization extensions), even across events like CPUs being powered off/on or suspended.

The main idea here is to turn some of the existing u-boot code into a separate section that can live in secure RAM (or a reserved page of memory), containing a secure monitor that will implement the PSCI operations. This code will still be alive when u-boot is long gone, hence the need for a piece of memory that will not be touched by the OS.

This patch series contains 4 parts:
- the first four patches are just bug fixes
- the next three refactor the HYP/non-secure code to allow relocation in secure memory
- the next three contain the generic PSCI code and DT infrastructure
- the last three implement the CPU_ON method of the Allwinner A20 (aka sun7i).

I realize the A20 u-boot code is not upstream yet (BTW is anyone actively working on that?), but hopefully that should give a good idea of how things are structured so far. The patches are against the sunxi u-boot tree as of today, and the first 10 patches will directly apply to mainline u-boot.

As for using this code, it goes like this:
sun7i# ext2load mmc 0:1 0x40008000 zImage ; ext2load mmc 0:1 0x60000000 sun7i-a20-cubietruck.dtb 2270120 bytes read in 117 ms (18.5 MiB/s)
9138 bytes read in 3 ms (2.9 MiB/s)
sun7i# fdt addr 0x60000000 ; fdt resize ; fdt set ethernet0 mac-address "[5a fe b0 07 b0 07]" sun7i# setenv bootargs console=ttyS0,115200 earlyprintk ip=dhcp root=/dev/nfs nfsroot=/backup/a20_root,tcp sun7i# bootz 0x40008000 - 0x60000000

The kernel now boots in HYP mode, finds its secondary CPU without any SMP code present in the kernel, and runs KVM out of the box. Hopefully, the Xen/ARM guys can do the same fairly easily.

This code has also been tested on a VExpress TC2, running KVM with all 5 CPUs, in order to make sure there was no obvious regression.

I'm wildly cross-posting this patch series, including to lists I'm not subscribed to. Please keep me on Cc for any comment you may have.

Cheers,

        M.

Marc Zyngier (13):
ARM: HYP/non-sec: fix alignment requirements for vectors
ARM: HYP/non-sec: move switch to non-sec to the last boot phase
ARM: HYP/non-sec: add a barrier after setting SCR.NS==1
ARM: non-sec: reset CNTVOFF to zero
ARM: add missing HYP mode constant
ARM: HYP/non-sec: add separate section for secure code
ARM: HYP/non-sec: allow relocation to secure RAM
ARM: HYP/non-sec: add generic ARMv7 PSCI code
ARM: HYP/non-sec: add the option for a second-stage monitor
ARM: HYP/non-sec/PSCI: emit DT nodes
sunxi: fix SRAM_B/SRAM_D memory map
sunxi: HYP/non-sec: add sun7i PSCI backend
sunxi: HYP/non-sec: configure CNTFRQ on all CPUs

 arch/arm/cpu/armv7/Makefile              |   5 +
 arch/arm/cpu/armv7/nonsec_virt.S         | 168 ++++++++++++++++---------------
 arch/arm/cpu/armv7/psci.S                | 113 +++++++++++++++++++++
 arch/arm/cpu/armv7/sunxi/Makefile        |   3 +
 arch/arm/cpu/armv7/sunxi/psci.S          | 124 +++++++++++++++++++++++
 arch/arm/cpu/armv7/sunxi/u-boot-psci.lds |  63 ++++++++++++
 arch/arm/cpu/armv7/virt-dt.c             | 102 +++++++++++++++++++
 arch/arm/cpu/armv7/virt-v7.c             |  59 ++++-------
 arch/arm/cpu/u-boot.lds                  |  30 ++++++
 arch/arm/include/asm/arch-sunxi/cpu.h    |   4 +-
 arch/arm/include/asm/armv7.h             |  11 +-
 arch/arm/include/asm/proc-armv/ptrace.h  |   2 +
 arch/arm/include/asm/psci.h              |  35 +++++++
 arch/arm/include/asm/secure.h            |  26 +++++
 arch/arm/lib/bootm.c                     |  26 ++---
 arch/arm/lib/interrupts.c                |   2 +-
 arch/arm/lib/sections.c                  |   2 +
 include/configs/sun7i.h                  |   7 ++
 18 files changed, 641 insertions(+), 141 deletions(-)
 create mode 100644 arch/arm/cpu/armv7/psci.S
 create mode 100644 arch/arm/cpu/armv7/sunxi/psci.S
 create mode 100644 arch/arm/cpu/armv7/sunxi/u-boot-psci.lds
 create mode 100644 arch/arm/cpu/armv7/virt-dt.c
 create mode 100644 arch/arm/include/asm/psci.h
 create mode 100644 arch/arm/include/asm/secure.h

1.8.4.3

Make sure the vectors are aligned on a 32 byte boundary, not the code that deals with it...

---
 arch/arm/cpu/armv7/nonsec_virt.S | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 24b4c18..29987cd 100644
@@ -14,6 +14,8 @@
 .arch_extension sec
 .arch_extension virt
 
+	.align	5				@ Minimal alignment for vectors
+
 /* the vector table for secure state and HYP mode */
 _monitor_vectors:
 	.word 0	/* reset */

@@ -32,7 +34,6 @@ _monitor_vectors:
  * to non-secure state.
  * We use only r0 and r1 here, due to constraints in the caller.
  */
-	.align	5
 _secure_monitor:
 	mrc	p15, 0, r1, c1, c1, 0		@ read SCR
 	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits

1.8.4.3

Looks like Masahiro Yamada's equivalent patch is now in the u-boot-arm tree.

Ian.

Having the switch to non-secure in the "prep" phase is causing all kind of troubles, as that stage can be called multiple times.

Instead, move the switch to non-secure to the last possible phase, when there is no turning back anymore.

---
 arch/arm/lib/bootm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index f476a89..f3da634 100644
@@ -234,7 +234,6 @@ static void boot_prep_linux(bootm_headers_t *images)
 		printf("FDT and ATAGS support not compiled in - hanging\n");
 		hang();
 	}
-	do_nonsec_virt_switch();
 }
 
 /* Subcommand: GO */

@@ -264,8 +263,10 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)
 	else
 		r2 = gd->bd->bi_boot_params;
 
-	if (!fake)
+	if (!fake) {
+		do_nonsec_virt_switch();
 		kernel_entry(0, machid, r2);
+	}
 }
 
 /* Main Entry point for arm bootm implementation

1.8.4.3

A CP15 instruction execution can be reordered, requiring an isb to be sure it is executed in program order.

---
 arch/arm/cpu/armv7/nonsec_virt.S | 1 +
 1 file changed, 1 insertion(+)

index 29987cd..648066f 100644
@@ -47,6 +47,7 @@ _secure_monitor:
 #endif
 
 	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)
+	isb
 
 #ifdef CONFIG_ARMV7_VIRT
 	mrceq	p15, 0, r0, c12, c0, 1		@ get MVBAR value

1.8.4.3

Before switching to non-secure, make sure that CNTVOFF is set to zero on all CPUs. Otherwise, kernel running in non-secure without HYP enabled (hence using virtual timers) may observe timers that are not synchronized, effectively seeing time going backward...

---
 arch/arm/cpu/armv7/nonsec_virt.S | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

index 648066f..5f81a4d 100644
@@ -39,10 +39,10 @@ _secure_monitor:
 	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
 	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
 
-#ifdef CONFIG_ARMV7_VIRT
 	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
 	and	r0, r0, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
 	cmp	r0, #(1 << CPUID_ARM_VIRT_SHIFT)
+#ifdef CONFIG_ARMV7_VIRT
 	orreq	r1, r1, #0x100			@ allow HVC instruction
 #endif
 

@@ -53,7 +53,14 @@ _secure_monitor:
 	mrceq	p15, 0, r0, c12, c0, 1		@ get MVBAR value
 	mcreq	p15, 4, r0, c12, c0, 0		@ write HVBAR
 #endif
+	bne	1f
 
+	@ Reset CNTVOFF to 0 before leaving monitor mode
+	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
+	ands	r0, r0, #CPUID_ARM_GENTIMER_MASK	@ test arch timer bits
+	movne	r0, #0
+	mcrrne	p15, 4, r0, r0, c14		@ Reset CNTVOFF to zero
+1:
 	movs	pc, lr				@ return to non-secure SVC
 
 _hyp_trap:

1.8.4.3

In order to be able to use the various mode constants (far more readable than random hex values), add the missing HYP and A values.

Also update arm/lib/interrupts.c to display HYP instead of an unknown value.

---
 arch/arm/include/asm/proc-armv/ptrace.h | 2 ++
 arch/arm/lib/interrupts.c               | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

index a060ee6..e66c381 100644
@@ -19,12 +19,14 @@
 #define IRQ_MODE	0x12
 #define SVC_MODE	0x13
 #define ABT_MODE	0x17
+#define HYP_MODE	0x1a
 #define UND_MODE	0x1b
 #define SYSTEM_MODE	0x1f
 #define MODE_MASK	0x1f
 #define T_BIT		0x20
 #define F_BIT		0x40
 #define I_BIT		0x80
+#define A_BIT		0x100
 #define CC_V_BIT	(1 << 28)
 #define CC_C_BIT	(1 << 29)
 #define CC_Z_BIT	(1 << 30)

index 603bf14..fa553c4 100644
@@ -103,7 +103,7 @@ void show_regs (struct pt_regs *regs)
 	"UK12_26",	"UK13_26",	"UK14_26",	"UK15_26",
 	"USER_32",	"FIQ_32",	"IRQ_32",	"SVC_32",
 	"UK4_32",	"UK5_32",	"UK6_32",	"ABT_32",
-	"UK8_32",	"UK9_32",	"UK10_32",	"UND_32",
+	"UK8_32",	"UK9_32",	"HYP_32",	"UND_32",
 	"UK12_32",	"UK13_32",	"UK14_32",	"SYS_32",
 	};
 

1.8.4.3

In anticipation of refactoring the HYP/non-secure code to run from secure RAM, add a new linker section that will contain that code.

Nothing is using it just yet.

---
 arch/arm/cpu/u-boot.lds | 30 ++++++++++++++++++++++++++++++
 arch/arm/lib/sections.c |  2 ++
 2 files changed, 32 insertions(+)

index 23bf030..8096efe 100644
@@ -7,6 +7,8 @@
  * SPDX-License-Identifier:	GPL-2.0+
  */
 
+#include <config.h>
+
 OUTPUT_FORMAT("elf32-littlearm", "elf32-littlearm", "elf32-littlearm")
 OUTPUT_ARCH(arm)
 ENTRY(_start)

@@ -22,6 +24,34 @@ SECTIONS
 		*(.text*)
 	}
 
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT) || defined(CONFIG_ARMV7_PSCI)
+
+#ifndef CONFIG_ARMV7_SECURE_BASE
+#define CONFIG_ARMV7_SECURE_BASE
+#endif
+
+	.__secure_start : {
+		. = ALIGN(0x1000);
+		*(.__secure_start)
+	}
+
+	.secure_text CONFIG_ARMV7_SECURE_BASE :
+		AT(ADDR(.__secure_start) + SIZEOF(.__secure_start))
+	{
+		*(._secure.text)
+	}
+
+	. = LOADADDR(.__secure_start) +
+		SIZEOF(.__secure_start) +
+		SIZEOF(.secure_text);
+
+	__secure_end_lma = .;
+	.__secure_end : AT(__secure_end_lma) {
+		*(.__secure_end)
+		LONG(0x1d1071c);	/* Must output something to reset LMA */
+	}
+#endif
+
 	. = ALIGN(4);
 	.rodata : { *(SORT_BY_ALIGNMENT(SORT_BY_NAME(.rodata*))) }
 

index e35687c..c141923 100644
@@ -25,3 +25,5 @@ char __image_copy_start[0] __attribute__((section(".__image_copy_start")));
 char __image_copy_end[0] __attribute__((section(".__image_copy_end")));
 char __rel_dyn_start[0] __attribute__((section(".__rel_dyn_start")));
 char __rel_dyn_end[0] __attribute__((section(".__rel_dyn_end")));
+char __secure_start[0] __attribute__((section(".__secure_start")));
+char __secure_end[0] __attribute__((section(".__secure_end")));

1.8.4.3

The current non-sec switching code suffers from one major issue:
it cannot run in secure RAM, as a large part of u-boot still needs to be run while we're switched to non-secure.

This patch reworks the whole HYP/non-secure strategy by:
- making sure the secure code is the *last* thing u-boot executes before entering the payload
- performing an exception return from secure mode directly into the payload
- allowing the code to be dynamically relocated to secure RAM before switching to non-secure.

This involves quite a bit of horrible code, specially as u-boot relocation is quite primitive.

---
 arch/arm/cpu/armv7/nonsec_virt.S | 161 +++++++++++++++++++--------------------
 arch/arm/cpu/armv7/virt-v7.c     |  59 +++++---------
 arch/arm/include/asm/armv7.h     |  10 ++-
 arch/arm/include/asm/secure.h    |  26 +++++++
 arch/arm/lib/bootm.c             |  21 ++---
 5 files changed, 136 insertions(+), 141 deletions(-)
 create mode 100644 arch/arm/include/asm/secure.h

index 5f81a4d..35f5463 100644
@@ -10,10 +10,13 @@
 #include <linux/linkage.h>
 #include <asm/gic.h>
 #include <asm/armv7.h>
+#include <asm/proc-armv/ptrace.h>
 
 .arch_extension sec
 .arch_extension virt
 
+	.pushsection ._secure.text, "ax"
+
 	.align	5				@ Minimal alignment for vectors
 
 /* the vector table for secure state and HYP mode */

@@ -23,51 +26,86 @@ _monitor_vectors:
 	adr pc, _secure_monitor
 	.word 0
 	.word 0
-	adr pc, _hyp_trap
+	.word 0
 	.word 0
 	.word 0
 
+.macro is_cpu_virt_capable	tmp
+	mrc	p15, 0, \tmp, c0, c1, 1		@ read ID_PFR1
+	and	\tmp, \tmp, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
+	cmp	\tmp, #(1 << CPUID_ARM_VIRT_SHIFT)
+.endm
+
 /*
  * secure monitor handler
  * U-boot calls this "software interrupt" in start.S
  * This is executed on a "smc" instruction, we use a "smc #0" to switch
  * to non-secure state.
- * We use only r0 and r1 here, due to constraints in the caller.
+ * r0, r1, r2: passed to the callee
+ * ip: target PC
  */
 _secure_monitor:
-	mrc	p15, 0, r1, c1, c1, 0		@ read SCR
-	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
-	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
+	mrc	p15, 0, r5, c1, c1, 0		@ read SCR
+	bic	r5, r5, #0x4e			@ clear IRQ, FIQ, EA, nET bits
+	orr	r5, r5, #0x31			@ enable NS, AW, FW bits
 
-	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
-	and	r0, r0, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
-	cmp	r0, #(1 << CPUID_ARM_VIRT_SHIFT)
+	mov	r6, #SVC_MODE			@ default mode is SVC
+	is_cpu_virt_capable r4
 #ifdef CONFIG_ARMV7_VIRT
-	orreq	r1, r1, #0x100			@ allow HVC instruction
+	orreq	r5, r5, #0x100			@ allow HVC instruction
+	moveq	r6, #HYP_MODE			@ Enter the kernel as HYP
 #endif
 
-	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)
+	mcr	p15, 0, r5, c1, c1, 0		@ write SCR (with NS bit set)
 	isb
 
-#ifdef CONFIG_ARMV7_VIRT
-	mrceq	p15, 0, r0, c12, c0, 1		@ get MVBAR value
-	mcreq	p15, 4, r0, c12, c0, 0		@ write HVBAR
-#endif
 	bne	1f
 
 	@ Reset CNTVOFF to 0 before leaving monitor mode
-	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
-	ands	r0, r0, #CPUID_ARM_GENTIMER_MASK	@ test arch timer bits
-	movne	r0, #0
-	mcrrne	p15, 4, r0, r0, c14		@ Reset CNTVOFF to zero
+	mrc	p15, 0, r4, c0, c1, 1		@ read ID_PFR1
+	ands	r4, r4, #CPUID_ARM_GENTIMER_MASK	@ test arch timer bits
+	movne	r4, #0
+	mcrrne	p15, 4, r4, r4, c14		@ Reset CNTVOFF to zero
 1:
-	movs	pc, lr				@ return to non-secure SVC
-
-_hyp_trap:
-	mrs	lr, elr_hyp	@ for older asm: .byte 0x00, 0xe3, 0x0e, 0xe1
-	mov pc, lr				@ do no switch modes, but
-						@ return to caller
-
+	mov	lr, ip
+	mov	ip, #(F_BIT | I_BIT | A_BIT)	@ Set A, I and F
+	tst	lr, #1				@ Check for Thumb PC
+	orrne	ip, ip, #T_BIT			@ Set T if Thumb
+	orr	ip, ip, r6			@ Slot target mode in
+	msr	spsr_cxfs, ip			@ Set full SPSR
+	movs	pc, lr				@ ERET to non-secure
+
+ENTRY(_do_nonsec_entry)
+	mov	ip, r0
+	mov	r0, r1
+	mov	r1, r2
+	mov	r2, r3
+	smc	#0
+ENDPROC(_do_nonsec_entry)
+
+.macro get_cbar_addr	addr
+#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
+	ldr	\addr, =CONFIG_ARM_GIC_BASE_ADDRESS
+#else
+	mrc	p15, 4, \addr, c15, c0, 0	@ read CBAR
+	bfc	\addr, #0, #15			@ clear reserved bits
+#endif
+.endm
+
+.macro get_gicd_addr	addr
+	get_cbar_addr	\addr
+	add	\addr, \addr, #GIC_DIST_OFFSET	@ GIC dist i/f offset
+.endm
+
+.macro get_gicc_addr	addr, tmp
+	get_cbar_addr	\addr
+	is_cpu_virt_capable \tmp
+	movne	\tmp, #GIC_CPU_OFFSET_A9	@ GIC CPU offset for A9
+	moveq	\tmp, #GIC_CPU_OFFSET_A15	@ GIC CPU offset for A15/A7
+	add	\addr, \addr, \tmp
+.endm
+
+#ifndef CONFIG_ARMV7_PSCI
 /*
  * Secondary CPUs start here and call the code for the core specific parts
  * of the non-secure and HYP mode transition. The GIC distributor specific

@@ -75,31 +113,21 @@ _hyp_trap:
  * Then they go back to wfi and wait to be woken up by the kernel again.
  */
 ENTRY(_smp_pen)
-	mrs	r0, cpsr
-	orr	r0, r0, #0xc0
-	msr	cpsr, r0			@ disable interrupts
-	ldr	r1, =_start
-	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
+	cpsid	i
+	cpsid	f
 
 	bl	_nonsec_init
-	mov	r12, r0				@ save GICC address
-#ifdef CONFIG_ARMV7_VIRT
-	bl	_switch_to_hyp
-#endif
-
-	ldr	r1, [r12, #GICC_IAR]		@ acknowledge IPI
-	str	r1, [r12, #GICC_EOIR]		@ signal end of interrupt
 
 	adr	r0, _smp_pen			@ do not use this address again
 	b	smp_waitloop			@ wait for IPIs, board specific
 ENDPROC(_smp_pen)
+#endif
 
 /*
  * Switch a core to non-secure state.
  *
  *  1. initialize the GIC per-core interface
  *  2. allow coprocessor access in non-secure modes
- *  3. switch the cpu mode (by calling "smc #0")
  *
  * Called from smp_pen by secondary cores and directly by the BSP.
  * Do not assume that the stack is available and only use registers

@@ -109,38 +137,23 @@ ENDPROC(_smp_pen)
  * though, but we check this in C before calling this function.
  */
 ENTRY(_nonsec_init)
-#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
-	ldr	r2, =CONFIG_ARM_GIC_BASE_ADDRESS
-#else
-	mrc	p15, 4, r2, c15, c0, 0		@ read CBAR
-	bfc	r2, #0, #15			@ clear reserved bits
-#endif
-	add	r3, r2, #GIC_DIST_OFFSET	@ GIC dist i/f offset
+	get_gicd_addr	r3
+
 	mvn	r1, #0				@ all bits to 1
 	str	r1, [r3, #GICD_IGROUPRn]	@ allow private interrupts
 
-	mrc	p15, 0, r0, c0, c0, 0		@ read MIDR
-	ldr	r1, =MIDR_PRIMARY_PART_MASK
-	and	r0, r0, r1			@ mask out variant and revision
-
-	ldr	r1, =MIDR_CORTEX_A7_R0P0 & MIDR_PRIMARY_PART_MASK
-	cmp	r0, r1				@ check for Cortex-A7
-
-	ldr	r1, =MIDR_CORTEX_A15_R0P0 & MIDR_PRIMARY_PART_MASK
-	cmpne	r0, r1				@ check for Cortex-A15
-
-	movne	r1, #GIC_CPU_OFFSET_A9		@ GIC CPU offset for A9
-	moveq	r1, #GIC_CPU_OFFSET_A15		@ GIC CPU offset for A15/A7
-	add	r3, r2, r1			@ r3 = GIC CPU i/f addr
+	get_gicc_addr	r3, r1
 
 	mov	r1, #1				@ set GICC_CTLR[enable]
 	str	r1, [r3, #GICC_CTLR]		@ and clear all other bits
 	mov	r1, #0xff
 	str	r1, [r3, #GICC_PMR]		@ set priority mask register
 
+	mrc	p15, 0, r0, c1, c1, 2
 	movw	r1, #0x3fff
-	movt	r1, #0x0006
-	mcr	p15, 0, r1, c1, c1, 2		@ NSACR = all copros to non-sec
+	movt	r1, #0x0004
+	orr	r0, r0, r1
+	mcr	p15, 0, r0, c1, c1, 2		@ NSACR = all copros to non-sec
 
 /* The CNTFRQ register of the generic timer needs to be
  * programmed in secure state. Some primary bootloaders / firmware

@@ -158,44 +171,24 @@ ENTRY(_nonsec_init)
 
 	adr	r1, _monitor_vectors
 	mcr	p15, 0, r1, c12, c0, 1		@ set MVBAR to secure vectors
-
-	mrc	p15, 0, ip, c12, c0, 0		@ save secure copy of VBAR
-
 	isb
-	smc	#0				@ call into MONITOR mode
-
-	mcr	p15, 0, ip, c12, c0, 0		@ write non-secure copy of VBAR
-
-	mov	r1, #1
-	str	r1, [r3, #GICC_CTLR]		@ enable non-secure CPU i/f
-	add	r2, r2, #GIC_DIST_OFFSET
-	str	r1, [r2, #GICD_CTLR]		@ allow private interrupts
 
 	mov	r0, r3				@ return GICC address
-
 	bx	lr
 ENDPROC(_nonsec_init)
 
 #ifdef CONFIG_SMP_PEN_ADDR
 /* void __weak smp_waitloop(unsigned previous_address); */
 ENTRY(smp_waitloop)
-	wfi
+	wfe
 	ldr	r1, =CONFIG_SMP_PEN_ADDR	@ load start address
 	ldr	r1, [r1]
 	cmp	r0, r1			@ make sure we dont execute this code
 	beq	smp_waitloop		@ again (due to a spurious wakeup)
-	mov	pc, r1
+	mov	r0, r1
+	b	_do_nonsec_entry
 ENDPROC(smp_waitloop)
 .weak smp_waitloop
 #endif
 
-ENTRY(_switch_to_hyp)
-	mov	r0, lr
-	mov	r1, sp				@ save SVC copy of LR and SP
-	isb
-	hvc #0			 @ for older asm: .byte 0x70, 0x00, 0x40, 0xe1
-	mov	sp, r1
-	mov	lr, r0				@ restore SVC copy of LR and SP
-
-	bx	lr
-ENDPROC(_switch_to_hyp)
+	.popsection

index 395444e..11476dd 100644
@@ -78,13 +78,17 @@ void v7_outer_cache_inval_range(u32 start, u32 end);
 
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 
-int armv7_switch_nonsec(void);
-int armv7_switch_hyp(void);
+int armv7_init_nonsec(void);
 
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
+void _do_nonsec_entry(void *target_pc, unsigned long r0,
+		      unsigned long r1, unsigned long r2);
 void _smp_pen(void);
-void _switch_to_hyp(void);
+
+extern char __secure_start[];
+extern char __secure_end[];
+
 #endif /* CONFIG_ARMV7_NONSEC || CONFIG_ARMV7_VIRT */
 
 #endif /* ! __ASSEMBLY__ */

new file mode 100644
index 0000000..effdb18
@@ -0,0 +1,26 @@
+#ifndef __ASM_SECURE_H
+#define __ASM_SECURE_H
+
+#include <config.h>
+
+#ifdef CONFIG_ARMV7_SECURE_BASE
+/*
+ * Warning, horror ahead.
+ *
+ * The target code lives in our "secure ram", but u-boot doesn't know
+ * that, and has blindly added reloc_off to every relocation
+ * entry. Gahh. Do the opposite conversion. This hack also prevents
+ * GCC from generating code veeners, which u-boot doesn't relocate at
+ * all...
+ */
+#define secure_ram_addr(_fn) ({						\
+			DECLARE_GLOBAL_DATA_PTR;			\
+			void *__fn = _fn;				\
+			typeof(_fn) *__tmp = (__fn - gd->reloc_off);	\
+			__tmp;						\
+		})
+#else
+#define secure_ram_addr(_fn)	(_fn)
+#endif
+
+#endif

index f3da634..408a774 100644
@@ -20,6 +20,7 @@
 #include <libfdt.h>
 #include <fdt_support.h>
 #include <asm/bootm.h>
+#include <asm/secure.h>
 #include <linux/compiler.h>
 
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)

@@ -185,19 +186,6 @@ static void setup_end_tag(bd_t *bd)
 
 __weak void setup_board_tags(struct tag **in_params) {}
 
-static void do_nonsec_virt_switch(void)
-{
-#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
-	if (armv7_switch_nonsec() == 0)
-#ifdef CONFIG_ARMV7_VIRT
-		if (armv7_switch_hyp() == 0)
-			debug("entered HYP mode\n");
-#else
-		debug("entered non-secure state\n");
-#endif
-#endif
-}
-
 /* Subcommand: PREP */
 static void boot_prep_linux(bootm_headers_t *images)
 {

@@ -264,8 +252,13 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)
 		r2 = gd->bd->bi_boot_params;
 
 	if (!fake) {
-		do_nonsec_virt_switch();
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
+		armv7_init_nonsec();
+		secure_ram_addr(_do_nonsec_entry)(kernel_entry,
+						  0, machid, r2);
+#else
 		kernel_entry(0, machid, r2);
+#endif
 	}
 }
 

1.8.4.3

Hi Marc,

It seems like patch 07/13 does not apply properly on top of current ARM. Can you have a look? If a rebased V3 is needed, you can remove patch 01 from the list (or if it is simpler to you, keep it in and I'll just ignore it when applying).

Amicalement,

Albert.

Hi Albert,

I don't think there was a V3 for this so far; correct?

Amicalement,

Albert.

Correct. I've been lazy. I'll try to repost something by the end of the week.

Thanks,

	M.

Jazz is not dead. It just smells funny...

Implement core support for PSCI. As this is generic code, it doesn't implement anything really useful (all the functions are returning Not Implemented).

---
 arch/arm/cpu/armv7/Makefile |   4 ++
 arch/arm/cpu/armv7/psci.S   | 113 ++++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/psci.h |  35 ++++++++++++++
 3 files changed, 152 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/psci.S
 create mode 100644 arch/arm/include/asm/psci.h

index 4069aa4..8027385 100644
@@ -24,6 +24,10 @@ obj-y	+= nonsec_virt.o
 obj-y	+= virt-v7.o
 endif
 
+ifneq ($(CONFIG_ARMV7_PSCI),)
+obj-y	+= psci.o
+endif
+
 obj-$(CONFIG_OMAP_COMMON) += omap-common/
 obj-$(CONFIG_TEGRA) += tegra-common/
 

new file mode 100644
index 0000000..704b4b0
@@ -0,0 +1,35 @@
+/*
+ * Copyright (C) 2013 - ARM Ltd
+ * Author: Marc Zyngier <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __ARM_PSCI_H__
+#define __ARM_PSCI_H__
+
+/* PSCI interface */
+#define ARM_PSCI_FN_BASE		0x95c1ba5e
+#define ARM_PSCI_FN(n)			(ARM_PSCI_FN_BASE + (n))
+
+#define ARM_PSCI_FN_CPU_SUSPEND		ARM_PSCI_FN(0)
+#define ARM_PSCI_FN_CPU_OFF		ARM_PSCI_FN(1)
+#define ARM_PSCI_FN_CPU_ON		ARM_PSCI_FN(2)
+#define ARM_PSCI_FN_MIGRATE		ARM_PSCI_FN(3)
+
+#define ARM_PSCI_RET_SUCCESS		0
+#define ARM_PSCI_RET_NI			(-1)
+#define ARM_PSCI_RET_INVAL		(-2)
+#define ARM_PSCI_RET_DENIED		(-3)
+
+#endif /* __ARM_PSCI_H__ */

1.8.4.3

Allow the switch to a second stage secure monitor just before switching to non-secure.

This allows a resident piece of firmware to be active once the kernel has been entered (the u-boot monitor is dead anyway, its pages being reused).

---
 arch/arm/cpu/armv7/nonsec_virt.S | 6 ++++++
 1 file changed, 6 insertions(+)

index 35f5463..6130a37 100644
@@ -45,6 +45,12 @@ _monitor_vectors:
  * ip: target PC
  */
 _secure_monitor:
+#ifdef CONFIG_ARMV7_PSCI
+	ldr	r5, =_psci_vectors		@ Switch to the next monitor
+	mcr	p15, 0, r5, c12, c0, 1
+	isb
+#endif
+
 	mrc	p15, 0, r5, c1, c1, 0		@ read SCR
 	bic	r5, r5, #0x4e			@ clear IRQ, FIQ, EA, nET bits
 	orr	r5, r5, #0x31			@ enable NS, AW, FW bits

1.8.4.3

Generate the PSCI node in the device tree.

Also add a reserve section for the "secure" code that lives in in normal RAM, so that the kernel knows it'd better not trip on it.

---
 arch/arm/cpu/armv7/Makefile  |   1 +
 arch/arm/cpu/armv7/virt-dt.c | 102 +++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h |   1 +
 arch/arm/lib/bootm.c         |   2 +
 4 files changed, 106 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/virt-dt.c

index 8027385..93a5242 100644
@@ -22,6 +22,7 @@ endif
 ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
 obj-y	+= nonsec_virt.o
 obj-y	+= virt-v7.o
+obj-y	+= virt-dt.o
 endif
 
 ifneq ($(CONFIG_ARMV7_PSCI),)

index 11476dd..323f282 100644
@@ -79,6 +79,7 @@ void v7_outer_cache_inval_range(u32 start, u32 end);
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 
 int armv7_init_nonsec(void);
+int armv7_update_dt(void *fdt);
 
 /* defined in assembly file */
 unsigned int _nonsec_init(void);

index 408a774..1709eec 100644
@@ -253,6 +253,8 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)
 
 	if (!fake) {
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
+		if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len)
+			armv7_update_dt(images->ft_addr);
 		armv7_init_nonsec();
 		secure_ram_addr(_do_nonsec_entry)(kernel_entry,
 						  0, machid, r2);

1.8.4.3

Move the B and D SRAM bank to their actual location (or at least where the documentation pretends they are).

---
 arch/arm/include/asm/arch-sunxi/cpu.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 378989c..41fb8dc 100644
@@ -31,8 +31,8 @@
 #define SUNXI_SRAM_A2_BASE		0x00004000	/* 16 kiB */
 #define SUNXI_SRAM_A3_BASE		0x00008000	/* 13 kiB */
 #define SUNXI_SRAM_A4_BASE		0x0000b400	/* 3 kiB */
-#define SUNXI_SRAM_D_BASE		0x01c00000
-#define SUNXI_SRAM_B_BASE		0x01c00000	/* 64 kiB (secure) */
+#define SUNXI_SRAM_D_BASE		0x00010000	/* 4 kiB */
+#define SUNXI_SRAM_B_BASE		0x00020000	/* 64 kiB (secure) */
 
 #define SUNXI_SRAMC_BASE		0x01c00000
 #define SUNXI_DRAMC_BASE		0x01c01000

1.8.4.3

So far, only supporting the CPU_ON method.
Other functions can be added later.

---
 arch/arm/cpu/armv7/sunxi/Makefile        |   3 +
 arch/arm/cpu/armv7/sunxi/psci.S          | 124 +++++++++++++++++++++++++++++++
 arch/arm/cpu/armv7/sunxi/u-boot-psci.lds |  63 ++++++++++++++++
 include/configs/sun7i.h                  |   6 ++
 4 files changed, 196 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/sunxi/psci.S
 create mode 100644 arch/arm/cpu/armv7/sunxi/u-boot-psci.lds

index f71a26d..0369049 100644
@@ -43,6 +43,9 @@ obj-y	+= cpu_info.o
 ifdef CONFIG_CMD_WATCHDOG
 obj-$(CONFIG_CMD_WATCHDOG)	+= cmd_watchdog.o
 endif
+ifdef CONFIG_ARMV7_PSCI
+obj-y	+= psci.o
+endif
 endif
 
 ifdef CONFIG_SPL_BUILD

index a6ede2a..c3c24fb 100644
@@ -38,6 +38,12 @@
 #define CONFIG_BOARD_POSTCLK_INIT 1
 #endif
 
+#define CONFIG_ARMV7_VIRT		1
+#define CONFIG_ARMV7_NONSEC		1
+#define CONFIG_ARMV7_PSCI		1
+#define CONFIG_ARMV7_PSCI_NR_CPUS	2
+#define CONFIG_ARMV7_SECURE_BASE	SUNXI_SRAM_B_BASE
+
 /*
  * Include common sunxi configuration where most the settings are
  */

1.8.4.3

CNTFRQ needs to be properly configured on all CPUs. Otherwise, virtual machines hoping to find valuable information on secondary CPUs will be disapointed...

---
 include/configs/sun7i.h | 1 +
 1 file changed, 1 insertion(+)

index c3c24fb..9098541 100644
@@ -43,6 +43,7 @@
 #define CONFIG_ARMV7_PSCI		1
 #define CONFIG_ARMV7_PSCI_NR_CPUS	2
 #define CONFIG_ARMV7_SECURE_BASE	SUNXI_SRAM_B_BASE
+#define CONFIG_SYS_CLK_FREQ		24000000
 
 /*
  * Include common sunxi configuration where most the settings are

1.8.4.3

Indeed, I booted with this series on a cb2 and Xen SMP Just Worked.

Thanks!

Ian.

Profit! ;-)

Thanks for testing!

	M.

Jazz is not dead. It just smells funny...

Hi,

Cool! Can you push the u-boot code you're using to your u-boot repo:
http://xenbits.xen.org/gitweb/?p=people/ianc/u-boot.git;a=summary

? that will save me reproducing your work.

Thanks & Regards,

Hans

I've just created
git://git.kernel.org/pub/scm/linux/kernel/git/maz/u-boot.git wip/psci

The mirrors seem a bit slow at the moment, but they should hopefully catch up.

	M.

Jazz is not dead. It just smells funny...

Done, see the devel/sunxi-psci branch.

Cheers,
Ian.

Hi Marc,

There are comments re 01/13 an 07/13. Should I expect a V3 on the U-Boot ML?

Amicalement,

Albert.

Hi Albert,

Yes. As I said yesterday, I plan to address the comments pretty quickly (during the weekend anyway), and repost a new version on the same mailing lists.

Thanks,

	M.

Jazz is not dead. It just smells funny...

Hi Marc Zyngier

Where could obtain this repository with these alterations to be able to test
virtualization with KVM on my cubieboard2'm using Kernel 3.14.rc4

I appreciate if you help

Thank you

Hi,

You're looking at an older patch series. The new one can be found there:
http://permalink.gmane.org/gmane.comp.hardware.netbook.arm.sunxi/7151

I also have a repository with this code at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/u-boot.git wip/psci

Cheers,

         M.

Who you jivin' with that Cosmik Debris?

Thank?s Marc

Error Compile

How could solve this compilation problem?
What am I doing wrong?

Have since I'm Very Grateful!

make Cubieboard2 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

arm-linux-gnueabihf-gcc -M -g -Os -ffunction-sections -fdata-sections
-fno-common -ffixed-r9 -msoft-float -D__KERNEL__ -DCONFIG_SYS_TEXT_BASE=0x4a000000 -I/root/cubie-test/util/u-boot-maz-wip-psci/include -I/root/cubie-test/util/u-boot-maz-wip-psci/arch/arm/include -fno-builtin -ffreestanding -nostdinc -isystem /usr/lib/gcc/arm-linux-gnueabihf/4.7/include -pipe -DCONFIG_ARM -D__ARM__ -marm -mno-thumb-interwork -mabi=aapcs-linux -mword-relocations
-march=armv7-a -MQ designware.o designware.c >.depend.designware
designware.c:21:3: error: #error "DesignWare Ether MAC requires PHYLIB - missing CONFIG_PHYLIB"
make[2]: *** Sem regra para processar o alvo .depend', necess?rio por built-in.o'. Pare.
make[2]: Saindo do diret?rio `/root/cubie-test/util/u-boot-maz-wip-psci/drivers/net' make[1]: ** [drivers/net/built-in.o] Erro 2
make[1]: Saindo do diret?rio `/root/cubie-test/util/u-boot-maz-wip-psci'
make: ** [Cubieboard2] Erro 2
root at vbi7:~/cubie-test/util/u-boot-maz-wip-psci#

_____________________________________________________________

Em ter?a-feira, 25 de fevereiro de 2014 11h04min55s UTC-3, maz escreveu: