ARM: Machine specific boot blobs

Add a Linux-specific pre-boot routine that matches the device- specific bootloaders behaviour. This is needed for modern Linux that expects the ARM PLL in SLCR to be a more even value (not 26).

Cc Alistair Francis <[email protected]>

Signed-off-by Peter Crosthwaite <[email protected]>

---
Changed since v1:
Left main loader address unchanged
Added comments for SLCR_WRITE macro (PMM review) Convert SLCR_WRITE impl. to use movw/movt (PMM review) Missing "-" for device-specific
Changed since RFC:
Use bootloader callback to load blob.
Change "firmware" to "board-setup" for consistency.

 hw/arm/xilinx_zynq.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

hw/arm/xilinx_zynq.c +42 -1

index 9f89483..82a9db8 100644
@@ -43,6 +43,45 @@ static const int dma_irqs[8] = {
     46, 47, 48, 49, 72, 73, 74, 75
 };
 
+#define BOARD_SETUP_ADDR        0x100
+
+#define SLCR_LOCK_OFFSET        0x004
+#define SLCR_UNLOCK_OFFSET      0x008
+#define SLCR_ARM_PLL_OFFSET     0x100
+
+#define SLCR_XILINX_UNLOCK_KEY  0xdf0d
+#define SLCR_XILINX_LOCK_KEY    0x767b
+
+#define ARMV7_IMM16(x) (extract32((x),  0, 12) | \
+                        extract32((x), 12,  4) << 16)
+
+/* Write immediate val to address r0 + addr. r0 should contain base offset
+ * of the SLCR block. Clobbers r1.
+ */
+
+#define SLCR_WRITE(addr, val) \
+    0xe3001000 + ARMV7_IMM16(extract32((val),  0, 16)), /* movw r1 ... */ \
+    0xe3401000 + ARMV7_IMM16(extract32((val), 16, 16)), /* movt r1 ... */ \
+    0xe5801000 + (addr)
+
+static void zynq_write_board_setup(ARMCPU *cpu,
+                                   const struct arm_boot_info *info)
+{
+    int n;
+    uint32_t board_setup_blob[] = {
+        0xe3a004f8, /* mov r0, #0xf8000000 */
+        SLCR_WRITE(SLCR_UNLOCK_OFFSET, SLCR_XILINX_UNLOCK_KEY),
+        SLCR_WRITE(SLCR_ARM_PLL_OFFSET, 0x00014008),
+        SLCR_WRITE(SLCR_LOCK_OFFSET, SLCR_XILINX_LOCK_KEY),
+        0xe12fff1e, /* bx lr */
+    };
+    for (n = 0; n < ARRAY_SIZE(board_setup_blob); n++) {
+        board_setup_blob[n] = tswap32(board_setup_blob[n]);
+    }
+    rom_add_blob_fixed("board-setup", board_setup_blob,
+                       sizeof(board_setup_blob), BOARD_SETUP_ADDR);
+}
+
 static struct arm_boot_info zynq_binfo = {};
 
 static void gem_init(NICInfo *nd, uint32_t base, qemu_irq irq)

@@ -252,6 +291,9 @@ static void zynq_init(MachineState *machine)
     zynq_binfo.nb_cpus = 1;
     zynq_binfo.board_id = 0xd32;
     zynq_binfo.loader_start = 0;
+    zynq_binfo.board_setup_addr = BOARD_SETUP_ADDR;
+    zynq_binfo.write_board_setup = zynq_write_board_setup;
+
     arm_load_kernel(ARM_CPU(first_cpu), &zynq_binfo);
 }
 
--

1.9.1

Add a flag that when set, will cause the primary CPU to start in secure mode, even if the overall boot in non-secure. This is useful for when

there is a board-setup blob that needs to run from secure mode, but device and secondary CPU init should still be done as-normal for a non- secure boot.

Signed-off-by Peter Crosthwaite <[email protected]>

---

 hw/arm/boot.c        | 3 ++-
 include/hw/arm/arm.h | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

hw/arm/boot.c +2 -1

index b0879a5..6680d45 100644
@@ -495,7 +495,8 @@ static void do_cpu_reset(void *opaque)
                 }
 
                 /* Set to non-secure if not a secure boot */
-                if (!info->secure_boot) {
+                if (!info->secure_boot &&
+                    (cs != first_cpu || !info->secure_board_setup)) {
                     /* Linux expects non-secure state */
                     env->cp15.scr_el3 |= SCR_NS;
                 }

include/hw/arm/arm.h +0 -0

index 9217b70..60dc919 100644

Quoted 2 lines · Peter Crosthwaite

Add a flag that when set, will cause the primary CPU to start in secure mode, even if the overall boot in non-secure. This is useful for when

"is non-secure".

Quoted 36 lines

there is a board-setup blob that needs to run from secure mode, but device and secondary CPU init should still be done as-normal for a non- secure boot.

Signed-off-by Peter Crosthwaite <[email protected]>

---

 hw/arm/boot.c        | 3 ++-
 include/hw/arm/arm.h | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

hw/arm/boot.c +2 -1

index b0879a5..6680d45 100644
@@ -495,7 +495,8 @@ static void do_cpu_reset(void *opaque)
                 }
 
                 /* Set to non-secure if not a secure boot */
-                if (!info->secure_boot) {
+                if (!info->secure_boot &&
+                    (cs != first_cpu || !info->secure_board_setup)) {
                     /* Linux expects non-secure state */
                     env->cp15.scr_el3 |= SCR_NS;
                 }

include/hw/arm/arm.h +3 -0

index 9217b70..60dc919 100644
@@ -97,6 +97,12 @@ struct arm_boot_info {
     hwaddr board_setup_addr;
     void (*write_board_setup)(ARMCPU *cpu,
                               const struct arm_boot_info *info);
+
+    /* If set, the board specific loader/setup blob will be run from secure
+     * mode, regardless of secure_boot. The blob becomes responsible for

include/hw/arm/arm.h struct arm_boot_info { · line 106

Quoted 4 lines

include/hw/arm/arm.h +3 -0

+     * changing to non-secure state if implementing a non-secure boot
+     */
+    bool secure_board_setup;
 };

I thought you were planning to have the generic code do the S->NS transition; but I guess it works better in the board code (we have to go up into Monitor and back down again, right?)

Is it an error for the board to set secure_board_setup if the CPU doesn't have EL3? (if so, worth mentioning in this comment; maybe assert?)

thanks
-- PMM

Quoted 4 lines · Peter Maydell

I thought you were planning to have the generic code do the S->NS transition; but I guess it works better in the board code (we have to go up into Monitor and back down again, right?)

Yes I had to change my mind on this one. The issue was that ARM arch doesn't guarantee a NS switch by simply modding SCR.NS inline and I wanted to follow this convention. The recommended way is via eret (presumably from monitor mode?). So to implement this for highbank I do a dummy SMC after the SCR.NS switch (from secure EL1). This can't be done generically as board-setup may or may-not install a functional monitor.

Quoted 4 lines

Is it an error for the board to set secure_board_setup if the CPU doesn't have EL3? (if so, worth mentioning in this comment; maybe assert?)

I don't like assert, as has_el3 is in theory is user modifiable (via either -cpu transplants or directly via -global). I think it is an error_exit().

Quoted 14 lines · Peter Crosthwaite

Quoted 4 lines · Peter Maydell

I thought you were planning to have the generic code do the S->NS transition; but I guess it works better in the board code (we have to go up into Monitor and back down again, right?)

Yes I had to change my mind on this one. The issue was that ARM arch doesn't guarantee a NS switch by simply modding SCR.NS inline and I wanted to follow this convention. The recommended way is via eret (presumably from monitor mode?). So to implement this for highbank I do a dummy SMC after the SCR.NS switch (from secure EL1). This can't be done generically as board-setup may or may-not install a functional monitor.

Quoted 8 lines

Quoted 4 lines

Is it an error for the board to set secure_board_setup if the CPU doesn't have EL3? (if so, worth mentioning in this comment; maybe assert?)

I don't like assert, as has_el3 is in theory is user modifiable (via either -cpu transplants or directly via -global). I think it is an error_exit().

The other question is what happens on a board like this if the user says -enable-kvm -cpu cortex-a15 ? Does that get us a CPU without the EL3 property? (I forget...) In any case it shouldn't be an error unless the board genuinely can't work with KVM at all, and if we're using KVM then the board blob definitely won't be running in Secure (and can't flip to Monitor mode either).

thanks
-- PMM

Quoted 7 lines · Peter Maydell

The other question is what happens on a board like this if the user says -enable-kvm -cpu cortex-a15 ? Does that get us a CPU without the EL3 property? (I forget...) In any case it shouldn't be an error unless the board genuinely can't work with KVM at all, and if we're using KVM then the board blob definitely won't be running in Secure (and can't flip to Monitor mode either).

I think all we can do is exit on !kvm and have the board if() the firmware blob for the same. What is supposed to actually happen when a virtualized guest running under KVM calls SMC? Does the VM have any say on what that SMC does or is that the property of the host OS? The latter suggests that Highbank Linux simply cannot be run under KVM.

Quoted 9 lines · Peter Crosthwaite

Quoted 7 lines · Peter Maydell

The other question is what happens on a board like this if the user says -enable-kvm -cpu cortex-a15 ? Does that get us a CPU without the EL3 property? (I forget...) In any case it shouldn't be an error unless the board genuinely can't work with KVM at all, and if we're using KVM then the board blob definitely won't be running in Secure (and can't flip to Monitor mode either).

Quoted 5 lines

I think all we can do is exit on !kvm and have the board if() the firmware blob for the same. What is supposed to actually happen when a virtualized guest running under KVM calls SMC? Does the VM have any say on what that SMC does or is that the property of the host OS? The latter suggests that Highbank Linux simply cannot be run under KVM.

The VM can intercept SMC, yes (HCR.TSC causes SMC to trap to Hyp, regardless of what the SCR settings are). I don't know whether KVM actually does trap it. Typically the need for SMC calls seems to have only been for A9 and earlier CPUs, though -- mostly the A15 managed to avoid the requirement, so it hasn't been necessary for KVM to deal with random board specific SMC handling. (Do we need the SMC handling for the A15 variant of the highbank model?)

thanks
-- PMM

Quoted 14 lines · Peter Crosthwaite

Quoted 4 lines · Peter Maydell

I thought you were planning to have the generic code do the S->NS transition; but I guess it works better in the board code (we have to go up into Monitor and back down again, right?)

Yes I had to change my mind on this one. The issue was that ARM arch doesn't guarantee a NS switch by simply modding SCR.NS inline and I wanted to follow this convention. The recommended way is via eret (presumably from monitor mode?). So to implement this for highbank I do a dummy SMC after the SCR.NS switch (from secure EL1). This can't be done generically as board-setup may or may-not install a functional monitor.

Quoted 4 lines

Quoted 4 lines

Is it an error for the board to set secure_board_setup if the CPU doesn't have EL3? (if so, worth mentioning in this comment; maybe assert?)

Quoted 4 lines

I don't like assert, as has_el3 is in theory is user modifiable (via either -cpu transplants or directly via -global). I think it is an error_exit().

The other question is what happens on a board like this if the user says -enable-kvm -cpu cortex-a15 ? Does that get us a CPU without the EL3 property? (I forget...) In any case it shouldn't be an error unless the board genuinely can't work with KVM at all, and if we're using KVM then the board blob definitely won't be running in Secure (and can't flip to Monitor mode either).

thanks
-- PMM

Quoted 7 lines · Peter Maydell

Quoted 5 lines · Peter Crosthwaite

Quoted 4 lines · Peter Maydell

Is it an error for the board to set secure_board_setup if the CPU doesn't have EL3? (if so, worth mentioning in this comment; maybe assert?)

Does the same apply to secure_boot flag as well? I can't see any guarding or erroring out there. secure_boot is simply ignored when there is no EL3, and the same if on EL3 will apply to this new flag as well. If we need to fix this, we should fix them both together.

Firstly, enable monitor mode and PSCI, both are which are features of this board.

In addition to PSCI, this board also uses SMC for cache maintainence ops. This means we need a secure monitor to catch these and nop them. Use the ARM boot board-setup feature to implement this. All traps to monitor mode implement the nop.

Signed-off-by Peter Crosthwaite <[email protected]>

---
Changed since v1:
fallthrough all of trap table to nop implementation use movw for table address
leave loader at 0
Move MVBAR (and blob to non-zero)
Split nop implementation from MVBAR setup set secure boot for board
implement NS switch in blob
Changed since RFC:
Use bootloader callback to load blob. Change "firmware" to "board-setup" for consistency. Tweak commit message.

 hw/arm/highbank.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 56 insertions(+), 10 deletions(-)

hw/arm/highbank.c +28 -0

index be04b27..f3578a3 100644
@@ -32,10 +32,55 @@
 #define SMP_BOOT_REG            0x40
 #define MPCORE_PERIPHBASE       0xfff10000
 
+#define MVBAR_ADDR              0x200
+
 #define NIRQ_GIC                160
 
 /* Board init.  */
 
+/* MVBAR_ADDR is limited by precision of movw */
+
+QEMU_BUILD_BUG_ON(MVBAR_ADDR >= (1 << 16));
+
+#define ARMV7_IMM16(x) (extract32((x),  0, 12) | \
+                        extract32((x), 12,  4) << 16)
+
+static void hb_write_board_setup(ARMCPU *cpu,
+                                 const struct arm_boot_info *info)
+{
+    int n;
+    uint32_t board_setup_blob[] = {
+        /* MVBAR_ADDR */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+#define ERET_ADDR (MVBAR_ADDR + 8 * sizeof(uint32_t))
+        0xe58fe008, /* save lr */
+        0xe8dfc000, /* exception return */
+        0,
+        0,

hw/arm/highbank.c +14 -10

+    for (n = 0; n < ARRAY_SIZE(board_setup_blob); n++) {
+        board_setup_blob[n] = tswap32(board_setup_blob[n]);
+    }
+    rom_add_blob_fixed("board-setup", board_setup_blob,
+                       sizeof(board_setup_blob), MVBAR_ADDR);
+}
+
 static void hb_write_secondary(ARMCPU *cpu, const struct arm_boot_info *info)
 {
     int n;

@@ -248,16 +293,13 @@ static void calxeda_init(MachineState *machine, enum cxmachines machine_id)
         cpuobj = object_new(object_class_get_name(oc));
         cpu = ARM_CPU(cpuobj);
 
-        /* By default A9 and A15 CPUs have EL3 enabled.  This board does not
-         * currently support EL3 so the CPU EL3 property is disabled before
-         * realization.
-         */
-        if (object_property_find(cpuobj, "has_el3", NULL)) {
-            object_property_set_bool(cpuobj, false, "has_el3", &err);
-            if (err) {
-                error_report_err(err);
-                exit(1);
-            }
+        object_property_set_int(cpuobj, QEMU_PSCI_CONDUIT_SMC,
+                                "psci-conduit", &error_abort);
+
+        if (n) {
+            /* Secondary CPUs start in PSCI powered-down state */
+            object_property_set_bool(cpuobj, true,
+                                     "start-powered-off", &error_abort);
         }
 
         if (object_property_find(cpuobj, "reset-cbar", NULL)) {

Quoted 66 lines · Peter Crosthwaite

Firstly, enable monitor mode and PSCI, both are which are features of this board.

In addition to PSCI, this board also uses SMC for cache maintainence ops. This means we need a secure monitor to catch these and nop them. Use the ARM boot board-setup feature to implement this. All traps to monitor mode implement the nop.

Signed-off-by Peter Crosthwaite <[email protected]>

---
Changed since v1:
fallthrough all of trap table to nop implementation use movw for table address
leave loader at 0
Move MVBAR (and blob to non-zero)
Split nop implementation from MVBAR setup set secure boot for board
implement NS switch in blob
Changed since RFC:
Use bootloader callback to load blob. Change "firmware" to "board-setup" for consistency. Tweak commit message.

 hw/arm/highbank.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 56 insertions(+), 10 deletions(-)

hw/arm/highbank.c +28 -0

index be04b27..f3578a3 100644
@@ -32,10 +32,55 @@
 #define SMP_BOOT_REG            0x40
 #define MPCORE_PERIPHBASE       0xfff10000
 
+#define MVBAR_ADDR              0x200
+
 #define NIRQ_GIC                160
 
 /* Board init.  */
 
+/* MVBAR_ADDR is limited by precision of movw */
+
+QEMU_BUILD_BUG_ON(MVBAR_ADDR >= (1 << 16));
+
+#define ARMV7_IMM16(x) (extract32((x),  0, 12) | \
+                        extract32((x), 12,  4) << 16)
+
+static void hb_write_board_setup(ARMCPU *cpu,
+                                 const struct arm_boot_info *info)
+{
+    int n;
+    uint32_t board_setup_blob[] = {
+        /* MVBAR_ADDR */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+        0xe320f000, /* nop */
+#define ERET_ADDR (MVBAR_ADDR + 8 * sizeof(uint32_t))
+        0xe58fe008, /* save lr */
+        0xe8dfc000, /* exception return */
+        0,
+        0,

hw/arm/highbank.c line 76

Quoted 10 lines

hw/arm/highbank.c +10 -0

+        0, /* exception return link will end up here */
+#define BOARD_SETUP_ADDR (ERET_ADDR + 5 * sizeof(uint32_t))
+        0xe3000000 + ARMV7_IMM16(MVBAR_ADDR), /* movw r0, MVBAR_ADDR */
+        0xee0c0f30, /* mcr p15, 0, r0, c12, c0, 1 - set mvbar */
+        0xee110f11, /* mrc p15, 0, r0, c1 , c1, 0 - get scr */
+        0xe3810001, /* orr r0, #1 - set NS */
+        0xee010f11, /* mcr p15, 0, r0, c1 , c1, 0 - set scr */
+        0xe1600070, /* smc - go to monitor mode to flush NS change */
+        0xe12fff1e, /* bx lr - return to caller */
+    };

This still confuses me. What I was expecting to see was something like:

    /* Monitor mode vector table; entry points which will only be reached
     * if the guest kernel is buggy are tight loops to avoid executing
     * random code by accident. The SMC vector entrypoint just returns,
     * making all SMC calls operate as NOPs.
     */
    0xwhatever, /* notused1: b notused1 */
    0xwhatever, /* notused2: b notused2 */
    0xwhatever, /* smc: movs pc, lr */
    0xwhatever, /* prefetch_abort: b prefetch_abort */
    0xwhatever, /* data_abort: b data_abort */
    0xwhatever, /* notused3: b notused3 */
    0xwhatever, /* irq: b irq */
    0xwhatever, /* fiq: b fiq */

/* Entry point for reset (set via board_setup_addr) */ [code here to set mvbar, etc]

Note that MOVS PC, LR is much simpler for exception return than RFE, because you don't need to mess about saving the LR and SPSR to memory in order to just read them back again. I'd have used ERET, which has a more explanatory mnemonic, but that only exists in CPUs with the virtualization extensions.

Quoted 33 lines

hw/arm/highbank.c +14 -10

+    for (n = 0; n < ARRAY_SIZE(board_setup_blob); n++) {
+        board_setup_blob[n] = tswap32(board_setup_blob[n]);
+    }
+    rom_add_blob_fixed("board-setup", board_setup_blob,
+                       sizeof(board_setup_blob), MVBAR_ADDR);
+}
+
 static void hb_write_secondary(ARMCPU *cpu, const struct arm_boot_info *info)
 {
     int n;

@@ -248,16 +293,13 @@ static void calxeda_init(MachineState *machine, enum cxmachines machine_id)
         cpuobj = object_new(object_class_get_name(oc));
         cpu = ARM_CPU(cpuobj);
 
-        /* By default A9 and A15 CPUs have EL3 enabled.  This board does not
-         * currently support EL3 so the CPU EL3 property is disabled before
-         * realization.
-         */
-        if (object_property_find(cpuobj, "has_el3", NULL)) {
-            object_property_set_bool(cpuobj, false, "has_el3", &err);
-            if (err) {
-                error_report_err(err);
-                exit(1);
-            }
+        object_property_set_int(cpuobj, QEMU_PSCI_CONDUIT_SMC,
+                                "psci-conduit", &error_abort);
+
+        if (n) {
+            /* Secondary CPUs start in PSCI powered-down state */
+            object_property_set_bool(cpuobj, true,
+                                     "start-powered-off", &error_abort);
         }
 

hw/arm/highbank.c static void calxeda_init(MachineState *machine, enum cxmachines machine_id) · line 427

Quoted 10 lines

hw/arm/highbank.c +4 -0

         if (object_property_find(cpuobj, "reset-cbar", NULL)) {

@@ -378,6 +420,10 @@ static void calxeda_init(MachineState *machine, enum cxmachines machine_id)
     highbank_binfo.loader_start = 0;
     highbank_binfo.write_secondary_boot = hb_write_secondary;
     highbank_binfo.secondary_cpu_reset_hook = hb_reset_secondary;
+    highbank_binfo.board_setup_addr = BOARD_SETUP_ADDR;
+    highbank_binfo.write_board_setup = hb_write_board_setup;
+    highbank_binfo.secure_board_setup = true;
+
     arm_load_kernel(ARM_CPU(first_cpu), &highbank_binfo);

If the user hands us "-cpu something-without-el3" I think we'll now go blithely ahead with the secure_board_setup with hilarious consquences, where previously we'd probably at least manage to try to boot the kernel. (You can argue that the user shouldn't be handing us silly arguments if you like.)

thanks
-- PMM

Quoted 15 lines · Peter Maydell

This still confuses me. What I was expecting to see was something like:

    /* Monitor mode vector table; entry points which will only be reached
     * if the guest kernel is buggy are tight loops to avoid executing
     * random code by accident. The SMC vector entrypoint just returns,
     * making all SMC calls operate as NOPs.
     */
    0xwhatever, /* notused1: b notused1 */
    0xwhatever, /* notused2: b notused2 */
    0xwhatever, /* smc: movs pc, lr */
    0xwhatever, /* prefetch_abort: b prefetch_abort */
    0xwhatever, /* data_abort: b data_abort */
    0xwhatever, /* notused3: b notused3 */
    0xwhatever, /* irq: b irq */
    0xwhatever, /* fiq: b fiq */

I don't get the difference between notused and prefetch_abort etc. The only one that is used is SMC. Which ones should tight loop and which ones should nop? Do we actually have to branch somewhere or can we get away with one-inst tight loop / nops as appropriate?

Hi,

This adds support for machine-specific primary boot blobs. This can be used to install little bits of firmware or boot code without having to throw the whole QEMU bootloader out and BYO (with device drivers and all).

It is then used to fix two boards, Zynq and Highbank, both which have small but critical expectations of pre-boot software setup.

Regards,
Peter

Changed since v1:
Addressed PMM review.
Added secure_board_setup flag (P4)
Added Zynq patch first, then Highbank
See indiv. patches for detailed change logs.

Peter Crosthwaite (5):
arm: boot: Adjust indentation of FIXUP comments
arm: boot: Add board specific setup code API
arm: xilinx_zynq: Add linux pre-boot
arm: boot: Add secure_board_setup flag
arm: highbank: Implement PSCI and dummy monitor