Skip to contentSkip to conversation

ARM: OMAP5: Couple of patches for KVM

PATCH New Santosh Shilimkar 6 participants 17 messages Nov 24, 2013 – Nov 27, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover ARM: OMAP5: Couple of patches for KVM

Couple of patches for OMAP5 machines towards KVM support.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>

Santosh Shilimkar (2):
ARM: dts: OMAP5: Add maintenance interrupt for virtualisation
ARM: OMAP5: Add HYP mode entry support for secondary CPUs

 arch/arm/boot/dts/omap5.dtsi       |    1 +
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 2 files changed, 8 insertions(+)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/2 ARM: dts: OMAP5: Add maintenance interrupt for virtualisation

Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Beno?t Cousson <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/boot/dts/omap5.dtsi |    1 +
 1 file changed, 1 insertion(+)
arch/arm/boot/dts/omap5.dtsi +1 -0
index fc3fad5..c9f1ae4 100644
@@ -74,6 +74,7 @@
 		      <0x48212000 0x1000>,
 		      <0x48214000 0x2000>,
 		      <0x48216000 0x2000>;
+		interrupts = <1 9 0x304>;
 	};
 
 	/*
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH 1/2 ARM: dts: OMAP5: Add maintenance interrupt for virtualisation

Quoted 17 lines · Santosh Shilimkar

Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Beno?t Cousson <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/boot/dts/omap5.dtsi |    1 +
 1 file changed, 1 insertion(+)
arch/arm/boot/dts/omap5.dtsi +0 -0
index fc3fad5..c9f1ae4 100644

arch/arm/boot/dts/omap5.dtsi line 77

Quoted 4 lines
arch/arm/boot/dts/omap5.dtsi +1 -0
@@ -74,6 +74,7 @@
 		      <0x48212000 0x1000>,
 		      <0x48214000 0x2000>,
 		      <0x48216000 0x2000>;
+		interrupts = <1 9 0x304>;

I should have used the GIC flags. Updated version end of the email.

Regards,
Santosh

Quoted 1 line

From 91cbd5f65ccd9a0780614fa7ab5505922bdce577 Mon Sep 17 00:00:00 2001

From: Santosh Shilimkar <[email protected]>
Date: Thu, 12 Sep 2013 15:53:13 -0400
Subject: [PATCH 1/2] ARM: dts: OMAP5: Add maintenance interrupt for
virtualisation

Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Beno?t Cousson <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/boot/dts/omap5.dtsi |    1 +
 1 file changed, 1 insertion(+)
arch/arm/boot/dts/omap5.dtsi +1 -0
index fc3fad5..907ab7b 100644
@@ -74,6 +74,7 @@
 		      <0x48212000 0x1000>,
 		      <0x48214000 0x2000>,
 		      <0x48216000 0x2000>;
+		interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_HIGH)>;
 	};
 
 	/*
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

In reply to Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +7 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA			0x48281800
+#define API_HYP_ENTRY				0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:	ldr	r2, =AUX_CORE_BOOT0_PA	@ read from AuxCoreBoot0
 	and	r4, r4, #0x0f
 	cmp	r0, r4
 	bne	wait
+#ifdef CONFIG_KVM_ARM_HOST
+	ldr	r12, =API_HYP_ENTRY
+	adr	r0, hyp_boot
+	smc	#0
+hyp_boot:
+#endif
 	b	secondary_startup
 END(omap5_secondary_startup)
 /*
Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

Quoted 30 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +1 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 10 lines
arch/arm/mach-omap2/omap-headsmp.S +6 -0
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

-Christoffer

#5 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

Regards,
Santosh

Regards,
Santosh

#6 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 4 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.

Quoted 4 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Quoted 8 lines

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

I agree, and PSCI is the obvious only correct answer to this.

We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.

However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.

-Christoffer

#7 lore raw

HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]

Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 4 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Quoted 7 lines

That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.

Fair enough.

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Quoted 8 lines

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

Quoted 9 lines

I agree, and PSCI is the obvious only correct answer to this.

We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.

However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

CC'ing few more folks and changing the subject line

Regards,
Santosh

#8 lore raw

HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]

Quoted 89 lines · Santosh Shilimkar
Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 4 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Quoted 6 lines

That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.

Fair enough.

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Quoted 8 lines

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

Quoted 8 lines

I agree, and PSCI is the obvious only correct answer to this.

We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.

However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Quoted 4 lines

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

Signature · 1 line
Catalin
#9 lore raw

HYP Kernel boot requirements

Quoted 95 lines · Catalin Marinas
Quoted 89 lines · Santosh Shilimkar
Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 4 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Quoted 6 lines

That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.

Fair enough.

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Quoted 8 lines

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

Quoted 8 lines

I agree, and PSCI is the obvious only correct answer to this.

We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.

However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Quoted 4 lines

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Quoted 6 lines

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

I will look at PSCI more closely and see what can be done here.

Regards,
Santosh

#10 lore raw

HYP Kernel boot requirements

Quoted 19 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Quoted 6 lines

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

Quoted 5 lines

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

Is it late to rewrite the OMAP5 firmware?

What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware.

Quoted 1 line

I will look at PSCI more closely and see what can be done here.

PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)?

Signature · 1 line
Catalin
#11 lore raw

HYP Kernel boot requirements

Quoted 20 lines · Catalin Marinas
Quoted 19 lines · Santosh Shilimkar
Quoted 18 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

Quoted 8 lines
Quoted 5 lines

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

Is it late to rewrite the OMAP5 firmware?

Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.

Quoted 8 lines

What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware.

Quoted 7 lines
Quoted 1 line

I will look at PSCI more closely and see what can be done here.

PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)?

Hmmm.. So thats rules out PSCI for OMAP5 then.

Keystone actually has nice way to deal with the problem since the boot-monitor code can be patched up and hence I had no trouble implementing the "CPU starts in HYP mode" requirement on it.

Regards,
Santosh

#12 lore raw

HYP Kernel boot requirements

Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Quoted 6 lines

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

Quoted 5 lines
Quoted 2 lines

Is it late to rewrite the OMAP5 firmware?

Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.

If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.

Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?

Cheers
---Dave

#13 lore raw

HYP Kernel boot requirements

Quoted 34 lines · Dave Martin
Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Quoted 6 lines

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

Quoted 5 lines
Quoted 2 lines

Is it late to rewrite the OMAP5 firmware?

Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.

Quoted 7 lines

If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.

Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?

On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs.

regards,
Santosh

#14 lore raw

HYP Kernel boot requirements

Quoted 39 lines · Santosh Shilimkar
Quoted 38 lines · Dave Martin
Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar

What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.

Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.

Quoted 6 lines

Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?

How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.

Quoted 5 lines
Quoted 2 lines

Is it late to rewrite the OMAP5 firmware?

Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.

If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.

Quoted 8 lines
Quoted 3 lines

Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?

On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs.

As last resort, you can still write a bunch of instructions acting as warm-bootloader, reserve a page and add them there. Code required is just a shim layer and should not require more than few bytes, after all it has just to check some variables and jump accordingly (+ calling smc if required).

Not pretty, agreed, but still better than patching the kernel IMHO (basically you will end up patching the bootloader - and possibly the dtb - instead, to install the shim mentioned above).

Lorenzo

#15 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

In reply to Santosh Shilimkar
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 3 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Absolutely not. We're way past the "one image per usage" model, and I fully expect the same kernel to boot both as a guest and a host. Otherwise, multiplatform is just a joke.

Quoted 11 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

I definitely agree that you need to consider PM as well. But we've decided a long time ago that we wouldn't take that kind of code in the kernel. Your CPU has to reach the kernel in HYP, and that is down to your boot protocol to take care off that.

There are a few options. Christoffer mentioned having a pen, and you also could implement PSCI fairly easily (I've recently posted a u-boot patch series to take care of this, though there is still a fair bit of work required).

Whatever the approach is, the bottom line remains: your CPU enters the kernel in HYP.

Cheers,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#16 lore raw

PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs

Quoted 72 lines · Marc Zyngier
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar

Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.

Hence patch the secondary CPU boot path for hyp mode etry.

Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
Cc Tony Lindgren <[email protected]>
---
 arch/arm/mach-omap2/omap-headsmp.S |    7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +5 -0
index 75e9295..4844dd8 100644
@@ -22,6 +22,7 @@
 
 /* Physical address needed since MMU not enabled yet on secondary core */
 #define AUX_CORE_BOOT0_PA                      0x48281800
+#define API_HYP_ENTRY                          0x102
 
 /*
  * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait:        ldr     r2, =AUX_CORE_BOOT0_PA  @ read from AuxCoreBoot0
        and     r4, r4, #0x0f
        cmp     r0, r4
        bne     wait
+#ifdef CONFIG_KVM_ARM_HOST
+       ldr     r12, =API_HYP_ENTRY
+       adr     r0, hyp_boot
+       smc     #0
Quoted 9 lines

arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50

Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2 -0
+hyp_boot:
+#endif
        b       secondary_startup
 END(omap5_secondary_startup)
 /*

hmm, this means that currently running this in a guest will fail to bring-up SMP, right?

Quoted 3 lines

Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?

Absolutely not. We're way past the "one image per usage" model, and I fully expect the same kernel to boot both as a guest and a host. Otherwise, multiplatform is just a joke.

Quoted 11 lines
Quoted 4 lines

Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?

Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.

In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.

I definitely agree that you need to consider PM as well. But we've decided a long time ago that we wouldn't take that kind of code in the kernel. Your CPU has to reach the kernel in HYP, and that is down to your boot protocol to take care off that.

Quoted 8 lines

There are a few options. Christoffer mentioned having a pen, and you also could implement PSCI fairly easily (I've recently posted a u-boot patch series to take care of this, though there is still a fair bit of work required).

Whatever the approach is, the bottom line remains: your CPU enters the kernel in HYP.

Our emails crossed. I understand the stand from you guys. Lets continue discussion on other thread.