ARM: OMAP5: Couple of patches for KVM
Subsystems (4)
cover ARM: OMAP5: Couple of patches for KVM
Couple of patches for OMAP5 machines towards KVM support.
Santosh Shilimkar (2):
ARM: dts: OMAP5: Add maintenance interrupt for virtualisation
ARM: OMAP5: Add HYP mode entry support for secondary CPUs
arch/arm/boot/dts/omap5.dtsi | 1 + arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 2 files changed, 8 insertions(+)
Signature · 1 line
1.7.9.5
PATCH 1/2 ARM: dts: OMAP5: Add maintenance interrupt for virtualisation
Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.
--- arch/arm/boot/dts/omap5.dtsi | 1 + 1 file changed, 1 insertion(+)
arch/arm/boot/dts/omap5.dtsi +1 -0
@@ -74,6 +74,7 @@ <0x48212000 0x1000>, <0x48214000 0x2000>, <0x48216000 0x2000>; + interrupts = <1 9 0x304>; }; /*
Signature · 1 line
1.7.9.5
PATCH 1/2 ARM: dts: OMAP5: Add maintenance interrupt for virtualisation
Quoted 17 lines · Santosh Shilimkar
Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Beno?t Cousson <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/boot/dts/omap5.dtsi | 1 + 1 file changed, 1 insertion(+)arch/arm/boot/dts/omap5.dtsi +0
-0
arch/arm/boot/dts/omap5.dtsi line 77
Quoted 4 lines
arch/arm/boot/dts/omap5.dtsi +1
-0@@ -74,6 +74,7 @@ <0x48212000 0x1000>, <0x48214000 0x2000>, <0x48216000 0x2000>; + interrupts = <1 9 0x304>;
I should have used the GIC flags. Updated version end of the email.
Regards,
Santosh
Quoted 1 line
From
91cbd5f65ccd9a0780614fa7ab5505922bdce577Mon Sep 17 00:00:00 2001
From: Santosh Shilimkar <[email protected]>
Date: Thu, 12 Sep 2013 15:53:13 -0400
Subject: [PATCH 1/2] ARM: dts: OMAP5: Add maintenance interrupt for
virtualisation
Add a maintenance IRQ using PPI 9 to OMAP5 device tree needed for virtualisation.
--- arch/arm/boot/dts/omap5.dtsi | 1 + 1 file changed, 1 insertion(+)
arch/arm/boot/dts/omap5.dtsi +1 -0
@@ -74,6 +74,7 @@ <0x48212000 0x1000>, <0x48214000 0x2000>, <0x48216000 0x2000>; + interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_HIGH)>; }; /*
Signature · 1 line
1.7.9.5
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
In reply to Santosh ShilimkarBoot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)
arch/arm/mach-omap2/omap-headsmp.S +7 -0
@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM
@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*
Signature · 1 line
1.7.9.5
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
Quoted 30 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +1
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 10 lines
arch/arm/mach-omap2/omap-headsmp.S +6
-0bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*
hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
-Christoffer
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0
Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ?
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
Regards,
Santosh
Regards,
Santosh
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 4 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?
That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.
Quoted 4 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Quoted 8 lines
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
I agree, and PSCI is the obvious only correct answer to this.
We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.
However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.
-Christoffer
HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]
Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 4 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?
Quoted 7 lines
That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.
Fair enough.
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Quoted 8 lines
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
Quoted 9 lines
I agree, and PSCI is the obvious only correct answer to this.
We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.
However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
CC'ing few more folks and changing the subject line
Regards,
Santosh
HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]
Quoted 89 lines · Santosh Shilimkar
Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 4 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?Quoted 6 lines
That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.
Fair enough.
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Quoted 8 lines
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
Quoted 8 lines
I agree, and PSCI is the obvious only correct answer to this.
We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.
However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Quoted 4 lines
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)?
Signature · 1 line
Catalin
HYP Kernel boot requirements
Quoted 95 lines · Catalin Marinas
Quoted 89 lines · Santosh Shilimkar
Quoted 52 lines · Christoffer Dall
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 4 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?Quoted 6 lines
That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution.
Fair enough.
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Quoted 8 lines
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
Quoted 8 lines
I agree, and PSCI is the obvious only correct answer to this.
We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas.
However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction.
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Quoted 4 lines
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Quoted 6 lines
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?
How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
I will look at PSCI more closely and see what can be done here.
Regards,
Santosh
HYP Kernel boot requirements
Quoted 19 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Quoted 6 lines
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?
Quoted 5 lines
How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
Is it late to rewrite the OMAP5 firmware?
What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware.
Quoted 1 line
I will look at PSCI more closely and see what can be done here.
PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)?
Signature · 1 line
Catalin
HYP Kernel boot requirements
Quoted 20 lines · Catalin Marinas
Quoted 19 lines · Santosh Shilimkar
Quoted 18 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?
Quoted 8 lines
Quoted 5 lines
How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
Is it late to rewrite the OMAP5 firmware?
Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.
Quoted 8 lines
What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware.
Quoted 7 lines
Quoted 1 line
I will look at PSCI more closely and see what can be done here.
PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)?
Hmmm.. So thats rules out PSCI for OMAP5 then.
Keystone actually has nice way to deal with the problem since the boot-monitor code can be patched up and hence I had no trouble implementing the "CPU starts in HYP mode" requirement on it.
Regards,
Santosh
HYP Kernel boot requirements
Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Quoted 6 lines
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
Quoted 5 lines
Quoted 2 lines
Is it late to rewrite the OMAP5 firmware?
Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.
If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.
Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?
Cheers
---Dave
HYP Kernel boot requirements
Quoted 34 lines · Dave Martin
Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Quoted 6 lines
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
Quoted 5 lines
Quoted 2 lines
Is it late to rewrite the OMAP5 firmware?
Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.
Quoted 7 lines
If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.
Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?
On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs.
regards,
Santosh
HYP Kernel boot requirements
Quoted 39 lines · Santosh Shilimkar
Quoted 38 lines · Dave Martin
Quoted 27 lines · Santosh Shilimkar
Quoted 26 lines · Catalin Marinas
Quoted 24 lines · Santosh Shilimkar
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Santosh Shilimkar
What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching.
Am not challenging the agreed direction but we need to solve the PM problem as well before making "all CPU runs boot-loader for HYP kernels" as a must have. At least its is a change in boot strategy from existing kernels.
Quoted 6 lines
Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g.
omap5_hyp_secondary_startup)?How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode.
Quoted 5 lines
Quoted 2 lines
Is it late to rewrite the OMAP5 firmware?
Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used.
If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload.
Quoted 8 lines
Quoted 3 lines
Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5?
On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs.
As last resort, you can still write a bunch of instructions acting as warm-bootloader, reserve a page and add them there. Code required is just a shim layer and should not require more than few bytes, after all it has just to check some variables and jump accordingly (+ calling smc if required).
Not pretty, agreed, but still better than patching the kernel IMHO (basically you will end up patching the bootloader - and possibly the dtb - instead, to install the shim mentioned above).
Lorenzo
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
In reply to Santosh ShilimkarQuoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 3 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?
Absolutely not. We're way past the "one image per usage" model, and I fully expect the same kernel to boot both as a guest and a host. Otherwise, multiplatform is just a joke.
Quoted 11 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
I definitely agree that you need to consider PM as well. But we've decided a long time ago that we wouldn't take that kind of code in the kernel. Your CPU has to reach the kernel in HYP, and that is down to your boot protocol to take care off that.
There are a few options. Christoffer mentioned having a pen, and you also could implement PSCI fairly easily (I've recently posted a u-boot patch series to take care of this, though there is still a fair bit of work required).
Whatever the approach is, the bottom line remains: your CPU enters the kernel in HYP.
Cheers,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
PATCH 2/2 ARM: OMAP5: Add HYP mode entry support for secondary CPUs
Quoted 72 lines · Marc Zyngier
Quoted 46 lines · Santosh Shilimkar
Quoted 36 lines · Christoffer Dall
Quoted 35 lines · Santosh Shilimkar
Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU.
Hence patch the secondary CPU boot path for hyp mode etry.
Cc Marc Zyngier <[email protected]>Cc Christoffer Dall <[email protected]>Cc Tony Lindgren <[email protected]>Signed-off-by Santosh Shilimkar <[email protected]>--- arch/arm/mach-omap2/omap-headsmp.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm/mach-omap2/omap-headsmp.S +5
-0@@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM@@ -38,6 +39,12 @@ wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0Quoted 9 lines
arch/arm/mach-omap2/omap-headsmp.S wait: ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 · line 50
Quoted 5 lines
arch/arm/mach-omap2/omap-headsmp.S +2
-0+hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /*hmm, this means that currently running this in a guest will fail to bring-up SMP, right?
Quoted 3 lines
Nope. Because the code under '
KVM_ARM_HOST' macro. Guest build will not enableCONFIG_KVM_ARM_HOSTand things should be fine then. Right ?Absolutely not. We're way past the "one image per usage" model, and I fully expect the same kernel to boot both as a guest and a host. Otherwise, multiplatform is just a joke.
Quoted 11 lines
Quoted 4 lines
Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel?
Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode.
In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must.
I definitely agree that you need to consider PM as well. But we've decided a long time ago that we wouldn't take that kind of code in the kernel. Your CPU has to reach the kernel in HYP, and that is down to your boot protocol to take care off that.
Quoted 8 lines
There are a few options. Christoffer mentioned having a pen, and you also could implement PSCI fairly easily (I've recently posted a u-boot patch series to take care of this, though there is still a fair bit of work required).
Whatever the approach is, the bottom line remains: your CPU enters the kernel in HYP.
Our emails crossed. I understand the stand from you guys. Lets continue discussion on other thread.