Initial support of Trusted Foundations on Tegra30
Subsystems (3)
v2 cover Initial support of Trusted Foundations on Tegra30
Hello,
This series of patches brings initial support of Trusted Foundations to Tegra30, that is to the consumer-grade Tegra30 devices which do not allow to easily replace the proprietary bootloader. Support is initial because this series implements only a proper CPU boot-up (main + secondary cores) and a basic L2 cache maintenance that is done using the TF firmware. Suspend-resume support is missing yet as I couldn't get it to work (CPU hangs on resume from suspend after awhile and seems that is related to inappropriately done cache maintenance during of suspend-resume using the firmware), it is work-in-progress for now.
This patchset is partially based on the work done by Micha? Miros?aw [0].
[0] https://www.spinics.net/lists/linux-tegra/msg30368.html
Changelog:
v2:
- The "Don't apply CPU erratas in insecure mode" patch got some cleanup, in particular resolved the messiness in __tegra_cpu_reset_handler_data.
- Added a comment to tf_cache_write_sec(), justifying the warning message.
Dmitry Osipenko (5):
ARM: trusted_foundations: Implement L2 cache initialization callback
ARM: trusted_foundations: Provide information about whether firmware
is registered
ARM: tegra: Setup L2 cache using Trusted Foundations firmware
ARM: tegra: Don't apply CPU erratas in insecure mode
ARM: tegra: Always boot CPU in ARM-mode
arch/arm/firmware/trusted_foundations.c | 32 ++++++++++++++++++++++ arch/arm/include/asm/trusted_foundations.h | 7 +++++ arch/arm/mach-tegra/reset-handler.S | 25 +++++++++-------- arch/arm/mach-tegra/reset.c | 5 +++- arch/arm/mach-tegra/reset.h | 9 ++++-- arch/arm/mach-tegra/sleep-tegra20.S | 4 +++ arch/arm/mach-tegra/tegra.c | 15 ++++++++++ 7 files changed, 82 insertions(+), 15 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/5 ARM: trusted_foundations: Implement L2 cache initialization callback
Implement L2 cache initialization firmware callback that should be invoked early in boot to enable cache HW.
--- arch/arm/firmware/trusted_foundations.c | 27 +++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm/firmware/trusted_foundations.c +27 -0
@@ -18,8 +18,13 @@#include <linux/init.h> #include <linux/of.h> #include <asm/firmware.h> +#include <asm/outercache.h> #include <asm/trusted_foundations.h> +#define TF_CACHE_MAINT 0xfffff100 + +#define TF_CACHE_INIT 1 +#define TF_SET_CPU_BOOT_ADDR_SMC 0xfffff200#define TF_CPU_PM 0xfffffffc
@@ -63,9 +68,31 @@ static int tf_prepare_idle(void)return 0 ; } +#ifdef CONFIG_CACHE_L2X0 +static void tf_cache_write_sec (unsigned long val,unsigned int reg) +{ +/* +* The L2X0 cache driver shouldn't invoke a write to a secure registers, +* though it's better to reinsure by printing a warning message. +*/ +pr_warn ("%s: Ignoring write [0x%x]: 0x%08lx\n" , __func__, reg, val); +} + +static int tf_init_cache (void ) +{ + outer_cache.write_sec = tf_cache_write_sec; +tf_generic_smc (TF_CACHE_MAINT ,TF_CACHE_INIT ,0 ); + +return 0 ; +} +#endif /* CONFIG_CACHE_L2X0 */ +static const struct firmware_ops trusted_foundations_ops = { .set_cpu_boot_addr = tf_set_cpu_boot_addr, .prepare_idle = tf_prepare_idle, +#ifdef CONFIG_CACHE_L2X0 + .l2x0_init = tf_init_cache, +#endif };void register_trusted_foundations (struct trusted_foundations_platform_data *pd)
Signature · 1 line
2.17.1
PATCH v2 2/5 ARM: trusted_foundations: Provide information about whether firmware is registered
In reply to Dmitry OsipenkoAdd a helper that provides information about whether Trusted Foundations firmware operations have been registered.
--- arch/arm/firmware/trusted_foundations.c | 5 +++++ arch/arm/include/asm/trusted_foundations.h | 7 +++++++ 2 files changed, 12 insertions(+)
arch/arm/firmware/trusted_foundations.c +5 -0
@@ -124,3 +124,8 @@ void of_register_trusted_foundations(void)panic ("Trusted Foundation: missing version-minor property\n" );register_trusted_foundations (&pdata); } + +bool trusted_foundations_registered (void ) +{ +return firmware_ops == &trusted_foundations_ops; +}
arch/arm/include/asm/trusted_foundations.h +7 -0
@@ -31,6 +31,7 @@#include <linux/of.h> #include <linux/cpu.h> #include <linux/smp.h> +#include <linux/types.h> struct trusted_foundations_platform_data {unsigned int version_major;
@@ -41,6 +42,7 @@ struct trusted_foundations_platform_data {void register_trusted_foundations (struct trusted_foundations_platform_data *pd);void of_register_trusted_foundations (void ); +bool trusted_foundations_registered (void );#else /* CONFIG_TRUSTED_FOUNDATIONS */
@@ -68,6 +70,11 @@ static inline void of_register_trusted_foundations(void)if (of_find_compatible_node (NULL ,NULL ,"tlm,trusted-foundations" ))register_trusted_foundations (NULL ); } + +static inline bool trusted_foundations_registered (void ) +{ +return false; +}#endif /* CONFIG_TRUSTED_FOUNDATIONS */ #endif
Signature · 1 line
2.17.1
PATCH v2 3/5 ARM: tegra: Setup L2 cache using Trusted Foundations firmware
In reply to Dmitry OsipenkoOn Tegra20/30 L2 cache must be initialized using firmware call if CPU is running in insecure mode. Initialize L2 cache and setup the outer-cache callbacks in early boot using the firmware API.
--- arch/arm/mach-tegra/tegra.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/arm/mach-tegra/tegra.c +15 -0
@@ -38,6 +38,7 @@#include <soc/tegra/fuse.h> #include <soc/tegra/pmc.h> +#include <asm/firmware.h> #include <asm/hardware/cache-l2x0.h> #include <asm/mach/arch.h> #include <asm/mach/time.h>
@@ -70,9 +71,23 @@ u32 tegra_uart_config[3] = { 0, }; +static void __init tegra_trusted_foundations_l2x0_cache_init (void ) +{ +if (IS_ENABLED (CONFIG_CACHE_L2X0 ) && +IS_ENABLED (CONFIG_ARCH_TEGRA_2x_SOC) && +of_machine_is_compatible ("nvidia,tegra20" )) +call_firmware_op (l2x0_init); + +if (IS_ENABLED (CONFIG_CACHE_L2X0 ) && +IS_ENABLED (CONFIG_ARCH_TEGRA_3x_SOC) && +of_machine_is_compatible ("nvidia,tegra30" )) +call_firmware_op (l2x0_init); +} +static void __init tegra_init_early (void ) {of_register_trusted_foundations (); +tegra_trusted_foundations_l2x0_cache_init ();tegra_cpu_reset_handler_init (); }
Signature · 1 line
2.17.1
PATCH v2 4/5 ARM: tegra: Don't apply CPU erratas in insecure mode
In reply to Dmitry OsipenkoCPU isn't allowed to touch secure registers while running under secure monitor. Hence skip applying CPU erratas in the reset handler if Trusted Foundations firmware presents.
--- arch/arm/mach-tegra/reset-handler.S | 24 ++++++++++++------------ arch/arm/mach-tegra/reset.c | 3 +++ arch/arm/mach-tegra/reset.h | 9 +++++++-- arch/arm/mach-tegra/sleep-tegra20.S | 4 ++++ 4 files changed, 26 insertions(+), 14 deletions(-)
arch/arm/mach-tegra/reset-handler.S +12 -12
@@ -29,8 +29,6 @@ #define PMC_SCRATCH41 0x140 -#define RESET_DATA(x) ((TEGRA_RESET_##x)*4) - #ifdef CONFIG_PM_SLEEP /* * tegra_resume
@@ -121,6 +119,12 @@ ENTRY(__tegra_cpu_reset_handler) cpsid aif, 0x13 @ SVC mode, interrupts disabled tegra_get_soc_id TEGRA_APB_MISC_BASE, r6 + + adr r12, __tegra_cpu_reset_handler_data + ldr r0, [r12, #RESET_DATA(TF_PRESENT)] + cmp r0, #0 + bne after_errata + #ifdef CONFIG_ARCH_TEGRA_2x_SOC t20_check: cmp r6, #TEGRA20
@@ -155,7 +159,6 @@ after_errata: and r10, r10, #0x3 @ R10 = CPU number mov r11, #1 mov r11, r11, lsl r10 @ R11 = CPU mask - adr r12, __tegra_cpu_reset_handler_data #ifdef CONFIG_SMP /* Does the OS know about this CPU? */
@@ -169,10 +172,9 @@ after_errata: cmp r6, #TEGRA20 bne 1f /* If not CPU0, don't let CPU0 reset CPU1 now that CPU1 is coming up. */ - mov32 r5, TEGRA_IRAM_BASE + TEGRA_IRAM_RESET_HANDLER_OFFSET mov r0, #CPU_NOT_RESETTABLE cmp r10, #0 - strneb r0, [r5, #__tegra20_cpu1_resettable_status_offset] + strneb r0, [r12, #RESET_DATA(RESETTABLE_STATUS)] 1: #endif
@@ -277,14 +279,12 @@ ENDPROC(__tegra_cpu_reset_handler) .align L1_CACHE_SHIFT .type __tegra_cpu_reset_handler_data, %object .globl __tegra_cpu_reset_handler_data + .globl __tegra_cpu_reset_handler_data_offset + .equ __tegra_cpu_reset_handler_data_offset, \ + . - __tegra_cpu_reset_handler_start __tegra_cpu_reset_handler_data: - .rept TEGRA_RESET_DATA_SIZE - .long 0 + .rept TEGRA_RESET_DATA_SIZE + .long 0 .endr - .globl __tegra20_cpu1_resettable_status_offset - .equ __tegra20_cpu1_resettable_status_offset, \ - . - __tegra_cpu_reset_handler_start - .byte 0 .align L1_CACHE_SHIFT - ENTRY(__tegra_cpu_reset_handler_end)
arch/arm/mach-tegra/reset.c +3 -0
@@ -24,6 +24,7 @@#include <asm/cacheflush.h> #include <asm/firmware.h> #include <asm/hardware/cache-l2x0.h> +#include <asm/trusted_foundations.h> #include "iomap.h" #include "irammap.h"
@@ -89,6 +90,8 @@ static void __init tegra_cpu_reset_handler_enable(void)void __init tegra_cpu_reset_handler_init (void ) { + __tegra_cpu_reset_handler_data[TEGRA_RESET_TF_PRESENT ] = +trusted_foundations_registered ();#ifdef CONFIG_SMP __tegra_cpu_reset_handler_data[TEGRA_RESET_MASK_PRESENT] =
arch/arm/mach-tegra/reset.h +7 -2
@@ -25,7 +25,11 @@#define TEGRA_RESET_STARTUP_SECONDARY 3#define TEGRA_RESET_STARTUP_LP2 4#define TEGRA_RESET_STARTUP_LP1 5 -#define TEGRA_RESET_DATA_SIZE 6 +#define TEGRA_RESET_RESETTABLE_STATUS 6 +#define TEGRA_RESET_TF_PRESENT 7 +#define TEGRA_RESET_DATA_SIZE 8 + +#define RESET_DATA (x) ((TEGRA_RESET_##x)*4)#ifndef __ASSEMBLY__
@@ -49,7 +53,8 @@ void __tegra_cpu_reset_handler_end(void); (u32)__tegra_cpu_reset_handler_start)))#define tegra20_cpu1_resettable_status \ (IO_ADDRESS(TEGRA_IRAM_BASE + TEGRA_IRAM_RESET_HANDLER_OFFSET + \ - (u32)__tegra20_cpu1_resettable_status_offset)) + ((u32)&__tegra_cpu_reset_handler_data[TEGRA_RESET_RESETTABLE_STATUS] - \ + (u32)__tegra_cpu_reset_handler_start)))#endif #define tegra_cpu_reset_handler_offset \
arch/arm/mach-tegra/sleep-tegra20.S +4 -0
@@ -28,6 +28,7 @@ #include <asm/cache.h> #include "irammap.h" +#include "reset.h" #include "sleep.h" #define EMC_CFG 0xc
@@ -54,6 +55,9 @@ #define APB_MISC_XM2CFGCPADCTRL2 0x8e4 #define APB_MISC_XM2CFGDPADCTRL2 0x8e8 +#define __tegra20_cpu1_resettable_status_offset \ + (__tegra_cpu_reset_handler_data_offset + RESET_DATA(RESETTABLE_STATUS)) + .macro pll_enable, rd, r_car_base, pll_base ldr \rd, [\r_car_base, #\pll_base] tst \rd, #(1 << 30)
Signature · 1 line
2.17.1
PATCH v2 5/5 ARM: tegra: Always boot CPU in ARM-mode
In reply to Dmitry OsipenkoCPU always jumps into the reset handler in ARM-mode from the Trusted Foundations firmware, hence make CPU to always jump into kernel in ARM-mode regardless of the firmware presence to support Thumb2 kernel + TF case.
--- arch/arm/mach-tegra/reset-handler.S | 1 + arch/arm/mach-tegra/reset.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/arm/mach-tegra/reset-handler.S +1 -0
@@ -113,6 +113,7 @@ ENTRY(__tegra_cpu_reset_handler_start) * must be position-independent. */ + .arm .align L1_CACHE_SHIFT ENTRY(__tegra_cpu_reset_handler)
arch/arm/mach-tegra/reset.c +1 -1
@@ -97,7 +97,7 @@ void __init tegra_cpu_reset_handler_init(void) __tegra_cpu_reset_handler_data[TEGRA_RESET_MASK_PRESENT ] = *((u32 *)cpu_possible_mask); __tegra_cpu_reset_handler_data[TEGRA_RESET_STARTUP_SECONDARY ] = -__pa_symbol ((void *)secondary_startup); +__pa_symbol ((void *)secondary_startup_arm);#endif #ifdef CONFIG_PM_SLEEP
Signature · 1 line
2.17.1
Good Afternoon,
I have tested these patches on the Ouya T3 device.
They work great to enable the L2 cache controller, however they do not respect explicitly disabling the L2 cache controller via the kernel config nor device tree.
With CONFIG_CACHE_L2X0 disabled, but CONFIG_TRUSTED_FOUNDATIONS enabled, the L2 cache controller is silently enabled and allows all four cores to boot.
One must also disable CONFIG_TRUSTED_FOUNDATIONS to stop the L2 cache controller from spinning up.
Quoted 43 lines · Dmitry Osipenko
Hello,
This series of patches brings initial support of Trusted Foundations to Tegra30, that is to the consumer-grade Tegra30 devices which do not allow to easily replace the proprietary bootloader. Support is initial because this series implements only a proper CPU boot-up (main + secondary cores) and a basic L2 cache maintenance that is done using the TF firmware. Suspend-resume support is missing yet as I couldn't get it to work (CPU hangs on resume from suspend after awhile and seems that is related to inappropriately done cache maintenance during of suspend-resume using the firmware), it is work-in-progress for now.
This patchset is partially based on the work done by Micha? Miros?aw [0].
[0] https://www.spinics.net/lists/linux-tegra/msg30368.html
Changelog:
v2:
- The "Don't apply CPU erratas in insecure mode" patch got some cleanup, in particular resolved the messiness in__tegra_cpu_reset_handler_data.- Added a comment to
tf_cache_write_sec(), justifying the warning message.Dmitry Osipenko (5):
ARM:trusted_foundations: Implement L2 cache initialization callback
ARM:trusted_foundations: Provide information about whether firmware
is registered
ARM: tegra: Setup L2 cache using Trusted Foundations firmware
ARM: tegra: Don't apply CPU erratas in insecure mode
ARM: tegra: Always boot CPU in ARM-modearch/arm/firmware/trusted_foundations.c | 32 ++++++++++++++++++++++ arch/arm/include/asm/trusted_foundations.h | 7 +++++ arch/arm/mach-tegra/reset-handler.S | 25 +++++++++-------- arch/arm/mach-tegra/reset.c | 5 +++- arch/arm/mach-tegra/reset.h | 9 ++++-- arch/arm/mach-tegra/sleep-tegra20.S | 4 +++ arch/arm/mach-tegra/tegra.c | 15 ++++++++++ 7 files changed, 82 insertions(+), 15 deletions(-)
Quoted 11 lines · On Friday, 29 June 2018 22:37:02 MSK Peter Geis wrote:
Good Afternoon,
I have tested these patches on the Ouya T3 device.
They work great to enable the L2 cache controller, however they do not respect explicitly disabling the L2 cache controller via the kernel config nor device tree.With
CONFIG_CACHE_L2X0disabled, butCONFIG_TRUSTED_FOUNDATIONSenabled, the L2 cache controller is silently enabled and allows all four cores to boot.
I don't see how cache could be enabled with CONFIG_CACHE_L2X0 disabled, there is no code to do that. Could you elaborate please?
Secondary cores do not depend on the cache state, disabled cache shouldn't prevent them to boot.
Quoted 5 lines
One must also disable
CONFIG_TRUSTED_FOUNDATIONSto stop the L2 cache controller from spinning up.Tested-by Peter Geis <[email protected]>
Quoted 13 lines · Dmitry Osipenko
Quoted 11 lines · On Friday, 29 June 2018 22:37:02 MSK Peter Geis wrote:
Good Afternoon,
I have tested these patches on the Ouya T3 device.
They work great to enable the L2 cache controller, however they do not respect explicitly disabling the L2 cache controller via the kernel config nor device tree.With
CONFIG_CACHE_L2X0disabled, butCONFIG_TRUSTED_FOUNDATIONSenabled, the L2 cache controller is silently enabled and allows all four cores to boot.
Quoted 5 lines
I don't see how cache could be enabled with
CONFIG_CACHE_L2X0disabled, there is no code to do that. Could you elaborate please?Secondary cores do not depend on the cache state, disabled cache shouldn't prevent them to boot.
On the untouched mainline kernel running on a Trusted Foundations T3 device, I observed the following indications:
With the L2 cache controller enabled, all four processor cores were enabled, but it would immediately panic for writing to a secure register from insecure mode.
With the L2 cache controller disabled, only the boot core is detected, but it successfully boots.
This is the issue that I inquired originally to you about.
With your patch running on the same device, the following is observed:
With the L2 controller enabled, all four cores are active, and the cache controller appears to function.
With the L2 controller disabled, but trusted foundations enabled, the L2 controller enabled kernel message is missing, however all four cores still enable.
After looking through the code a little more deeply, I see you modified the reset handler for handling offline cores.
I am wondering if you fixed an additional issue inadvertently.
The reason I discovered this is I am working with kexec as a bootloader. On a device without trusted foundations, kexec works without issue. On a device with trusted foundations and your patch, I found that disabling the l2 cache controller and trusted foundations allow kexec to occur.
I haven't tried an either/or scenario, though now you have me thinking I should.
Quoted 10 lines
Quoted 6 lines
One must also disable
CONFIG_TRUSTED_FOUNDATIONSto stop the L2 cache controller from spinning up.Tested-by Peter Geis <[email protected]>
Quoted 28 lines · On Monday, 2 July 2018 21:53:08 MSK Peter Geis wrote:
Quoted 12 lines · Dmitry Osipenko
Quoted 10 lines · On Friday, 29 June 2018 22:37:02 MSK Peter Geis wrote:
Good Afternoon,
I have tested these patches on the Ouya T3 device.
They work great to enable the L2 cache controller, however they do not respect explicitly disabling the L2 cache controller via the kernel config nor device tree.With
CONFIG_CACHE_L2X0disabled, butCONFIG_TRUSTED_FOUNDATIONSenabled, the L2 cache controller is silently enabled and allows all four cores to boot.Quoted 5 lines
I don't see how cache could be enabled with
CONFIG_CACHE_L2X0disabled, there is no code to do that. Could you elaborate please?Secondary cores do not depend on the cache state, disabled cache shouldn't prevent them to boot.
On the untouched mainline kernel running on a Trusted Foundations T3 device, I observed the following indications:
With the L2 cache controller enabled, all four processor cores were enabled, but it would immediately panic for writing to a secure register from insecure mode.
With the L2 cache controller disabled, only the boot core is detected, but it successfully boots.
This is the issue that I inquired originally to you about.
Quoted 7 lines
With your patch running on the same device, the following is observed:
With the L2 controller enabled, all four cores are active, and the cache controller appears to function.
With the L2 controller disabled, but trusted foundations enabled, the L2 controller enabled kernel message is missing, however all four cores still enable.
This is the correct behaviour.
Quoted 4 lines
After looking through the code a little more deeply, I see you modified the reset handler for handling offline cores.
I am wondering if you fixed an additional issue inadvertently.
CPU will fail to boot if it tries to apply erratas via accessing the secure registers. I've changed the reset handler to skip erratas on T20/30 if trusted foundations present, see "Don't apply CPU erratas in insecure mode" patch. This is an intentional change.
Quoted 8 lines
The reason I discovered this is I am working with kexec as a bootloader. On a device without trusted foundations, kexec works without issue. On a device with trusted foundations and your patch, I found that disabling the l2 cache controller and trusted foundations allow kexec to occur.
I haven't tried an either/or scenario, though now you have me thinking I should.
Secondary cores are dying in the reset handler in a case of disabled L2 + disabled TF. Looks like there is something wrong in regards to stopping secondary CPU cores / flushing CPU caches, the suspend-resume isn't working right now because of it and likely that kexec is suffering from the same issue.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
One must also disable
CONFIG_TRUSTED_FOUNDATIONSto stop the L2 cache controller from spinning up.Tested-by Peter Geis <[email protected]>
Thanks for testing!
Kexec works for me on T30 with the following kernel config options on top of the tegra_defconfig:
CONFIG_SMP=nCONFIG_CPU_IDLE=nCONFIG_TEGRA_IOMMU_SMMU=n
CPU stopping isn't working correctly with the trusted foundations and it's not obvious what is wrong. I hope we'll fix it at some point.
Re: [PATCH v2 1/5] ARM: trusted_foundations: Implement L2 cache initialization callback
In reply to Dmitry OsipenkoQuoted 7 lines · On Tuesday, 19 June 2018 14:00:23 MSK Dmitry Osipenko wrote:
Implement L2 cache initialization firmware callback that should be invoked early in boot to enable cache HW.
Signed-off-by Dmitry Osipenko <[email protected]>--- arch/arm/firmware/trusted_foundations.c | 27 +++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
Russell, could you please take a look at the arm/firmware/ patches and ACK them if they are good to you?