Skip to contentSkip to conversation

imx7d: CPU core issue in secure mode

U-Boot Tobias Junghans 5 participants 6 messages Jul 4, 2019 – Jul 31, 2019

lore
Share & tools
lore
#0 lore raw

Hi,

I'm trying to get an imx7d-based Colibris board running in secure mode in order to be able to use the CAAM, especially the HWRNG. However it seems like it's currently not possible to boot a mainline kernel (4.19) in secure mode with both CPU cores powered up, likely due to the missing PSCI firmware in secure mode. When booting in nonsecure mode the kernel recognizes both CPU cores while CAAM isn't working. Basically it's the same issue as discussed at

https://www.spinics.net/lists/u-boot-v2/msg33873.html

I'm using the latest mainline U-Boot (2019.07-rc4) with CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?

Thank you and best regards

Tobias

#1 lore raw

+ Stefan Agner

Quoted 23 lines · Tobias Junghans

Hi,

I'm trying to get an imx7d-based Colibris board running in secure mode in order to be able to use the CAAM, especially the HWRNG. However it seems like it's currently not possible to boot a mainline kernel (4.19) in secure mode with both CPU cores powered up, likely due to the missing PSCI firmware in secure mode. When booting in nonsecure mode the kernel recognizes both CPU cores while CAAM isn't working. Basically it's the same issue as discussed at

https://www.spinics.net/lists/u-boot-v2/msg33873.html

I'm using the latest mainline U-Boot (2019.07-rc4) with CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?

Thank you and best regards

Tobias

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot

#2 lore raw
In reply to Tobias Junghans

+ Peng

Hi Tobias, Peng,

Quoted 17 lines · Tobias Junghans

Hi,

I'm trying to get an imx7d-based Colibris board running in secure mode in order to be able to use the CAAM, especially the HWRNG. However it seems like it's currently not possible to boot a mainline kernel (4.19) in secure mode with both CPU cores powered up, likely due to the missing PSCI firmware in secure mode. When booting in nonsecure mode the kernel recognizes both CPU cores while CAAM isn't working. Basically it's the same issue as discussed at

https://www.spinics.net/lists/u-boot-v2/msg33873.html

I'm using the latest mainline U-Boot (2019.07-rc4) with CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?

Thank you and best regards

Quoted 7 lines

Tobias

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://lists.denx.de/listinfo/u-boot

I might be mistaken, but AFAIK there was on-going work done by Peng Fan regarding
proper CAAM initialization in the OP-TEE and further usage in the mainline kernel.

As I understood, the initial initialization of the jobrings is done in OP-TEE (which is booted before U-boot) in secure world, and then linux kernel, running in normal world, should be able to use it.
Regarding PSCI, frankly, I have no idea who particularly should provide it's support here: U-boot or OP-TEE (taking into account that in this setup U-boot is booted in non-secure PL2, so OP-TEE is the only one, who is able to provide secure
runtime services, so-called secure monitor).

BTW, I also saw some setups, where similar things to do the same in U-boot (when it's booted in secure mode),
which also does have it's own implementation of secure monitor(subsequently PSCI) and CAAM driver,
which probably does the same type of initialization, as in OP-TEE.

Peng,
Could you please provide some comments regarding this? Thanks!

Signature · 8 lines
Best regards - Freundliche Grüsse - Meilleures salutations

Igor Opaniuk

mailto: igor.opaniuk at gmail.com
skype: igor.opanyuk
+380 (93) 836 40 67
http://ua.linkedin.com/in/iopaniuk
#3 lore raw
Quoted 28 lines

Subject: Re: [U-Boot] imx7d: CPU core issue in secure mode

+ Peng

Hi Tobias, Peng,

Quoted 12 lines · Tobias Junghans

Hi,

I'm trying to get an imx7d-based Colibris board running in secure mode in order to be able to use the CAAM, especially the HWRNG. However it seems like it's currently not possible to boot a mainline kernel (4.19) in secure mode with both CPU cores powered up, likely due to the missing PSCI firmware in secure mode. When booting in nonsecure mode the kernel recognizes both CPU cores while CAAM isn't working. Basically it's the same issue as discussed at

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.

Quoted 1 line

spinics.net%2Flists%2Fu-boot-v2%2Fmsg33873.html&data=02%7C01%7 Cpen

Quoted 1 line

g.fan%40nxp.com%7C69f453b8841a47775d7608d705edd3ee%7C686ea1d3b c2b4c6fa

Quoted 1 line
Quoted 7 lines

92cd99c5c301635%7C0%7C0%7C636984391331231662&sdata=MtD5x 15k3vvgBMr

Quoted 4 lines

vqBaZBY9G8AFD0WuE9J8XxIP%2Fz%2Bk%3D&reserved=0

I'm using the latest mainline U-Boot (2019.07-rc4) with CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about

this issue?

Try "setenv bootm_boot_mode nonsec" in U-Boot stage.

Quoted 17 lines
Quoted 11 lines

Thank you and best regards

Tobias

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist

s.denx.de%2Flistinfo%2Fu-boot&data=02%7C01%7Cpeng.fan%40nxp.co m%7C

Quoted 1 line

69f453b8841a47775d7608d705edd3ee%7C686ea1d3bc2b4c6fa92cd99c5c30 1635%7C

Quoted 1 line
Quoted 7 lines

0%7C0%7C636984391331231662&sdata=Ra4mzQpiZpANam1gyhhsy2g WMHNH3JRNr

Quoted 1 line

ryP%2BPOiqsM%3D&reserved=0

I might be mistaken, but AFAIK there was on-going work done by Peng Fan regarding proper CAAM initialization in the OP-TEE and further usage in the mainline kernel.

Silvano was doing the CAAM part in OP-TEE.

Quoted 9 lines

As I understood, the initial initialization of the jobrings is done in OP-TEE (which is booted before U-boot) in secure world, and then linux kernel, running in normal world, should be able to use it.
Regarding PSCI, frankly, I have no idea who particularly should provide it's support here: U-boot or OP-TEE (taking into account that in this setup U-boot is booted in non-secure PL2, so OP-TEE is the only one, who is able to provide secure runtime services, so-called secure monitor).

Quoted 8 lines

BTW, I also saw some setups, where similar things to do the same in U-boot (when it's booted in secure mode), which also does have it's own implementation of secure monitor(subsequently PSCI) and CAAM driver, which probably does the same type of initialization, as in OP-TEE.

Peng,
Could you please provide some comments regarding this? Thanks!

There is psci services in U-Boot too. If want non-secure kernel without OP-TEE, Need set "setenv bootm_boot_mode nonsec " in U-Boot stage.
If want run OP-TEE, not set the env.

Regards,
Peng.

Quoted 14 lines

--
Best regards - Freundliche Grüsse - Meilleures salutations

Igor Opaniuk

mailto: igor.opaniuk at gmail.com
skype: igor.opanyuk
+380 (93) 836 40 67
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fua.linke
din.com%2Fin%2Fiopaniuk&data=02%7C01%7Cpeng.fan%40nxp.com%7 C69f453b8841a47775d7608d705edd3ee%7C686ea1d3bc2b4c6fa92cd99c5c3 01635%7C0%7C0%7C636984391331231662&sdata=%2B8TlRt9QP6mV wMhc3TtHxaZdM%2FvSx09Jz%2BpFhJOlgvg%3D&reserved=0

#4 lore raw

Hi Peng,

Am Freitag, 12. Juli 2019, 05:38:21 CEST schrieb Peng Fan:

Quoted 1 line

Try "setenv bootm_boot_mode nonsec" in U-Boot stage.

Unfortunately this does not help. I tried the following setups:

CONFIG_SECURE_BOOT=y
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
CONFIG_FSL_CAAM=y
CONFIG_SYS_FSL_HAS_SEC=y
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_COMPAT=4
CONFIG_SYS_FSL_SEC_LE=y

Booting with bootm_boot_mode=nonsec

U-Boot 2019.07 (Jul 12 2019 - 10:02:31 +0200)
CPU: Freescale i.MX7D rev1.3 1000 MHz (running at 792 MHz)
..
SEC0: RNG instantiated
..

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing 
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
..
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.0
..
[    0.002872] CPU: Testing write buffer coherency: ok
[    0.003224] CPU0: update cpu_capacity 1024
[    0.003234] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004687] smp: Bringing up secondary CPUs ...
[    0.005424] CPU1: update cpu_capacity 1024
[    0.005432] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.005553] smp: Brought up 1 node, 2 CPUs
[    0.005568] CPU: All CPU(s) started in HYP mode.
[    0.005571] CPU: Virtualization extensions available.
..
[    0.185229] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185240] caam 30900000.caam: job rings = 3, qi = 0
[    0.186894] caam_jr 30901000.jr0: failed to flush job ring 0
[    0.192721] caam_jr: probe of 30901000.jr0 failed with error -5
[    0.192846] caam_jr 30902000.jr1: failed to flush job ring 1
[    0.198796] caam_jr: probe of 30902000.jr1 failed with error -5
[    0.198989] caam_jr 30903000.jr1: failed to flush job ring 2
[    0.204957] caam_jr: probe of 30903000.jr1 failed with error -5
[    0.212619] Job Ring Device allocation for transform failed

Same configuration with

setenv bootm_boot_mode=sec

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing 
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.002866] CPU: Testing write buffer coherency: ok
[    0.003217] CPU0: update cpu_capacity 1024
[    0.003226] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004673] smp: Bringing up secondary CPUs ...
[    0.005174] smp: Brought up 1 node, 1 CPU
[    0.005188] CPU: All CPU(s) started in SVC mode.
..
[    0.185631] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185643] caam 30900000.caam: job rings = 3, qi = 0
[    0.196909] caam algorithms registered in /proc/crypto
[    0.199620] caam_jr 30901000.jr0: registering rng-caam

=> only 1 CPU core up.

Now I tried to disable the CAAM driver and secure boot support in U-Boot

# CONFIG_SECURE_BOOT is not set
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
# CONFIG_FSL_CAAM is not set
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_LE=y

and booting with bootm_boot_mode=nonsec

[    0.185233] caam 30900000.caam: Entropy delay = 3200
[    0.212342] caam 30900000.caam: failed to acquire DECO 0
[    0.217689] caam 30900000.caam: failed to instantiate RNG

=> both CPU cores are up in nonsec mode

Am I missing something?

Thank you and best regards

Tobias

#5 lore raw

[Adding Bryan and Breno]

Hi Bryan,

I think you worked on allowing the CAAM driver in Linux to work on i.MX7D running in non-secure when you created:
commit 22191ac35344 ("drivers/crypto/fsl: assign job-rings to non-TrustZone")

It was reverted later by Breno as it broke secure boot.

If I understand correctly the current solution is to let OP-TEE deal with job-rings initialization.

Is there any other alternative to use the mainline kernel CAAM driver in non-secure if someone is not using OP-TEE?

Thanks,

Fabio Estevam

Quoted 131 lines · Tobias Junghans

Hi Peng,

Am Freitag, 12. Juli 2019, 05:38:21 CEST schrieb Peng Fan:

Quoted 1 line

Try "setenv bootm_boot_mode nonsec" in U-Boot stage.

Unfortunately this does not help. I tried the following setups:

CONFIG_SECURE_BOOT=y
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
CONFIG_FSL_CAAM=y
CONFIG_SYS_FSL_HAS_SEC=y
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_COMPAT=4
CONFIG_SYS_FSL_SEC_LE=y

Booting with bootm_boot_mode=nonsec

U-Boot 2019.07 (Jul 12 2019 - 10:02:31 +0200)
CPU: Freescale i.MX7D rev1.3 1000 MHz (running at 792 MHz)
..
SEC0: RNG instantiated
..

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
..
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.0
..
[    0.002872] CPU: Testing write buffer coherency: ok
[    0.003224] CPU0: update cpu_capacity 1024
[    0.003234] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004687] smp: Bringing up secondary CPUs ...
[    0.005424] CPU1: update cpu_capacity 1024
[    0.005432] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.005553] smp: Brought up 1 node, 2 CPUs
[    0.005568] CPU: All CPU(s) started in HYP mode.
[    0.005571] CPU: Virtualization extensions available.
..
[    0.185229] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185240] caam 30900000.caam: job rings = 3, qi = 0
[    0.186894] caam_jr 30901000.jr0: failed to flush job ring 0
[    0.192721] caam_jr: probe of 30901000.jr0 failed with error -5
[    0.192846] caam_jr 30902000.jr1: failed to flush job ring 1
[    0.198796] caam_jr: probe of 30902000.jr1 failed with error -5
[    0.198989] caam_jr 30903000.jr1: failed to flush job ring 2
[    0.204957] caam_jr: probe of 30903000.jr1 failed with error -5
[    0.212619] Job Ring Device allocation for transform failed

Same configuration with

setenv bootm_boot_mode=sec

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.002866] CPU: Testing write buffer coherency: ok
[    0.003217] CPU0: update cpu_capacity 1024
[    0.003226] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004673] smp: Bringing up secondary CPUs ...
[    0.005174] smp: Brought up 1 node, 1 CPU
[    0.005188] CPU: All CPU(s) started in SVC mode.
..
[    0.185631] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185643] caam 30900000.caam: job rings = 3, qi = 0
[    0.196909] caam algorithms registered in /proc/crypto
[    0.199620] caam_jr 30901000.jr0: registering rng-caam

=> only 1 CPU core up.

Now I tried to disable the CAAM driver and secure boot support in U-Boot

# CONFIG_SECURE_BOOT is not set
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
# CONFIG_FSL_CAAM is not set
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_LE=y

and booting with bootm_boot_mode=nonsec

[    0.185233] caam 30900000.caam: Entropy delay = 3200
[    0.212342] caam 30900000.caam: failed to acquire DECO 0
[    0.217689] caam 30900000.caam: failed to instantiate RNG

=> both CPU cores are up in nonsec mode

Am I missing something?

Thank you and best regards

Tobias