Skip to contentSkip to conversation

ARM: fdt: copy TF-A reserved memory into fdt passed to Linux

PATCH In review Alexander Sverdlin 4 participants 10 messages Jun 13 – Jun 26 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: fdt: copy TF-A reserved memory into fdt passed to Linux

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

[1] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/49754
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

 arch/arm/lib/bootm-fdt.c | 55 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm/lib/bootm-fdt.c +55 -0
index 2671f9a0ebf..19d917943c1 100644
@@ -14,7 +14,9 @@
  * Copyright (C) 2001  Erik Mouw ([email protected])
  */
 
+#include <dm/ofnode.h>
 #include <fdt_support.h>
+#include <linux/ioport.h>
 #ifdef CONFIG_ARMV7_NONSEC
 #include <asm/armv7.h>
 #endif
@@ -24,6 +26,52 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
+static int tfa_copy_reserved_memory(void *new_blob)
+{
+#ifdef CONFIG_OF_CONTROL
+	ofnode node, subnode;
+
+	/*
+	 * TF-A for several platforms inserts its memory region as
+	 * reserved-memory node
+	 */
+	node = ofnode_path("/reserved-memory");
+	if (!ofnode_valid(node))
+		return 0;
+
+	ofnode_for_each_subnode(subnode, node) {
+		struct fdt_memory carveout;
+		struct resource res;
+		const char *name;
+		int ret;
+
+		name = ofnode_get_name(subnode);
+		if (!name)
+			return -FDT_ERR_BADSTRUCTURE;
+
+		/* only handle TF-A reservations */
+		if (strncmp(name, "tf-a", 4))
+			continue;
+
+		/* check if this subnode has a reg property */
+		ret = ofnode_read_resource(subnode, 0, &res);
+		if (ret)
+			continue;
+
+		carveout.start = res.start,
+		carveout.end = res.end,
+
+		ret = fdtdec_add_reserved_memory(new_blob, "tf-a", &carveout,
+						 NULL, 0, NULL,
+						 FDTDEC_RESERVED_MEMORY_NO_MAP);
+		if (ret < 0)
+			return ret;
+	}
+#endif
+
+	return 0;
+}
+
 #ifdef CONFIG_FMAN_ENET
 __weak int fdt_update_ethernet_dt(void *blob)
 {
@@ -56,6 +104,13 @@ int arch_fixup_fdt(void *blob)
 		return ret;
 #endif
 
+	ret = tfa_copy_reserved_memory(blob);
+	if (ret) {
+		printf("ERROR: transfer of TF-A nodes to new fdt failed: %s\n",
+		       fdt_strerror(ret));
+		return ret;
+	}
+
 #ifdef CONFIG_ARMV8_SPIN_TABLE
 	ret = spin_table_update_dt(blob);
 	if (ret)
Signature · 1 line
2.54.0
#1 lore raw

Hi,

Quoted 6 lines · Alexander Sverdlin

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

To be fair we've been adding the reserved memory regions statically in the Linux device-trees to mitigate the issue.

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Also note that the usual way to deal with this is to not load any device-tree when booting the kernel, which will implicitly let U-Boot use its current device-tree for Linux (with the modifications brought by tf-a).

But of course I agree that it is very desirable to "forward" the device-tree modifications to the kernel device-tree so we are not stuck with whatever device-tree U-Boot was built with.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

RaspberryPi seems to be using "atf@0". Generally speaking the property is a free-form argument to fdt_add_reserved_memory in tf-a and I don't think we can have a common way to match them.

Introducing a Kconfig property for the prefix would be a satisfying solution in my opinion.

Quoted 9 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

Quoted 6 lines

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

Thanks for looking at this!

Like I said, I guess the same needs to be done for the cpuidle psci nodes.

Quoted 22 lines

[1] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/49754
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

 arch/arm/lib/bootm-fdt.c | 55 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm/lib/bootm-fdt.c +2 -0
index 2671f9a0ebf..19d917943c1 100644
@@ -14,7 +14,9 @@
  * Copyright (C) 2001  Erik Mouw ([email protected])
  */
 
+#include <dm/ofnode.h>
 #include <fdt_support.h>
+#include <linux/ioport.h>
 #ifdef CONFIG_ARMV7_NONSEC
 #include <asm/armv7.h>
 #endif

arch/arm/lib/bootm-fdt.c line 29

Quoted 4 lines
arch/arm/lib/bootm-fdt.c +1 -0
@@ -24,6 +26,52 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
+static int tfa_copy_reserved_memory(void *new_blob)

I would keep a fdt-related prefix to the name, so something like:
fdt_update_tfa_reserved_memory or fdt_update_bl31_reserved_memory

arch/arm/lib/bootm-fdt.c line 36

Quoted 7 lines
arch/arm/lib/bootm-fdt.c +7 -0
+{
+#ifdef CONFIG_OF_CONTROL
+	ofnode node, subnode;
+
+	/*
+	 * TF-A for several platforms inserts its memory region as
+	 * reserved-memory node

This will always be the case, I don't think we need a comment about it.

Quoted 6 lines
arch/arm/lib/bootm-fdt.c +6 -0
+	 */
+	node = ofnode_path("/reserved-memory");
+	if (!ofnode_valid(node))
+		return 0;
+
+	ofnode_for_each_subnode(subnode, node) {

arch/arm/lib/bootm-fdt.c line 52

Quoted 10 lines
arch/arm/lib/bootm-fdt.c +10 -0
+		struct fdt_memory carveout;
+		struct resource res;
+		const char *name;
+		int ret;
+
+		name = ofnode_get_name(subnode);
+		if (!name)
+			return -FDT_ERR_BADSTRUCTURE;
+
+		/* only handle TF-A reservations */

Please capitalize the first letter and end with a dot (same for all comments).

Quoted 6 lines
arch/arm/lib/bootm-fdt.c +6 -0
+		if (strncmp(name, "tf-a", 4))
+			continue;
+
+		/* check if this subnode has a reg property */
+		ret = ofnode_read_resource(subnode, 0, &res);
+		if (ret)

arch/arm/lib/bootm-fdt.c line 68

Quoted 10 lines
arch/arm/lib/bootm-fdt.c +10 -0
+			continue;
+
+		carveout.start = res.start,
+		carveout.end = res.end,
+
+		ret = fdtdec_add_reserved_memory(new_blob, "tf-a", &carveout,
+						 NULL, 0, NULL,
+						 FDTDEC_RESERVED_MEMORY_NO_MAP);
+		if (ret < 0)
+			return ret;

Looks good otherwise.

Quoted 10 lines
arch/arm/lib/bootm-fdt.c +6 -0
+	}
+#endif
+
+	return 0;
+}
+
 #ifdef CONFIG_FMAN_ENET
 __weak int fdt_update_ethernet_dt(void *blob)
 {

arch/arm/lib/bootm-fdt.c int arch_fixup_fdt(void *blob) · line 110

Quoted 7 lines
arch/arm/lib/bootm-fdt.c +4 -0
@@ -56,6 +104,13 @@ int arch_fixup_fdt(void *blob)
 		return ret;
 #endif
 
+	ret = tfa_copy_reserved_memory(blob);
+	if (ret) {
+		printf("ERROR: transfer of TF-A nodes to new fdt failed: %s\n",
+		       fdt_strerror(ret));

This is very unlikely, not sure it is worth an error message. Other calls in the same function are silent.

Quoted 9 lines
arch/arm/lib/bootm-fdt.c +3 -0
+		return ret;
+	}
+
 #ifdef CONFIG_ARMV8_SPIN_TABLE
 	ret = spin_table_update_dt(blob);
 	if (ret)
Signature · 1 line
2.54.0

All the best,

Paul

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
#2 lore raw

Hi Paul,

Quoted 9 lines · Paul Kocialkowski
Quoted 6 lines

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

To be fair we've been adding the reserved memory regions statically in the Linux device-trees to mitigate the issue.

once for H616, but it could be the only SoC among ARM64 platforms doing this and discouraged for A133:
https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Do you refer to some unmerged code? Didn't find it in the current TF-A sources...

Quoted 4 lines

Also note that the usual way to deal with this is to not load any device-tree when booting the kernel, which will implicitly let U-Boot use its current device-tree for Linux (with the modifications brought by tf-a).

?!
We definitely want to load the very device tree coming in the FIT image and pass it to the kernel from this FIT image. Sometimes people would have several DTs to chose from. The thing in U-Boot is basically to get U-Boot up and running. OF_UPSTREAM is rather to reduce the traffic on the U-Boot mailing list and maintainers effort, but in most of the cases we shall expect this DT to be not from the kernel we actually load.

Quoted 12 lines

But of course I agree that it is very desirable to "forward" the device-tree modifications to the kernel device-tree so we are not stuck with whatever device-tree U-Boot was built with.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving  memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

Quoted 6 lines

RaspberryPi seems to be using "atf@0". Generally speaking the property is a free-form argument to fdt_add_reserved_memory in tf-a and I don't think we can have a common way to match them.

Introducing a Kconfig property for the prefix would be a satisfying solution in my opinion.

This was a very conservative patch solving the A133 case, but actually I don't see anything wrong with just copying all the reserved areas from the U-Boot live tree to the device tree we are going to pass to the kernel. Maybe fdtdec_add_reserved_memory() needs to be taught to detect overlapping ranges and extend them properly, or maybe yet another function has to be created for this purpose, to avoid duplicated reserved-memory nodes by all means, but would also solve the PSCI cpuidle issue, as well as potentially Raspi case of TF-A side and simplify TI K3 on U-Boot side.

Quoted 16 lines
Quoted 9 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

Quoted 6 lines

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

Quoted 4 lines

Thanks for looking at this!

Like I said, I guess the same needs to be done for the cpuidle psci nodes.

See above, maybe there is a way to carefully copy all /reserved-memory nodes? Maybe this full copy shall be configurable, but with a proper overlapping-aware implementation maybe even a Kconfig option is not required...

Signature · 1 line
Alexander Sverdlin.
#3 lore raw

Hi Alexander,

Quoted 10 lines · Alexander Sverdlin

Hi Paul,

Quoted 7 lines · Paul Kocialkowski
Quoted 6 lines

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

Quoted 6 lines
Quoted 2 lines

To be fair we've been adding the reserved memory regions statically in the Linux device-trees to mitigate the issue.

once for H616, but it could be the only SoC among ARM64 platforms doing this and discouraged for A133:
https://lore.kernel.org/all/[email protected]/

Yes of course it would be better to have a proper solution for this. I was only mentioning this for context.

Quoted 10 lines
Quoted 7 lines

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Do you refer to some unmerged code? Didn't find it in the current TF-A sources...

fdt_add_cpu_idle_states is in common/fdt_fixup.c and it is called by the common allwinner code in plat/allwinner/common/sunxi_prepare_dtb.c based on the sunxi_psci_is_scpi check (which currently always returns false for A133).

Quoted 5 lines
Quoted 4 lines

Also note that the usual way to deal with this is to not load any device-tree when booting the kernel, which will implicitly let U-Boot use its current device-tree for Linux (with the modifications brought by tf-a).

Quoted 7 lines

?!
We definitely want to load the very device tree coming in the FIT image and pass it to the kernel from this FIT image. Sometimes people would have several DTs to chose from. The thing in U-Boot is basically to get U-Boot up and running. OF_UPSTREAM is rather to reduce the traffic on the U-Boot mailing list and maintainers effort, but in most of the cases we shall expect this DT to be not from the kernel we actually load.

Yes maybe "usual way" was not the best way to put it. "Current workaround" would maybe have been better wording. Anyway we agree that this is not a satisfying situation.

Quoted 19 lines
Quoted 12 lines

But of course I agree that it is very desirable to "forward" the device-tree modifications to the kernel device-tree so we are not stuck with whatever device-tree U-Boot was built with.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving  memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

Quoted 6 lines

RaspberryPi seems to be using "atf@0". Generally speaking the property is a free-form argument to fdt_add_reserved_memory in tf-a and I don't think we can have a common way to match them.

Introducing a Kconfig property for the prefix would be a satisfying solution in my opinion.

Quoted 7 lines

This was a very conservative patch solving the A133 case, but actually I don't see anything wrong with just copying all the reserved areas from the U-Boot live tree to the device tree we are going to pass to the kernel. Maybe fdtdec_add_reserved_memory() needs to be taught to detect overlapping ranges and extend them properly, or maybe yet another function has to be created for this purpose, to avoid duplicated reserved-memory nodes by all means,

Yes maybe a more generic approach that goes over each reserved memory node and decides whether it needs to be copied or not would be best. I would still be satisfied with a Kconfig prefix though.

Quoted 2 lines

but would also solve the PSCI cpuidle issue, as well
as potentially Raspi case of TF-A side and simplify TI K3 on U-Boot side.

Well the idle-state node is not in reserved-memory, it is part of the cpus node, but it would be fairly easy to copy it over if no such node already exists.

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

Quoted 8 lines
Quoted 4 lines

Thanks for looking at this!

Like I said, I guess the same needs to be done for the cpuidle psci nodes.

See above, maybe there is a way to carefully copy all /reserved-memory nodes? Maybe this full copy shall be configurable, but with a proper overlapping-aware implementation maybe even a Kconfig option is not required...

It is separate from reserved-memory.

All the best,

Paul

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
#4 lore raw

Hi Paul,

Quoted 8 lines · Paul Kocialkowski
Quoted 8 lines
Quoted 7 lines

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Quoted 7 lines
Quoted 2 lines

Do you refer to some unmerged code? Didn't find it in the current TF-A sources...

fdt_add_cpu_idle_states is in common/fdt_fixup.c and it is called by the common allwinner code in plat/allwinner/common/sunxi_prepare_dtb.c based on the sunxi_psci_is_scpi check (which currently always returns false for A133).

sorry, I misunderstood you initially, I though there are some data structures of the SCP firmware which are being overwritten by Linux, but it turns out, you refer to idle-states node and compatible = "arm,idle-state" subnodes... But contrary to /reserved-memory nodes for TF-A, these are fixed for a SoC model, even if you'd like to load SCP firmware at a different address, right?

It looks to me that all other machines/SoCs do define this stuff statically in the Linux DT.

I have a feeling, that idle-states node is more like a "contract" or an API between FW and Linux and is supposed to change less frequently, while loading TF-A to a completely different address would be transparent to Linux, as long as we properly copy /reserved-memory nodes.

Signature · 1 line
Alexander Sverdlin.
#5 lore raw

Hi Alexander,

Quoted 19 lines · Alexander Sverdlin

Hi Paul,

Quoted 8 lines · Paul Kocialkowski
Quoted 8 lines
Quoted 7 lines

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Quoted 7 lines
Quoted 2 lines

Do you refer to some unmerged code? Didn't find it in the current TF-A sources...

fdt_add_cpu_idle_states is in common/fdt_fixup.c and it is called by the common allwinner code in plat/allwinner/common/sunxi_prepare_dtb.c based on the sunxi_psci_is_scpi check (which currently always returns false for A133).

Quoted 5 lines

sorry, I misunderstood you initially, I though there are some data structures of the SCP firmware which are being overwritten by Linux, but it turns out, you refer to idle-states node and compatible = "arm,idle-state" subnodes... But contrary to /reserved-memory nodes for TF-A, these are fixed for a SoC model, even if you'd like to load SCP firmware at a different address, right?

The nodes are only inserted by TF-A when the SCP firmware is present. They should not be there without it so we cannot just have them always present in the static device-tree declarations.

Quoted 2 lines

It looks to me that all other machines/SoCs do define this stuff statically in the Linux DT.

I guess most of them don't consider the SCP firmware as optional but make it a hard requirement. For Allwinner it is optional (and it's very nice that way).

Quoted 4 lines

I have a feeling, that idle-states node is more like a "contract" or an API between FW and Linux and is supposed to change less frequently, while loading TF-A to a completely different address would be transparent to Linux, as long as we properly copy /reserved-memory nodes.

You could see it that way yes.

All the best,

Paul

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
#6 lore raw
In reply to Alexander Sverdlin

Hi Alexander,

thanks for taking care of solving the problem upstream, but ... ;-)

Quoted 33 lines · Alexander Sverdlin

Hi Paul,

Quoted 9 lines · Paul Kocialkowski
Quoted 6 lines

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

To be fair we've been adding the reserved memory regions statically in the Linux device-trees to mitigate the issue.

once for H616, but it could be the only SoC among ARM64 platforms doing this and discouraged for A133:
https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

But another thing we do overwrite current is the cpu idle states, which are added by fdt_add_cpu_idle_states in tf-a. These are only set when the SCP firmware is available (which is checked at run-time) and they are never propagated to the final device-tree. Including the definitions statically would result in cpu idle calls done even without the SCP firmware, which would probably fail (although maybe some states can still be supported).

Do you refer to some unmerged code? Didn't find it in the current TF-A sources...

Quoted 4 lines

Also note that the usual way to deal with this is to not load any device-tree when booting the kernel, which will implicitly let U-Boot use its current device-tree for Linux (with the modifications brought by tf-a).

Quoted 7 lines

?!
We definitely want to load the very device tree coming in the FIT image and pass it to the kernel from this FIT image. Sometimes people would have several DTs to chose from. The thing in U-Boot is basically to get U-Boot up and running. OF_UPSTREAM is rather to reduce the traffic on the U-Boot mailing list and maintainers effort, but in most of the cases we shall expect this DT to be not from the kernel we actually load.

Sorry, buy I completely and strongly disagree here. This concept of "let me load the DT that shipped with the kernel I am going to load" was ill-fated from the very beginning, but back 15 years ago it was just the only practical way of handling things, because the bootloaders were not ready. I understand that this is how embedded people use to do things ever since, but I don't think we should continue doing so.

By definition firmware (the combination of SPL, TF-A, U-Boot, crust in our case) is device specific, so the board DT living in firmware is a good match. As you said, U-Boot needs the DT anyway, and since the DT describes the hardware, it's the same DT the kernel gets. Following this idea, there should be one best (golden) DT for a board at any point in time, probably the one from the latest kernel tree. And that's the one that goes into the firmware image, and that both U-Boot and the kernel use.

For distro boot this is basically essential, and for UEFI based boot there is actually no other choice. And I like to treat the classic embedded boot methods more as special cases of the more generic ones.

I see a point of loading a custom DT during development, to play around with nodes and properties, but I think there are better ways to handle this:
- Do the DT changes in the U-Boot tree, and load u-boot-sunxi-with-spl.bin via FEL. That's what I typically do.
- Put your changes in DT overlays, and load and apply them in U-Boot:

Quoted 3 lines

fdt copy $fdtcontroladdr $fdt_addr_r
load mmc 0 fdtoverlay_addr_r my_ovl.dtbo
fdt apply $fdtoverlay_addr_r

- Patch your changes in using U-Boot's fdt command:

Quoted 2 lines

fdt copy $fdtcontroladdr $fdt_addr_r
fdt set /path/to/node your-property <42>

And even if you really want to completely load your custom DT, this is probably changed anyway, so you could patch in the things that need adjustment, like the reserved memory nodes.

Cheers,
Andre

Quoted 53 lines
Quoted 12 lines

But of course I agree that it is very desirable to "forward" the device-tree modifications to the kernel device-tree so we are not stuck with whatever device-tree U-Boot was built with.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving  memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

Quoted 6 lines

RaspberryPi seems to be using "atf@0". Generally speaking the property is a free-form argument to fdt_add_reserved_memory in tf-a and I don't think we can have a common way to match them.

Introducing a Kconfig property for the prefix would be a satisfying solution in my opinion.

This was a very conservative patch solving the A133 case, but actually I don't see anything wrong with just copying all the reserved areas from the U-Boot live tree to the device tree we are going to pass to the kernel. Maybe fdtdec_add_reserved_memory() needs to be taught to detect overlapping ranges and extend them properly, or maybe yet another function has to be created for this purpose, to avoid duplicated reserved-memory nodes by all means, but would also solve the PSCI cpuidle issue, as well as potentially Raspi case of TF-A side and simplify TI K3 on U-Boot side.

Quoted 16 lines
Quoted 9 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

Quoted 6 lines

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

Quoted 4 lines

Thanks for looking at this!

Like I said, I guess the same needs to be done for the cpuidle psci nodes.

See above, maybe there is a way to carefully copy all /reserved-memory nodes? Maybe this full copy shall be configurable, but with a proper overlapping-aware implementation maybe even a Kconfig option is not required...

#7 lore raw

Hi Andre,

Quoted 16 lines · Andre Przywara
Quoted 7 lines · Alexander Sverdlin

?!
We definitely want to load the very device tree coming in the FIT image and pass it to the kernel from this FIT image. Sometimes people would have several DTs to chose from. The thing in U-Boot is basically to get U-Boot up and running. OF_UPSTREAM is rather to reduce the traffic on the U-Boot mailing list and maintainers effort, but in most of the cases we shall expect this DT to be not from the kernel we actually load.

Sorry, buy I completely and strongly disagree here. This concept of "let me load the DT that shipped with the kernel I am going to load" was ill-fated from the very beginning, but back 15 years ago it was just the only practical way of handling things, because the bootloaders were not ready. I understand that this is how embedded people use to do things ever since, but I don't think we should continue doing so.

Quoted 7 lines

By definition firmware (the combination of SPL, TF-A, U-Boot, crust in our case) is device specific, so the board DT living in firmware is a good match. As you said, U-Boot needs the DT anyway, and since the DT describes the hardware, it's the same DT the kernel gets. Following this idea, there should be one best (golden) DT for a board at any point in time, probably the one from the latest kernel tree. And that's the one that goes into the firmware image, and that both U-Boot and the kernel use.

I think this is a vast debate that has no clear consensus. Yes there are arguments in favor of sticking everything with the boot software, very much like the ACPI approach of x86. But anyone who worked with ACPI also knows the pain of having to deal with a hardware description that is not shipped with the kernel. A large part of it is due to proprietary bootloaders and the impossibility of modification.

With a default approach that encourages shipping the device-tree with the bootloader, we can be guaranteed that we will see proprietary bootloaders that will provide broken device-trees that we can never change in the future and no way to load a different one. On the other hand if we make it clear that the device-tree is tied to the kernel then we can safely assume that bootloaders will provide a way to load a revised device-tree, which has been the case so far.

It may not affect sunxi where we have a fully free and upstream boot chain, but it will definitely happen at some point on some platforms and we can imagine the worst case scenario with a device that has a signed proprietary bootloader that will only load its non-standard vendor-customized device-tree for its downstream kernel. This will make it extremely painful to just support an upstream kernel later on.

And one could also argue that U-Boot does not need to have the same device-tree as Linux (it could be a subset that works for multiple boards for instance, or just have less units defined -- something I've seen a few vendors do). It can also have specific bits that are irrelevant to Linux (and not necessarily specified in common bindings) added to it (the -u-boot.dtsi files). So I don't think we can really talk about a single canonical source for the device-tree. This might also be the case for other systems (maybe FreeBSD) that might need to tweak the device-tree a bit at times.

In my opinion the device-tree remains related to the software that consumes it, even though we have a lot of sharing in place. And in fact the device-trees are still stored separately by each project and there is no single unified canonical source that would work for everybody.

And to state the obvious, many (if not most) users still expect to load the device-tree from the kernel and this is not going to change overnight, so deciding to avoid this use case for new features like passing tf-a nodes does not seem very reasonable to me.

And honestly it also makes the development process on the device-tree side rather painful, having to reflash the bootloader everytime a new change needs to be tested. I find it much more convenient to have clearly separated device-trees for each project.

So I definitely strongly support the idea of doing something to pass the tf-a node changes from device-tree to the other.

Quoted 3 lines

For distro boot this is basically essential, and for UEFI based boot there is actually no other choice. And I like to treat the classic embedded boot methods more as special cases of the more generic ones.

I think both uses cases are working fine with a separate device-tree.

Quoted 8 lines

I see a point of loading a custom DT during development, to play around with nodes and properties, but I think there are better ways to handle this:
- Do the DT changes in the U-Boot tree, and load u-boot-sunxi-with-spl.bin via FEL. That's what I typically do.
- Put your changes in DT overlays, and load and apply them in U-Boot:

Quoted 3 lines

fdt copy $fdtcontroladdr $fdt_addr_r
load mmc 0 fdtoverlay_addr_r my_ovl.dtbo
fdt apply $fdtoverlay_addr_r

Quoted 7 lines

- Patch your changes in using U-Boot's fdt command:

Quoted 2 lines

fdt copy $fdtcontroladdr $fdt_addr_r
fdt set /path/to/node your-property <42>

And even if you really want to completely load your custom DT, this is probably changed anyway, so you could patch in the things that need adjustment, like the reserved memory nodes.

I still find this process rather painful and I'm sure a lot of people would agree. Completely changing the development workflow of a large number of people does not feel very reasonable to me either.

And to be fair, we currently support both approaches, so people who prefer to deal with a single device-tree in u-boot can totally do that.

But I don't think we should force everybody to do this anytime soon.

All the best,

Paul

Quoted 57 lines

Cheers,
Andre

Quoted 53 lines
Quoted 12 lines

But of course I agree that it is very desirable to "forward" the device-tree modifications to the kernel device-tree so we are not stuck with whatever device-tree U-Boot was built with.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving  memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

Quoted 6 lines

RaspberryPi seems to be using "atf@0". Generally speaking the property is a free-form argument to fdt_add_reserved_memory in tf-a and I don't think we can have a common way to match them.

Introducing a Kconfig property for the prefix would be a satisfying solution in my opinion.

This was a very conservative patch solving the A133 case, but actually I don't see anything wrong with just copying all the reserved areas from the U-Boot live tree to the device tree we are going to pass to the kernel. Maybe fdtdec_add_reserved_memory() needs to be taught to detect overlapping ranges and extend them properly, or maybe yet another function has to be created for this purpose, to avoid duplicated reserved-memory nodes by all means, but would also solve the PSCI cpuidle issue, as well as potentially Raspi case of TF-A side and simplify TI K3 on U-Boot side.

Quoted 16 lines
Quoted 9 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

Quoted 6 lines

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

Quoted 4 lines

Thanks for looking at this!

Like I said, I guess the same needs to be done for the cpuidle psci nodes.

See above, maybe there is a way to carefully copy all /reserved-memory nodes? Maybe this full copy shall be configurable, but with a proper overlapping-aware implementation maybe even a Kconfig option is not required...

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
#8 lore raw
In reply to Alexander Sverdlin

Hi Alexander,

thanks for taking care of this upstream, and doing it in a generic (not Allwinner specific way)!
As you can see from that other reply, I am not a fan of those propagations from the control DT to something loaded, as this looks like a can of worms and I think that's unnecessary. I might be talked into accepting this for the reserved memory regions, since they are somewhat crucial, and also generic.

Quoted 7 lines · Alexander Sverdlin

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

And others like Allwinner simply forget to do it, which results in Linux overwriting TF-A and crashing.

Quoted 7 lines

Unfortunately seems that the things are not much better on TF-A side and there is no universal way to get the reserved memory region across platforms. But there is at least a most common way in TF-A, namely reserving memory range in the FDT, in particular:
- Allwinner ("tf-a@40000000" node)
- ARM FPGA ("tf-a@80000000" node)
- Xilinx ("tf-a" node)

Regardless of the general approach, as Paul already mentioned: the node names are by design arbitrary, and no meaning should be derived from them. So either we just copy all reserved nodes (why not?), or we look at the no-map property, which tells us that this region must not be mapped, to avoid speculation by the core into this memory area. Which would fault if this memory is secure-only. So all secure memory should have the no-map property, though not all no-map tagged regions must be secure-only.

One actual nice way of abstracting this is to look at U-Boot's Logical Memory Blocks (LMBs), which is U-Boot's crude way of dealing with memory regions. There is already code in place that puts no-map reserved regions from the control DT into those LMBs. If you type "bdinfo" on the U-Boot prompt, you should see them. If you create reserved memory regions based on the LMBs, that would be even more generic.

Quoted 8 lines

While this patch aims to improve the situation for Allwinner platforms, it's deliberately adding more generic code to pave the potential way of unification for other platforms.

Note that fdtdec_add_reserved_memory() has a check for an already existing carveout with exactly matching boundaries and will not create a duplicate even if the name doesn't match. It would not however detect an already existing bigger carveout fully containing the one requested.

Finding larger blocks should be quite easy, although there are theoretical corner cases which get nasty. I wonder if the LMB parsing code already solves that problem?

Cheers,
Andre

Quoted 96 lines

---
The patch has been developed to faciliate Allwinner A133 SoC support, where most of the work currently happens on TF-A [1] and Linux [2] sides, but I wanted to send this patch upfront to get the first feedback and because already supported H616 SoC would already benefit from the patch.

[1] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/49754
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

  arch/arm/lib/bootm-fdt.c | 55 ++++++++++++++++++++++++++++++++++++++++
  1 file changed, 55 insertions(+)
arch/arm/lib/bootm-fdt.c +55 -0
index 2671f9a0ebf..19d917943c1 100644
@@ -14,7 +14,9 @@
   * Copyright (C) 2001  Erik Mouw ([email protected])
   */
 
+#include <dm/ofnode.h>
  #include <fdt_support.h>
+#include <linux/ioport.h>
  #ifdef CONFIG_ARMV7_NONSEC
  #include <asm/armv7.h>
  #endif
@@ -24,6 +26,52 @@
 
  DECLARE_GLOBAL_DATA_PTR;
 
+static int tfa_copy_reserved_memory(void *new_blob)
+{
+#ifdef CONFIG_OF_CONTROL
+	ofnode node, subnode;
+
+	/*
+	 * TF-A for several platforms inserts its memory region as
+	 * reserved-memory node
+	 */
+	node = ofnode_path("/reserved-memory");
+	if (!ofnode_valid(node))
+		return 0;
+
+	ofnode_for_each_subnode(subnode, node) {
+		struct fdt_memory carveout;
+		struct resource res;
+		const char *name;
+		int ret;
+
+		name = ofnode_get_name(subnode);
+		if (!name)
+			return -FDT_ERR_BADSTRUCTURE;
+
+		/* only handle TF-A reservations */
+		if (strncmp(name, "tf-a", 4))
+			continue;
+
+		/* check if this subnode has a reg property */
+		ret = ofnode_read_resource(subnode, 0, &res);
+		if (ret)
+			continue;
+
+		carveout.start = res.start,
+		carveout.end = res.end,
+
+		ret = fdtdec_add_reserved_memory(new_blob, "tf-a", &carveout,
+						 NULL, 0, NULL,
+						 FDTDEC_RESERVED_MEMORY_NO_MAP);
+		if (ret < 0)
+			return ret;
+	}
+#endif
+
+	return 0;
+}
+
  #ifdef CONFIG_FMAN_ENET
  __weak int fdt_update_ethernet_dt(void *blob)
  {
@@ -56,6 +104,13 @@ int arch_fixup_fdt(void *blob)
  		return ret;
  #endif
 
+	ret = tfa_copy_reserved_memory(blob);
+	if (ret) {
+		printf("ERROR: transfer of TF-A nodes to new fdt failed: %s\n",
+		       fdt_strerror(ret));
+		return ret;
+	}
+
  #ifdef CONFIG_ARMV8_SPIN_TABLE
  	ret = spin_table_update_dt(blob);
  	if (ret)
#9 lore raw
In reply to Alexander Sverdlin

Hi Alexander,

Quoted 6 lines · Alexander Sverdlin

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Currently some ARM-based platforms reserve TF-A memory in their own ways:
- Mediatek gets BL31 region via SMC call in ft_system_setup()
- K3 uses CONFIG_K3_ATF_LOAD_ADDR, effectively in ft_system_setup()

On Rockchip, we reserve the first 2MiB of DRAM. c.f. arch/arm/mach-rockchip/sdram.c. I haven't checked how it makes it to the Linux FDT but I'm assuming it does otherwise it'd be broken in interesting ways. Don't ask us how we'll handle TF-A changing its location or amount of RAM. We also need to support Rockchip's TF-A blob which... doesn't support FDT (hence NO_PLATFORM_PARAM=y for most SoCs by default).

When xPL passes an FDT to TF-A (BL31; CONFIG=NO_PLATFORM_PARAM=n), TF-A passes it to OP-TEE OS (BL32) and OP-TEE OS writes its reserved area to the FDT before going back to U-Boot BL33.

U-Boot reads this (if CONFIG_OPTEE_LIB=y + CONFIG_OF_LIBFDT=y which are forced if you built with TEE variable present in your environment) and propagates the reserved areas to the kernel FDT, c.f. optee_copy_firmware_node(). So you probably want to hook yourself to wherever this is called.

Doing a quick grep in TF-A sources, it seems Alwinner is patching the FDT (and I'm now concerned not seeing Rockchip in there.... some more work to do I guess) which I assume is what prompted this series :)

Cheers,
Quentin