Skip to contentSkip to conversation

bcm4908: Fix secondary CPU initialization

PATCH Applied Sam Edwards 2 participants 7 messages Oct 5, 2024 – Nov 24, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover bcm4908: Fix secondary CPU initialization

Hello list,

This is v2 of my previous patch [1] for resolving a problem preventing secondary CPU(s) from coming up on bcm4908. After some discussion, I decided to try dropping the reserved memory region from 64K to only 4K. Looks like it works!

Changes v1->v2:
- Reduce 64K reserved region to 4K
- Style change to the reg property to use hex instead of decimal
- Slight rephrasing to the commit message
- Add a new patch that also moves the cpu-release-addr into this reserved memory region

Cheers,
Sam

[1]: https://lore.kernel.org/lkml/[email protected]/T/

Sam Edwards (2):
arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

 .../boot/dts/broadcom/bcmbca/bcm4908.dtsi      | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.2
#1 lore raw

PATCH v2 1/2 arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches smp_prepare_cpus(), the secondary CPUs may become inaccessible.

This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.

Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the first 4KiB of physical memory.

---
 arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +10 -0
index 8b924812322c..c51b92387fad 100644
@@ -68,6 +68,16 @@ l2: l2-cache0 {
 		};
 	};
 
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+
+		cfe-stub@0 {
+			reg = <0x0 0x0 0x0 0x1000>;
+		};
+	};
+
 	axi@81000000 {
 		compatible = "simple-bus";
 		#address-cells = <1>;
Signature · 1 line
2.44.2
maintainer
#2 lore raw

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

Quoted 11 lines · Sam Edwards

The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches smp_prepare_cpus(), the secondary CPUs may become inaccessible.

This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.

If we supported CPU hotplug on those platforms (do we?) then it actually does matter that this memory remains protected, and it cannot be reclaimed. This does not invalidate the commit message and I will take it as-is, but it it is not memory that we can necessarily reclaim that easily, if we did things properly.

Quoted 30 lines

Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the first 4KiB of physical memory.

---
  arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
  1 file changed, 10 insertions(+)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +10 -0
index 8b924812322c..c51b92387fad 100644
@@ -68,6 +68,16 @@ l2: l2-cache0 {
  		};
  	};
 
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+
+		cfe-stub@0 {
+			reg = <0x0 0x0 0x0 0x1000>;
+		};
+	};
+
  	axi@81000000 {
  		compatible = "simple-bus";
  		#address-cells = <1>;
Signature · 1 line
Florian
#3 lore raw

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

Quoted 14 lines · Florian Fainelli
Quoted 11 lines · Sam Edwards

The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches smp_prepare_cpus(), the secondary CPUs may become inaccessible.

This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.

Quoted 5 lines

If we supported CPU hotplug on those platforms (do we?) then it actually does matter that this memory remains protected, and it cannot be reclaimed. This does not invalidate the commit message and I will take it as-is, but it it is not memory that we can necessarily reclaim that easily, if we did things properly.

I am looking at only one build of CFE, so don't take what I'm saying as gospel, but as I understand it:

CFE implements only the spin-table method, which isn't dynamic. Once the kernel calls for the secondary CPUs to be released, it cannot shut them down and bring them up again, so they have no need to reenter the spin stub.

With U-Boot+ATF, it's of course a different story: ATF needs to stay resident in memory to implement PSCI (which _is_ dynamic), but U-Boot inserts/overwrites the necessary DT structures to protect ATF and tell Linux to use PSCI, so we don't need to be thinking about that here.

But, again, that's just my understanding. If there's a version of CFE out there that loads ATF/similar or implements PSCI itself, then, well, that'd invalidate this patch('s commit message). I doubt such a variant of CFE exists, but you're much better equipped to confirm or refute that possibility, given your familiarity with the platform and internal details.

Best wishes,
Sam

Quoted 35 lines
Quoted 21 lines

Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the first 4KiB of physical memory.

---
  arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++
  1 file changed, 10 insertions(+)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +3 -0
index 8b924812322c..c51b92387fad 100644
@@ -68,6 +68,16 @@ l2: l2-cache0 {
              };
      };
 
+     reserved-memory {
+             #address-cells = <2>;
+             #size-cells = <2>;

arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi l2: l2-cache0 { · line 83

Quoted 10 lines
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +7 -0
+             ranges;
+
+             cfe-stub@0 {
+                     reg = <0x0 0x0 0x0 0x1000>;
+             };
+     };
+
      axi@81000000 {
              compatible = "simple-bus";
              #address-cells = <1>;

--
Florian

maintainer Applied
#4 lore raw

Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area

In reply to Sam Edwards

From: Florian Fainelli <[email protected]>

Quoted 12 lines · Sam Edwards

The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches smp_prepare_cpus(), the secondary CPUs may become inaccessible.

This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.

Quoted 5 lines

Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the first 4KiB of physical memory.

---

Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks! --
Florian

#5 lore raw

PATCH v2 2/2 arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

In reply to Sam Edwards

The cpu-release-addr property is relevant only when the "spin-table" enable method is used. It is the physical address where the bootloader expects Linux to write the secondary CPU entry point's physical address. On this platform, only the CFE bootloader uses this method: U-Boot uses PSCI instead.

CFE actually walks the FDT to learn this address, so we're free to put it wherever we want. We only need to make sure that it goes in a reserved-memory block so that writing to it during early boot does not risk conflicting with an unrelated memory allocation: this was not done.

Since the previous patch reserved the first page of memory for CFE's secondary-CPU init stub, which is actually much smaller than a page, just put this address at the end of that page and it shall be so protected.

---
 arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +4 -4
index c51b92387fad..613ba7ee43d6 100644
@@ -30,7 +30,7 @@ cpu0: cpu@0 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x0>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -39,7 +39,7 @@ cpu1: cpu@1 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x1>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -48,7 +48,7 @@ cpu2: cpu@2 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x2>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
@@ -57,7 +57,7 @@ cpu3: cpu@3 {
 			compatible = "brcm,brahma-b53";
 			reg = <0x3>;
 			enable-method = "spin-table";
-			cpu-release-addr = <0x0 0xfff8>;
+			cpu-release-addr = <0x0 0xff8>;
 			next-level-cache = <&l2>;
 		};
 
Signature · 1 line
2.44.2
maintainer Applied
#6 lore raw

Re: [PATCH v2 2/2] arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr

From: Florian Fainelli <[email protected]>

Quoted 11 lines · Sam Edwards

The cpu-release-addr property is relevant only when the "spin-table" enable method is used. It is the physical address where the bootloader expects Linux to write the secondary CPU entry point's physical address. On this platform, only the CFE bootloader uses this method: U-Boot uses PSCI instead.

CFE actually walks the FDT to learn this address, so we're free to put it wherever we want. We only need to make sure that it goes in a reserved-memory block so that writing to it during early boot does not risk conflicting with an unrelated memory allocation: this was not done.

Quoted 7 lines

Since the previous patch reserved the first page of memory for CFE's secondary-CPU init stub, which is actually much smaller than a page, just put this address at the end of that page and it shall be so protected.

---

Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks! --
Florian