bcm4908: Fix secondary CPU initialization
Subsystems (2)
v2 cover bcm4908: Fix secondary CPU initialization
Hello list,
This is v2 of my previous patch [1] for resolving a problem preventing secondary CPU(s) from coming up on bcm4908. After some discussion, I decided to try dropping the reserved memory region from 64K to only 4K. Looks like it works!
Changes v1->v2:
- Reduce 64K reserved region to 4K
- Style change to the reg property to use hex instead of decimal
- Slight rephrasing to the commit message
- Add a new patch that also moves the cpu-release-addr into this reserved memory region
Cheers,
Sam
[1]: https://lore.kernel.org/lkml/[email protected]/T/
Sam Edwards (2):
arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr
.../boot/dts/broadcom/bcmbca/bcm4908.dtsi | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.2
PATCH v2 1/2 arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches smp_prepare_cpus(), the secondary CPUs may become inaccessible.
This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.
Therefore, add a /reserved-memory node to bcm4908.dtsi to protect the first 4KiB of physical memory.
--- arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +10 -0
@@ -68,6 +68,16 @@ l2: l2-cache0 { }; }; + reserved-memory { + #address-cells = <2>; + #size-cells = <2>; + ranges; + + cfe-stub@0 { + reg = <0x0 0x0 0x0 0x1000>; + }; + }; + axi@81000000 { compatible = "simple-bus"; #address-cells = <1>;
Signature · 1 line
2.44.2
Florian Fainelli
maintainerRe: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
Quoted 11 lines · Sam Edwards
The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches
smp_prepare_cpus(), the secondary CPUs may become inaccessible.This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.
If we supported CPU hotplug on those platforms (do we?) then it actually does matter that this memory remains protected, and it cannot be reclaimed. This does not invalidate the commit message and I will take it as-is, but it it is not memory that we can necessarily reclaim that easily, if we did things properly.
Quoted 30 lines
Therefore, add a /reserved-memory node to
bcm4908.dtsito protect the first 4KiB of physical memory.Signed-off-by Sam Edwards <[email protected]>--- arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +10
-0@@ -68,6 +68,16 @@ l2: l2-cache0 { }; }; + reserved-memory { + #address-cells = <2>; + #size-cells = <2>; + ranges; + + cfe-stub@0 { + reg = <0x0 0x0 0x0 0x1000>; + }; + }; + axi@81000000 { compatible = "simple-bus"; #address-cells = <1>;
Signature · 1 line
Florian
Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
Quoted 14 lines · Florian Fainelli
Quoted 11 lines · Sam Edwards
The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches
smp_prepare_cpus(), the secondary CPUs may become inaccessible.This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.
Quoted 5 lines
If we supported CPU hotplug on those platforms (do we?) then it actually does matter that this memory remains protected, and it cannot be reclaimed. This does not invalidate the commit message and I will take it as-is, but it it is not memory that we can necessarily reclaim that easily, if we did things properly.
I am looking at only one build of CFE, so don't take what I'm saying as gospel, but as I understand it:
CFE implements only the spin-table method, which isn't dynamic. Once the kernel calls for the secondary CPUs to be released, it cannot shut them down and bring them up again, so they have no need to reenter the spin stub.
With U-Boot+ATF, it's of course a different story: ATF needs to stay resident in memory to implement PSCI (which _is_ dynamic), but U-Boot inserts/overwrites the necessary DT structures to protect ATF and tell Linux to use PSCI, so we don't need to be thinking about that here.
But, again, that's just my understanding. If there's a version of CFE out there that loads ATF/similar or implements PSCI itself, then, well, that'd invalidate this patch('s commit message). I doubt such a variant of CFE exists, but you're much better equipped to confirm or refute that possibility, given your familiarity with the platform and internal details.
Best wishes,
Sam
Quoted 35 lines
Quoted 21 lines
Therefore, add a /reserved-memory node to
bcm4908.dtsito protect the first 4KiB of physical memory.Signed-off-by Sam Edwards <[email protected]>--- arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +3
-0@@ -68,6 +68,16 @@ l2: l2-cache0 { }; }; + reserved-memory { + #address-cells = <2>; + #size-cells = <2>;arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi l2: l2-cache0 { · line 83
Quoted 10 lines
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +7
-0+ ranges; + + cfe-stub@0 { + reg = <0x0 0x0 0x0 0x1000>; + }; + }; + axi@81000000 { compatible = "simple-bus"; #address-cells = <1>;--
Florian
Re: [PATCH v2 1/2] arm64: dts: broadcom: bcmbca: bcm4908: Reserve CFE stub area
In reply to Sam EdwardsFrom: Florian Fainelli <[email protected]>
Quoted 12 lines · Sam Edwards
The CFE bootloader places a stub program in the first page of physical memory to hold the secondary CPUs until the boot CPU writes the release address, but does not splice a /reserved-memory node into the FDT to protect it. If Linux overwrites this program before execution reaches
smp_prepare_cpus(), the secondary CPUs may become inaccessible.This is only a problem with CFE, and then only until the secondary CPUs are brought online. Ideally, there would be some hypothetical mechanism we could use to indicate that this area of memory is sensitive only during boot. But as there is none, and since it is such a small amount of memory, it is easiest to reserve it unconditionally.
Quoted 5 lines
Therefore, add a /reserved-memory node to
bcm4908.dtsito protect the first 4KiB of physical memory.Signed-off-by Sam Edwards <[email protected]>---
Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks! --
Florian
PATCH v2 2/2 arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr
In reply to Sam EdwardsThe cpu-release-addr property is relevant only when the "spin-table" enable method is used. It is the physical address where the bootloader expects Linux to write the secondary CPU entry point's physical address. On this platform, only the CFE bootloader uses this method: U-Boot uses PSCI instead.
CFE actually walks the FDT to learn this address, so we're free to put it wherever we want. We only need to make sure that it goes in a reserved-memory block so that writing to it during early boot does not risk conflicting with an unrelated memory allocation: this was not done.
Since the previous patch reserved the first page of memory for CFE's secondary-CPU init stub, which is actually much smaller than a page, just put this address at the end of that page and it shall be so protected.
--- arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/arm64/boot/dts/broadcom/bcmbca/bcm4908.dtsi +4 -4
@@ -30,7 +30,7 @@ cpu0: cpu@0 { compatible = "brcm,brahma-b53"; reg = <0x0>; enable-method = "spin-table"; - cpu-release-addr = <0x0 0xfff8>; + cpu-release-addr = <0x0 0xff8>; next-level-cache = <&l2>; };
@@ -39,7 +39,7 @@ cpu1: cpu@1 { compatible = "brcm,brahma-b53"; reg = <0x1>; enable-method = "spin-table"; - cpu-release-addr = <0x0 0xfff8>; + cpu-release-addr = <0x0 0xff8>; next-level-cache = <&l2>; };
@@ -48,7 +48,7 @@ cpu2: cpu@2 { compatible = "brcm,brahma-b53"; reg = <0x2>; enable-method = "spin-table"; - cpu-release-addr = <0x0 0xfff8>; + cpu-release-addr = <0x0 0xff8>; next-level-cache = <&l2>; };
@@ -57,7 +57,7 @@ cpu3: cpu@3 { compatible = "brcm,brahma-b53"; reg = <0x3>; enable-method = "spin-table"; - cpu-release-addr = <0x0 0xfff8>; + cpu-release-addr = <0x0 0xff8>; next-level-cache = <&l2>; };
Signature · 1 line
2.44.2
Re: [PATCH v2 2/2] arm64: dts: broadcom: bcmbca: bcm4908: Protect cpu-release-addr
From: Florian Fainelli <[email protected]>
Quoted 11 lines · Sam Edwards
The
cpu-release-addrproperty is relevant only when the "spin-table" enable method is used. It is the physical address where the bootloader expects Linux to write the secondary CPU entry point's physical address. On this platform, only the CFE bootloader uses this method: U-Boot uses PSCI instead.CFE actually walks the FDT to learn this address, so we're free to put it wherever we want. We only need to make sure that it goes in a reserved-memory block so that writing to it during early boot does not risk conflicting with an unrelated memory allocation: this was not done.
Quoted 7 lines
Since the previous patch reserved the first page of memory for CFE's secondary-CPU init stub, which is actually much smaller than a page, just put this address at the end of that page and it shall be so protected.
Signed-off-by Sam Edwards <[email protected]>---
Applied to https://github.com/Broadcom/stblinux/commits/devicetree-arm64/next, thanks! --
Florian