arm64, thunder: Enable Cavium Thunder SoC Family

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Quoted 5 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Just out of interest, does raising the current maximum limit actually break any existing code? If not, then doing this as two patches doesn't seem worth it.

Quoted 10 lines · Will Deacon

Quoted 5 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Just out of interest, does raising the current maximum limit actually break any existing code? If not, then doing this as two patches doesn't seem worth it.

Increasing to 64 should be fine from the perspective of cpu mask implementation. Memory foot print should be the same already as this uses long which is 64 bit. So this wouldn't hurt.

However, I felt a bit uncomfortable having a dependency here to enabling 64 cpus and getting this patch set upstream. Support for more than 32 cpus is not well tested yet and there still might be problems with e.g. interrupt delivery or topology.

Quoted 12 lines · Robert Richter

Quoted 10 lines · Will Deacon

Quoted 5 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Just out of interest, does raising the current maximum limit actually break any existing code? If not, then doing this as two patches doesn't seem worth it.

Quoted 8 lines

Increasing to 64 should be fine from the perspective of cpu mask implementation. Memory foot print should be the same already as this uses long which is 64 bit. So this wouldn't hurt.

However, I felt a bit uncomfortable having a dependency here to enabling 64 cpus and getting this patch set upstream. Support for more than 32 cpus is not well tested yet and there still might be problems with e.g. interrupt delivery or topology.

All I mean is, if the kernel doesn't explode on existing systems by changing the upper limit to 64, then we should do that. If you're not comfortable that the Thunder code can handle that, then leave the thunder default as 32, like you do in the current patch. It just seems odd not to change the maximum number, since it's an arbitrary limit from my perspective.

Quoted 5 lines · Will Deacon

All I mean is, if the kernel doesn't explode on existing systems by changing the upper limit to 64, then we should do that. If you're not comfortable that the Thunder code can handle that, then leave the thunder default as 32, like you do in the current patch. It just seems odd not to change the maximum number, since it's an arbitrary limit from my perspective.

FWIW my Juno happily boots with a NR_CPUS=64 kernel.

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 arch/arm64/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)

arch/arm64/Kconfig +5 -0

index fd4e81a4e1ce..77fb82b0f684 100644
@@ -134,6 +134,11 @@ source "kernel/Kconfig.freezer"
 
 menu "Platform selection"
 
+config ARCH_THUNDER
+	bool "Cavium Inc. Thunder SoC Family"
+	help
+	  This enables support for Cavium's Thunder Family of SoCs.
+
 config ARCH_VEXPRESS
 	bool "ARMv8 software model (Versatile Express)"
 	select ARCH_REQUIRE_GPIOLIB

Quoted 20 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 arch/arm64/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)

arch/arm64/Kconfig +0 -0

index fd4e81a4e1ce..77fb82b0f684 100644
@@ -134,6 +134,11 @@ source "kernel/Kconfig.freezer"
 
 menu "Platform selection"
 

Quoted 9 lines

arch/arm64/Kconfig +5 -0

+config ARCH_THUNDER
+	bool "Cavium Inc. Thunder SoC Family"
+	help
+	  This enables support for Cavium's Thunder Family of SoCs.
+
 config ARCH_VEXPRESS
 	bool "ARMv8 software model (Versatile Express)"
 	select ARCH_REQUIRE_GPIOLIB

arch/arm64/Kconfig config NR_CPUS · line 265

Quoted 5 lines

arch/arm64/Kconfig +1 -0

@@ -256,6 +261,7 @@ config NR_CPUS
 	range 2 32
 	depends on SMP
 	# These have to remain sorted largest to smallest
+	default "32" if ARCH_THUNDER
 	default "8"

Why do you need ARCH_THUNDER? If it's just to change this default,

please don't bother - we don't do this kind of thing on x86, so why should it be done here? Just ensure that NR_CPUS is appropriately adjusted. Alternatively, look at how x86 deals with this (with X86_BIGSMP / MAXSMP).

Quoted 21 lines · Russell King - ARM Linux

Quoted 20 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 arch/arm64/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)

arch/arm64/Kconfig +0 -0

index fd4e81a4e1ce..77fb82b0f684 100644
@@ -134,6 +134,11 @@ source "kernel/Kconfig.freezer"
 
 menu "Platform selection"
 

Quoted 16 lines

Quoted 9 lines

arch/arm64/Kconfig +5 -0

+config ARCH_THUNDER
+	bool "Cavium Inc. Thunder SoC Family"
+	help
+	  This enables support for Cavium's Thunder Family of SoCs.
+
 config ARCH_VEXPRESS
 	bool "ARMv8 software model (Versatile Express)"
 	select ARCH_REQUIRE_GPIOLIB

arch/arm64/Kconfig config NR_CPUS · line 265

Quoted 5 lines

arch/arm64/Kconfig +1 -0

@@ -256,6 +261,7 @@ config NR_CPUS
 	range 2 32
 	depends on SMP
 	# These have to remain sorted largest to smallest
+	default "32" if ARCH_THUNDER
 	default "8"

Why do you need ARCH_THUNDER? If it's just to change this default,

No, we need it just to enable all our drivers on the SoC. We want to enable the SoC by using defconfig + ARCH_THUNDER. As said in my other mail, I put it here to be able to base all other thunder driver patch sets on this initial base patch set. Otherwise this particular patch and also the dtb patch need to be shipped with all other driver patch sets. This might lead to duplicate submissions of the same patch.

With doing defconfig + ARCH_THUNDER we also want to enable the max number of cpus that is currently supported. I only enable 32 cpus since booting more cpus is untested. There might be problems at the 32 cpu boundary. Just setting it to 64 does not mean a kernel will actually boot more than 32 cpus. But if it will be ack'ed, I would be fine to set NR_CPUS to 32 or 64 in general and independent from ARCH_THUNDER.

For simplicity I better drop setting NR_CPUS in this patch.

Quoted 4 lines

please don't bother - we don't do this kind of thing on x86, so why should it be done here? Just ensure that NR_CPUS is appropriately adjusted. Alternatively, look at how x86 deals with this (with X86_BIGSMP / MAXSMP).

Quoted 54 lines · Robert Richter

Quoted 21 lines · Russell King - ARM Linux

Quoted 20 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

Increase maximum numbers of cpus to 32. This relates to current maximal possible cpu number. Increasing this to 64 cpus will be a separate patch not part of this enablement patches.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 arch/arm64/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)

arch/arm64/Kconfig +0 -0

index fd4e81a4e1ce..77fb82b0f684 100644
@@ -134,6 +134,11 @@ source "kernel/Kconfig.freezer"
 
 menu "Platform selection"
 

Quoted 16 lines

Quoted 9 lines

arch/arm64/Kconfig +5 -0

+config ARCH_THUNDER
+	bool "Cavium Inc. Thunder SoC Family"
+	help
+	  This enables support for Cavium's Thunder Family of SoCs.
+
 config ARCH_VEXPRESS
 	bool "ARMv8 software model (Versatile Express)"
 	select ARCH_REQUIRE_GPIOLIB

arch/arm64/Kconfig config NR_CPUS · line 265

Quoted 5 lines

arch/arm64/Kconfig +1 -0

@@ -256,6 +261,7 @@ config NR_CPUS
 	range 2 32
 	depends on SMP
 	# These have to remain sorted largest to smallest
+	default "32" if ARCH_THUNDER
 	default "8"

Why do you need ARCH_THUNDER? If it's just to change this default,

No, we need it just to enable all our drivers on the SoC. We want to enable the SoC by using defconfig + ARCH_THUNDER. As said in my other mail, I put it here to be able to base all other thunder driver patch sets on this initial base patch set. Otherwise this particular patch and also the dtb patch need to be shipped with all other driver patch sets. This might lead to duplicate submissions of the same patch.

With doing defconfig + ARCH_THUNDER we also want to enable the max number of cpus that is currently supported. I only enable 32 cpus since booting more cpus is untested. There might be problems at the 32 cpu boundary. Just setting it to 64 does not mean a kernel will actually boot more than 32 cpus. But if it will be ack'ed, I would be fine to set NR_CPUS to 32 or 64 in general and independent from ARCH_THUNDER.

Quoted 1 line

For simplicity I better drop setting NR_CPUS in this patch.

So, ARM64 will get a big long list of "config ARCH_foo" options just to stuff lots of broken select statements into the configuration. Yes, this may have been the norm with ARM, but it's turned out to be more of a problem than a solution, especially as it keeps causing Kconfig warnings when things change in the rest of the kernel tree.

The same is true with defconfigs - Linus threatened to delete all ARM defconfigs except one at one point.

Quoted 19 lines · Russell King - ARM Linux

Quoted 17 lines · Robert Richter

On 05.09.14 10:32:40, Russell King - ARM Linux wrote:
No, we need it just to enable all our drivers on the SoC. We want to enable the SoC by using defconfig + ARCH_THUNDER. As said in my other mail, I put it here to be able to base all other thunder driver patch sets on this initial base patch set. Otherwise this particular patch and also the dtb patch need to be shipped with all other driver patch sets. This might lead to duplicate submissions of the same patch.

With doing defconfig + ARCH_THUNDER we also want to enable the max number of cpus that is currently supported. I only enable 32 cpus since booting more cpus is untested. There might be problems at the 32 cpu boundary. Just setting it to 64 does not mean a kernel will actually boot more than 32 cpus. But if it will be ack'ed, I would be fine to set NR_CPUS to 32 or 64 in general and independent from ARCH_THUNDER.

For simplicity I better drop setting NR_CPUS in this patch.

Quoted 5 lines

So, ARM64 will get a big long list of "config ARCH_foo" options just to stuff lots of broken select statements into the configuration. Yes, this may have been the norm with ARM, but it's turned out to be more of a problem than a solution, especially as it keeps causing Kconfig warnings when things change in the rest of the kernel tree.

Agreed; this seems more pain than it is worth.

Quoted 2 lines

The same is true with defconfigs - Linus threatened to delete all ARM defconfigs except one at one point.

IMO we should continue doing what we've done so far and make the ARM64 defconfig work on everything it can by default, no ARCH_* necessary. That's what most people will build and test and we shouldn't get platform-specific code (well, drivers) breaking the single image.

For the extreme configurations (really tiny or really big) custom configuration being necessary is fine. That doesn't have to involve ARCH_* config options.

If you want to build a custom config then you should have an idea of what you need. ARCH_* options are only necessary if someone wants a kernel tuned for a specific SoC but doesn't know anything about that SoC.

A common pattern these days is to do dependencies like

arch/*/Kconfig:

	config ARCH_FOO
	bool "Enable support for Foo platform"
	help
	  ...

drivers/*/Kconfig

	config SUBSYS_FOO
	bool "SUBSYS driver for Foo"
	depends on ARCH_FOO || COMPILE_TEST
	depends on OF && REGULATOR && GENERIC_PHY # or whatever

That way we can enable everything in the defconfig, but someone who likes to build a more specialized kernel can disable the other platforms and won't get the drivers that are specific to those.

I personally think this is a bit more verbose than what we need, but I don't strongly object doing it that way.

The code size really should not matter much on ARM64 though: it's unlikely we will see a lot of systems with less than a few gigabytes of memory, and I expect that a generic kernel would be e.g. 6 MB instead of 4 MB for a platform specific kernel.

Quoted 10 lines

A common pattern these days is to do dependencies like

arch/*/Kconfig:
config ARCH_FOO
bool "Enable support for Foo platform"
help
...

Quoted 5 lines

drivers/*/Kconfig

      config SUBSYS_FOO
      bool "SUBSYS driver for Foo"
      depends on ARCH_FOO || COMPILE_TEST
      depends on OF && REGULATOR && GENERIC_PHY # or whatever

Russell's comments w.r.t. Kconfig warnings when config names change still holds regardless of select vs depends on.

Quoted 5 lines

That way we can enable everything in the defconfig, but someone who likes to build a more specialized kernel can disable the other platforms and won't get the drivers that are specific to those.

Quoted 2 lines

I personally think this is a bit more verbose than what we need, but I don't strongly object doing it that way.

You'd still be able to do this without ARCH_FOO, though you would need to know which drivers are necessary for a particular SoC. That seems to be the way things are handled on x86; I don't recall having to select support for specific machines there, just the individual drivers.

Quoted 16 lines · On Friday 05 September 2014 15:22:46 Mark Rutland wrote:

Quoted 15 lines

A common pattern these days is to do dependencies like

arch/*/Kconfig:
config ARCH_FOO
bool "Enable support for Foo platform"
help
...

drivers/*/Kconfig

      config SUBSYS_FOO
      bool "SUBSYS driver for Foo"
      depends on ARCH_FOO || COMPILE_TEST
      depends on OF && REGULATOR && GENERIC_PHY # or whatever

Quoted 2 lines

Russell's comments w.r.t. Kconfig warnings when config names change still holds regardless of select vs depends on.

Yes, that's what I wrote in my reply as well.

Quoted 12 lines

Quoted 7 lines

That way we can enable everything in the defconfig, but someone who likes to build a more specialized kernel can disable the other platforms and won't get the drivers that are specific to those.

I personally think this is a bit more verbose than what we need, but I don't strongly object doing it that way.

You'd still be able to do this without ARCH_FOO, though you would need to know which drivers are necessary for a particular SoC. That seems to be the way things are handled on x86; I don't recall having to select support for specific machines there, just the individual drivers.

The main difference is that there are very few drivers on x86 that are specific to one of the two chip makers. Almost everything is a PCI device that can actually be plugged in anywhere.

On ARM64 there is going to be a lot of stuff that really makes sense only for one of the 50 licensees.

This is well hidden on x86, but each vendor has a config option. For AMD systems we have had our own option to disable vendor specific code, see CPU_SUP_AMD for example. Disabling this will remove all AMD specific code in the kernel. Of course this is enabled per default.

With ARCH_THUNDER I intended to do the same (you could name this also SOC_SUP_CAVIUM or so but I kept the current naming scheme). In patch 4/4 I have added the option to defconfig. This enables this per default and nobody has to deal with any option manually, just running make defconfig is fine.

Also, at least to disable building the dtb file for foo, you will need ARCH_FOO too. How else would you deal with dtb files then?

Having ARCH_FOO might not be necessary for drivers. One could enable drivers manually, but this option is still a good reference for the drivers needed by foo. At some point you will carry tons of enabled drivers in your defconfig and you don't know which platform actually is using it. For generic drivers this might be fine. But in my point of few, each soc specific driver should have an soc specific option too. Then you easily can remove an soc from the defconfig.

Quoted 17 lines · Arnd Bergmann

Quoted 16 lines · On Friday 05 September 2014 15:22:46 Mark Rutland wrote:

Quoted 15 lines

A common pattern these days is to do dependencies like

arch/*/Kconfig:
config ARCH_FOO
bool "Enable support for Foo platform"
help
...

drivers/*/Kconfig

      config SUBSYS_FOO
      bool "SUBSYS driver for Foo"
      depends on ARCH_FOO || COMPILE_TEST
      depends on OF && REGULATOR && GENERIC_PHY # or whatever

Quoted 17 lines

Quoted 2 lines

Russell's comments w.r.t. Kconfig warnings when config names change still holds regardless of select vs depends on.

Yes, that's what I wrote in my reply as well.

Quoted 12 lines

Quoted 7 lines

That way we can enable everything in the defconfig, but someone who likes to build a more specialized kernel can disable the other platforms and won't get the drivers that are specific to those.

I personally think this is a bit more verbose than what we need, but I don't strongly object doing it that way.

You'd still be able to do this without ARCH_FOO, though you would need to know which drivers are necessary for a particular SoC. That seems to be the way things are handled on x86; I don't recall having to select support for specific machines there, just the individual drivers.

This is well hidden on x86, but each vendor has a config option. For AMD systems we have had our own option to disable vendor specific code, see CPU_SUP_AMD for example. Disabling this will remove all AMD specific code in the kernel. Of course this is enabled per default.

With ARCH_THUNDER I intended to do the same (you could name this also SOC_SUP_CAVIUM or so but I kept the current naming scheme). In patch 4/4 I have added the option to defconfig. This enables this per default and nobody has to deal with any option manually, just running make defconfig is fine.

Also, at least to disable building the dtb file for foo, you will need ARCH_FOO too. How else would you deal with dtb files then?

arch/arm64/boot/dts/thunder-88xx.dts

Quoted 5 lines · On Friday 05 September 2014 09:46:43 Robert Richter wrote:

arch/arm64/boot/dts/thunder-88xx.dts new file +0 -0

new file mode 100644
index 000000000000..4cc1d059d2d9

Most people end up having a soc.dtsi file and a board.dsi file including the former and adding the machine-specific properties.

From: Radha Mohan Chintakuntla <[email protected]>

This patch adds documentation for the devicetree bindings used by the DT files of Cavium Thunder SoC platforms.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 Documentation/devicetree/bindings/arm/cavium-thunder.txt | 10 ++++++++++
 Documentation/devicetree/bindings/arm/cpus.txt           |  1 +
 2 files changed, 11 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/cavium-thunder.txt

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +0 -0

new file mode 100644
index 000000000000..6f63a5866902

Quoted 23 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

This patch adds documentation for the devicetree bindings used by the DT files of Cavium Thunder SoC platforms.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 Documentation/devicetree/bindings/arm/cavium-thunder.txt | 10 ++++++++++
 Documentation/devicetree/bindings/arm/cpus.txt           |  1 +
 2 files changed, 11 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/cavium-thunder.txt

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +4 -0

new file mode 100644
index 000000000000..6f63a5866902
@@ -0,0 +1,10 @@
+Cavium Thunder platform device tree bindings
+--------------------------------------------
+
+Boards with Cavium's Thunder SoC shall have following properties.

Documentation/devicetree/bindings/arm/cavium-thunder.txt line 10

Quoted 6 lines

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +6 -0

+
+Root Node
+---------
+Required root node properties:
+
+  - compatible = "cavium,thunder-88xx";

Is this file necessary? We don't have one for xgene afaict.

Quoted 24 lines · Will Deacon

Quoted 23 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

This patch adds documentation for the devicetree bindings used by the DT files of Cavium Thunder SoC platforms.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 Documentation/devicetree/bindings/arm/cavium-thunder.txt | 10 ++++++++++
 Documentation/devicetree/bindings/arm/cpus.txt           |  1 +
 2 files changed, 11 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/cavium-thunder.txt

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +4 -0

new file mode 100644
index 000000000000..6f63a5866902
@@ -0,0 +1,10 @@
+Cavium Thunder platform device tree bindings
+--------------------------------------------
+
+Boards with Cavium's Thunder SoC shall have following properties.

Quoted 8 lines

Documentation/devicetree/bindings/arm/cavium-thunder.txt line 10

Quoted 6 lines

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +6 -0

+
+Root Node
+---------
+Required root node properties:
+
+  - compatible = "cavium,thunder-88xx";

Is this file necessary? We don't have one for xgene afaict.

Hmm, looking at Documentation/devicetree/bindings/arm/calxeda.txt there is something similar.

I am fine with both, keeping or removing it.

Would leave this to the DT maintainers.

Quoted 37 lines · Robert Richter

Quoted 24 lines · Will Deacon

Quoted 23 lines · Robert Richter

From: Radha Mohan Chintakuntla <[email protected]>

This patch adds documentation for the devicetree bindings used by the DT files of Cavium Thunder SoC platforms.

Signed-off-by Radha Mohan Chintakuntla <[email protected]>

Signed-off-by Robert Richter <[email protected]>

---
 Documentation/devicetree/bindings/arm/cavium-thunder.txt | 10 ++++++++++
 Documentation/devicetree/bindings/arm/cpus.txt           |  1 +
 2 files changed, 11 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/cavium-thunder.txt

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +4 -0

new file mode 100644
index 000000000000..6f63a5866902
@@ -0,0 +1,10 @@
+Cavium Thunder platform device tree bindings
+--------------------------------------------
+
+Boards with Cavium's Thunder SoC shall have following properties.

Quoted 8 lines

Documentation/devicetree/bindings/arm/cavium-thunder.txt line 10

Quoted 6 lines

Documentation/devicetree/bindings/arm/cavium-thunder.txt new file +6 -0

+
+Root Node
+---------
+Required root node properties:
+
+  - compatible = "cavium,thunder-88xx";

Is this file necessary? We don't have one for xgene afaict.

Hmm, looking at Documentation/devicetree/bindings/arm/calxeda.txt there is something similar.

Quoted 3 lines

I am fine with both, keeping or removing it.

Would leave this to the DT maintainers.

Having a required root node compatible is fine, though I would only expect it to be necessary for errata workarounds if we have no other way of detecting things.

That said, compatible strings shouldn't have wildcards, so "cavium,thunder-88xx" should be replaced with something more specific. Later boards can claim compatibility with that (but should have their own strings too).

From: Robert Richter <[email protected]>

This initial patches enable Cavium Thunder SoC Family. The patches add Kconfig and devicetree support and then add Thunder to the defconfig.

The Thunder system needs more enablement patches for subsystems and devices, this includes network, ahci, gicv3/gicv3-its, pci, smmu, kvm. We will send separate patch sets for these. All of them base on this initial patches.

I sent a separate patch set independently that introduces support for dts vendor subdirs. After both patch sets have been applied I will send a patch in addition that moves the dts file into a subdirectory.

Patches are available here: