SMP support for Armada XP

Hello,

Waiting for Russell decision for patch 4, I made minors changes in the series. They are purely cosmetic for patch 4, and for the first patch, I removed the redundant code as Will have asked last week.

The purpose of this patch set is to add the SMP support for the Armada XP SoCs. Beside the SMP support itself brought by the last 3 patches, this patch set also adds the support for the coherency fabric unit and the power management service unit.

The coherency fabric is responsible for ensuring hardware coherency between all CPUs and between CPUs and I/O masters. This unit is also available for Armada 370 and will be used in an incoming patch set for hardware I/O cache coherency.

The power management service unit is responsible for powering down and waking up CPUs and other SOC units.

The original code is from Yehuda Yitschak, it was reworked by myself and reviewed by Yehuda.

This patch set is based on 3.7-rc6 and depends one the framework clock support which should be being included in the marvell branch maintained by Jason Cooper. The git branch called ArmadaXP-SMP-for-3.8-V4 is also available at
https://github.com/MISL-EBU-System-SW/mainline-public.git.

Changelog:

V3->V4:
- Rebased on to v3.7-rc6
- the function to setup the coherency of the CPU is now share between SMP and coherency part and no more duplicate. Most of the code of headsmp.S have been moved in coherency_ll.S and the redundant code have been deleted from coherency.c.
- Fixed cosmetic inconsistency in code added to proc-v7, only macros are used and no more immediate values.
- Added reviewed-by and acked-by for the patch 4.

V2 -> V3:
- Rebased on to v3.7-rc5
- removed the unnecessary used of __iomem flag for a pointer to a function. the second parameter of armada_xp_boot_cpu is now just a void *.
- removed the dsb() call in armada_xp_boot_cpu(), as the following writel calls dsb if needed.
- Change the PJ4B mask for proc_info in arch/arm/mm/proc-v7.S to be less restrictive, the last 4 bits contain revision information
- Merge the multiple 'or' instruction in a single instruction in __v7_pj4b_setup

V1 -> V2:
- Rebased on to v3.7-rc3
- Fixed typos found by Alexandre Belloni
- Added clk_prepare_enable() before getting rate clk in set_secondary_cpus_clock()
- Add explanation in the binding documentation about the per-CPU interrupt registers: the address of the virtual register must be used.
- Removed the armada_xp prefix in the coherency.c file to be more compliant with the name convention of the other files.
- Coherency_init is now called from armada_370_xp_dt_init() and is no more an early_init() call. As the device tree is not available from an early_init(), it was useless to call coherency_init() so early. The need to be able to call some function very early during the boot were already resolved by using the hard code address of the register.

Yehuda Yitschak (5):
arm: mvebu: Added support for coherency fabric in mach-mvebu
arm: mvebu: Added initial support for power managmement service unit
arm: mvebu: Added IPI support via doorbells
arm: mm: Added support for PJ4B cpu and init routines
arm: mvebu: Added SMP support for Armada XP

 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   12 +-
 .../devicetree/bindings/arm/armada-370-xp-pmsu.txt |   20 ++++
 .../devicetree/bindings/arm/coherency-fabric.txt   |   16 +++
 arch/arm/boot/dts/armada-370-xp.dtsi               |    5 +
 arch/arm/boot/dts/armada-xp.dtsi                   |   12 +-
 arch/arm/configs/mvebu_defconfig                   |    3 +
 arch/arm/mach-mvebu/Kconfig                        |    3 +-
 arch/arm/mach-mvebu/Makefile                       |    4 +-
 arch/arm/mach-mvebu/armada-370-xp.c                |    3 +
 arch/arm/mach-mvebu/armada-370-xp.h                |   10 ++
 arch/arm/mach-mvebu/coherency.c                    |   80 +++++++++++++
 arch/arm/mach-mvebu/coherency.h                    |   24 ++++
 arch/arm/mach-mvebu/coherency_ll.S                 |   48 ++++++++
 arch/arm/mach-mvebu/common.h                       |    6 +
 arch/arm/mach-mvebu/headsmp.S                      |   50 ++++++++
 arch/arm/mach-mvebu/hotplug.c                      |   30 +++++
 arch/arm/mach-mvebu/irq-armada-370-xp.c            |   92 ++++++++++++++-
 arch/arm/mach-mvebu/platsmp.c                      |  124 ++++++++++++++++++++
 arch/arm/mach-mvebu/pmsu.c                         |   75 ++++++++++++
 arch/arm/mach-mvebu/pmsu.h                         |   16 +++
 arch/arm/mm/Kconfig                                |    4 +
 arch/arm/mm/proc-v7.S                              |   66 +++++++++++
 22 files changed, 691 insertions(+), 12 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/arm/armada-370-xp-pmsu.txt
 create mode 100644 Documentation/devicetree/bindings/arm/coherency-fabric.txt
 create mode 100644 arch/arm/mach-mvebu/coherency.c
 create mode 100644 arch/arm/mach-mvebu/coherency.h
 create mode 100644 arch/arm/mach-mvebu/coherency_ll.S
 create mode 100644 arch/arm/mach-mvebu/headsmp.S
 create mode 100644 arch/arm/mach-mvebu/hotplug.c
 create mode 100644 arch/arm/mach-mvebu/platsmp.c
 create mode 100644 arch/arm/mach-mvebu/pmsu.c
 create mode 100644 arch/arm/mach-mvebu/pmsu.h

1.7.9.5

From: Yehuda Yitschak <[email protected]>

---
 .../devicetree/bindings/arm/coherency-fabric.txt   |   16 ++++
 arch/arm/boot/dts/armada-370-xp.dtsi               |    5 ++
 arch/arm/mach-mvebu/Makefile                       |    2 +-
 arch/arm/mach-mvebu/coherency.c                    |   80 ++++++++++++++++++++
 arch/arm/mach-mvebu/coherency.h                    |   24 ++++++
 arch/arm/mach-mvebu/coherency_ll.S                 |   48 ++++++++++++
 arch/arm/mach-mvebu/common.h                       |    2 +
 7 files changed, 176 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/arm/coherency-fabric.txt
 create mode 100644 arch/arm/mach-mvebu/coherency.c
 create mode 100644 arch/arm/mach-mvebu/coherency.h
 create mode 100644 arch/arm/mach-mvebu/coherency_ll.S

new file mode 100644
index 0000000..2bfbf67
@@ -0,0 +1,16 @@
+Coherency fabric
+----------------
+Available on Marvell SOCs: Armada 370 and Armada XP
+
+Required properties:
+
+- compatible: "marvell,coherency-fabric"
+- reg: Should contain,coherency fabric registers location and length.
+
+Example:
+
+coherency-fabric at d0020200 {
+	compatible = "marvell,coherency-fabric";
+	reg = <0xd0020200 0xb0>;
+};
+

index 94b4b9e..b0d075b 100644
@@ -36,6 +36,11 @@
 	      interrupt-controller;
 	};
 
+	coherency-fabric at d0020200 {
+		compatible = "marvell,coherency-fabric";
+		reg = <0xd0020200 0xb0>;
+	};
+
 	soc {
 		#address-cells = <1>;
 		#size-cells = <1>;

index 57f996b..5ce4b42 100644
@@ -2,4 +2,4 @@ ccflags-$(CONFIG_ARCH_MULTIPLATFORM) := -I$(srctree)/$(src)/include \
 	-I$(srctree)/arch/arm/plat-orion/include
 
 obj-y += system-controller.o
-obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o irq-armada-370-xp.o addr-map.o
+obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o irq-armada-370-xp.o addr-map.o coherency.o coherency_ll.o

new file mode 100644
index 0000000..2f42813
@@ -0,0 +1,24 @@
+/*
+ * arch/arm/mach-mvebu/include/mach/coherency.h
+ *
+ *
+ * Coherency fabric (Aurora) support for Armada 370 and XP platforms.
+ *
+ * Copyright (C) 2012 Marvell
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2.  This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#ifndef __MACH_370_XP_COHERENCY_H
+#define __MACH_370_XP_COHERENCY_H
+
+#ifdef CONFIG_SMP
+int coherency_get_cpu_count(void);
+#endif
+
+int set_cpu_coherent(int cpu_id, int smp_group_id);
+int coherency_init(void);
+
+#endif	/* __MACH_370_XP_COHERENCY_H */

index 281fab3..ea08919 100644
@@ -21,4 +21,6 @@ void mvebu_clocks_init(void);
 void armada_370_xp_init_irq(void);
 void armada_370_xp_handle_irq(struct pt_regs *regs);
 
+
+int armada_370_xp_coherency_init(void);
 #endif

1.7.9.5

Hi Gregory,

Thanks for turning this around so quickly! I still have a few comments on your assembly, but you've got the right idea:

[...]

I was a little too quick I think: I have just passed
a few hours to fix a bug which appeared when I have added the HWIOCC patches. But I see that you have already found it!

I guess

Yes I have change it already, and I don't change r0 any more as it is the base register: I was lucky that ARMADA_XP_CFB_CFG_REG_OFFSET was equal to 0!

True! And it was a bug in fact.
When I read the datasheet on PCS I stuck on the expression register 4, which is r3 and not r4! :(
Now it's fixed and of course now I use r2

yes I have already fixed it too.

Well, why not.

Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

From: Yehuda Yitschak <[email protected]>

---
 .../devicetree/bindings/arm/armada-370-xp-pmsu.txt |   20 ++++++
 arch/arm/boot/dts/armada-xp.dtsi                   |    6 ++
 arch/arm/mach-mvebu/Makefile                       |    2 +-
 arch/arm/mach-mvebu/common.h                       |    1 +
 arch/arm/mach-mvebu/pmsu.c                         |   75 ++++++++++++++++++++
 arch/arm/mach-mvebu/pmsu.h                         |   16 +++++
 6 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/devicetree/bindings/arm/armada-370-xp-pmsu.txt
 create mode 100644 arch/arm/mach-mvebu/pmsu.c
 create mode 100644 arch/arm/mach-mvebu/pmsu.h

new file mode 100644
index 0000000..926b4d6
@@ -0,0 +1,20 @@
+Power Management Service Unit(PMSU)
+-----------------------------------
+Available on Marvell SOCs: Armada 370 and Armada XP
+
+Required properties:
+
+- compatible: "marvell,armada-370-xp-pmsu"
+
+- reg: Should contain PMSU registers location and length. First pair
+  for the per-CPU SW Reset Control registers, second pair for the
+  Power Management Service Unit.
+
+Example:
+
+armada-370-xp-pmsu at d0022000 {
+	compatible = "marvell,armada-370-xp-pmsu";
+	reg = <0xd0022100 0x430>,
+	      <0xd0020800 0x20>;
+};
+

index a564b52..f521ed8 100644
@@ -27,6 +27,12 @@
 		    <0xd0021870 0x58>;
 	};
 
+	armada-370-xp-pmsu at d0022000 {
+		compatible = "marvell,armada-370-xp-pmsu";
+		reg = <0xd0022100 0x430>,
+		      <0xd0020800 0x20>;
+	};
+
 	cpus {
 	    #address-cells = <1>;
 	    #size-cells = <0>;

index 5ce4b42..2e3ec11 100644
@@ -2,4 +2,4 @@ ccflags-$(CONFIG_ARCH_MULTIPLATFORM) := -I$(srctree)/$(src)/include \
 	-I$(srctree)/arch/arm/plat-orion/include
 
 obj-y += system-controller.o
-obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o irq-armada-370-xp.o addr-map.o coherency.o coherency_ll.o
+obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o irq-armada-370-xp.o addr-map.o coherency.o coherency_ll.o pmsu.o

index ea08919..74ee0b2 100644
@@ -23,4 +23,5 @@ void armada_370_xp_handle_irq(struct pt_regs *regs);
 
 
 int armada_370_xp_coherency_init(void);
+int armada_370_xp_pmsu_init(void);
 #endif

new file mode 100644
index 0000000..07a737c
@@ -0,0 +1,16 @@
+/*
+ * Power Management Service Unit (PMSU) support for Armada 370/XP platforms.
+ *
+ * Copyright (C) 2012 Marvell
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2.  This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#ifndef __MACH_MVEBU_PMSU_H
+#define __MACH_MVEBU_PMSU_H
+
+int armada_xp_boot_cpu(unsigned int cpu_id, void *phys_addr);
+
+#endif	/* __MACH_370_XP_PMSU_H */

1.7.9.5

From: Yehuda Yitschak <[email protected]>

---
 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   12 ++-
 arch/arm/boot/dts/armada-xp.dtsi                   |    2 +-
 arch/arm/mach-mvebu/armada-370-xp.h                |   10 +++
 arch/arm/mach-mvebu/irq-armada-370-xp.c            |   92 ++++++++++++++++++--
 4 files changed, 106 insertions(+), 10 deletions(-)

index 70c0dc5..61df564 100644
@@ -6,9 +6,15 @@ Required properties:
 - interrupt-controller: Identifies the node as an interrupt controller.
 - #interrupt-cells: The number of cells to define the interrupts. Should be 1.
   The cell is the IRQ number
+
 - reg: Should contain PMIC registers location and length. First pair
   for the main interrupt registers, second pair for the per-CPU
-  interrupt registers
+  interrupt registers. For this last pair, to be compliant with SMP
+  support, the "virtual" must be use (For the record, these registers
+  automatically map to the interrupt controller registers of the
+  current CPU)
+
+
 
 Example:
 

@@ -18,6 +24,6 @@ Example:
               #address-cells = <1>;
               #size-cells = <1>;
               interrupt-controller;
-              reg = <0xd0020000 0x1000>,
-                    <0xd0021000 0x1000>;
+              reg = <0xd0020a00 0x1d0>,
+                    <0xd0021070 0x58>;
         };

index f521ed8..531619f 100644
@@ -24,7 +24,7 @@
 
 	mpic: interrupt-controller at d0020000 {
 	      reg = <0xd0020a00 0x1d0>,
-		    <0xd0021870 0x58>;
+		    <0xd0021070 0x58>;
 	};
 
 	armada-370-xp-pmsu at d0022000 {

index aac9beb..dce590d 100644
@@ -19,4 +19,14 @@
 #define ARMADA_370_XP_REGS_VIRT_BASE	IOMEM(0xfeb00000)
 #define ARMADA_370_XP_REGS_SIZE		SZ_1M
 
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_SMP
+#include <linux/cpumask.h>
+
+void armada_mpic_send_doorbell(const struct cpumask *mask, unsigned int irq);
+void armada_xp_mpic_smp_cpu_init(void);
+#endif
+#endif
+
 #endif /* __MACH_ARMADA_370_XP_H */

1.7.9.5

From: Yehuda Yitschak <[email protected]>

---
 arch/arm/mach-mvebu/Kconfig |    2 +-
 arch/arm/mm/Kconfig         |    4 +++
 arch/arm/mm/proc-v7.S       |   66 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 1 deletion(-)

index 17d246b..9bfaa0c 100644
@@ -22,7 +22,7 @@ config MVEBU_CLK_CPU
 config MACH_ARMADA_370_XP
 	bool
 	select ARMADA_370_XP_TIMER
-	select CPU_V7
+	select CPU_PJ4B
 
 config MACH_ARMADA_370
 	bool "Marvell Armada 370 boards"

index 94186b6..3fd629d 100644
@@ -352,6 +352,10 @@ config CPU_PJ4
 	select ARM_THUMBEE
 	select CPU_V7
 
+config CPU_PJ4B
+	bool
+	select CPU_V7
+
 # ARMv6
 config CPU_V6
 	bool "Support ARM V6 processor" if ARCH_INTEGRATOR || MACH_REALVIEW_EB || MACH_REALVIEW_PBX

1.7.9.5

From: Yehuda Yitschak <[email protected]>

1. added smp init functions in platsmp.c
2. added secondary cpu entry point in headsmp.S
3. added hotplog initial support in hotplug.c
4. added SMP support for PJ4B cpu

---
 arch/arm/boot/dts/armada-xp.dtsi    |    4 ++
 arch/arm/configs/mvebu_defconfig    |    3 +
 arch/arm/mach-mvebu/Kconfig         |    1 +
 arch/arm/mach-mvebu/Makefile        |    2 +
 arch/arm/mach-mvebu/armada-370-xp.c |    3 +
 arch/arm/mach-mvebu/common.h        |    3 +
 arch/arm/mach-mvebu/headsmp.S       |   50 ++++++++++++++
 arch/arm/mach-mvebu/hotplug.c       |   30 +++++++++
 arch/arm/mach-mvebu/platsmp.c       |  124 +++++++++++++++++++++++++++++++++++
 9 files changed, 220 insertions(+)
 create mode 100644 arch/arm/mach-mvebu/headsmp.S
 create mode 100644 arch/arm/mach-mvebu/hotplug.c
 create mode 100644 arch/arm/mach-mvebu/platsmp.c

index 531619f..7f968dc 100644
@@ -38,24 +38,28 @@
 	    #size-cells = <0>;
 
 	    cpu at 0 {
+		device_type = "cpu";
 	       	compatible = "marvell,sheeva-v7";
 		reg = <0>;
 		clocks = <&cpuclk 0>;
 	    };
 
 	    cpu at 1 {
+		device_type = "cpu";
 		compatible = "marvell,sheeva-v7";
 		reg = <1>;
 		clocks = <&cpuclk 1>;
 	    };
 
 	    cpu at 2 {
+		device_type = "cpu";
 		compatible = "marvell,sheeva-v7";
 		reg = <2>;
 		clocks = <&cpuclk 2>;
 	    };
 
 	    cpu at 3 {
+		device_type = "cpu";
 		compatible = "marvell,sheeva-v7";
 		reg = <3>;
 		clocks = <&cpuclk 3>;

index 3458752..da598d3 100644
@@ -12,6 +12,9 @@ CONFIG_ARCH_MVEBU=y
 CONFIG_MACH_ARMADA_370=y
 CONFIG_MACH_ARMADA_XP=y
 # CONFIG_CACHE_L2X0 is not set
+# CONFIG_SWP_EMULATE is not set
+CONFIG_SMP=y
+# CONFIG_LOCAL_TIMERS is not set
 CONFIG_AEABI=y
 CONFIG_HIGHMEM=y
 # CONFIG_COMPACTION is not set

index 9bfaa0c..d70afe3 100644
@@ -22,6 +22,7 @@ config MVEBU_CLK_CPU
 config MACH_ARMADA_370_XP
 	bool
 	select ARMADA_370_XP_TIMER
+	select HAVE_SMP
 	select CPU_PJ4B
 
 config MACH_ARMADA_370

index 2e3ec11..5dcb369 100644
@@ -3,3 +3,5 @@ ccflags-$(CONFIG_ARCH_MULTIPLATFORM) := -I$(srctree)/$(src)/include \
 
 obj-y += system-controller.o
 obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o irq-armada-370-xp.o addr-map.o coherency.o coherency_ll.o pmsu.o
+obj-$(CONFIG_SMP)                += platsmp.o headsmp.o
+obj-$(CONFIG_HOTPLUG_CPU)        += hotplug.o

index 2af6ce5..66befa1 100644
@@ -22,6 +22,7 @@
 #include <asm/mach/time.h>
 #include "armada-370-xp.h"
 #include "common.h"
+#include "coherency.h"
 
 static struct map_desc armada_370_xp_io_desc[] __initdata = {
 	{

@@ -50,6 +51,7 @@ struct sys_timer armada_370_xp_timer = {
 static void __init armada_370_xp_dt_init(void)
 {
 	of_platform_populate(NULL, of_default_bus_match_table, NULL, NULL);
+	coherency_init();
 }
 
 static const char * const armada_370_xp_dt_board_dt_compat[] = {

@@ -59,6 +61,7 @@ static const char * const armada_370_xp_dt_board_dt_compat[] = {
 };
 
 DT_MACHINE_START(ARMADA_XP_DT, "Marvell Aramada 370/XP (Device Tree)")
+	.smp		= smp_ops(armada_xp_smp_ops),
 	.init_machine	= armada_370_xp_dt_init,
 	.map_io		= armada_370_xp_map_io,
 	.init_irq	= armada_370_xp_init_irq,

index 74ee0b2..86484bb 100644
@@ -21,7 +21,10 @@ void mvebu_clocks_init(void);
 void armada_370_xp_init_irq(void);
 void armada_370_xp_handle_irq(struct pt_regs *regs);
 
+void armada_xp_cpu_die(unsigned int cpu);
 
 int armada_370_xp_coherency_init(void);
 int armada_370_xp_pmsu_init(void);
+void armada_xp_secondary_startup(void);
+extern struct smp_operations armada_xp_smp_ops;
 #endif

new file mode 100644
index 0000000..b228b6a
@@ -0,0 +1,30 @@
+/*
+ * Symmetric Multi Processing (SMP) support for Armada XP
+ *
+ * Copyright (C) 2012 Marvell
+ *
+ * Lior Amsalem <[email protected]>
+ * Gregory CLEMENT <[email protected]>
+ * Thomas Petazzoni <[email protected]>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2.  This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/smp.h>
+#include <asm/proc-fns.h>
+
+/*
+ * platform-specific code to shutdown a CPU
+ *
+ * Called with IRQs disabled
+ */
+void __ref armada_xp_cpu_die(unsigned int cpu)
+{
+	cpu_do_idle();
+
+	/* We should never return from idle */
+	panic("mvebu: cpu %d unexpectedly exit from shutdown\n", cpu);
+}

1.7.9.5

I will try it.

I prefer to keep them separate, as we can use coherency without SMP. That's why I create the coherency_ll.S file instead of putting everything in this file.

Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Hmm, can you elaborate a bit more on coherency without SMP please? Do these controls also affect IO coherency?

Will

I meant Hardware IO coherency which is the purpose on the other patch set:
"Add hardware I/O coherency support for Armada 370/XP"

Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Dear Will Deacon,

The coherency unit does coherency between CPUs, but also between the CPU and the devices for DMA. So for example the MV78230 is a single-core variant of the Armada XP, so having CONFIG_SMP for it is useless, but you still want the CPU to join the coherency fabric in order to provide coherency with I/O.

Thanks to the coherency unit, we have an almost DMA coherent architecture (see the HW I/O coherency patches from Gr?gory for details).

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Ok, gotcha -- I didn't realise it was all controlled by the same register on the CPU side.

Thanks for the explanation,

Will