Skip to contentSkip to conversation

MCPM: clarify boot CPU situation wrt CCI

PATCH New Nicolas Pitre 4 participants 11 messages Jun 24, 2014 – Jun 26, 2014

Subsystems (4)
lore
Share & tools
lore
#1 lore raw

PATCH 1/3 ARM: MCPM: provide infrastructure to allow for MCPM loopback

The kernel already has the responsibility to handle resources such as the
CCI when hotplugging CPUs, during the booting of secondary CPUs, and when
resuming from suspend/idle. It would be more coherent and less confusing
if the CCI for the boot CPU (or cluster) was also initialized by the kernel rather than expecting the firmware/bootloader to do it and only in that case. After all, the kernel
has all the necessary code already and the bootloader shouldn't have to
care at all.

The CCI may be turned on only when the cache is off. Leveraging the CPU suspend code to loop back through the low-level MCPM entry point is all that is needed to properly turn on the CCI from the kernel by using the same code as for secondary boot.

Let's provide a generic MCPM loopback function that can be invoked by backend initialization code to set things (CCI or similar) on the boot CPU just as it is done for the other CPUs.

---
 arch/arm/common/mcpm_entry.c | 52 ++++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/mcpm.h  | 16 ++++++++++++++
 2 files changed, 68 insertions(+)
arch/arm/common/mcpm_entry.c +52 -0
index f91136ab44..5e7284a3f8 100644
@@ -12,11 +12,13 @@
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/irqflags.h>
+#include <linux/cpu_pm.h>
 
 #include <asm/mcpm.h>
 #include <asm/cacheflush.h>
 #include <asm/idmap.h>
 #include <asm/cputype.h>
+#include <asm/suspend.h>
 
 extern unsigned long mcpm_entry_vectors[MAX_NR_CLUSTERS][MAX_CPUS_PER_CLUSTER];
 
@@ -146,6 +148,56 @@ int mcpm_cpu_powered_up(void)
 	return 0;
 }
 
+#ifdef CONFIG_ARM_CPU_SUSPEND
+
+static int __init nocache_trampoline(unsigned long _arg)
+{
+	void (*cache_disable)(void) = (void *)_arg;
+	unsigned int mpidr = read_cpuid_mpidr();
+	unsigned int cpu = MPIDR_AFFINITY_LEVEL(mpidr, 0);
+	unsigned int cluster = MPIDR_AFFINITY_LEVEL(mpidr, 1);
+	phys_reset_t phys_reset;
+
+	mcpm_set_entry_vector(cpu, cluster, cpu_resume);
+	setup_mm_for_reboot();
+
+	__mcpm_cpu_going_down(cpu, cluster);
+	BUG_ON(!__mcpm_outbound_enter_critical(cpu, cluster));
+	cache_disable();
+	__mcpm_outbound_leave_critical(cluster, CLUSTER_DOWN);
+	__mcpm_cpu_down(cpu, cluster);
+
+	phys_reset = (phys_reset_t)(unsigned long)virt_to_phys(cpu_reset);
+	phys_reset(virt_to_phys(mcpm_entry_point));
+	BUG();
+}
+	
+int __init mcpm_loopback(void (*cache_disable)(void))
+{
+	int ret;
+
+	/*
+	 * We're going to soft-restart the current CPU through the
+	 * low-level MCPM code by leveraging the suspend/resume
+	 * infrastructure. Let's play it safe by using cpu_pm_enter()
+	 * in case the CPU init code path resets the VFP or similar.
+	 */
+	local_irq_disable();
+	local_fiq_disable();
+	ret = cpu_pm_enter();
+	if (!ret) {
+		ret = cpu_suspend((unsigned long)cache_disable, nocache_trampoline);
+		cpu_pm_exit();
+	}
+	local_fiq_enable();
+	local_irq_enable();
+	if (ret)
+		pr_err("%s returned %d\n", __func__, ret);
+	return ret;
+}
+
+#endif
+
 struct sync_struct mcpm_sync;
 
 /*
arch/arm/include/asm/mcpm.h +16 -0
index 94060adba1..ff73affd45 100644
@@ -217,6 +217,22 @@ int __mcpm_cluster_state(unsigned int cluster);
 int __init mcpm_sync_init(
 	void (*power_up_setup)(unsigned int affinity_level));
 
+/**
+ * mcpm_loopback - make a run through the MCPM low-level code
+ *
+ * @cache_disable: pointer to function performing cache disabling
+ *
+ * This exercises the MCPM machinery by soft resetting the CPU and branching
+ * to the MCPM low-level entry code before returning to the caller.  
+ * The @cache_disable function must do the necessary cache disabling to
+ * let the regular kernel init code turn it back on as if the CPU was
+ * hotplugged in. The MCPM state machine is set as if the cluster was
+ * initialized meaning the power_up_setup callback passed to mcpm_sync_init()
+ * will be invoked for all affinity levels. This may be useful to initialize
+ * some resources such as enabling the CCI that requires the cache to be off, or simply for testing purposes.
+ */
+int __init mcpm_loopback(void (*cache_disable)(void));
+
 void __init mcpm_smp_set_ops(void);
 
 #else
Signature · 1 line
1.8.4.108.g55ea5f6
#2 lore raw

PATCH 1/3 ARM: MCPM: provide infrastructure to allow for MCPM loopback

Nicolas,

Quoted 4 lines · Nicolas Pitre

The kernel already has the responsibility to handle resources such as the
CCI when hotplugging CPUs, during the booting of secondary CPUs, and when
resuming from suspend/idle. It would be more coherent and less confusing
if the CCI for the boot CPU (or cluster) was also initialized by the kernel rather than expecting the

nit: wrap long line?

Quoted 13 lines

firmware/bootloader to do it and only in that case. After all, the kernel has all the necessary code already and the bootloader shouldn't have to care at all.

The CCI may be turned on only when the cache is off. Leveraging the CPU suspend code to loop back through the low-level MCPM entry point is all that is needed to properly turn on the CCI from the kernel by using the same code as for secondary boot.

Let's provide a generic MCPM loopback function that can be invoked by backend initialization code to set things (CCI or similar) on the boot CPU just as it is done for the other CPUs.

Quoted 5 lines
---
 arch/arm/common/mcpm_entry.c | 52 ++++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/mcpm.h  | 16 ++++++++++++++
 2 files changed, 68 insertions(+)

Thank you very much for posting! With your series I'm able to boot all 8 cores on exynos5420-peach-pit and exynos5800-peach-pi sitting on my desk.

I will note that git yelled about whitespace damage on theis patch:

# pwclient git-am 4406301
Applying patch #4406301 using 'git am'
Description: [1/3] ARM: MCPM: provide infrastructure to allow for MCPM loopback
Applying: ARM: MCPM: provide infrastructure to allow for MCPM loopback
/b/tip/src/third_party/kernel/3.8/.git/rebase-apply/patch:51: trailing
whitespace.

/b/tip/src/third_party/kernel/3.8/.git/rebase-apply/patch:95: trailing whitespace.
* to the MCPM low-level entry code before returning to the caller.
warning: 2 lines add whitespace errors.

#3 lore raw

PATCH 2/3 ARM: TC2: test the MCPM loopback during boot

In reply to Nicolas Pitre

This is not strictly needed on TC2 but still a good thing to exercise that code.

---
 arch/arm/mach-vexpress/tc2_pm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/arm/mach-vexpress/tc2_pm.c +17 -0
index b743a0ae02..54a9fff77c 100644
@@ -323,6 +323,21 @@ static void __naked tc2_pm_power_up_setup(unsigned int affinity_level)
 "	b	cci_enable_port_for_self ");
 }
 
+static void __init tc2_cache_off(void)
+{
+	pr_info("TC2: disabling cache during MCPM loopback test\n");
+	if (read_cpuid_part_number() == ARM_CPU_PART_CORTEX_A15) {
+		/* disable L2 prefetching on the Cortex-A15 */
+		asm volatile(
+		"mcr	p15, 1, %0, c15, c0, 3 \n\t"
+		"isb	\n\t"
+		"dsb	"
+		: : "r" (0x400) );
+	}
+	v7_exit_coherency_flush(all);
+	cci_disable_port_by_cpu(read_cpuid_mpidr());
+}
+
 static int __init tc2_pm_init(void)
 {
 	int ret, irq;
@@ -370,6 +385,8 @@ static int __init tc2_pm_init(void)
 	ret = mcpm_platform_register(&tc2_pm_power_ops);
 	if (!ret) {
 		mcpm_sync_init(tc2_pm_power_up_setup);
+		/* test if we can (re)enable the CCI on our own */
+		BUG_ON(mcpm_loopback(tc2_cache_off) != 0);
 		pr_info("TC2 power management initialized\n");
 	}
 	return ret;
Signature · 1 line
1.8.4.108.g55ea5f6
#4 lore raw

PATCH 3/3 ARM: exynos: activate the CCI on boot CPU/cluster with the MCPM loopback

In reply to Nicolas Pitre

The Chromebook firmware doesn't enable the CCI for the boot cpu, and arguably it shouldn't have to either. Let's have the kernel handle the CCI on its own for the boot CPU the same way it does it for secondary CPUs by using the MCPM loopback.

---
 arch/arm/mach-exynos/mcpm-exynos.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/arm/mach-exynos/mcpm-exynos.c +15 -0
index 0498d0b887..0c839f94ec 100644
@@ -290,6 +290,19 @@ static void __naked exynos_pm_power_up_setup(unsigned int affinity_level)
 	"b	cci_enable_port_for_self");
 }
 
+static void __init exynos_cache_off(void)
+{
+	if (read_cpuid_part_number() == ARM_CPU_PART_CORTEX_A15) {
+		/* disable L2 prefetching on the Cortex-A15 */
+		asm volatile(
+		"mcr	p15, 1, %0, c15, c0, 3\n\t"
+		"isb\n\t"
+		"dsb"
+		: : "r" (0x400));
+	}
+	exynos_v7_exit_coherency_flush(all);
+}
+
 static const struct of_device_id exynos_dt_mcpm_match[] = {
 	{ .compatible = "samsung,exynos5420" },
 	{ .compatible = "samsung,exynos5800" },
@@ -333,6 +346,8 @@ static int __init exynos_mcpm_init(void)
 	ret = mcpm_platform_register(&exynos_power_ops);
 	if (!ret)
 		ret = mcpm_sync_init(exynos_pm_power_up_setup);
+	if (!ret)
+		ret = mcpm_loopback(exynos_cache_off); /* turn on the CCI */
 	if (ret) {
 		iounmap(ns_sram_base_addr);
 		return ret;
Signature · 1 line
1.8.4.108.g55ea5f6
#5 lore raw

PATCH 3/3 ARM: exynos: activate the CCI on boot CPU/cluster with the MCPM loopback

Quoted 7 lines · Nicolas Pitre

The Chromebook firmware doesn't enable the CCI for the boot cpu, and arguably it shouldn't have to either. Let's have the kernel handle the CCI on its own for the boot CPU the same way it does it for secondary CPUs by using the MCPM loopback.

---

Tested on top of next-20140623. Verified that all 8 cores are coming up on Exynos5800 based Peach-pi board.

Quoted 37 lines
 arch/arm/mach-exynos/mcpm-exynos.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/arm/mach-exynos/mcpm-exynos.c +15 -0
index 0498d0b887..0c839f94ec 100644
@@ -290,6 +290,19 @@ static void __naked exynos_pm_power_up_setup(unsigned int affinity_level)
 	"b	cci_enable_port_for_self");
 }
 
+static void __init exynos_cache_off(void)
+{
+	if (read_cpuid_part_number() == ARM_CPU_PART_CORTEX_A15) {
+		/* disable L2 prefetching on the Cortex-A15 */
+		asm volatile(
+		"mcr	p15, 1, %0, c15, c0, 3\n\t"
+		"isb\n\t"
+		"dsb"
+		: : "r" (0x400));
+	}
+	exynos_v7_exit_coherency_flush(all);
+}
+
 static const struct of_device_id exynos_dt_mcpm_match[] = {
 	{ .compatible = "samsung,exynos5420" },
 	{ .compatible = "samsung,exynos5800" },
@@ -333,6 +346,8 @@ static int __init exynos_mcpm_init(void)
 	ret = mcpm_platform_register(&exynos_power_ops);
 	if (!ret)
 		ret = mcpm_sync_init(exynos_pm_power_up_setup);
+	if (!ret)
+		ret = mcpm_loopback(exynos_cache_off); /* turn on the CCI */
 	if (ret) {
 		iounmap(ns_sram_base_addr);
 		return ret;
Signature · 1 line
Tushar Behera
#6 lore raw

PATCH 3/3 ARM: exynos: activate the CCI on boot CPU/cluster with the MCPM loopback

In reply to Nicolas Pitre

Nicolas,

Quoted 9 lines · Nicolas Pitre

The Chromebook firmware doesn't enable the CCI for the boot cpu, and arguably it shouldn't have to either. Let's have the kernel handle the CCI on its own for the boot CPU the same way it does it for secondary CPUs by using the MCPM loopback.

---
 arch/arm/mach-exynos/mcpm-exynos.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

Thank you very much for posting! With your series I'm able to boot all 8 cores on exynos5420-peach-pit and exynos5800-peach-pi sitting on my desk.

#7 lore raw

PATCH 3/3 ARM: exynos: activate the CCI on boot CPU/cluster with the MCPM loopback

Quoted 5 lines · Doug Anderson

Thank you very much for posting! With your series I'm able to boot all 8 cores on exynos5420-peach-pit and exynos5800-peach-pi sitting on my desk.

Thanks to all. I've submitted those patches, with minor nits fixed, to RMK's system as patches 8081 to 8083.

Nicolas

#8 lore raw

cover MCPM: clarify boot CPU situation wrt CCI

In reply to Nicolas Pitre
Quoted 7 lines · Nicolas Pitre

For the background story, please see:

http://news.gmane.org/group/gmane.linux.kernel.samsung-soc/thread=32807

I sat on those patches for a while but they are the best I could think of in terms of implementation. To ease merging I suggest I collect all the ACK's and Tested-by's and submit them all at once to RMK's patch system.

I tested this series on Exynos5 Peach Pi (a.k.a. 13" ARM Chromebook 2) and verified all 8 cores are booting up.

Kevin

P.S. FYI for anyone else wanting to try this on peach-pi, I still need a couple other out of tree clock related patches in order to boot fully. Until the final versions are merged, I'm using these:

- Doug's aclk66 clock removal for earlyprintk support
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/262225.html
- Leave mau_epll enabled
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/262259.html

#9 lore raw

cover MCPM: clarify boot CPU situation wrt CCI

Kevin,

Quoted 2 lines · Kevin Hilman

- Leave mau_epll enabled
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/262259.html

Can you drop your mau_epll one and instead pick just
<https://patchwork.kernel.org/patch/4333581/>?

I noticed that Tushar has only been pushing for that patch to be applied and I'm going to guess that the "leave mau_epll enabled" isn't needed. I just tested myself and confirmed that things are happy.

#10 lore raw

cover MCPM: clarify boot CPU situation wrt CCI

Quoted 6 lines · Doug Anderson

Kevin,

Quoted 2 lines · Kevin Hilman

- Leave mau_epll enabled
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/262259.html

Quoted 6 lines

Can you drop your mau_epll one and instead pick just
<https://patchwork.kernel.org/patch/4333581/>?

I noticed that Tushar has only been pushing for that patch to be applied and I'm going to guess that the "leave mau_epll enabled" isn't needed. I just tested myself and confirmed that things are happy.

I replied on the other thread too, but just to be thorough...

Picking the above patch alone, I still get the boot hang. Adding 'clk_ignore_unused' to the command line gets things booting again, which suggest there's still a missing piece of the puzzle.

Kevin