ARM: TC2 big.LITTLE CPU idle driver

This patch is v2 of a previous posting:

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/186352.html

v2 changes:

- removed duplicated idle statistics and irq enabling
- use MPIDR masks to compute cluster and cpu ids

This patch series provides the implementation of CPU idle driver for the TC2 ARM big.LITTLE SoC. It is based and dependent on Nico's branch

git://git.linaro.org/people/nico/linux mcpm+tc2

and relative pull request

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/185411.html

First two patches in the series simply implement a method to disable the GIC CPU IF and add the respective call in the MCPM TC2 back-end. Details are explained in the commit logs.

Patch 3 implements the TC2 CPU idle driver, that paves the way for a generic ARM idle driver, MCPM based (but PSCI can be easily integrated as well) for all upcoming big.LITTLE systems.

The CPU idle driver is built upon the multiple drivers CPU idle infrastructure to define different target residencies for different clusters.

Current driver matches the DT compatible string defining a TC2 testchip core tile, but in the future will be augmented with a proper match table to match against all machines that can rely on this driver to implement CPU idle capabilities.

This CPU idle driver integrates all existing PM kernel concepts recently implemented for ARM, MCPM, CPU PM notifiers and cpu_suspend and lays the foundation for a reference implementation of a generic CPU idle driver, since the driver as it stands is completely generic, C-states definition notwithstanding.

C-state definition for the driver should be made dynamic so that the driver can become completely generic and decoupled from static C-states definition.

The driver has been tested, obviously on the TC2 testchip, with different Linux systems ranging from simple busybox to Android and Ubuntu rootfs, through millions of C-state iterations randomly triggered by the aforementioned root filesystem environments.

Comments and review very welcome.

With thanks,
Lorenzo

Lorenzo Pieralisi (2):
ARM: vexpress: tc2: disable GIC CPU IF in tc2_pm_suspend
cpuidle: big.LITTLE: vexpress-TC2 CPU idle driver

Nicolas Pitre (1):
drivers: irq-chip: irq-gic: introduce gic_cpu_if_down()

 MAINTAINERS                          |   9 ++
 arch/arm/mach-vexpress/tc2_pm.c      |   2 +
 drivers/cpuidle/Kconfig              |  10 ++
 drivers/cpuidle/Makefile             |   1 +
 drivers/cpuidle/cpuidle-big_little.c | 175 +++++++++++++++++++++++++++++++++++
 drivers/irqchip/irq-gic.c            |   6 ++
 include/linux/irqchip/arm-gic.h      |   1 +
 7 files changed, 204 insertions(+)
 create mode 100644 drivers/cpuidle/cpuidle-big_little.c

1.8.2.2

From: Nicolas Pitre <[email protected]>

When processors are about to hit low power states, the assertion of standbywfi signal, triggered by the wfi instruction, is essential to entering low power modes. If an IRQ is pending on the processor at the time wfi is issued, the wfi instruction completes and the processor restarts execution without asserting the standbywfi signal. Depending on the platform power controller HW this behaviour can be acceptable or not; if this behaviour must be prevented software should be provided with a way to disable the routing of interrupts to the core IRQ pins.

On systems where raw GIC distributor interrupts are connected to the power controller as wake-up events (hence the power controller still senses IRQs and can wake up cores upon IRQ pending), the GIC CPU interface can be disabled on power down, so that the GIC CPU IF output is gated and wfi cannot complete, thereby preventing the standbywfi issue.

This patch adds a simple function to the GIC driver that allows to disable the GIC CPU IF from power down procedures.

---
 drivers/irqchip/irq-gic.c       | 6 ++++++
 include/linux/irqchip/arm-gic.h | 1 +
 2 files changed, 7 insertions(+)

index ee7c503..d0e9480 100644
@@ -453,6 +453,12 @@ static void gic_cpu_init(struct gic_chip_data *gic)
 	writel_relaxed(1, base + GIC_CPU_CTRL);
 }
 
+void gic_cpu_if_down(void)
+{
+	void __iomem *cpu_base = gic_data_cpu_base(&gic_data[0]);
+	writel_relaxed(0, cpu_base + GIC_CPU_CTRL);
+}
+
 #ifdef CONFIG_CPU_PM
 /*
  * Saves the GIC distributor registers during suspend or idle.  Must be called

index 3e203eb..0e5d9ec 100644
@@ -66,6 +66,7 @@ extern struct irq_chip gic_arch_extn;
 void gic_init_bases(unsigned int, int, void __iomem *, void __iomem *,
 		    u32 offset, struct device_node *);
 void gic_cascade_irq(unsigned int gic_nr, unsigned int irq);
+void gic_cpu_if_down(void);
 
 static inline void gic_init(unsigned int nr, int start,
 			    void __iomem *dist , void __iomem *cpu)

1.8.2.2

To prevent cores from exiting wfi when they are about to be shut down the GIC CPU IF must be disabled so that the GIC CPU IF IRQ output line is not asserted to the cores. wfi completion must be prevented since, in absence of coordinating HW logic, if the power controller receives a standbywfi signal but in the meantime the processor restarts executing owing to a pending IRQ, the core might be reset when running in a non-quiescent state (eg with pending load/store transactions)

Raw GIC distributor IRQ signals are routed to the power controller, that is capable of taking core out of reset on pending IRQs even if their GIC CPU IF is disabled, thus keeping the normal wfi behaviour.

GIC CPU IF is restored upon CPU wake-up by the respective MCPM API consumers (ie CPU idle driver and suspend to RAM thread).

---
 arch/arm/mach-vexpress/tc2_pm.c | 2 ++
 1 file changed, 2 insertions(+)

index dfb55d4..3303ac6 100644
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/spinlock.h>
 #include <linux/errno.h>
+#include <linux/irqchip/arm-gic.h>
 
 #include <asm/mcpm.h>
 #include <asm/proc-fns.h>

@@ -211,6 +212,7 @@ static void tc2_pm_suspend(u64 residency)
 	cpu = MPIDR_AFFINITY_LEVEL(mpidr, 0);
 	cluster = MPIDR_AFFINITY_LEVEL(mpidr, 1);
 	ve_spc_set_resume_addr(cluster, cpu, virt_to_phys(mcpm_entry_point));
+	gic_cpu_if_down();
 	tc2_pm_down(residency);
 }
 

1.8.2.2

The big.LITTLE architecture is composed of two clusters of cpus. One cluster contains less powerful but more energy efficient processors and the other cluster groups the powerful but energy-intensive cpus.

The TC2 testchip implements two clusters of CPUs (A7 and A15 clusters in a big.LITTLE configuration) connected through a CCI interconnect that manages coherency of their respective L2 caches and intercluster distributed virtual memory messages (DVM).

TC2 testchip integrates a power controller that manages cores resets, wake-up IRQs and cluster low-power states. Power states are managed at cluster level, which means that voltage is removed from a cluster iff all cores in a cluster are in a wfi state. Single cores can enter a reset state which is identical to wfi in terms of power consumption but simplifies the way cluster states are entered.

This patch provides a multiple driver CPU idle implementation for TC2 which paves the way for a generic big.LITTLE idle driver for all upcoming big.LITTLE based systems on chip.

The driver relies on the MCPM infrastructure to coordinate and manage core power states; in particular MCPM allows to suspend specific cores and hides the CPUs coordination required to shut-down clusters of CPUs.

Power down sequences for the respective clusters are implemented in the MCPM TC2 backend, with all code needed to clean caches and exit coherency.

The multiple driver CPU idle infrastructure allows to define different C-states for big and little cores, determined at boot by checking the part id of the possible CPUs and initializing the respective logical masks in the big and little drivers.

Current big.little systems are composed of A7 and A15 clusters, as implemented in TC2, but in the future that may change and the driver will have evolve to retrieve what is a 'big' cpu and what is a 'little' cpu in order to build the correct topology.

---
 MAINTAINERS                          |   9 ++
 drivers/cpuidle/Kconfig              |  10 ++
 drivers/cpuidle/Makefile             |   1 +
 drivers/cpuidle/cpuidle-big_little.c | 175 +++++++++++++++++++++++++++++++++++
 4 files changed, 195 insertions(+)
 create mode 100644 drivers/cpuidle/cpuidle-big_little.c

index bf61e04..01f1b3d 100644
@@ -2263,6 +2263,15 @@ F:	drivers/cpufreq/arm_big_little.h
 F:	drivers/cpufreq/arm_big_little.c
 F:	drivers/cpufreq/arm_big_little_dt.c
 
+CPUIDLE DRIVER - ARM BIG LITTLE
+M:      Lorenzo Pieralisi <[email protected]>
+M:      Daniel Lezcano <[email protected]>
+L:      linux-pm at vger.kernel.org
+L:      linux-arm-kernel at lists.infradead.org
+T:      git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git
+S:      Maintained
+F:      drivers/cpuidle/cpuidle-big_little.c
+
 CPUIDLE DRIVERS
 M:	Rafael J. Wysocki <[email protected]>
 M:	Daniel Lezcano <[email protected]>

index 0e2cd5c..0f86587 100644
@@ -42,6 +42,16 @@ config CPU_IDLE_ZYNQ
 	help
 	  Select this to enable cpuidle on Xilinx Zynq processors.
 
+config CPU_IDLE_BIG_LITTLE
+	bool "Support for ARM big.LITTLE processors"
+	depends on ARCH_VEXPRESS_TC2_PM
+	select ARM_CPU_SUSPEND
+	select CPU_IDLE_MULTIPLE_DRIVERS
+	help
+	  Select this option to enable CPU idle driver for big.LITTLE based
+	  ARM systems. Driver manages CPUs coordination through MCPM and
+	  define different C-states for little and big cores through the
+	  multiple CPU idle drivers infrastructure.
 endif
 
 config ARCH_NEEDS_CPU_IDLE_COUPLED

index 8767a7b..3b6445c 100644
@@ -8,3 +8,4 @@ obj-$(CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED) += coupled.o
 obj-$(CONFIG_CPU_IDLE_CALXEDA) += cpuidle-calxeda.o
 obj-$(CONFIG_ARCH_KIRKWOOD) += cpuidle-kirkwood.o
 obj-$(CONFIG_CPU_IDLE_ZYNQ) += cpuidle-zynq.o
+obj-$(CONFIG_CPU_IDLE_BIG_LITTLE) += cpuidle-big_little.o

1.8.2.2

Hi Lorenzo,

I don't think this patch is still in RFC but ready for merging if nobody complains :)

Thanks
-- Daniel

 <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Yes, you are right, I should have dropped RFC by now. Just one last doubt I have and it is not really related to this patch but to the MCPM API. I think that:

void mcpm_cpu_suspend()
void mcpm_cpu_power_down()

should fail gracefully with a return value instead of hitting a BUG_ON if the platform_ops pointer is not initialized (ie likely to be a missing SCC node in DT for TC2).

It is just a heads-up, thoughts appreciated, I really really do not want to make this driver dependent on the vexpress SCC compatible string (or better on the presence of an SCC DT node in the dts file), the compatible string we are using should be correct.

Thanks,
Lorenzo