cpufreq: cppc: Add support for frequency invariance

This patch attempts to make it generic enough so other parts of the kernel can also provide their own implementation of scale_freq_tick() callback, which is called by the scheduler periodically to update the per-cpu freq_scale variable.

The implementations now need to provide 'struct scale_freq_data' for the CPUs for which they have hardware counters available, and a callback gets registered for each possible CPU in a per-cpu variable.

The arch specific (or ARM AMU) counters are updated to adapt to this and they take the highest priority if they are available, i.e. they will be used instead of CPPC based counters for example.

The special code to rebuild the sched domains, in case invariance status change for the system, is moved out of arm64 specific code and is added to arch_topology.c.

Note that this also defines SCALE_FREQ_SOURCE_CPUFREQ but doesn't use it and it is added to show that cpufreq is also acts as source of information for FIE and will be used by default if no other counters are supported for a platform.

Reviewed-by Ionela Voinescu <[email protected]>

Tested-by Ionela Voinescu <[email protected]>

Signed-off-by Viresh Kumar <[email protected]>

---
 arch/arm64/include/asm/topology.h |  10 +--
 arch/arm64/kernel/topology.c      | 105 +++++++++++-------------------
 drivers/base/arch_topology.c      |  85 ++++++++++++++++++++++--
 include/linux/arch_topology.h     |  14 +++-
 4 files changed, 134 insertions(+), 80 deletions(-)

drivers/base/arch_topology.c +41 -3

index de8587cc119e..8f62dbf93f67 100644
@@ -21,17 +21,94 @@
 #include <linux/sched.h>
 #include <linux/smp.h>
 
+static DEFINE_PER_CPU(struct scale_freq_data *, sft_data);
+static struct cpumask scale_freq_counters_mask;
+static bool scale_freq_invariant;
+
+static bool supports_scale_freq_counters(const struct cpumask *cpus)
+{
+	return cpumask_subset(cpus, &scale_freq_counters_mask);
+}
+
 bool topology_scale_freq_invariant(void)
 {
 	return cpufreq_supports_freq_invariance() ||
-	       arch_freq_counters_available(cpu_online_mask);
+	       supports_scale_freq_counters(cpu_online_mask);
 }
 
-__weak bool arch_freq_counters_available(const struct cpumask *cpus)
+static void update_scale_freq_invariant(bool status)
 {
-	return false;
+	if (scale_freq_invariant == status)
+		return;
+
+	/*
+	 * Task scheduler behavior depends on frequency invariance support,
+	 * either cpufreq or counter driven. If the support status changes as
+	 * a result of counter initialisation and use, retrigger the build of
+	 * scheduling domains to ensure the information is propagated properly.
+	 */
+	if (topology_scale_freq_invariant() == status) {
+		scale_freq_invariant = status;
+		rebuild_sched_domains_energy();
+	}
 }
+
+void topology_set_scale_freq_source(struct scale_freq_data *data,
+				    const struct cpumask *cpus)
+{
+	struct scale_freq_data *sfd;
+	int cpu;
+
+	/*
+	 * Avoid calling rebuild_sched_domains() unnecessarily if FIE is
+	 * supported by cpufreq.
+	 */
+	if (cpumask_empty(&scale_freq_counters_mask))
+		scale_freq_invariant = topology_scale_freq_invariant();
+
+	for_each_cpu(cpu, cpus) {
+		sfd = per_cpu(sft_data, cpu);
+

drivers/base/arch_topology.c +9 -0

+
+void topology_clear_scale_freq_source(enum scale_freq_source source,
+				      const struct cpumask *cpus)
+{
+	struct scale_freq_data *sfd;
+	int cpu;
+
+	for_each_cpu(cpu, cpus) {
+		sfd = per_cpu(sft_data, cpu);

Quoted 1 line · Viresh Kumar

+EXPORT_SYMBOL_GPL(topology_set_scale_freq_source);

I don't get why you need to export this in this patch. The arm64 topology code is never built as a module.

Quoted 1 line

+EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source);

Same here.

Quoted 1 line

+EXPORT_SYMBOL_GPL(freq_scale);

And here.

This one probably wants a less generic name as well if it's going to be exported.

Quoted 6 lines · Will Deacon

Quoted 1 line · Viresh Kumar

+EXPORT_SYMBOL_GPL(topology_set_scale_freq_source);

I don't get why you need to export this in this patch. The arm64 topology code is never built as a module.

Quoted 7 lines

Quoted 1 line

+EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source);

Same here.

Quoted 1 line

+EXPORT_SYMBOL_GPL(freq_scale);

And here.

After this patch, any part of the kernel can use these helpers/variables to run their own implementation of tick-freq-scale and so this patch looked to be the right place for that to me.

And the second patch in the series updates the CPPC cpufreq driver (tristate) to use these exported symbols, so we have the first user who needs the exported symbols as well.

Quoted 2 lines

This one probably wants a less generic name as well if it's going to be exported.

x86 names it arch_freq_scale, perhaps we should stick to that instead.

This patch attempts to make it generic enough so other parts of the kernel can also provide their own implementation of scale_freq_tick() callback, which is called by the scheduler periodically to update the per-cpu freq_scale variable.

The implementations now need to provide 'struct scale_freq_data' for the CPUs for which they have hardware counters available, and a callback gets registered for each possible CPU in a per-cpu variable.

The arch specific (or ARM AMU) counters are updated to adapt to this and they take the highest priority if they are available, i.e. they will be used instead of CPPC based counters for example.

The special code to rebuild the sched domains, in case invariance status change for the system, is moved out of arm64 specific code and is added to arch_topology.c.

Note that this also defines SCALE_FREQ_SOURCE_CPUFREQ but doesn't use it and it is added to show that cpufreq is also acts as source of information for FIE and will be used by default if no other counters are supported for a platform.

Reviewed-by Ionela Voinescu <[email protected]>

Tested-by Ionela Voinescu <[email protected]>

Signed-off-by Viresh Kumar <[email protected]>

---
 arch/arm64/include/asm/topology.h |  10 +--
 arch/arm64/kernel/topology.c      | 105 +++++++++++-------------------
 drivers/base/arch_topology.c      |  85 ++++++++++++++++++++++--
 include/linux/arch_topology.h     |  14 +++-
 4 files changed, 134 insertions(+), 80 deletions(-)

arch/arm64/include/asm/topology.h +1 -9

index 3b8dca4eb08d..ec2db3419c41 100644
@@ -17,17 +17,9 @@ int pcibus_to_node(struct pci_bus *bus);
 #include <linux/arch_topology.h>
 
 void update_freq_counters_refs(void);
-void topology_scale_freq_tick(void);
-
-#ifdef CONFIG_ARM64_AMU_EXTN
-/*
- * Replace task scheduler's default counter-based
- * frequency-invariance scale factor setting.
- */
-#define arch_scale_freq_tick topology_scale_freq_tick
-#endif /* CONFIG_ARM64_AMU_EXTN */
 
 /* Replace task scheduler's default frequency-invariant accounting */
+#define arch_scale_freq_tick topology_scale_freq_tick
 #define arch_set_freq_scale topology_set_freq_scale
 #define arch_scale_freq_capacity topology_get_freq_scale
 #define arch_scale_freq_invariant topology_scale_freq_invariant

arch/arm64/kernel/topology.c +39 -66

index e08a4126453a..47fca7376c93 100644
@@ -199,12 +199,47 @@ static int freq_inv_set_max_ratio(int cpu, u64 max_rate, u64 ref_rate)
 	return 0;
 }
 
-static DEFINE_STATIC_KEY_FALSE(amu_fie_key);
-#define amu_freq_invariant() static_branch_unlikely(&amu_fie_key)
+static void amu_scale_freq_tick(void)
+{
+	u64 prev_core_cnt, prev_const_cnt;
+	u64 core_cnt, const_cnt, scale;
+
+	prev_const_cnt = this_cpu_read(arch_const_cycles_prev);
+	prev_core_cnt = this_cpu_read(arch_core_cycles_prev);
+
+	update_freq_counters_refs();
+
+	const_cnt = this_cpu_read(arch_const_cycles_prev);
+	core_cnt = this_cpu_read(arch_core_cycles_prev);
+
+	if (unlikely(core_cnt <= prev_core_cnt ||
+		     const_cnt <= prev_const_cnt))
+		return;
+
+	/*
+	 *	    /\core    arch_max_freq_scale
+	 * scale =  ------- * --------------------
+	 *	    /\const   SCHED_CAPACITY_SCALE
+	 *
+	 * See validate_cpu_freq_invariance_counters() for details on
+	 * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT.
+	 */
+	scale = core_cnt - prev_core_cnt;
+	scale *= this_cpu_read(arch_max_freq_scale);
+	scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT,
+			  const_cnt - prev_const_cnt);
+
+	scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE);
+	this_cpu_write(freq_scale, (unsigned long)scale);
+}
+
+static struct scale_freq_data amu_sfd = {
+	.source = SCALE_FREQ_SOURCE_ARCH,
+	.set_freq_scale = amu_scale_freq_tick,
+};
 
 static void amu_fie_setup(const struct cpumask *cpus)
 {
-	bool invariant;
 	int cpu;
 
 	/* We are already set since the last insmod of cpufreq driver */

@@ -221,25 +256,10 @@ static void amu_fie_setup(const struct cpumask *cpus)
 
 	cpumask_or(amu_fie_cpus, amu_fie_cpus, cpus);
 
-	invariant = topology_scale_freq_invariant();
-
-	/* We aren't fully invariant yet */
-	if (!invariant && !cpumask_equal(amu_fie_cpus, cpu_present_mask))
-		return;
-
-	static_branch_enable(&amu_fie_key);
+	topology_set_scale_freq_source(&amu_sfd, amu_fie_cpus);
 
 	pr_debug("CPUs[%*pbl]: counters will be used for FIE.",
 		 cpumask_pr_args(cpus));
-
-	/*
-	 * Task scheduler behavior depends on frequency invariance support,
-	 * either cpufreq or counter driven. If the support status changes as
-	 * a result of counter initialisation and use, retrigger the build of
-	 * scheduling domains to ensure the information is propagated properly.
-	 */
-	if (!invariant)
-		rebuild_sched_domains_energy();
 }
 
 static int init_amu_fie_callback(struct notifier_block *nb, unsigned long val,

@@ -283,53 +303,6 @@ static int __init init_amu_fie(void)
 }
 core_initcall(init_amu_fie);
 
-bool arch_freq_counters_available(const struct cpumask *cpus)
-{
-	return amu_freq_invariant() &&
-	       cpumask_subset(cpus, amu_fie_cpus);
-}
-
-void topology_scale_freq_tick(void)
-{
-	u64 prev_core_cnt, prev_const_cnt;
-	u64 core_cnt, const_cnt, scale;
-	int cpu = smp_processor_id();
-
-	if (!amu_freq_invariant())
-		return;
-
-	if (!cpumask_test_cpu(cpu, amu_fie_cpus))
-		return;
-
-	prev_const_cnt = this_cpu_read(arch_const_cycles_prev);
-	prev_core_cnt = this_cpu_read(arch_core_cycles_prev);
-
-	update_freq_counters_refs();
-
-	const_cnt = this_cpu_read(arch_const_cycles_prev);
-	core_cnt = this_cpu_read(arch_core_cycles_prev);
-
-	if (unlikely(core_cnt <= prev_core_cnt ||
-		     const_cnt <= prev_const_cnt))
-		return;
-
-	/*
-	 *	    /\core    arch_max_freq_scale
-	 * scale =  ------- * --------------------
-	 *	    /\const   SCHED_CAPACITY_SCALE
-	 *
-	 * See validate_cpu_freq_invariance_counters() for details on
-	 * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT.
-	 */
-	scale = core_cnt - prev_core_cnt;
-	scale *= this_cpu_read(arch_max_freq_scale);
-	scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT,
-			  const_cnt - prev_const_cnt);
-
-	scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE);
-	this_cpu_write(freq_scale, (unsigned long)scale);
-}
-
 #ifdef CONFIG_ACPI_CPPC_LIB
 #include <acpi/cppc_acpi.h>
 

drivers/base/arch_topology.c +81 -4

index de8587cc119e..8f62dbf93f67 100644
@@ -21,17 +21,94 @@
 #include <linux/sched.h>
 #include <linux/smp.h>
 
+static DEFINE_PER_CPU(struct scale_freq_data *, sft_data);
+static struct cpumask scale_freq_counters_mask;
+static bool scale_freq_invariant;
+
+static bool supports_scale_freq_counters(const struct cpumask *cpus)
+{
+	return cpumask_subset(cpus, &scale_freq_counters_mask);
+}
+
 bool topology_scale_freq_invariant(void)
 {
 	return cpufreq_supports_freq_invariance() ||
-	       arch_freq_counters_available(cpu_online_mask);
+	       supports_scale_freq_counters(cpu_online_mask);
 }
 
-__weak bool arch_freq_counters_available(const struct cpumask *cpus)
+static void update_scale_freq_invariant(bool status)
 {
-	return false;
+	if (scale_freq_invariant == status)
+		return;
+
+	/*
+	 * Task scheduler behavior depends on frequency invariance support,
+	 * either cpufreq or counter driven. If the support status changes as
+	 * a result of counter initialisation and use, retrigger the build of
+	 * scheduling domains to ensure the information is propagated properly.
+	 */
+	if (topology_scale_freq_invariant() == status) {
+		scale_freq_invariant = status;
+		rebuild_sched_domains_energy();
+	}
 }
+
+void topology_set_scale_freq_source(struct scale_freq_data *data,
+				    const struct cpumask *cpus)
+{
+	struct scale_freq_data *sfd;
+	int cpu;
+
+	/*
+	 * Avoid calling rebuild_sched_domains() unnecessarily if FIE is
+	 * supported by cpufreq.
+	 */
+	if (cpumask_empty(&scale_freq_counters_mask))
+		scale_freq_invariant = topology_scale_freq_invariant();
+
+	for_each_cpu(cpu, cpus) {
+		sfd = per_cpu(sft_data, cpu);
+
+		/* Use ARCH provided counters whenever possible */
+		if (!sfd || sfd->source != SCALE_FREQ_SOURCE_ARCH) {
+			per_cpu(sft_data, cpu) = data;
+			cpumask_set_cpu(cpu, &scale_freq_counters_mask);
+		}
+	}
+
+	update_scale_freq_invariant(true);
+}
+EXPORT_SYMBOL_GPL(topology_set_scale_freq_source);
+
+void topology_clear_scale_freq_source(enum scale_freq_source source,
+				      const struct cpumask *cpus)
+{
+	struct scale_freq_data *sfd;
+	int cpu;
+
+	for_each_cpu(cpu, cpus) {
+		sfd = per_cpu(sft_data, cpu);
+
+		if (sfd && sfd->source == source) {
+			per_cpu(sft_data, cpu) = NULL;
+			cpumask_clear_cpu(cpu, &scale_freq_counters_mask);
+		}
+	}
+
+	update_scale_freq_invariant(false);
+}
+EXPORT_SYMBOL_GPL(topology_clear_scale_freq_source);
+
+void topology_scale_freq_tick(void)
+{
+	struct scale_freq_data *sfd = *this_cpu_ptr(&sft_data);
+
+	if (sfd)
+		sfd->set_freq_scale();
+}
+
 DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE;
+EXPORT_SYMBOL_GPL(freq_scale);
 
 void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq,
 			     unsigned long max_freq)

@@ -47,7 +124,7 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq,
 	 * want to update the scale factor with information from CPUFREQ.
 	 * Instead the scale factor will be updated from arch_scale_freq_tick.
 	 */
-	if (arch_freq_counters_available(cpus))
+	if (supports_scale_freq_counters(cpus))
 		return;
 
 	scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq;

include/linux/arch_topology.h +3 -1

index 0f6cd6b73a61..3bcfba5c21a7 100644
@@ -34,7 +34,19 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq,
 			     unsigned long max_freq);
 bool topology_scale_freq_invariant(void);
 
-bool arch_freq_counters_available(const struct cpumask *cpus);
+enum scale_freq_source {
+	SCALE_FREQ_SOURCE_CPUFREQ = 0,
+	SCALE_FREQ_SOURCE_ARCH,

include/linux/arch_topology.h +0 -0

 DECLARE_PER_CPU(unsigned long, thermal_pressure);
 

Signature · 1 line

2.25.0.rc1.19.g042ed3e048af

include/linux/arch_topology.h void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, · line 35

Quoted 7 lines · Viresh Kumar

include/linux/arch_topology.h +0 -0

index 0f6cd6b73a61..3bcfba5c21a7 100644
@@ -34,7 +34,19 @@ void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq,
 			     unsigned long max_freq);
 bool topology_scale_freq_invariant(void);

Quoted 5 lines

include/linux/arch_topology.h +3 -1

-bool arch_freq_counters_available(const struct cpumask *cpus);
+enum scale_freq_source {
+	SCALE_FREQ_SOURCE_CPUFREQ = 0,
+	SCALE_FREQ_SOURCE_ARCH,

include/linux/arch_topology.h void topology_set_freq_scale(const struct cpumask *cpus, unsigned long cur_freq, · line 49

Quoted 10 lines

include/linux/arch_topology.h +10 -0

+};
+
+struct scale_freq_data {
+	enum scale_freq_source source;
+	void (*set_freq_scale)(void);
+};
+
+void topology_scale_freq_tick(void);
+void topology_set_scale_freq_source(struct scale_freq_data *data, const struct cpumask *cpus);
+void topology_clear_scale_freq_source(enum scale_freq_source source, const struct cpumask *cpus);

Nit: can you split these lines?

The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking.

Normally, this scaling factor can be obtained directly with the help of the cpufreq drivers as they know the exact frequency the hardware is running at. But that isn't the case for CPPC cpufreq driver.

Another way of obtaining that is using the arch specific counter support, which is already present in kernel, but that hardware is optional for platforms.

This patch updates the CPPC driver to register itself with the topology core to provide its own implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which gets called by the scheduler on every tick. Note that the arch specific counters have higher priority than CPPC counters, if available, though the CPPC driver doesn't need to have any special handling for that.

On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we reach here from hard-irq context), which then schedules a normal work item and cppc_scale_freq_workfn() updates the per_cpu freq_scale variable based on the counter updates since the last tick.

To allow platforms to disable frequency invariance support if they want,

this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by default.

This also exports sched_setattr_nocheck() as the CPPC driver can be built as a module.

Cc Ionela Voinescu <[email protected]>

Cc [email protected]

Signed-off-by Viresh Kumar <[email protected]>

---
 drivers/cpufreq/Kconfig.arm    |   9 ++
 drivers/cpufreq/cppc_cpufreq.c | 244 +++++++++++++++++++++++++++++++--
 include/linux/arch_topology.h  |   1 +
 kernel/sched/core.c            |   1 +
 4 files changed, 243 insertions(+), 12 deletions(-)

drivers/cpufreq/Kconfig.arm +0 -0

index e65e0a43be64..a3e2d6dfea70 100644

drivers/cpufreq/Kconfig.arm +3 -0

+
+	  If in doubt, say N.
+
 config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM
 	tristate "Allwinner nvmem based SUN50I CPUFreq driver"
 	depends on ARCH_SUNXI

drivers/cpufreq/cppc_cpufreq.c +0 -0

index 8a482c434ea6..c4580a37a1b1 100644

drivers/cpufreq/cppc_cpufreq.c +61 -0

+static void cppc_freq_invariance_policy_init(struct cpufreq_policy *policy,
+					     struct cppc_cpudata *cpu_data)
+{
+	struct cppc_freq_invariance *cppc_fi;
+	int i;
+
+	if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+		return;
+
+	for_each_cpu(i, policy->cpus) {
+		cppc_fi = &per_cpu(cppc_freq_inv, i);
+		cppc_fi->cpu = i;
+		cppc_fi->cpu_data = cpu_data;
+		kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn);
+		init_irq_work(&cppc_fi->irq_work, cppc_irq_work);
+	}
+}
+
+static void cppc_freq_invariance_exit(void)
+{
+	struct cppc_freq_invariance *cppc_fi;
+	int i;
+
+	if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+		return;
+
+	topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, cpu_present_mask);
+
+	for_each_possible_cpu(i) {
+		cppc_fi = &per_cpu(cppc_freq_inv, i);
+		irq_work_sync(&cppc_fi->irq_work);
+	}
+
+	kthread_destroy_worker(kworker_fie);
+	kworker_fie = NULL;
+}
+
+static void __init cppc_freq_invariance_init(void)
+{
+	struct cppc_perf_fb_ctrs fb_ctrs = {0};
+	struct cppc_freq_invariance *cppc_fi;
+	struct sched_attr attr = {
+		.size		= sizeof(struct sched_attr),
+		.sched_policy	= SCHED_DEADLINE,
+		.sched_nice	= 0,
+		.sched_priority	= 0,
+		/*
+		 * Fake (unused) bandwidth; workaround to "fix"
+		 * priority inheritance.
+		 */
+		.sched_runtime	= 1000000,
+		.sched_deadline = 10000000,
+		.sched_period	= 10000000,
+	};
+	int i, ret;
+
+	if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+		return;
+
+	kworker_fie = kthread_create_worker(0, "cppc_fie");
+	if (IS_ERR(kworker_fie))

drivers/cpufreq/cppc_cpufreq.c +10 -0

+	for_each_possible_cpu(i) {
+		cppc_fi = &per_cpu(cppc_freq_inv, i);
+
+		/* A policy failed to initialize, abort */
+		if (unlikely(!cppc_fi->cpu_data))
+			return cppc_freq_invariance_exit();
+
+		ret = cppc_get_perf_ctrs(i, &fb_ctrs);
+		if (ret) {
+			pr_warn("%s: failed to read perf counters: %d\n",

Hey,

Quoted 18 lines · Viresh Kumar

The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking.

Normally, this scaling factor can be obtained directly with the help of the cpufreq drivers as they know the exact frequency the hardware is running at. But that isn't the case for CPPC cpufreq driver.

Another way of obtaining that is using the arch specific counter support, which is already present in kernel, but that hardware is optional for platforms.

This patch updates the CPPC driver to register itself with the topology core to provide its own implementation (cppc_scale_freq_tick()) of topology_scale_freq_tick() which gets called by the scheduler on every tick. Note that the arch specific counters have higher priority than CPPC counters, if available, though the CPPC driver doesn't need to have any special handling for that.

Quoted 6 lines

On an invocation of cppc_scale_freq_tick(), we schedule an irq work (since we reach here from hard-irq context), which then schedules a normal work item and cppc_scale_freq_workfn() updates the per_cpu freq_scale variable based on the counter updates since the last tick.

To allow platforms to disable frequency invariance support if they want,

^
this CPPC counter-based frequency invariance support..

(disabling this config will not disable cpufreq or arch counter-based FIE)

Quoted 21 lines

this is all done under CONFIG_ACPI_CPPC_CPUFREQ_FIE, which is enabled by default.

This also exports sched_setattr_nocheck() as the CPPC driver can be built as a module.

Cc Ionela Voinescu <[email protected]>

Cc [email protected]

Signed-off-by Viresh Kumar <[email protected]>

---
 drivers/cpufreq/Kconfig.arm    |   9 ++
 drivers/cpufreq/cppc_cpufreq.c | 244 +++++++++++++++++++++++++++++++--
 include/linux/arch_topology.h  |   1 +
 kernel/sched/core.c            |   1 +
 4 files changed, 243 insertions(+), 12 deletions(-)

drivers/cpufreq/Kconfig.arm +0 -0

index e65e0a43be64..a3e2d6dfea70 100644

drivers/cpufreq/Kconfig.arm config ACPI_CPPC_CPUFREQ · line 24

Quoted 6 lines

drivers/cpufreq/Kconfig.arm +3 -0

@@ -19,6 +19,15 @@ config ACPI_CPPC_CPUFREQ
 
        If in doubt, say N.
 
+config ACPI_CPPC_CPUFREQ_FIE
+     bool "Frequency Invariance support for CPPC cpufreq driver"
+     depends on ACPI_CPPC_CPUFREQ

It also depends on GENERIC_ARCH_TOPOLOGY.

drivers/cpufreq/Kconfig.arm config ACPI_CPPC_CPUFREQ · line 27

Quoted 3 lines

drivers/cpufreq/Kconfig.arm +3 -0

+     default y
+     help
+       This enables frequency invariance support for CPPC cpufreq driver.

^^^^^^^^^^^^^^^^^^^^^^^^ s//based on CPPC counters.

.. or more detailed: This extends frequency invariance support in the CPPC cpufreq driver, by using CPPC delivered and reference performance counters.

Quoted 10 lines

drivers/cpufreq/Kconfig.arm +3 -0

+
+       If in doubt, say N.
+
 config ARM_ALLWINNER_SUN50I_CPUFREQ_NVMEM
      tristate "Allwinner nvmem based SUN50I CPUFreq driver"
      depends on ARCH_SUNXI

drivers/cpufreq/cppc_cpufreq.c +0 -0

index 8a482c434ea6..c4580a37a1b1 100644

[..]

Quoted 61 lines

drivers/cpufreq/cppc_cpufreq.c +61 -0

+static void cppc_freq_invariance_policy_init(struct cpufreq_policy *policy,
+                                          struct cppc_cpudata *cpu_data)
+{
+     struct cppc_freq_invariance *cppc_fi;
+     int i;
+
+     if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+             return;
+
+     for_each_cpu(i, policy->cpus) {
+             cppc_fi = &per_cpu(cppc_freq_inv, i);
+             cppc_fi->cpu = i;
+             cppc_fi->cpu_data = cpu_data;
+             kthread_init_work(&cppc_fi->work, cppc_scale_freq_workfn);
+             init_irq_work(&cppc_fi->irq_work, cppc_irq_work);
+     }
+}
+
+static void cppc_freq_invariance_exit(void)
+{
+     struct cppc_freq_invariance *cppc_fi;
+     int i;
+
+     if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+             return;
+
+     topology_clear_scale_freq_source(SCALE_FREQ_SOURCE_CPPC, cpu_present_mask);
+
+     for_each_possible_cpu(i) {
+             cppc_fi = &per_cpu(cppc_freq_inv, i);
+             irq_work_sync(&cppc_fi->irq_work);
+     }
+
+     kthread_destroy_worker(kworker_fie);
+     kworker_fie = NULL;
+}
+
+static void __init cppc_freq_invariance_init(void)
+{
+     struct cppc_perf_fb_ctrs fb_ctrs = {0};
+     struct cppc_freq_invariance *cppc_fi;
+     struct sched_attr attr = {
+             .size           = sizeof(struct sched_attr),
+             .sched_policy   = SCHED_DEADLINE,
+             .sched_nice     = 0,
+             .sched_priority = 0,
+             /*
+              * Fake (unused) bandwidth; workaround to "fix"
+              * priority inheritance.
+              */
+             .sched_runtime  = 1000000,
+             .sched_deadline = 10000000,
+             .sched_period   = 10000000,
+     };
+     int i, ret;
+
+     if (cppc_cpufreq_driver.get == hisi_cppc_cpufreq_get_rate)
+             return;
+
+     kworker_fie = kthread_create_worker(0, "cppc_fie");
+     if (IS_ERR(kworker_fie))

drivers/cpufreq/cppc_cpufreq.c static struct cppc_workaround_oem_info wa_info[] = { · line 223

Quoted 10 lines

drivers/cpufreq/cppc_cpufreq.c +10 -0

+             return;
+
+     ret = sched_setattr_nocheck(kworker_fie->task, &attr);
+     if (ret) {
+             pr_warn("%s: failed to set SCHED_DEADLINE: %d\n", __func__,
+                     ret);
+             kthread_destroy_worker(kworker_fie);
+             return;
+     }
+

Nit: to me it makes more sense to move the code below to cppc_freq_invariance_policy_init(). It seems a bit strange to do part of the initialization of the per-cpu information there, and part here. But I do understand the reasons for it. Moving the code below would also save some cycles going through the CPUs again and will mimic the frequency invariance setup process in the arm64 topology, where we do amu_fie_setup() at policy creation time.

It's not a big deal so I'll leave it up to you.

Quoted 10 lines

drivers/cpufreq/cppc_cpufreq.c +10 -0

+     for_each_possible_cpu(i) {
+             cppc_fi = &per_cpu(cppc_freq_inv, i);
+
+             /* A policy failed to initialize, abort */
+             if (unlikely(!cppc_fi->cpu_data))
+                     return cppc_freq_invariance_exit();
+
+             ret = cppc_get_perf_ctrs(i, &fb_ctrs);
+             if (ret) {
+                     pr_warn("%s: failed to read perf counters: %d\n",

drivers/cpufreq/cppc_cpufreq.c static struct cppc_workaround_oem_info wa_info[] = { · line 243

Quoted 10 lines

drivers/cpufreq/cppc_cpufreq.c +10 -0

+                             __func__, ret);
+                     return cppc_freq_invariance_exit();
+             }
+
+             cppc_fi->prev_perf_fb_ctrs = fb_ctrs;
+     }
+
+     /* Register for freq-invariance */
+     topology_set_scale_freq_source(&cppc_sftd, cpu_present_mask);
+}

After another very quick round of testing:

Reviewed-by Ionela Voinescu <[email protected]>

Tested-by Ionela Voinescu <[email protected]>

I did not get the chance to test on ThunderX2 yet, but if you are happy with your testing on it, I won't delay this any further.

Thanks,
Ionela.