Skip to contentSkip to conversation

powercap/dtpm: Add the DTPM framework

PATCH Superseded Daniel Lezcano 2 participants 7 messages Nov 23, 2020 – Nov 26, 2020

Subsystems (5)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover powercap/dtpm: Add the DTPM framework

The density of components greatly increased the last decade bringing a numerous number of heating sources which are monitored by more than 20 sensors on recent SoC. The skin temperature, which is the case temperature of the device, must stay below approximately 45°C in order to comply with the legal requirements.

The skin temperature is managed as a whole by an user space daemon, which is catching the current application profile, to allocate a power budget to the different components where the resulting heating effect will comply with the skin temperature constraint.

This technique is called the Dynamic Thermal Power Management.

The Linux kernel does not provide any unified interface to act on the power of the different devices. Currently, the thermal framework is changed to export artificially the performance states of different devices via the cooling device software component with opaque values. This change is done regardless of the in-kernel logic to mitigate the temperature. The user space daemon uses all the available knobs to act on the power limit and those differ from one platform to another.

This series provides a Dynamic Thermal Power Management framework to provide an unified way to act on the power of the devices.

Changelog:
V3:
- Fixed power-limit computation in addition with the hotplugging
- Improved the encapsulation
- Added specific ops for the leaves of the tree
- Simplified API and self-encapsulation
- Fixed documentation and generated it to check the content
V2:
- Fixed indentation
- Fixed typos in comments
- Fixed missing kfree for dtpm_cpu
- Capitalize letters in the Kconfig description
- Reduced name description
- Stringified section name
- Added more debug traces in the code
- Removed duplicate initialization in the dtpm cpu

Daniel Lezcano (4):
units: Add Watt units
Documentation/powercap/dtpm: Add documentation for dtpm powercap/drivers/dtpm: Add API for dynamic thermal power management powercap/drivers/dtpm: Add CPU energy model based support

 Documentation/power/index.rst         |   1 +
 Documentation/power/powercap/dtpm.rst | 210 ++++++++++++
 drivers/powercap/Kconfig              |  13 +
 drivers/powercap/Makefile             |   2 +
 drivers/powercap/dtpm.c               | 441 ++++++++++++++++++++++++++
 drivers/powercap/dtpm_cpu.c           | 252 +++++++++++++++
 include/asm-generic/vmlinux.lds.h     |  11 +
 include/linux/cpuhotplug.h            |   1 +
 include/linux/dtpm.h                  |  78 +++++
 include/linux/units.h                 |   4 +
 10 files changed, 1013 insertions(+)
 create mode 100644 Documentation/power/powercap/dtpm.rst
 create mode 100644 drivers/powercap/dtpm.c
 create mode 100644 drivers/powercap/dtpm_cpu.c
 create mode 100644 include/linux/dtpm.h

--
2.17.1

#1 lore raw

PATCH v3 1/4 units: Add Watt units

As there are the temperature units, let's add the Watt macros definition.

---
 include/linux/units.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/units.h +4 -0
index aaf716364ec3..92c234e71cab 100644
@@ -4,6 +4,10 @@
 
 #include <linux/kernel.h>
 
+#define MILLIWATT_PER_WATT	1000L
+#define MICROWATT_PER_MILLIWATT	1000L
+#define MICROWATT_PER_WATT	1000000L
+
 #define ABSOLUTE_ZERO_MILLICELSIUS -273150
 
 static inline long milli_kelvin_to_millicelsius(long t)
Signature · 1 line
2.17.1
#2 lore raw

PATCH v3 2/4 Documentation/powercap/dtpm: Add documentation for dtpm

In reply to Daniel Lezcano

The dynamic thermal and power management is a technique to dynamically adjust the power consumption of different devices in order to ensure a global thermal constraint.

An userspace daemon is usually monitoring the temperature and the power to take immediate action on the device.

The DTPM framework provides an unified API to userspace to act on the power.

Document this framework.

---
 Documentation/power/index.rst         |   1 +
 Documentation/power/powercap/dtpm.rst | 210 ++++++++++++++++++++++++++
 2 files changed, 211 insertions(+)
 create mode 100644 Documentation/power/powercap/dtpm.rst
Documentation/power/index.rst +1 -0
index ced8a8007434..a0f5244fb427 100644
@@ -30,6 +30,7 @@ Power Management
     userland-swsusp
 
     powercap/powercap
+    powercap/dtpm
 
     regulator/consumer
     regulator/design
Documentation/power/powercap/dtpm.rst new file +210 -0

210 lines not on this page · raw message

Signature · 1 line
2.17.1
#3 lore raw

PATCH v3 3/4 powercap/drivers/dtpm: Add API for dynamic thermal power management

In reply to Daniel Lezcano

On the embedded world, the complexity of the SoC leads to an increasing number of hotspots which need to be monitored and mitigated as a whole in order to prevent the temperature to go above the normative and legally stated 'skin temperature'.

Another aspect is to sustain the performance for a given power budget, for example virtual reality where the user can feel dizziness if the GPU performance is capped while a big CPU is processing something else. Or reduce the battery charging because the dissipated power is too high compared with the power consumed by other devices.

The userspace is the most adequate place to dynamically act on the different devices by limiting their power given an application
profile: it has the knowledge of the platform.

These userspace daemons are in charge of the Dynamic Thermal Power Management (DTPM).

Nowadays, the dtpm daemons are abusing the thermal framework as they act on the cooling device state to force a specific and arbitrary state without taking care of the governor decisions. Given the closed loop of some governors that can confuse the logic or directly enter in a decision conflict.

As the number of cooling device support is limited today to the CPU and the GPU, the dtpm daemons have little control on the power dissipation of the system. The out of tree solutions are hacking around here and there in the drivers, in the frameworks to have control on the devices. The common solution is to declare them as cooling devices.

There is no unification of the power limitation unit, opaque states are used.

This patch provides a way to create a hierarchy of constraints using the powercap framework. The devices which are registered as power limit-able devices are represented in this hierarchy as a tree. They are linked together with intermediate nodes which are just there to propagate the constraint to the children.

The leaves of the tree are the real devices, the intermediate nodes are virtual, aggregating the children constraints and power characteristics.

Each node have a weight on a 2^10 basis, in order to reflect the percentage of power distribution of the children's node. This percentage is used to dispatch the power limit to the children.

The weight is computed against the max power of the siblings.

This simple approach allows to do a fair distribution of the power limit.

---
 drivers/powercap/Kconfig          |   6 +
 drivers/powercap/Makefile         |   1 +
 drivers/powercap/dtpm.c           | 441 ++++++++++++++++++++++++++++++
 include/asm-generic/vmlinux.lds.h |  11 +
 include/linux/dtpm.h              |  76 +++++
 5 files changed, 535 insertions(+)
 create mode 100644 drivers/powercap/dtpm.c
 create mode 100644 include/linux/dtpm.h
drivers/powercap/Kconfig +6 -0
index bc228725346b..cc1953bd8bed 100644
@@ -43,4 +43,10 @@ config IDLE_INJECT
 	  CPUs for power capping. Idle period can be injected
 	  synchronously on a set of specified CPUs or alternatively
 	  on a per CPU basis.
+
+config DTPM
+	bool "Power capping for Dynamic Thermal Power Management"
+	help
+	  This enables support for the power capping for the dynamic
+	  thermal power management userspace engine.
 endif
drivers/powercap/Makefile +1 -0
index 7255c94ec61c..6482ac52054d 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_DTPM) += dtpm.o
 obj-$(CONFIG_POWERCAP)	+= powercap_sys.o
 obj-$(CONFIG_INTEL_RAPL_CORE) += intel_rapl_common.o
 obj-$(CONFIG_INTEL_RAPL) += intel_rapl_msr.o
drivers/powercap/dtpm.c new file +441 -0

441 lines not on this page · raw message

include/asm-generic/vmlinux.lds.h +11 -0
index b2b3d81b1535..b3e4e0740089 100644
@@ -316,6 +316,16 @@
 #define THERMAL_TABLE(name)
 #endif
 
+#ifdef CONFIG_DTPM
+#define DTPM_TABLE()							\
+	. = ALIGN(8);							\
+	__dtpm_table = .;						\
+	KEEP(*(__dtpm_table))						\
+	__dtpm_table_end = .;
+#else
+#define DTPM_TABLE()
+#endif
+
 #define KERNEL_DTB()							\
 	STRUCT_ALIGN();							\
 	__dtb_start = .;						\
@@ -733,6 +743,7 @@
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(timer)						\
 	THERMAL_TABLE(governor)						\
+	DTPM_TABLE()							\
 	EARLYCON_TABLE()						\
 	LSM_TABLE()							\
 	EARLY_LSM_TABLE()						\
include/linux/dtpm.h new file +76 -0

76 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

PATCH v3 4/4 powercap/drivers/dtpm: Add CPU energy model based support

In reply to Daniel Lezcano

With the powercap dtpm controller, we are able to plug devices with power limitation features in the tree.

The following patch introduces the CPU power limitation based on the energy model and the performance states.

The power limitation is done at the performance domain level. If some CPUs are unplugged, the corresponding power will be subtracted from the performance domain total power.

It is up to the platform to initialize the dtpm tree and add the CPU.

Here is an example to create a simple tree with one root node called "pkg" and the CPU's performance domains.

static int dtpm_register_pkg(struct dtpm_descr *descr)

{
	struct dtpm *pkg;
	int ret;

	pkg = dtpm_alloc(NULL);
	if (!pkg)
		return -ENOMEM;

	ret = dtpm_register(descr->name, pkg, descr->parent);
	if (ret)
		return ret;

	return dtpm_register_cpu(pkg);
}
static struct dtpm_descr descr = {
	.name = "pkg",
	.init = dtpm_register_pkg,
};

DTPM_DECLARE(descr);

---
 drivers/powercap/Kconfig    |   7 +
 drivers/powercap/Makefile   |   1 +
 drivers/powercap/dtpm_cpu.c | 252 ++++++++++++++++++++++++++++++++++++
 include/linux/cpuhotplug.h  |   1 +
 include/linux/dtpm.h        |   2 +
 5 files changed, 263 insertions(+)
 create mode 100644 drivers/powercap/dtpm_cpu.c
drivers/powercap/Kconfig +7 -0
index cc1953bd8bed..20b4325c6161 100644
@@ -49,4 +49,11 @@ config DTPM
 	help
 	  This enables support for the power capping for the dynamic
 	  thermal power management userspace engine.
+
+config DTPM_CPU
+	bool "Add CPU power capping based on the energy model"
+	depends on DTPM && ENERGY_MODEL
+	help
+	  This enables support for CPU power limitation based on
+	  energy model.
 endif
drivers/powercap/Makefile +1 -0
index 6482ac52054d..fabcf388a8d3 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 obj-$(CONFIG_DTPM) += dtpm.o
+obj-$(CONFIG_DTPM_CPU) += dtpm_cpu.o
 obj-$(CONFIG_POWERCAP)	+= powercap_sys.o
 obj-$(CONFIG_INTEL_RAPL_CORE) += intel_rapl_common.o
 obj-$(CONFIG_INTEL_RAPL) += intel_rapl_msr.o
drivers/powercap/dtpm_cpu.c new file +252 -0

252 lines not on this page · raw message

include/linux/cpuhotplug.h +1 -0
index bc56287a1ed1..72fd8db62342 100644
@@ -192,6 +192,7 @@ enum cpuhp_state {
 	CPUHP_AP_ONLINE_DYN_END		= CPUHP_AP_ONLINE_DYN + 30,
 	CPUHP_AP_X86_HPET_ONLINE,
 	CPUHP_AP_X86_KVM_CLK_ONLINE,
+	CPUHP_AP_DTPM_CPU_ONLINE,
 	CPUHP_AP_ACTIVE,
 	CPUHP_ONLINE,
 };
include/linux/dtpm.h +2 -0
index 84b087b4ac01..723fe51f1628 100644
@@ -73,4 +73,6 @@ void dtpm_unregister(struct dtpm *dtpm);
 
 int dtpm_register(const char *name, struct dtpm *dtpm, struct dtpm *parent);
 
+int dtpm_register_cpu(struct dtpm *parent);
+
 #endif
Signature · 1 line
2.17.1
#5 lore raw

Re: [PATCH v3 4/4] powercap/drivers/dtpm: Add CPU energy model based support

Hi Daniel,

Quoted 3 lines · Daniel Lezcano

With the powercap dtpm controller, we are able to plug devices with power limitation features in the tree.

[snip]

drivers/powercap/dtpm_cpu.c line 134

Quoted 5 lines
drivers/powercap/dtpm_cpu.c new file +5 -0
+
+static void pd_release(struct dtpm *dtpm)
+{
+	struct dtpm_cpu *dtpm_cpu = dtpm->private;
+

Maybe it's worth to add:
------------------->8----------------
if (freq_qos_request_active(&dtpm_cpu->qos_req))

	freq_qos_remove_request(&dtpm_cpu->qos_req);

-------------------8<---------------

If we are trying to unregister dtpm in error path due to freq_qos registration failure, a warning would be emitted from freq_qos.

Quoted 3 lines
drivers/powercap/dtpm_cpu.c new file +3 -0
+	freq_qos_remove_request(&dtpm_cpu->qos_req);
+	kfree(dtpm_cpu);
+}

[snip]

Quoted 38 lines
drivers/powercap/dtpm_cpu.c new file +38 -0
+
+static int cpuhp_dtpm_cpu_online(unsigned int cpu)
+{
+	struct dtpm *dtpm;
+	struct dtpm_cpu *dtpm_cpu;
+	struct cpufreq_policy *policy;
+	struct em_perf_domain *pd;
+	char name[CPUFREQ_NAME_LEN];
+	int ret;
+
+	policy = cpufreq_cpu_get(cpu);
+
+	if (!policy)
+		return 0;
+
+	pd = em_cpu_get(cpu);
+	if (!pd)
+		return -EINVAL;
+
+	dtpm = per_cpu(dtpm_per_cpu, cpu);
+	if (dtpm)
+		return power_add(dtpm, pd);
+
+	dtpm = dtpm_alloc(&dtpm_ops);
+	if (!dtpm)
+		return -EINVAL;
+
+	dtpm_cpu = kzalloc(sizeof(dtpm_cpu), GFP_KERNEL);
+	if (!dtpm_cpu) {
+		kfree(dtpm);
+		return -ENOMEM;
+	}
+
+	dtpm->private = dtpm_cpu;
+	dtpm_cpu->cpu = cpu;
+
+	for_each_cpu(cpu, policy->related_cpus)
+		per_cpu(dtpm_per_cpu, cpu) = dtpm;

drivers/powercap/dtpm_cpu.c line 221

Quoted 10 lines
drivers/powercap/dtpm_cpu.c new file +10 -0
+
+	sprintf(name, "cpu%d", dtpm_cpu->cpu);
+
+	ret = dtpm_register(name, dtpm, __parent);
+	if (ret)
+		goto out_kfree_dtpm_cpu;
+
+	ret = power_add(dtpm, pd);
+	if (ret)
+		goto out_power_sub;

Shouldn't we call dtpm_unregister() instead?
The dtpm_unregister() would remove the zone, which IIUC we are currently missing.

drivers/powercap/dtpm_cpu.c line 227

Quoted 6 lines
drivers/powercap/dtpm_cpu.c new file +6 -0
+
+	ret = freq_qos_add_request(&policy->constraints,
+				   &dtpm_cpu->qos_req, FREQ_QOS_MAX,
+				   pd->table[pd->nr_perf_states - 1].frequency);
+	if (ret)
+		goto out_dtpm_unregister;

Could this trigger different steps, starting from out_power_sub_v2 below?

drivers/powercap/dtpm_cpu.c line 235

Quoted 8 lines
drivers/powercap/dtpm_cpu.c new file +8 -0
+
+	return 0;
+
+out_dtpm_unregister:
+	dtpm_unregister(dtpm);
+	dtpm_cpu = NULL; /* Already freed by the release ops */
+out_power_sub:
+	power_sub(dtpm, pd);

I would change the order of these two above into something like:

out_power_sub_v2:

	power_sub(dtpm, pd);
out_dtpm_unregister_v2:
	dtpm_unregister(dtpm);
	dtpm_cpu = NULL;

drivers/powercap/dtpm_cpu.c line 242

Quoted 7 lines
drivers/powercap/dtpm_cpu.c new file +7 -0
+out_kfree_dtpm_cpu:
+	for_each_cpu(cpu, policy->related_cpus)
+		per_cpu(dtpm_per_cpu, cpu) = NULL;
+	kfree(dtpm_cpu);
+
+	return ret;
+}

IIUC power_sub() would decrement the power and set it to 0 for that dtmp, then the dtpm_unregister() would also try to decrement the power, but by the value of 0. So it should be safe.

Regards,
Lukasz

#6 lore raw

Re: [PATCH v3 4/4] powercap/drivers/dtpm: Add CPU energy model based support

Hi Lukasz,

Quoted 21 lines · Lukasz Luba

Hi Daniel,

Quoted 3 lines · Daniel Lezcano

With the powercap dtpm controller, we are able to plug devices with power limitation features in the tree.

[snip]

drivers/powercap/dtpm_cpu.c line 134

Quoted 5 lines
drivers/powercap/dtpm_cpu.c new file +5 -0
+
+static void pd_release(struct dtpm *dtpm)
+{
+    struct dtpm_cpu *dtpm_cpu = dtpm->private;
+

Maybe it's worth to add:
------------------->8----------------
if (freq_qos_request_active(&dtpm_cpu->qos_req))
    freq_qos_remove_request(&dtpm_cpu->qos_req);
-------------------8<---------------

Quoted 2 lines

If we are trying to unregister dtpm in error path due to freq_qos registration failure, a warning would be emitted from freq_qos.

Ah yes, good point.

Quoted 69 lines
Quoted 3 lines

+    freq_qos_remove_request(&dtpm_cpu->qos_req);
+    kfree(dtpm_cpu);
+}

[snip]

Quoted 38 lines
drivers/powercap/dtpm_cpu.c new file +38 -0
+
+static int cpuhp_dtpm_cpu_online(unsigned int cpu)
+{
+    struct dtpm *dtpm;
+    struct dtpm_cpu *dtpm_cpu;
+    struct cpufreq_policy *policy;
+    struct em_perf_domain *pd;
+    char name[CPUFREQ_NAME_LEN];
+    int ret;
+
+    policy = cpufreq_cpu_get(cpu);
+
+    if (!policy)
+        return 0;
+
+    pd = em_cpu_get(cpu);
+    if (!pd)
+        return -EINVAL;
+
+    dtpm = per_cpu(dtpm_per_cpu, cpu);
+    if (dtpm)
+        return power_add(dtpm, pd);
+
+    dtpm = dtpm_alloc(&dtpm_ops);
+    if (!dtpm)
+        return -EINVAL;
+
+    dtpm_cpu = kzalloc(sizeof(dtpm_cpu), GFP_KERNEL);
+    if (!dtpm_cpu) {
+        kfree(dtpm);
+        return -ENOMEM;
+    }
+
+    dtpm->private = dtpm_cpu;
+    dtpm_cpu->cpu = cpu;
+
+    for_each_cpu(cpu, policy->related_cpus)
+        per_cpu(dtpm_per_cpu, cpu) = dtpm;

drivers/powercap/dtpm_cpu.c line 221

Quoted 10 lines
drivers/powercap/dtpm_cpu.c new file +10 -0
+
+    sprintf(name, "cpu%d", dtpm_cpu->cpu);
+
+    ret = dtpm_register(name, dtpm, __parent);
+    if (ret)
+        goto out_kfree_dtpm_cpu;
+
+    ret = power_add(dtpm, pd);
+    if (ret)
+        goto out_power_sub;

Shouldn't we call dtpm_unregister() instead?
The dtpm_unregister() would remove the zone, which IIUC we are currently missing.

Quoted 6 lines

+
+    ret = freq_qos_add_request(&policy->constraints,
+                   &dtpm_cpu->qos_req, FREQ_QOS_MAX,
+                   pd->table[pd->nr_perf_states - 1].frequency); +    if (ret)
+        goto out_dtpm_unregister;

Could this trigger different steps, starting from out_power_sub_v2 below?

Quoted 10 lines

drivers/powercap/dtpm_cpu.c line 235

Quoted 8 lines
drivers/powercap/dtpm_cpu.c new file +8 -0
+
+    return 0;
+
+out_dtpm_unregister:
+    dtpm_unregister(dtpm);
+    dtpm_cpu = NULL; /* Already freed by the release ops */
+out_power_sub:
+    power_sub(dtpm, pd);

I would change the order of these two above into something like:

Ok, I'll revisit the rollback routine.

Quoted 6 lines

out_power_sub_v2:
    power_sub(dtpm, pd);
out_dtpm_unregister_v2:
    dtpm_unregister(dtpm);
    dtpm_cpu = NULL;

Quoted 11 lines

drivers/powercap/dtpm_cpu.c line 242

Quoted 7 lines
drivers/powercap/dtpm_cpu.c new file +7 -0
+out_kfree_dtpm_cpu:
+    for_each_cpu(cpu, policy->related_cpus)
+        per_cpu(dtpm_per_cpu, cpu) = NULL;
+    kfree(dtpm_cpu);
+
+    return ret;
+}

IIUC power_sub() would decrement the power and set it to 0 for that dtmp, then the dtpm_unregister() would also try to decrement the power, but by the value of 0. So it should be safe.

Right.

Thanks for the review

Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog