Skip to contentSkip to conversation

RAPL (Running Average Power Limit) driver

PATCH Superseded Jacob Pan 5 participants 23 messages Apr 2, 2013 – Apr 5, 2013 x86 platform drivers

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

cover RAPL (Running Average Power Limit) driver

RAPL(Running Average Power Limit) interface provides platform software with the ability to monitor, control, and get notifications on SOC power consumptions. Since its first appearance on Sandy Bridge, more features have being added to extend its usage. In RAPL, platforms are divided into domains for fine grained control. These domains include package, DRAM controller, CPU core (Power Plane 0), graphics uncore (power plane 1), etc.

The purpose of this driver is to expose RAPL for userspace consumption. Overall, RAPL fits in the generic thermal layer in that platform level power capping and monitoring are mainly used for thermal management and thermal layer provides the abstracted interface needed to have portable applications.

Specifically, userspace is presented with per domain cooling device with sysfs links to its kobject. Although RAPL domain provides many parameters for fine tuning, long term power limit is exposed as the single knob via cooling device state. Whereas the rest of the parameters are still accessible via the linked kobject. This simplifies the interface for both simple and advanced use cases.

DETAILS
=======
1. sysfs layout

As an x86 platform driver, RAPL driver binds with supported CPU ids during probing phase. Once domains are discovered, kobjets are created for each domain which are also linked with cooling devices after its registration with the generic thermal layer.

e.g.package RAPL domain registered as cooling device #15, link "device" back to its kobject.

/sys/class/thermal/cooling_device15/

├── cur_state
├── device -> ../../../platform/intel_rapl/rapl_domains/package
├── max_state
├── power
├── subsystem -> ../../../../class/thermal
├── type
└── uevent

In driver's private sysfs area, domains kobjects are grouped under a kset which exposes global data. /sys/devices/platform/intel_rapl/

├── driver -> ../../../bus/platform/drivers/intel_rapl
├── power
├── rapl_domains
│   ├── package
│   │   └── thermal_cooling
-> ../../../../virtual/thermal/cooling_device15  
│   ├── power_plane_0
│   │   └── thermal_cooling
-> ../../../../virtual/thermal/cooling_device16  
│   └── power_plane_1
│       └── thermal_cooling
-> ../../../../virtual/thermal/cooling_device18  
└── subsystem -> ../../../bus/platform

2. per domain parameters

These are the fine tuning parameters only used by advanced power/thermal management applications. Refer to Intel SDM ch14 for details.

root@chromoly:/sys/class/thermal/cooling_device15/device# grep . *
domain_name:package
energy:924228
lock:0
max_power:0
max_window:0
min_power:0
pl1_clamp:1
pl1_enable:1
pl2_clamp:0
pl2_enable:1
power:2276
power_limit1:12000
power_limit2:31250
thermal_spec_power:17000
throttle_time:
time_window1:28000
time_window2:0

3. event notifications

RAPL driver uses eventfd to provide userspace notifications on selected events. A file node called "event_control" is created for each RAPL domain. User can write control file descriptor, eventfd descriptor, and threshold to event_control file. Then, user application can use poll/select or blocking read to get notifications from the driver. Multiple events are allowed for each domain but only a single threshold is accepted.

4. Usage Examples (assume the topology in the sysfs layout above)

- set power limit to package domain (whole SOC package) to 6w

root@chromoly:~# echo 6000
	> /sys/class/thermal/cooling_device15/cur_state  

- set power limit to pp1 domain (graphics) to 4w

root@chromoly:~# echo 4000
	> /sys/class/thermal/cooling_device18/cur_state  

- check the current power usage in mWatts of pp1 domain

root@chromoly:~# cat  /sys/class/thermal/cooling_device18/cur_state 
61

- set event notification when power consumption of graphics unit crosses 5w.
root@chromoly:~#
event_fd_listener /sys/class/thermal/cooling_device18/device/power 5000
(event_fd_listener opens control file power and creates an eventfd, then write efd, cfd, threshold to event_control file of the given domain)

Caveats:

1. Package power limit events are supported by legacy thermal reporting mechanism, which uses local APIC thermal vector to generate interrupts when targeted P-states are not honored by the HW/FW. This is tied to machine check reporting. Until RAPL is used, this notification is a rare exception. When RAPL power limit is set artifically low, this notification could result in unwanted interrupts for each power limit excursion. Therefore, RAPL driver attempts to turn off the power limit notification interrupt when user sets a power limit.

2. By Intel Software Developer's Manual, RAPL interface can report max/min power for certain domains. But in reality HW often reports 0 for max/min power. RAPL driver tackles this problem by using thermal specification power or current power limit1 when max power information is not available. The result is that the max_state of a RAPL cooling device can be based on thermal spec power or power limit 1.

3. Since RAPL is backed by FW. In case of FW failure or plain lack of support, setting RAPL power limit could result in silent failure. I don't have a good solution for that.

4. Data polling starts only when the following items are set
- power limit
- events

Jacob Pan (1):
Introduce Intel RAPL cooling device driver

 drivers/platform/x86/Kconfig      |    8 +
 drivers/platform/x86/Makefile     |    1 +
 drivers/platform/x86/intel_rapl.c | 1323 +++++++++++++++++++++++++++++++++++++
 drivers/platform/x86/intel_rapl.h |  249 +++++++
 4 files changed, 1581 insertions(+)
 create mode 100644 drivers/platform/x86/intel_rapl.c
 create mode 100644 drivers/platform/x86/intel_rapl.h
Signature · 1 line
1.7.9.5
author
#1 lore raw

PATCH 1/1 Introduce Intel RAPL cooling device driver

RAPL(Running Average Power Limit) interface provides platform software with the ability to monitor, control, and get notifications on SOC power consumptions. Since its first appearance on Sandy Bridge, more features have being added to extend its usage. In RAPL, platforms are divided into domains for fine grained control. These domains include package, DRAM controller, CPU core (Power Plane 0), graphics uncore (power plane 1), etc.

The purpose of this driver is to expose RAPL for userspace consumption. Overall, RAPL fits in the generic thermal layer in that platform level power capping and monitoring are mainly used for thermal management and thermal layer provides the abstracted interface needed to have portable applications.

Specifically, userspace is presented with per domain cooling device with sysfs links to its kobject. Although RAPL domain provides many parameters for fine tuning, long term power limit is exposed as the single knob via cooling device state. Whereas the rest of the parameters are still accessible via the linked kobject. This simplifies the interface for both simple and advanced use cases.

Eventfd is used to provide notifications to the userspace. At per domain level, use can choose any event capable parameters to register for threshold crossing notifications. This is shamelessly "borrowed" from cgroup with some trimming/fitting.

Zhang, Rui's initial RAPL driver was used as a reference and starting point. Many thanks.
https://lkml.org/lkml/2011/5/26/93

Unlike the patch above, which is mainly for monitoring, this driver focus on the control and usability by user applications.

---
 drivers/platform/x86/Kconfig      |    8 +
 drivers/platform/x86/Makefile     |    1 +
 drivers/platform/x86/intel_rapl.c | 1323 +++++++++++++++++++++++++++++++++++++
 drivers/platform/x86/intel_rapl.h |  249 +++++++
 4 files changed, 1581 insertions(+)
 create mode 100644 drivers/platform/x86/intel_rapl.c
 create mode 100644 drivers/platform/x86/intel_rapl.h
drivers/platform/x86/Kconfig +8 -0
index 3338437..34bcd52 100644
@@ -781,4 +781,12 @@ config APPLE_GMUX
 	  graphics as well as the backlight. Currently only backlight
 	  control is supported by the driver.
 
+config INTEL_RAPL
+	tristate "Intel RAPL Support"
+	depends on X86 && THERMAL
+	default y
+	---help---
+	  RAPL, AKA, Running Average Power Limit provides mechanisms to enforce
+	  and monitor per domain power consumption limits of supported Intel CPUs.
+
 endif # X86_PLATFORM_DEVICES
drivers/platform/x86/Makefile +1 -0
index ace2b38..a80c0f4 100644
@@ -51,3 +51,4 @@ obj-$(CONFIG_INTEL_OAKTRAIL)	+= intel_oaktrail.o
 obj-$(CONFIG_SAMSUNG_Q10)	+= samsung-q10.o
 obj-$(CONFIG_APPLE_GMUX)	+= apple-gmux.o
 obj-$(CONFIG_CHROMEOS_LAPTOP)	+= chromeos_laptop.o
+obj-$(CONFIG_INTEL_RAPL)	+= intel_rapl.o
drivers/platform/x86/intel_rapl.c new file +1323 -0

1,323 lines not on this page · raw message

drivers/platform/x86/intel_rapl.h new file +249 -0

249 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#2 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 6 lines · Jacob Pan
drivers/platform/x86/intel_rapl.c new file +0 -0
new file mode 100644
index 0000000..56ee928

drivers/platform/x86/intel_rapl.c line 48

Quoted 10 lines
drivers/platform/x86/intel_rapl.c new file +9 -0
@@ -0,0 +1,1323 @@
+/*
+ *  intel_rapl.c - Intel Running Average Power Limit Driver for MSR based
+ *                 RAPL interface
+ *
+ *  Copyright (c) 2013, Intel Corporation.
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#include <linux/sysfs.h>
+
 
+#include "../../../fs/sysfs/sysfs.h"

What does this driver need from ^^^^^^ that file, which says:

* fs/sysfs/sysfs.h - sysfs internal header file

and should that be moved to include/linux/sysfs.h ?

Signature · 1 line
~Randy
#3 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 16 lines · Randy Dunlap
Quoted 14 lines · Jacob Pan
drivers/platform/x86/intel_rapl.c new file +8 -0
new file mode 100644
index 0000000..56ee928
@@ -0,0 +1,1323 @@
+/*
+ *  intel_rapl.c - Intel Running Average Power Limit Driver for MSR based
+ *                 RAPL interface
+ *
+ *  Copyright (c) 2013, Intel Corporation.
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#include <linux/sysfs.h>
+
Quoted 8 lines

drivers/platform/x86/intel_rapl.c line 48

Quoted 1 line
drivers/platform/x86/intel_rapl.c new file +1 -0
+#include "../../../fs/sysfs/sysfs.h"

What does this driver need from ^^^^^^ that file, which says:

* fs/sysfs/sysfs.h - sysfs internal header file

and should that be moved to include/linux/sysfs.h ?

See my other complain about this already, I beat you to it :)

greg k-h

#4 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Jacob Pan
Quoted 7 lines · Jacob Pan

RAPL(Running Average Power Limit) interface provides platform software with the ability to monitor, control, and get notifications on SOC power consumptions. Since its first appearance on Sandy Bridge, more features have being added to extend its usage. In RAPL, platforms are divided into domains for fine grained control. These domains include package, DRAM controller, CPU core (Power Plane 0), graphics uncore (power plane 1), etc.

trivia only:

drivers/platform/x86/intel_rapl.c

Quoted 1 line
drivers/platform/x86/intel_rapl.c +0 -0

[]

drivers/platform/x86/intel_rapl.c line 77

Quoted 6 lines
drivers/platform/x86/intel_rapl.c +6 -0
+static char *rapl_domain_names[] = {
+	"package",
+	"power_plane_0",
+	"power_plane_1",
+	"dram",
+};

const

drivers/platform/x86/intel_rapl.c line 82

Quoted 3 lines
drivers/platform/x86/intel_rapl.c +3 -0
+static void rapl_init_domains(void)
+{
+	int i, j = 0;

I think this below is unpleasant to read.
Maybe add a temporary pointer and get rid of j?

	typeof rapl_domains *t = rapl_domains;

(not typeof, but the actual type)

drivers/platform/x86/intel_rapl.c line 90

Quoted 8 lines
drivers/platform/x86/intel_rapl.c +8 -0
+
+	for (i = 0; i < RAPL_DOMAIN_MAX; i++) {
+		unsigned int mask = rg_data.domain_map & (1 << i);
+
+		switch (mask) {
+		case 1 << RAPL_DOMAIN_PKG:
+			rapl_domains[j].name =
+				rapl_domain_names[RAPL_DOMAIN_PKG];
			t->name = rapl_domain_names[RAPL_DOMAIN_PKG];
			etc...

[]

drivers/platform/x86/intel_rapl.c line 153

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+		if (mask)
+			j++;
			t++;

drivers/platform/x86/intel_rapl.c line 202

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+/* in the order of enum rapl_primitives */
+static struct rapl_primitive_info rpi[] = {

const?

drivers/platform/x86/intel_rapl.c line 206

Quoted 4 lines
drivers/platform/x86/intel_rapl.c +4 -0
+	/* name, mask, shift, msr index, unit divisor*/
+	PRIMITIVE_INFO_INIT(energy, ENERGY_STATUS_MASK, 0,
+			RAPL_DOMAIN_MSR_STATUS, ENERGY_UNIT,
+			RAPL_PRIMITIVE_EVENT_CAP),

[]

drivers/platform/x86/intel_rapl.c line 260

Quoted 3 lines
drivers/platform/x86/intel_rapl.c +3 -0
+static int rapl_read_data_raw(struct rapl_domain *domain,
+			struct rapl_primitive_info *rp, bool xlate, u64 *data)
+{

[]

drivers/platform/x86/intel_rapl.c line 273

Quoted 1 line
drivers/platform/x86/intel_rapl.c +1 -0
+	/* specical-case pkg lock bit since pkg domain uses a different bit */

special-case

[]

Quoted 37 lines
drivers/platform/x86/intel_rapl.c +37 -0
+static ssize_t store_event_control(struct rapl_domain *rd, const char *buf,
+				size_t size)
+{
+	unsigned int efd, cfd, new_threshold;
+	struct file *efile = NULL;
+	struct file *cfile = NULL;
+	int ret = 0;
+	int prim;
+	struct rapl_event *ep;
+	u64 val;
+
+	if (sscanf(buf, "%u %u %u", &efd, &cfd, &new_threshold) != 3)
+		return -EINVAL;
+
+	efile = eventfd_fget(efd);
+	if (IS_ERR(efile)) {
+		ret = PTR_ERR(efile);
+		pr_err("failed to get eventfd file %d\n", efd);
+		goto done;
+	}
+	cfile = fget(cfd);
+	if (!cfile) {
+		ret = -EBADF;
+		fput(efile);
+		goto done;
+	}
+	/* check if the cfile belongs to the same rapl domain */
+	if (strcmp(rd->kobj.sd->s_name,
+			cfile->f_dentry->d_parent->d_name.name)) {
+		pr_debug("cfile does not belong to domain %s\n",
+			rd->kobj.sd->s_name);
+		ret = -EINVAL;
+		goto exit_cleanup_fds;
+	}
+	prim = primitive_name_to_entry(
+		(const char *)cfile->f_dentry->d_name.name);
+	if (prim < 0) {

drivers/platform/x86/intel_rapl.c line 634

Quoted 10 lines
drivers/platform/x86/intel_rapl.c +10 -0
+		pr_err("failed lookup primitive id for control file %s\n",
+			cfile->f_dentry->d_name.name);
+		ret = -EINVAL;
+		goto exit_cleanup_fds;
+	}
+	if (!(rpi[prim].flag & RAPL_PRIMITIVE_EVENT_CAP)) {
+		pr_info("Invalid control file %d\n", prim);
+		ret = -EINVAL;
+		goto exit_cleanup_fds;
+	}

Perhaps all these pr_<levels> should be pr_err as they all lead to a failure path.

[]

drivers/platform/x86/intel_rapl.c line 1159

Quoted 3 lines
drivers/platform/x86/intel_rapl.c +3 -0
+static int stop_periodic_polling(void)
+{
+	if (true == polling_started) {

You don't need a comparison,

	if (polling_started) {

is perhaps better.

[]

drivers/platform/x86/intel_rapl.c line 1216

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+static int rapl_detect_domains(void)
+{

[]

drivers/platform/x86/intel_rapl.c line 1239

Quoted 1 line
drivers/platform/x86/intel_rapl.c +1 -0
+	pr_info("Found %d vaild RAPL domains\n", rg_data.nr_domains);

valid

drivers/platform/x86/intel_rapl.c

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+	rapl_domains = kzalloc(sizeof(struct rapl_domain) * rg_data.nr_domains,
+			GFP_KERNEL);

kcalloc

drivers/platform/x86/intel_rapl.c line 1243

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+	if (NULL == rapl_domains) {
+		pr_err("Failed to allocate memory for rapl domain\n");

You don't need OOM messages as the memory
subsystem already does a dump_stack on OOM.

[]

drivers/platform/x86/intel_rapl.c line 1267

Quoted 5 lines
drivers/platform/x86/intel_rapl.c +4 -0
+static int __init intel_rapl_init(void)
+{
 
+	rd_data = kzalloc(sizeof(struct rapl_domain_data) * rg_data.nr_domains,
+			GFP_KERNEL);

kcalloc again.

drivers/platform/x86/intel_rapl.c line 1269

Quoted 2 lines
drivers/platform/x86/intel_rapl.c +2 -0
+	if (NULL == rd_data) {
+		pr_err("Failed to allocate memory for rapl domain data\n");

OOM too.

#5 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Jacob Pan
Quoted 48 lines · Jacob Pan

RAPL(Running Average Power Limit) interface provides platform software with the ability to monitor, control, and get notifications on SOC power consumptions. Since its first appearance on Sandy Bridge, more features have being added to extend its usage. In RAPL, platforms are divided into domains for fine grained control. These domains include package, DRAM controller, CPU core (Power Plane 0), graphics uncore (power plane 1), etc.

The purpose of this driver is to expose RAPL for userspace consumption. Overall, RAPL fits in the generic thermal layer in that platform level power capping and monitoring are mainly used for thermal management and thermal layer provides the abstracted interface needed to have portable applications.

Specifically, userspace is presented with per domain cooling device with sysfs links to its kobject. Although RAPL domain provides many parameters for fine tuning, long term power limit is exposed as the single knob via cooling device state. Whereas the rest of the parameters are still accessible via the linked kobject. This simplifies the interface for both simple and advanced use cases.

Eventfd is used to provide notifications to the userspace. At per domain level, use can choose any event capable parameters to register for threshold crossing notifications. This is shamelessly "borrowed" from cgroup with some trimming/fitting.

Zhang, Rui's initial RAPL driver was used as a reference and starting point. Many thanks.
https://lkml.org/lkml/2011/5/26/93

Unlike the patch above, which is mainly for monitoring, this driver focus on the control and usability by user applications.

---
 drivers/platform/x86/Kconfig      |    8 +
 drivers/platform/x86/Makefile     |    1 +
 drivers/platform/x86/intel_rapl.c | 1323 +++++++++++++++++++++++++++++++++++++
 drivers/platform/x86/intel_rapl.h |  249 +++++++
 4 files changed, 1581 insertions(+)
 create mode 100644 drivers/platform/x86/intel_rapl.c
 create mode 100644 drivers/platform/x86/intel_rapl.h
drivers/platform/x86/Kconfig +0 -0
index 3338437..34bcd52 100644

drivers/platform/x86/Kconfig config APPLE_GMUX · line 789

Quoted 9 lines
drivers/platform/x86/Kconfig +6 -0
@@ -781,4 +781,12 @@ config APPLE_GMUX
 	  graphics as well as the backlight. Currently only backlight
 	  control is supported by the driver.
 
+config INTEL_RAPL
+	tristate "Intel RAPL Support"
+	depends on X86 && THERMAL
+	default y
+	---help---
+	  RAPL, AKA, Running Average Power Limit provides mechanisms to enforce

RAPL, aka Running Average Power Limit, provides mechanisms to enforce

or a.k.a., but not AKA

or even:
RAPL (Running Average Power Limit) provides mechanisms to enforce

Quoted 3 lines
drivers/platform/x86/Kconfig +2 -0
+	  and monitor per domain power consumption limits of supported Intel CPUs.
+
 endif # X86_PLATFORM_DEVICES
Signature · 1 line
~Randy
#6 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Jacob Pan
Quoted 34 lines · Jacob Pan

RAPL(Running Average Power Limit) interface provides platform software with the ability to monitor, control, and get notifications on SOC power consumptions. Since its first appearance on Sandy Bridge, more features have being added to extend its usage. In RAPL, platforms are divided into domains for fine grained control. These domains include package, DRAM controller, CPU core (Power Plane 0), graphics uncore (power plane 1), etc.

The purpose of this driver is to expose RAPL for userspace consumption. Overall, RAPL fits in the generic thermal layer in that platform level power capping and monitoring are mainly used for thermal management and thermal layer provides the abstracted interface needed to have portable applications.

Specifically, userspace is presented with per domain cooling device with sysfs links to its kobject. Although RAPL domain provides many parameters for fine tuning, long term power limit is exposed as the single knob via cooling device state. Whereas the rest of the parameters are still accessible via the linked kobject. This simplifies the interface for both simple and advanced use cases.

Eventfd is used to provide notifications to the userspace. At per domain level, use can choose any event capable parameters to register for threshold crossing notifications. This is shamelessly "borrowed" from cgroup with some trimming/fitting.

Zhang, Rui's initial RAPL driver was used as a reference and starting point. Many thanks.
https://lkml.org/lkml/2011/5/26/93

Unlike the patch above, which is mainly for monitoring, this driver focus on the control and usability by user applications.

Quoted 8 lines
---
 drivers/platform/x86/Kconfig      |    8 +
 drivers/platform/x86/Makefile     |    1 +
 drivers/platform/x86/intel_rapl.c | 1323 +++++++++++++++++++++++++++++++++++++
 drivers/platform/x86/intel_rapl.h |  249 +++++++
 4 files changed, 1581 insertions(+)
 create mode 100644 drivers/platform/x86/intel_rapl.c
 create mode 100644 drivers/platform/x86/intel_rapl.h

You are creating new sysfs files, yet you forgot to document them in Documentation/ABI, please do so as part of this patch.

Quoted 13 lines
drivers/platform/x86/intel_rapl.c new file +10 -0
@@ -0,0 +1,1323 @@
+/*
+ *  intel_rapl.c - Intel Running Average Power Limit Driver for MSR based
+ *                 RAPL interface
+ *
+ *  Copyright (c) 2013, Intel Corporation.
+ * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or (at

drivers/platform/x86/intel_rapl.c line 20

Quoted 10 lines
drivers/platform/x86/intel_rapl.c new file +10 -0
+ *  your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful, but
+ *  WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ *  General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License along
+ *  with this program; if not, write to the Free Software Foundation, Inc.,
+ *  59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.

You really want to track the FSF's address for the next 40 years and keep your file up to date? I didn't think so...

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

Kids these days...

Surely one of the internal Intel code reviews caught this when you ran it through that process?

I'm not even going to review the rest of this file, please fix that up first, before sending it out again. And, when you do so, please cc: me.

Oh, I couldn't help myself:

drivers/platform/x86/intel_rapl.c line 909

Quoted 6 lines
drivers/platform/x86/intel_rapl.c new file +6 -0
+static void rapl_domain_kobj_release(struct kobject *kobj)
+{
+	struct rapl_domain *rd = kobj_to_rapl_domain(kobj);
+
+	complete(&rd->kobj_unregister);
+}

No, that's not how you do a release function (hint, you can lock up your driver from userspace by holding your file open...) Free the memory here, where it is supposed to be free.

And, one final complaint, never use "raw" kobjects, for loads of good reasons, not the least being you just prevented userspace from seeing what is happening with your devices. Use the driver model, that's what it is there for, if you need "sub children", or subdirectories.

greg k-h

#7 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 2 lines · Greg KH

I'm not even going to review the rest of this file, please fix that up first, before sending it out again. And, when you do so, please cc: me.

It's like a train wreck, you just can't tear away your eyes...

You race with userspace in the creation of your sysfs files. Your device is announced to userspace and then, later on, you create the sysfs files. This means that tools like libudev never sees your files, so the attributes aren't read properly by loads of userspace programs that you want to have read them.

Please fix that before resending this as well.

greg k-h

author
#8 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Greg KH
Quoted 11 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

I did feel unsure about this but i saw some precedence in the kernel. Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

Signature · 3 lines
Thanks,

Jacob
#9 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 8 lines · Jacob Pan
Quoted 5 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Quoted 7 lines
Quoted 6 lines

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

I did feel unsure about this but i saw some precedence in the kernel.

Someone else is doing this with the sysfs api? I don't see any other code in Linus's tree doing this at the moment, where did you see this? Let me know and I'll fix it up right away.

Quoted 3 lines

Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

What do you mean by this? What exactly are you trying to do? No normal driver code should _ever_ call sysfs functions directly, nor should they ever care about sysfs internals.

And, odds are, you didn't test your code as a module, right, as any internal sysfs function that you could get from this .h file, wouldn't be exported for a module to use, unless I missed one somewhere?

greg k-h

author
#10 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 15 lines · On Tue, 2 Apr 2013 16:48:05 -0700 Greg KH <[email protected]> wrote:
Quoted 14 lines · Jacob Pan
Quoted 11 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

Quoted 7 lines
Quoted 2 lines

I did feel unsure about this but i saw some precedence in the kernel.

Someone else is doing this with the sysfs api? I don't see any other code in Linus's tree doing this at the moment, where did you see this? Let me know and I'll fix it up right away.

no, i did not mean sysfs api. I mean include internal header files via #include ../../
e.g.in drivers/usb/image/microtek.c

#include "../../scsi/scsi.h"
#include <scsi/scsi_host.h>

Quoted 8 lines
Quoted 3 lines

Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

What do you mean by this? What exactly are you trying to do? No normal driver code should _ever_ call sysfs functions directly, nor should they ever care about sysfs internals.

i did not call sysfs internal calls, just need to use struct sysfs_dirent {}

to do the following sanity check against user passed event control file, it is still not a 100% strong check.

	/* check if the cfile belongs to the same rapl domain */
	if (strcmp(rd->kobj.sd->s_name,
			cfile->f_dentry->d_parent->d_name.name)) {
		pr_debug("cfile does not belong to domain %s\n",
			rd->kobj.sd->s_name);
		ret = -EINVAL;
		goto exit_cleanup_fds;
	}
Quoted 4 lines

And, odds are, you didn't test your code as a module, right, as any internal sysfs function that you could get from this .h file, wouldn't be exported for a module to use, unless I missed one somewhere?

I did run the driver as module since i didn't use sysfs internal functions, just the struct. I may be hitting a corner case here, but for drivers who need to discover sysfs hierarchy would it be useful to expose some info in struct sysfs_dirent{}?

Quoted 6 lines

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe platform-driver-x86" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

[Jacob Pan]

Signature · 3 lines
Thanks,

Jacob
#11 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 25 lines · Jacob Pan
Quoted 15 lines · On Tue, 2 Apr 2013 16:48:05 -0700 Greg KH <[email protected]> wrote:
Quoted 14 lines · Jacob Pan
Quoted 11 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

Quoted 7 lines
Quoted 2 lines

I did feel unsure about this but i saw some precedence in the kernel.

Someone else is doing this with the sysfs api? I don't see any other code in Linus's tree doing this at the moment, where did you see this? Let me know and I'll fix it up right away.

Quoted 6 lines

no, i did not mean sysfs api. I mean include internal header files via #include ../../
e.g.in drivers/usb/image/microtek.c

#include "../../scsi/scsi.h"
#include <scsi/scsi_host.h>

That is because this is a scsi host driver. Your code is not part of sysfs itself.

Quoted 13 lines
Quoted 8 lines
Quoted 3 lines

Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

What do you mean by this? What exactly are you trying to do? No normal driver code should _ever_ call sysfs functions directly, nor should they ever care about sysfs internals.

i did not call sysfs internal calls, just need to use struct sysfs_dirent {}

to do the following sanity check against user passed event control file, it is still not a 100% strong check.

Quoted 8 lines
	/* check if the cfile belongs to the same rapl domain */
	if (strcmp(rd->kobj.sd->s_name,
			cfile->f_dentry->d_parent->d_name.name)) {
		pr_debug("cfile does not belong to domain %s\n",
			rd->kobj.sd->s_name);
		ret = -EINVAL;
		goto exit_cleanup_fds;
	}

This made it through a code review at Intel? Seriously? Come on, there's just so much wrong here, I don't know where to begin.

Hint, if you find yourself caring about the internals of sysfs in a device driver, you are doing something so wrong it's not funny. Do you see _any_ other driver doing anything like this? What makes this driver so special that it can do unexpected, and totally different things with sysfs?

Quoted 8 lines
Quoted 4 lines

And, odds are, you didn't test your code as a module, right, as any internal sysfs function that you could get from this .h file, wouldn't be exported for a module to use, unless I missed one somewhere?

I did run the driver as module since i didn't use sysfs internal functions, just the struct. I may be hitting a corner case here, but for drivers who need to discover sysfs hierarchy would it be useful to expose some info in struct sysfs_dirent{}?

No, not at all, why would a driver ever care about that? Somehow we have gotten by for the past 10+ years without needing it, why is your driver so different than the thousands of other Linux drivers?

greg k-h

author
#12 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 59 lines · On Wed, 3 Apr 2013 09:30:52 -0700 Greg KH <[email protected]> wrote:
Quoted 26 lines · Jacob Pan
Quoted 16 lines · On Tue, 2 Apr 2013 16:48:05 -0700 Greg KH <[email protected]> wrote:
Quoted 15 lines · Jacob Pan
Quoted 12 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

drivers/platform/x86/intel_rapl.c line 48

Quoted 2 lines
drivers/platform/x86/intel_rapl.c new file +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

Quoted 7 lines
Quoted 2 lines

I did feel unsure about this but i saw some precedence in the kernel.

Someone else is doing this with the sysfs api? I don't see any other code in Linus's tree doing this at the moment, where did you see this? Let me know and I'll fix it up right away.

Quoted 6 lines

no, i did not mean sysfs api. I mean include internal header files via #include ../../
e.g.in drivers/usb/image/microtek.c

#include "../../scsi/scsi.h"
#include <scsi/scsi_host.h>

That is because this is a scsi host driver. Your code is not part of sysfs itself.

Quoted 13 lines
Quoted 8 lines
Quoted 3 lines

Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

What do you mean by this? What exactly are you trying to do? No normal driver code should _ever_ call sysfs functions directly, nor should they ever care about sysfs internals.

i did not call sysfs internal calls, just need to use struct sysfs_dirent {}

to do the following sanity check against user passed event control file, it is still not a 100% strong check.

Quoted 8 lines
	/* check if the cfile belongs to the same rapl domain */
	if (strcmp(rd->kobj.sd->s_name,
			cfile->f_dentry->d_parent->d_name.name)) {
		pr_debug("cfile does not belong to domain %s\n",
			rd->kobj.sd->s_name);
		ret = -EINVAL;
		goto exit_cleanup_fds;
	}
Quoted 9 lines

This made it through a code review at Intel? Seriously? Come on, there's just so much wrong here, I don't know where to begin.

Hint, if you find yourself caring about the internals of sysfs in a device driver, you are doing something so wrong it's not funny. Do you see _any_ other driver doing anything like this? What makes this driver so special that it can do unexpected, and totally different things with sysfs?

I admit that my knowledge in this area are limited. I appreciate your help and straightforward comments.

Perhaps the reason is that not many drivers use eventfd and its way to arm event thresholds. The userspace passed an eventfd, a file descriptor, and a threshold value to the RAPL driver. In order to authenticate the control file descriptor is a valid, I need to check
1. the control file is capable of generating events, e.g. it can be an energy counter but not a constant power_limit1
2. the control file belongs to the same RAPL domain since the name are reused in all RAPL domains. e.g. all domains have energy counter and events are per domain.

I used the sysfs directory check for #2. It is just an extra check. But I think can drop that check and user pick events based on its name string. The fact that user writes to the per domain control file implies the domain specificness of the event.

In similar eventfd usage in cgroup, it has its own fs so i does the check based on file ops of the control files.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

And, odds are, you didn't test your code as a module, right, as any internal sysfs function that you could get from this .h file, wouldn't be exported for a module to use, unless I missed one somewhere?

I did run the driver as module since i didn't use sysfs internal functions, just the struct. I may be hitting a corner case here, but for drivers who need to discover sysfs hierarchy would it be useful to expose some info in struct sysfs_dirent{}?

Quoted 5 lines

No, not at all, why would a driver ever care about that? Somehow we have gotten by for the past 10+ years without needing it, why is your driver so different than the thousands of other Linux drivers?

greg k-h

ditto, I will drop that extra check.

Signature · 3 lines
Thanks,

Jacob
#13 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 71 lines · Jacob Pan
Quoted 59 lines · On Wed, 3 Apr 2013 09:30:52 -0700 Greg KH <[email protected]> wrote:
Quoted 26 lines · Jacob Pan
Quoted 16 lines · On Tue, 2 Apr 2013 16:48:05 -0700 Greg KH <[email protected]> wrote:
Quoted 15 lines · Jacob Pan
Quoted 12 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

Review

Quoted 2 lines
(quoted diff) +2 -0
+#include "intel_rapl.h"
+#include "../../../fs/sysfs/sysfs.h"  

WTF?

Oh, that's a sure sign you are not doing something properly, if you think it's ok to muck around with the internals of sysfs.

There's a reason that file is "private", why do you think it's ok to use it directly? Did you just think that I somehow "forgot" to put it in the proper include directory?

Quoted 7 lines
Quoted 2 lines

I did feel unsure about this but i saw some precedence in the kernel.

Someone else is doing this with the sysfs api? I don't see any other code in Linus's tree doing this at the moment, where did you see this? Let me know and I'll fix it up right away.

Quoted 6 lines

no, i did not mean sysfs api. I mean include internal header files via #include ../../
e.g.in drivers/usb/image/microtek.c

#include "../../scsi/scsi.h"
#include <scsi/scsi_host.h>

That is because this is a scsi host driver. Your code is not part of sysfs itself.

Quoted 13 lines
Quoted 8 lines
Quoted 3 lines

Anyway, I needed a way to validate a userspace file passed to rapl driver belong to the same sysfs directory. I will look for alternative ways.

What do you mean by this? What exactly are you trying to do? No normal driver code should _ever_ call sysfs functions directly, nor should they ever care about sysfs internals.

i did not call sysfs internal calls, just need to use struct sysfs_dirent {}

to do the following sanity check against user passed event control file, it is still not a 100% strong check.

Quoted 8 lines
	/* check if the cfile belongs to the same rapl domain */
	if (strcmp(rd->kobj.sd->s_name,
			cfile->f_dentry->d_parent->d_name.name)) {
		pr_debug("cfile does not belong to domain %s\n",
			rd->kobj.sd->s_name);
		ret = -EINVAL;
		goto exit_cleanup_fds;
	}
Quoted 9 lines

This made it through a code review at Intel? Seriously? Come on, there's just so much wrong here, I don't know where to begin.

Hint, if you find yourself caring about the internals of sysfs in a device driver, you are doing something so wrong it's not funny. Do you see _any_ other driver doing anything like this? What makes this driver so special that it can do unexpected, and totally different things with sysfs?

Quoted 5 lines

I admit that my knowledge in this area are limited. I appreciate your help and straightforward comments.

Perhaps the reason is that not many drivers use eventfd and its way to arm event thresholds.

sysfs should not be using eventfs, that's the issue here. Don't do that, sysfs has other apis to do this type of thing, if you somehow think it is something that you really need to do (hint, I doubt it...)

greg k-h

author
#14 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Greg KH
Quoted 4 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

And, one final complaint, never use "raw" kobjects, for loads of good reasons, not the least being you just prevented userspace from seeing what is happening with your devices. Use the driver model, that's what it is there for, if you need "sub children", or subdirectories.

I chose to use to kobjects for the reason that userspace can see the device linking more clearly.
Let me try to paraphrase, I have two options:
1. if I use the platform device model instead of raw kobjects, I would have one platform device for each rapl domain. Then link individual platform device with the generic thermal layer sysfs.

2. In the current patch, I have one platform driver, then expose per domain kobject that can be linked to the generic thermal layer. Common attributes of all domains are grouped under the kset.

I did consider both options. I thought using #2 option is better since it allow user to discover the topologies easier by following the sysfs link. If i use use #1, it would be hard to expose the common attributes and more code too. Perhpas I misread Documentation/kobjects.txt which i thought kobject/kset are perfect for presenting situation like this.

Signature · 3 lines
Thanks,

Jacob
#15 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 9 lines · Jacob Pan
Quoted 4 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

And, one final complaint, never use "raw" kobjects, for loads of good reasons, not the least being you just prevented userspace from seeing what is happening with your devices. Use the driver model, that's what it is there for, if you need "sub children", or subdirectories.

I chose to use to kobjects for the reason that userspace can see the device linking more clearly.

Then use a 'struct device'.

Quoted 4 lines

Let me try to paraphrase, I have two options:
1. if I use the platform device model instead of raw kobjects, I would have one platform device for each rapl domain. Then link individual platform device with the generic thermal layer sysfs.

I don't understand.

Quoted 3 lines

2. In the current patch, I have one platform driver, then expose per domain kobject that can be linked to the generic thermal layer. Common attributes of all domains are grouped under the kset.

Then use a 'struct device' that is attached to that kset, no need to use a platform device, as that's not what is needed here, right?

Quoted 6 lines

I did consider both options. I thought using #2 option is better since it allow user to discover the topologies easier by following the sysfs link. If i use use #1, it would be hard to expose the common attributes and more code too. Perhpas I misread Documentation/kobjects.txt which i thought kobject/kset are perfect for presenting situation like this.

No, kobjects are for things not using the device tree, like filesystems or other stuff. You are in the device tree, so use a 'struct device'. If you use a "raw" kobject, you do not notify userspace what is going on, and you loose out on a bunch of other stuff that the driver model provides you.

But do you even need to use anything at all?

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

greg k-h

author
#16 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 41 lines · On Tue, 2 Apr 2013 17:13:00 -0700 Greg KH <[email protected]> wrote:
Quoted 10 lines · Jacob Pan
Quoted 5 lines · On Tue, 2 Apr 2013 16:00:42 -0700 Greg KH <[email protected]> wrote:

And, one final complaint, never use "raw" kobjects, for loads of good reasons, not the least being you just prevented userspace from seeing what is happening with your devices. Use the driver model, that's what it is there for, if you need "sub children", or subdirectories.

I chose to use to kobjects for the reason that userspace can see the device linking more clearly.

Then use a 'struct device'.

Quoted 4 lines

Let me try to paraphrase, I have two options:
1. if I use the platform device model instead of raw kobjects, I would have one platform device for each rapl domain. Then link individual platform device with the generic thermal layer sysfs.

I don't understand.

Quoted 3 lines

2. In the current patch, I have one platform driver, then expose per domain kobject that can be linked to the generic thermal layer. Common attributes of all domains are grouped under the kset.

Then use a 'struct device' that is attached to that kset, no need to use a platform device, as that's not what is needed here, right?

Quoted 6 lines

I did consider both options. I thought using #2 option is better since it allow user to discover the topologies easier by following the sysfs link. If i use use #1, it would be hard to expose the common attributes and more code too. Perhpas I misread Documentation/kobjects.txt which i thought kobject/kset are perfect for presenting situation like this.

No, kobjects are for things not using the device tree, like filesystems or other stuff. You are in the device tree, so use a 'struct device'. If you use a "raw" kobject, you do not notify userspace what is going on, and you loose out on a bunch of other stuff that the driver model provides you.

Quoted 7 lines

But do you even need to use anything at all?

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

greg k-h

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 > cur_state 

For #2, to give userspace complete control of the RAPL interface, which is not generic, I put them under the device private sysfs area.

root@chromoly:/sys/class/thermal/cooling_device15/device# echo 1000 > time_window1 

As you mentioned about using device tree vs. fs, and how kobject are used for fs. I do have the need to go between a generic thermal sysfs and the true device tree. This is the reason why I used kobjects and link them between device tree and its thermal sysfs representation.

e.g. a RAPL package cooling device linked with its platform device kobj. (device is linked with rapl_domains/package, the line is too long)

root@chromoly:/sys/class/thermal# ls -l cooling_device15/
total 0
-rw-r--r-- 1 root root 4096 Apr  2 15:03 cur_state
lrwxrwxrwx 1 root root    0 Apr  2 21:28 device
-> ../../../platform/intel_rapl/rapl_domains/package
-r--r--r-- 1 root root 4096 Apr  2 15:03 max_state
drwxr-xr-x 2 root root    0 Apr  2 21:28 power
lrwxrwxrwx 1 root root    0 Apr  2 15:03 subsystem
-> ../../../../class/thermal
-r--r--r-- 1 root root 4096 Apr  2 15:03 type
-rw-r--r-- 1 root root 4096 Apr  2 15:03 uevent

For userspace which is not satisfied with the simple use case of a single knob for setting power limit, it can follow the link to find the device tree entry. Then get access to the complete knobs, including event notifications.

I wanted to use generic thermal fs in that it gives a simple representation of the most usable feature of RAPL. Basically, RAPL domains can be enumerated by thermal control apps just as ACPI fans or processor cooling.

Signature · 3 lines
Thanks,

Jacob
#17 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 14 lines · Jacob Pan
Quoted 3 lines

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

Quoted 4 lines

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 > cur_state 

Great, how about submitting that functionality as patch 1 of your series? That seems like a very "normal" thermal driver, right?

Quoted 3 lines

For #2, to give userspace complete control of the RAPL interface, which is not generic, I put them under the device private sysfs area.

root@chromoly:/sys/class/thermal/cooling_device15/device# echo 1000 > time_window1 

I totally fail to understand the difference. What do you want to show to userspace that can't be expressed through the thermal interface today? Perhaps the thermal interface could be expanded to provide more functionality that you need? Why create a one-off API that will never be used again and require userspace programs to be written just to handle this one type of device?

Quoted 4 lines

As you mentioned about using device tree vs. fs, and how kobject are used for fs. I do have the need to go between a generic thermal sysfs and the true device tree. This is the reason why I used kobjects and link them between device tree and its thermal sysfs representation.

I don't understand your leap to using kobjects.

Quoted 12 lines

e.g. a RAPL package cooling device linked with its platform device kobj. (device is linked with rapl_domains/package, the line is too long)

root@chromoly:/sys/class/thermal# ls -l cooling_device15/
total 0
-rw-r--r-- 1 root root 4096 Apr  2 15:03 cur_state
lrwxrwxrwx 1 root root    0 Apr  2 21:28 device
-> ../../../platform/intel_rapl/rapl_domains/package
-r--r--r-- 1 root root 4096 Apr  2 15:03 max_state
drwxr-xr-x 2 root root    0 Apr  2 21:28 power
lrwxrwxrwx 1 root root    0 Apr  2 15:03 subsystem
-> ../../../../class/thermal

Review

Quoted 2 lines
(quoted diff) +0 -2
-r--r--r-- 1 root root 4096 Apr  2 15:03 type
-rw-r--r-- 1 root root 4096 Apr  2 15:03 uevent

I still don't understand. What are you adding here, the device symlink? Or something else?

Quoted 4 lines

For userspace which is not satisfied with the simple use case of a single knob for setting power limit, it can follow the link to find the device tree entry. Then get access to the complete knobs, including event notifications.

And what is in that device directory? What is rapl_domains? Why isn't that a normal 'struct device'?

Still confused.

greg k-h

author
#18 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 15 lines · On Wed, 3 Apr 2013 09:35:09 -0700 Greg KH <[email protected]> wrote:
Quoted 14 lines · Jacob Pan
Quoted 3 lines

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

Quoted 9 lines
Quoted 5 lines

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 >
cur_state 

Great, how about submitting that functionality as patch 1 of your series? That seems like a very "normal" thermal driver, right?

yes, that would be a normal thermal cooling device driver. I will do that first. Thanks for the suggestion.

Quoted 8 lines
Quoted 4 lines

For #2, to give userspace complete control of the RAPL interface, which is not generic, I put them under the device private sysfs area. root@chromoly:/sys/class/thermal/cooling_device15/device# echo 1000 > time_window1

I totally fail to understand the difference. What do you want to show to userspace that can't be expressed through the thermal interface today?

The difference is one single knob (long term power limit) in the thermal interface vs multiple fine grained control and data in the complete RAPL interface.

Here is what a complete RAPL interface for package domain looks like.

root@chromoly:/sys/class/thermal/cooling_device15/device# grep . *
domain_name:package
energy:22396031
lock:0
max_power:0
max_window:0
min_power:0
pl1_clamp:0
pl1_enable:1
pl2_clamp:0
pl2_enable:1
power:7841
power_limit1:25000
power_limit2:31250
thermal_spec_power:17000
throttle_time: 
time_window1:28000
time_window2:0
Quoted 2 lines

Perhaps the thermal interface could be expanded to provide
more functionality that you need?

yes, some of them such as limits. But not all the data in the list above are suitable for thermal interface. That is why I am trying to balance between abstracted generic data and RAPL specific data while still allow linking between the two.

The way I envisioned how a thermal/power management app would use is:
1. go through generic thermal layer sysfs and find available RAPL domains
2. if the app wants to do more fine grained control, it follows the device symlink to locate the RAPL domain specific sysfs area.

Quoted 4 lines

Why create a one-off API that will
never be used again and require userspace programs to be written just to handle this one type of device?

why is that a one-off API? RAPL interface is maintained identical across Intel CPUs after Sandy Bridges. I agree with you that it is still one type of device with some of its data unique. Should i create a RAPL class device?

Quoted 8 lines
Quoted 5 lines

As you mentioned about using device tree vs. fs, and how kobject are used for fs. I do have the need to go between a generic thermal sysfs and the true device tree. This is the reason why I used kobjects and link them between device tree and its thermal sysfs representation.

I don't understand your leap to using kobjects.

I use kobjects mainly for its symlink to allow userspace locate the 'true' device behind generic thermal layers' cooling device.

Quoted 19 lines
Quoted 13 lines

e.g. a RAPL package cooling device linked with its platform device kobj. (device is linked with rapl_domains/package, the line is too long)

root@chromoly:/sys/class/thermal# ls -l cooling_device15/
total 0
-rw-r--r-- 1 root root 4096 Apr  2 15:03 cur_state
lrwxrwxrwx 1 root root    0 Apr  2 21:28 device
-> ../../../platform/intel_rapl/rapl_domains/package
-r--r--r-- 1 root root 4096 Apr  2 15:03 max_state
drwxr-xr-x 2 root root    0 Apr  2 21:28 power
lrwxrwxrwx 1 root root    0 Apr  2 15:03 subsystem
-> ../../../../class/thermal

Review

Quoted 2 lines
(quoted diff) +0 -2
-r--r--r-- 1 root root 4096 Apr  2 15:03 type
-rw-r--r-- 1 root root 4096 Apr  2 15:03 uevent

I still don't understand. What are you adding here, the device symlink? Or something else?

Quoted 6 lines
Quoted 4 lines

For userspace which is not satisfied with the simple use case of a single knob for setting power limit, it can follow the link to find the device tree entry. Then get access to the complete knobs, including event notifications.

And what is in that device directory?

the device directory contains the complete RAPL interface representation. paste the example of package domain again.

root@chromoly:/sys/class/thermal/cooling_device15/device# grep . *
domain_name:package
energy:22396031
lock:0
max_power:0
max_window:0
min_power:0
pl1_clamp:0
pl1_enable:1
pl2_clamp:0
pl2_enable:1
power:7841
power_limit1:25000
power_limit2:31250
thermal_spec_power:17000
grep: throttle_time: Input/output error
time_window1:28000
time_window2:0
Quoted 3 lines

What is rapl_domains? Why
isn't that a normal 'struct device'?

RAPL domains can be viewed as sub devices of RAPL interface. On a given platform they can be a complete or partial list of
package, power plane 0 (processor core), power plane 1 (graphics), dram controller, pch, etc.

Yes, I can create a RAPL class and create_device for each domain based on 'struct device'. I can still use the kobj symlink to link to generic thermal layer. I just thought it is an overkill as compared to simply using kobjects and kset.

I don't understand the benefit of using 'struct device' in this case. RAPL interface as a whole has its device, I am not creating standalone kobjects.

Quoted 3 lines

Still confused.

greg k-h

[Jacob Pan]

Signature · 3 lines
Thanks,

Jacob
#19 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 24 lines · Jacob Pan
Quoted 21 lines · On Wed, 3 Apr 2013 09:35:09 -0700 Greg KH <[email protected]> wrote:
Quoted 19 lines · Jacob Pan
Quoted 3 lines

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 >
cur_state 
Quoted 5 lines
Quoted 3 lines

Great, how about submitting that functionality as patch 1 of your series? That seems like a very "normal" thermal driver, right?

yes, that would be a normal thermal cooling device driver. I will do that first. Thanks for the suggestion.

Do that first, get it merged, and then let's work on the second part. The patch for that will be much more obvious as to what you are attempting to do.

Quoted 6 lines
Quoted 2 lines

Perhaps the thermal interface could be expanded to provide
more functionality that you need?

yes, some of them such as limits. But not all the data in the list above are suitable for thermal interface. That is why I am trying to balance between abstracted generic data and RAPL specific data while still allow linking between the two.

What is not in the existing interface? And as this is a thermal device, why can't you add them there?

Quoted 5 lines

The way I envisioned how a thermal/power management app would use is:
1. go through generic thermal layer sysfs and find available RAPL domains
2. if the app wants to do more fine grained control, it follows the device symlink to locate the RAPL domain specific sysfs area.

So any application will have to know all of the device-specific attributes? That totally defeats the purpose of a generic api that the kernel is providing. You are creating device-specific apis that will not work over the long-run (i.e. next 5-10 years.) Please don't do that unless you have exhausted _all_ other alternatives.

So, get your first driver accepted, using the in-kernel thermal api, and then, if you still feel you wish to do device-specific extensions, we can discuss that then.

greg k-h

author
#20 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 28 lines · On Fri, 5 Apr 2013 13:23:09 -0700 Greg KH <[email protected]> wrote:
Quoted 27 lines · Jacob Pan
Quoted 24 lines · On Wed, 3 Apr 2013 09:35:09 -0700 Greg KH <[email protected]> wrote:
Quoted 14 lines · Jacob Pan
Quoted 3 lines

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

Quoted 5 lines

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 >
cur_state 

Great, how about submitting that functionality as patch 1 of your series? That seems like a very "normal" thermal driver, right?

Quoted 7 lines
Quoted 2 lines

yes, that would be a normal thermal cooling device driver. I will do that first. Thanks for the suggestion.

Do that first, get it merged, and then let's work on the second part. The patch for that will be much more obvious as to what you are attempting to do.

Sorry I was too busy to work on v2 before seeing this. I agree I need to simplify the interface, I just need to come up with a more intelligent way to abstract that and do the best guesses for the user. Hopefully, v2 will serve as a confirmation on the comments I got from v1. i.e. kobject->struct device, removed dependencies on sysfs internal data struct, etc.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Perhaps the thermal interface could be expanded to provide
more functionality that you need?

yes, some of them such as limits. But not all the data in the list above are suitable for thermal interface. That is why I am trying to balance between abstracted generic data and RAPL specific data while still allow linking between the two.

What is not in the existing interface? And as this is a thermal device, why can't you add them there?

existing interface has only cur_state and max_state, I have been working with Rui (thermal maintainer) on adding more knobs for cooling devices, such as limit low, limit high, event control. I believe they can all be added.

What is not appropriate for thermal interface are the things like energy counters, accumulated throttle time, time constants that used by the internal control algorithm.

Quoted 7 lines
Quoted 6 lines

The way I envisioned how a thermal/power management app would use
is: 1. go through generic thermal layer sysfs and find available RAPL domains
2. if the app wants to do more fine grained control, it follows the device symlink to locate the RAPL domain specific sysfs area.

Quoted 6 lines

So any application will have to know all of the device-specific attributes? That totally defeats the purpose of a generic api that the kernel is providing. You are creating device-specific apis that will not work over the long-run (i.e. next 5-10 years.) Please don't do that unless you have exhausted _all_ other alternatives.

I agree we should improve the abstraction to avoid using device specific attributes. Unfortunately, on the other side power tuning tends to be very platform specific.

Quoted 4 lines

So, get your first driver accepted, using the in-kernel thermal api, and then, if you still feel you wish to do device-specific extensions, we can discuss that then.

Agreed. I just want to make sure the rapl class and per domain 'struct device' layout in v2 is what you suggested. I avoided using raw kobjects and sysfs internals.

The challenge for the simplified/abstract model is that the driver would have to guess what is best for the user.
e.g. when user selects power limit of 8200mw for the core power plane. Besides setting that limit, the driver will decide the following for the user:
- dynamics of the control, rise time, overshoot, steady state error
- whether or not allow P/T states to go below OS request value
- correlation with instantaneous power limit

I believe it is all doable, just need to strike a balance and for different platforms. But I do believe it is a better interface for most generic applications.

Quoted 6 lines

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe platform-driver-x86" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

[Jacob Pan]

Signature · 3 lines
Thanks,

Jacob
#21 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

Quoted 38 lines · Jacob Pan
Quoted 28 lines · On Fri, 5 Apr 2013 13:23:09 -0700 Greg KH <[email protected]> wrote:
Quoted 27 lines · Jacob Pan
Quoted 24 lines · On Wed, 3 Apr 2013 09:35:09 -0700 Greg KH <[email protected]> wrote:
Quoted 14 lines · Jacob Pan
Quoted 3 lines

Let's step back and start over, what exactly are you trying to tell userspace? What data do you have that you need to express to it? How do you want userspace to see/use it?

It is a good idea to step back and let me explain what I wanted to do here for userspace.

I have two kinds of applications that might use this driver.
1. simple use case where user sets a power limit for a RAPL domain. e.g. set graphics unit power limit to 7w
2. advanced use case where use can do fine tuning on top of simple power limit,e.g. the dynamic response parameters of power control logic, event notifications, etc.

Quoted 5 lines

For #1, this driver register with the abstract generic thermal layer (/sys/class/thermal) and presents itself as a set of cooling devices with a single knob per domain for power limits.

root@chromoly:/sys/class/thermal/cooling_device15# echo 7000 >
cur_state 

Great, how about submitting that functionality as patch 1 of your series? That seems like a very "normal" thermal driver, right?

Quoted 7 lines
Quoted 2 lines

yes, that would be a normal thermal cooling device driver. I will do that first. Thanks for the suggestion.

Do that first, get it merged, and then let's work on the second part. The patch for that will be much more obvious as to what you are attempting to do.

Quoted 6 lines

Sorry I was too busy to work on v2 before seeing this. I agree I need to simplify the interface, I just need to come up with a more intelligent way to abstract that and do the best guesses for the user. Hopefully, v2 will serve as a confirmation on the comments I got from v1. i.e. kobject->struct device, removed dependencies on sysfs internal data struct, etc.

I'm not going to review that part of the code, sorry, as it's about to be ripped out anyway :)

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Perhaps the thermal interface could be expanded to provide
more functionality that you need?

yes, some of them such as limits. But not all the data in the list above are suitable for thermal interface. That is why I am trying to balance between abstracted generic data and RAPL specific data while still allow linking between the two.

Quoted 6 lines
Quoted 2 lines

What is not in the existing interface? And as this is a thermal device, why can't you add them there?

existing interface has only cur_state and max_state, I have been working with Rui (thermal maintainer) on adding more knobs for cooling devices, such as limit low, limit high, event control. I believe they can all be added.

Great, then you will not need any of the driver-specific sysfs files, struct device usage, or kobject mess, so your code should be a lot smaller.

greg k-h

#22 lore raw

Re: [PATCH 1/1] Introduce Intel RAPL cooling device driver

In reply to Jacob Pan

drivers/platform/x86/Kconfig config APPLE_GMUX · line 786

Quoted 9 lines · Jacob Pan
drivers/platform/x86/Kconfig +3 -0
@@ -781,4 +781,12 @@ config APPLE_GMUX
 	  graphics as well as the backlight. Currently only backlight
 	  control is supported by the driver.
 
+config INTEL_RAPL
+	tristate "Intel RAPL Support"
+	depends on X86 && THERMAL

drivers/platform/x86/Kconfig is sourced from drivers/platform/Kconfig. The relevant line is wrapped in an "if X86" and "endif" pair. So X86 is superfluous here.
"

drivers/platform/x86/Kconfig config APPLE_GMUX · line 792

Quoted 6 lines
drivers/platform/x86/Kconfig +5 -0
+	default y
+	---help---
+	  RAPL, AKA, Running Average Power Limit provides mechanisms to enforce
+	  and monitor per domain power consumption limits of supported Intel CPUs.
+
 endif # X86_PLATFORM_DEVICES

Paul Bolle