pm qos infrastructure and interface

The following patch is a generalization of the latency.c implementation done by Arjan last year. It provides infrastructure for more than one parameter, and exposes a user mode interface for processes to register pm_qos expectations of processes.

This interface provides a kernel and user mode interface for registering performance expectations by drivers, subsystems and user space applications on one of the parameters.

Currently we have {cpu_dma_latency, network_latency, network_throughput} as the initial set of pm_qos parameters.

The infrastructure exposes multiple misc device nodes one per implemented parameter. The set of parameters implement is defined by pm_qos_power_init() and pm_qos_params.h. This is done because having the available parameters being runtime configurable or changeable from a driver was seen as too easy to abuse.

For each parameter a list of performance requirements is maintained along with an aggregated target value. The aggregated target value is updated with changes to the requirement list or elements of the list. Typically the aggregated target value is simply the max or min of the requirement values held in the parameter list elements.

pm_qos_add_requirement(param_id, name, target_value):
Will insert a named element in the list for that identified PM_QOS parameter with the target value. Upon change to this list the new target is recomputed and any registered notifiers are called only if the target value is now different.

pm_qos_update_requirement(param_id, name, new_target_value):
Will search the list identified by the param_id for the named list element and then update its target value, calling the notification tree if the aggregated target is changed. with that name is already registered.

pm_qos_remove_requirement(param_id, name):
Will search the identified list for the named element and remove it, after removal it will update the aggregate target and call the notification tree if the target was changed as a result of removing the named requirement.

Only processes can register a pm_qos requirement. To provide for automatic cleanup for process the interface requires the process to register its parameter requirements in the following way:

To register the default pm_qos target for the specific parameter, the process must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]

As long as the device node is held open that process has a registered requirement on the parameter. The name of the requirement is "process_<PID>" derived from the current->pid from within the open system call.

To change the requested target value the process needs to write a s32 value to the open device node. This translates to a pm_qos_update_requirement call.

To remove the user mode request for a target value simply close the device node.

--mgross

-----------

diff -urN -X linux-2.6.23-rc8/Documentation/dontdiff linux-2.6.23-rc8/Documentation/pm_qos_interface.txt linux-2.6.23-rc8-qos/Documentation/pm_qos_interface.txt
@@ -0,0 +1,59 @@
+PM quality of Service interface.
+
+This interface provides a kernel and user mode interface for registering
+performance expectations by drivers, subsystems and user space applications on
+one of the parameters.
+
+Currently we have {cpu_dma_latency, network_latency, network_throughput} as the
+initial set of pm_qos parameters.
+
+The infrastructure exposes multiple misc device nodes one per implemented
+parameter.  The set of parameters implement is defined by pm_qos_power_init()
+and pm_qos_params.h.  This is done because having the available parameters
+being runtime configurable or changeable from a driver was seen as too easy to
+abuse.
+
+For each parameter a list of performance requirements is maintained along with
+an aggregated target value.  The aggregated target value is updated with
+changes to the requirement list or elements of the list.  Typically the
+aggregated target value is simply the max or min of the requirement values held
+in the parameter list elements.
+
+From kernel mode the use of this interface is simple:
+pm_qos_add_requirement(param_id, name, target_value):
+Will insert a named element in the list for that identified PM_QOS parameter
+with the target value.  Upon change to this list the new target is recomputed
+and any registered notifiers are called only if the target value is now
+different.
+
+pm_qos_update_requirement(param_id, name, new_target_value):
+Will search the list identified by the param_id for the named list element and
+then update its target value, calling the notification tree if the aggregated
+target is changed.  with that name is already registered.
+
+pm_qos_remove_requirement(param_id, name):
+Will search the identified list for the named element and remove it, after
+removal it will update the aggregate target and call the notification tree if
+the target was changed as a result of removing the named requirement.
+
+
+From user mode:
+Only processes can register a pm_qos requirement.  To provide for automatic
+cleanup for process the interface requires the process to register its
+parameter requirements in the following way:
+
+To register the default pm_qos target for the specific parameter, the process
+must open one of /dev/[cpu_dma_latency, network_latency, network_throughput]
+
+As long as the device node is held open that process has a registered
+requirement on the parameter.  The name of the requirement is "process_<PID>"
+derived from the current->pid from within the open system call.
+
+To change the requested target value the process needs to write a s32 value to
+the open device node.  This translates to a pm_qos_update_requirement call.
+
+To remove the user mode request for a target value simply close the device
+node.
+
+
+

diff -urN -X linux-2.6.23-rc8/Documentation/dontdiff linux-2.6.23-rc8/include/linux/pm_qos_params.h linux-2.6.23-rc8-qos/include/linux/pm_qos_params.h
@@ -0,0 +1,35 @@
+/* interface for the pm_qos_power infrastructure of the linux kernel.
+ *
+ * Mark Gross
+ */
+#include <linux/list.h>
+#include <linux/notifier.h>
+#include <linux/miscdevice.h>
+
+struct requirement_list {
+	struct list_head list;
+	union {
+		s32 value;
+		s32 usec;
+		s32 kbps;
+	};
+	char *name;
+};
+
+#define PM_QOS_RESERVED 0
+#define PM_QOS_CPU_DMA_LATENCY 1
+#define PM_QOS_NETWORK_LATENCY 2
+#define PM_QOS_NETWORK_THROUGHPUT 3
+
+#define PM_QOS_NUM_CLASSES 4
+#define PM_QOS_DEFAULT_VALUE -1
+
+int pm_qos_add_requirement(int qos, char *name, s32 value);
+int pm_qos_update_requirement(int qos, char *name, s32 new_value);
+void pm_qos_remove_requirement(int qos, char *name);
+
+int pm_qos_requirement(int qos);
+
+int pm_qos_add_notifier(int qos, struct notifier_block *notifier);
+int pm_qos_remove_notifier(int qos, struct notifier_block *notifier);
+

diff -urN -X linux-2.6.23-rc8/Documentation/dontdiff linux-2.6.23-rc8/kernel/Makefile linux-2.6.23-rc8-qos/kernel/Makefile
@@ -9,7 +9,7 @@
 	    rcupdate.o extable.o params.o posix-timers.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o latency.o nsproxy.o srcu.o die_notifier.o \
-	    utsname.o
+	    utsname.o pm_qos_params.o
 
 obj-$(CONFIG_STACKTRACE) += stacktrace.o
 obj-y += time/

I'm a bit surprised that this change appears to have no configurability. If one has set CONFIG_PM=n (for example), shouldn't it all go away?

I'm a bit surprised that this change appears to have no configurability. If one has set CONFIG_PM=n (for example), shouldn't it all go away?

We considered that as an option but as latency.c didn't offer it I didn't either.

I could see the user mode interface portion of the implementation be made as a compile time option but the kernel infrastructure will continue to be needed by at least cpu-idel, pcm_native.c and ipw2100.

You know it could make sense to have the user mode interface part of the patch as configurable or a build time dependent of sysfs and misc device support for the linux-tiny guys. Is it practical to make a linux-tiny without the sysfs infrastructure needed to make a misc device? I'll ask on the linux-tiny list.

--mgross

We considered that as an option but as latency.c didn't offer it I didn't either.

I could see the user mode interface portion of the implementation be made as a compile time option but the kernel infrastructure will continue to be needed by at least cpu-idel, pcm_native.c and ipw2100.

You know it could make sense to have the user mode interface part of the patch as configurable or a build time dependent of sysfs and misc device support for the linux-tiny guys. Is it practical to make a linux-tiny without the sysfs infrastructure needed to make a misc device? I'll ask on the linux-tiny list.

--mgross

I suppose it's a matter of taste, but at least personally I believe that userspace exposed ABI's shouldn't be configurable as a starting point (if the code delta is 100's of kilobytes it's a different topic); what point is talking about a stable ABI if it's there or not half the time... that's the ultimate instability.

I suppose it's a matter of taste, but at least personally I believe that userspace exposed ABI's shouldn't be configurable as a starting point (if the code delta is 100's of kilobytes it's a different topic); what point is talking about a stable ABI if it's there or not half the time... that's the ultimate instability.

Hi Mark,

What about cpu_throughput{_min,_max}, as being something considered to be proportional to the CPU frequency? This way, the cpufreq policy notifiers might be able to utilize the pm_qos infrastructure; but maybe even also the userspace interface (at least the min freq/max freq one)... Haven't thought this through, but maybe you (or someone else) has.

Best,
Dominik

PS: Each time I've tried to get back into business during the past months, something got in between -- current issue is my terminally broken notebook...

Its good to hear from you. Will you be at the ELC conference in Linz next week?

I've only thought it though enough to choose to avoid cpufreq interactions.

Sadly core frequency is not proportional to throughput on X86 processors. I don't know how one would reliably quantify cpu throughput in this context, other than defining latencies.

I could see something like this to prevent cpufreq throttling at bad times, but how common of an issue is this any more?

Are we wasting watts without such information? Is CPUFREQ still making some applications behave poorly? Ondemand and cpufreq are fairly mature now and I don't hear of such issues any more.

--mgross

Hi Mark,

Well it's not exactly throughput, but the CPU frequency surely has an influence on it and also affects the quality of the service provided...

Hopefully none :) I was just wondering whether this generalization would make sense in the big scheme of things (i.e. grand plan of unified power management)...

Unfortunately not.

Best,
Dominik

This is true. I just worry going down this path would be a rat-hole.

FWIW I suppose one could use define bogo-mips as the throughput parameter and use that as away for a CPUFREQ driver to use a way to constrain the throttling. This way if an application that knows it needs higher throughputs and can't deal with the latencies of the governor changing core voltage and frequencies, it could register itself to PM_QOS.

In theory this could open a window to more aggressive governor policies
for saving power. I can see how that could work well for specific
workloads but not so much for desktop / general purpose workloads..

Maybe you are right.

Is there interest in anyone creating a new, or enhanced CPUFREQ governor that takes advantage of a PM_QOS bogo-mips throughput parameter to limit how low of a P-state to drop into?

If so I'd be happy to work with you to see what we can accomplish.

--mgross