Skip to contentSkip to conversation

cpuidle : Add debugfs support for cpuidle core

RFC In review Abhishek Goel 3 participants 5 messages Dec 17, 2019 – Dec 20, 2019 CPU idle time management framework

lore
Share & tools
lore
#0 lore raw

RFC 1/1 cpuidle : Add debugfs support for cpuidle core

Up until now, we did not have a way to tune cpuidle attribute like residency in kernel. This patch adds support for debugfs in cpuidle core. Thereby providing support for tuning cpuidle attributes like residency in kernel at runtime.
For example: Tuning residency at runtime can be used to quantify governors decision making as governor uses residency as one of the parameter to take decision about the state that needs to be entered while idling.

Currently, Only residency have been added in debugfs framework. We can later on add attributes like latency too.

---
 drivers/cpuidle/Kconfig           |   8 ++
 drivers/cpuidle/Makefile          |   1 +
 drivers/cpuidle/cpuidle-debugfs.c | 188 ++++++++++++++++++++++++++++++
 drivers/cpuidle/cpuidle.c         |  19 ++-
 drivers/cpuidle/cpuidle.h         |  24 ++++
 include/linux/cpuidle.h           |   3 +
 6 files changed, 241 insertions(+), 2 deletions(-)
 create mode 100644 drivers/cpuidle/cpuidle-debugfs.c
drivers/cpuidle/Kconfig +8 -0
index 88727b7c0d59..630a5a90744b 100644
@@ -71,6 +71,14 @@ config HALTPOLL_CPUIDLE
          before halting in the guest (more efficient than polling in the
          host via halt_poll_ns for some scenarios).
 
+config CPUIDLE_DEBUG_FS
+	bool "cpuidle state debugging information in debugfs"
+	default y
+	depends on DEBUG_FS
+	help
+	Include cpuidle state debugging information in debugfs. This is mostly
+	useful for kernel developers, but it does not incur any cost at runtime.
+
 endif
 
 config ARCH_NEEDS_CPU_IDLE_COUPLED
drivers/cpuidle/Makefile +1 -0
index ee70d5cc5b99..8b5a3f4fe7ec 100644
@@ -8,6 +8,7 @@ obj-$(CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED) += coupled.o
 obj-$(CONFIG_DT_IDLE_STATES)		  += dt_idle_states.o
 obj-$(CONFIG_ARCH_HAS_CPU_RELAX)	  += poll_state.o
 obj-$(CONFIG_HALTPOLL_CPUIDLE)		  += cpuidle-haltpoll.o
+obj-$(CONFIG_CPUIDLE_DEBUG_FS)		  += cpuidle-debugfs.o
 
 ##################################################################################
 # ARM SoC drivers
drivers/cpuidle/cpuidle-debugfs.c new file +188 -0

188 lines not on this page · raw message

drivers/cpuidle/cpuidle.c +17 -2
index 569dbac443bd..33d7fb25d879 100644
@@ -23,6 +23,7 @@
 #include <linux/suspend.h>
 #include <linux/tick.h>
 #include <trace/events/power.h>
+#include <linux/debugfs.h>
 
 #include "cpuidle.h"
 
@@ -32,6 +33,10 @@ DEFINE_PER_CPU(struct cpuidle_device, cpuidle_dev);
 DEFINE_MUTEX(cpuidle_lock);
 LIST_HEAD(cpuidle_detected_devices);
 
+#ifdef CONFIG_DEBUG_FS
+struct dentry *cpuidle_debugfs_root;
+#endif
+
 static int enabled_devices;
 static int off __read_mostly;
 static int initialized __read_mostly;
@@ -666,9 +671,12 @@ EXPORT_SYMBOL_GPL(cpuidle_unregister_device);
  */
 void cpuidle_unregister(struct cpuidle_driver *drv)
 {
-	int cpu;
+	int cpu, i;
 	struct cpuidle_device *device;
 
+	for (i = 0; i < drv->state_count; i++)
+		cpuidle_state_debugfs_unregister(i);
+
 	for_each_cpu(cpu, drv->cpumask) {
 		device = &per_cpu(cpuidle_dev, cpu);
 		cpuidle_unregister_device(device);
@@ -692,7 +700,7 @@ EXPORT_SYMBOL_GPL(cpuidle_unregister);
 int cpuidle_register(struct cpuidle_driver *drv,
 		     const struct cpumask *const coupled_cpus)
 {
-	int ret, cpu;
+	int ret, cpu, i;
 	struct cpuidle_device *device;
 
 	ret = cpuidle_register_driver(drv);
@@ -724,6 +732,9 @@ int cpuidle_register(struct cpuidle_driver *drv,
 		break;
 	}
 
+	for (i = 0; i < drv->state_count; i++)
+		cpuidle_state_debugfs_register(i);
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(cpuidle_register);
@@ -774,6 +785,10 @@ static int __init cpuidle_init(void)
 
 	latency_notifier_init(&cpuidle_latency_notifier);
 
+#ifdef CONFIG_DEBUG_FS
+	cpuidle_debugfs_root = debugfs_create_dir("cpuidle", NULL);
+#endif
+
 	return 0;
 }
 
drivers/cpuidle/cpuidle.h +24 -0
index 9f336af17fa6..781a9d0f4b43 100644
@@ -37,6 +37,30 @@ extern void cpuidle_remove_device_sysfs(struct cpuidle_device *device);
 extern int cpuidle_add_sysfs(struct cpuidle_device *dev);
 extern void cpuidle_remove_sysfs(struct cpuidle_device *dev);
 
+#ifdef CONFIG_CPUIDLE_DEBUG_FS
+
+#include <linux/seq_file.h>
+
+extern struct dentry *cpuidle_debugfs_root;
+
+struct cpuidle_debugfs_attr {
+	const char *name;
+	umode_t mode;
+	int (*show)(void *, struct seq_file *);
+	ssize_t (*write)(void *, const char __user *, size_t, loff_t *);
+};
+
+extern void cpuidle_state_debugfs_register(int index);
+extern void cpuidle_state_debugfs_unregister(int index);
+#else
+void cpuidle_state_debugfs_register(int index)
+{
+}
+void cpuidle_state_debugfs_unregister(int index)
+{
+}
+#endif
+
 #ifdef CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED
 bool cpuidle_state_is_coupled(struct cpuidle_driver *drv, int state);
 int cpuidle_coupled_state_verify(struct cpuidle_driver *drv);
include/linux/cpuidle.h +3 -0
index 2dbe46b7c213..fc4f2d7d7419 100644
@@ -70,6 +70,9 @@ struct cpuidle_state {
 	void (*enter_s2idle) (struct cpuidle_device *dev,
 			      struct cpuidle_driver *drv,
 			      int index);
+#ifdef CONFIG_CPUIDLE_DEBUG_FS
+	struct dentry	*debugfs_dir;
+#endif
 };
 
 /* Idle State Flags */
Signature · 1 line
2.17.1
#1 lore raw
Quoted 5 lines · Abhishek Goel

Up until now, we did not have a way to tune cpuidle attribute like residency in kernel. This patch adds support for debugfs in cpuidle core. Thereby providing support for tuning cpuidle attributes like residency in kernel at runtime.

This is not a good idea in my view, for a couple of reasons.

First off, if the target residency of an idle state is changed, it effectively becomes a different one and all of the statistics regarding it become outdated at that point. Synchronizing that would be a pain.

Next, governors may get confused if idle state parameters are changed on the fly. In particular, the statistics collected by the teo governor depend on the target residencies of idle states, so if one of them changes, the governor needs to be reloaded.

Next, idle states are expected to be ordered by the target residency (and by the exit latency), so their parameters cannot be allowed to change freely anyway.

Finally, the idle state parameters are expected to reflect the properties of the hardware, which wouldn't hold any more if they were allowed to change at any time.

Quoted 3 lines

For example: Tuning residency at runtime can be used to quantify governors decision making as governor uses residency as one of the parameter to take decision about the state that needs to be entered while idling.

IMO it would be better to introduce a testing cpuidle driver with an artificial set of idle states (or even such that the set of idle states to be used by it can be defined by the user e.g. via module parameters) for this purpose.

#2 lore raw
Quoted 21 lines · Rafael J. Wysocki
Quoted 5 lines · Abhishek Goel

Up until now, we did not have a way to tune cpuidle attribute like residency in kernel. This patch adds support for debugfs in cpuidle core. Thereby providing support for tuning cpuidle attributes like residency in kernel at runtime.

This is not a good idea in my view, for a couple of reasons.

First off, if the target residency of an idle state is changed, it effectively becomes a different one and all of the statistics regarding it become outdated at that point. Synchronizing that would be a pain.

Next, governors may get confused if idle state parameters are changed on the fly. In particular, the statistics collected by the teo governor depend on the target residencies of idle states, so if one of them changes, the governor needs to be reloaded.

Quoted 7 lines

Next, idle states are expected to be ordered by the target residency (and by the exit latency), so their parameters cannot be allowed to change freely anyway.

Finally, the idle state parameters are expected to reflect the properties of the hardware, which wouldn't hold any more if they were allowed to change at any time.

Certainly does sound like a headache.

Quoted 8 lines
Quoted 3 lines

For example: Tuning residency at runtime can be used to quantify governors decision making as governor uses residency as one of the parameter to take decision about the state that needs to be entered while idling.

IMO it would be better to introduce a testing cpuidle driver with an artificial set of idle states (or even such that the set of idle states to be used by it can be defined by the user e.g. via module parameters) for this purpose.

The motivation for this patch isn't really a desire to test / tune the governor. It's intended to allow working around a performance problem caused by using high-latency idle states on some interrupt heavy GPU workload. The interrupts occur around ~30ms apart which is long enough for the governor to put the CPU into the deeper states and over the course of long job the additional wakeup latency adds up. The initial fix someone came up with was cooking the residency values so the high-latency states had a residency of +50ms to prevent the govenor from using them. However, that fix is supposed to go into a bit of firmware I maintain and I'm not terribly happy with the idea. I'm fairly sure that ~30ms value is workload dependent and personally I don't think firmware should be making up numbers to trick specific kernel versions into doing specific things.

My impression is the right solution is to have the GPU driver set a PM QoS constraint on the CPUs receiving interrupts while a job is on-going. However, interrupt latency sensitivity isn't something that's unique to GPUs so I'm wondering it it makes sense to have the governor factor in interrupt traffic when deciding what state to use. Is that something that's been tried before?

Oliver

#3 lore raw
Quoted 56 lines · Oliver O'Halloran
Quoted 21 lines · Rafael J. Wysocki
Quoted 5 lines · Abhishek Goel

Up until now, we did not have a way to tune cpuidle attribute like residency in kernel. This patch adds support for debugfs in cpuidle core. Thereby providing support for tuning cpuidle attributes like residency in kernel at runtime.

This is not a good idea in my view, for a couple of reasons.

First off, if the target residency of an idle state is changed, it effectively becomes a different one and all of the statistics regarding it become outdated at that point. Synchronizing that would be a pain.

Next, governors may get confused if idle state parameters are changed on the fly. In particular, the statistics collected by the teo governor depend on the target residencies of idle states, so if one of them changes, the governor needs to be reloaded.

Quoted 7 lines

Next, idle states are expected to be ordered by the target residency (and by the exit latency), so their parameters cannot be allowed to change freely anyway.

Finally, the idle state parameters are expected to reflect the properties of the hardware, which wouldn't hold any more if they were allowed to change at any time.

Certainly does sound like a headache.

Quoted 8 lines
Quoted 3 lines

For example: Tuning residency at runtime can be used to quantify governors decision making as governor uses residency as one of the parameter to take decision about the state that needs to be entered while idling.

IMO it would be better to introduce a testing cpuidle driver with an artificial set of idle states (or even such that the set of idle states to be used by it can be defined by the user e.g. via module parameters) for this purpose.

The motivation for this patch isn't really a desire to test / tune the governor. It's intended to allow working around a performance problem caused by using high-latency idle states on some interrupt heavy GPU workload. The interrupts occur around ~30ms apart which is long enough for the governor to put the CPU into the deeper states and over the course of long job the additional wakeup latency adds up. The initial fix someone came up with was cooking the residency values so the high-latency states had a residency of +50ms to prevent the govenor from using them. However, that fix is supposed to go into a bit of firmware I maintain and I'm not terribly happy with the idea. I'm fairly sure that ~30ms value is workload dependent and personally I don't think firmware should be making up numbers to trick specific kernel versions into doing specific things.

Quoted 3 lines

My impression is the right solution is to have the GPU driver set a PM QoS constraint on the CPUs receiving interrupts while a job is on-going.

Yes, that would address the GPU problem.

Quoted 4 lines

However, interrupt latency sensitivity isn't something that's unique to GPUs so I'm wondering it it makes sense to have the governor factor in interrupt traffic when deciding what state to use. Is that something that's been tried before?

Yes, that is in the works.

The existing governors should take interrupts into account too in the form of the expected idle duration corrections, but that may not be particularly precise. If the governor currently in use (I guess menu) doesn't to that, you may try an alternative one (e.g. teo).

That said, work is in progress on taking the actual interrupt frequency into account in idle duration prediction.

Thanks!

author
#4 lore raw

Hi Rafael,

Quoted 59 lines · Rafael J. Wysocki
Quoted 49 lines · Oliver O'Halloran
Quoted 18 lines · Rafael J. Wysocki
Quoted 4 lines · Abhishek Goel

Up until now, we did not have a way to tune cpuidle attribute like residency in kernel. This patch adds support for debugfs in cpuidle core. Thereby providing support for tuning cpuidle attributes like residency in kernel at runtime.

This is not a good idea in my view, for a couple of reasons.

First off, if the target residency of an idle state is changed, it effectively becomes a different one and all of the statistics regarding it become outdated at that point. Synchronizing that would be a pain.

Next, governors may get confused if idle state parameters are changed on the fly. In particular, the statistics collected by the teo governor depend on the target residencies of idle states, so if one of them changes, the governor needs to be reloaded.

Quoted 7 lines

Next, idle states are expected to be ordered by the target residency (and by the exit latency), so their parameters cannot be allowed to change freely anyway.

Finally, the idle state parameters are expected to reflect the properties of the hardware, which wouldn't hold any more if they were allowed to change at any time.

Certainly does sound like a headache.

Quoted 7 lines
Quoted 3 lines

For example: Tuning residency at runtime can be used to quantify governors decision making as governor uses residency as one of the parameter to take decision about the state that needs to be entered while idling.

IMO it would be better to introduce a testing cpuidle driver with an artificial set of idle states (or even such that the set of idle states to be used by it can be defined by the user e.g. via module parameters) for this purpose.

The motivation for this patch isn't really a desire to test / tune the governor. It's intended to allow working around a performance problem caused by using high-latency idle states on some interrupt heavy GPU workload. The interrupts occur around ~30ms apart which is long enough for the governor to put the CPU into the deeper states and over the course of long job the additional wakeup latency adds up. The initial fix someone came up with was cooking the residency values so the high-latency states had a residency of +50ms to prevent the govenor from using them. However, that fix is supposed to go into a bit of firmware I maintain and I'm not terribly happy with the idea. I'm fairly sure that ~30ms value is workload dependent and personally I don't think firmware should be making up numbers to trick specific kernel versions into doing specific things.

Quoted 3 lines

My impression is the right solution is to have the GPU driver set a PM QoS constraint on the CPUs receiving interrupts while a job is on-going.

Yes, that would address the GPU problem.

Quoted 4 lines

However, interrupt latency sensitivity isn't something that's unique to GPUs so I'm wondering it it makes sense to have the governor factor in interrupt traffic when deciding what state to use. Is that something that's been tried before?

Quoted 6 lines

Yes, that is in the works.

The existing governors should take interrupts into account too in the form of the expected idle duration corrections, but that may not be particularly precise. If the governor currently in use (I guess menu) doesn't to that, you may try an alternative one (e.g. teo).

For this particular case, I tried TEO but it did not solve the issue.

Quoted 4 lines

That said, work is in progress on taking the actual interrupt frequency into account in idle duration prediction.

Thanks!

Could you please point out to the patch looking into device interrupt frequency for cpuidle.

Thanks.