cpu: pseries: Offline state framework.
Subsystems (3)
v2 cover cpu: pseries: Offline state framework.
Hi,
This is the version 2 of the patch series to provide a cpu-offline framework that enables the administrators choose the state the offline CPU must be put into when multiple such states are exposed by the underlying architecture.
Version 1 of the Patch can be found here:
http://lkml.org/lkml/2009/8/6/236
The patch-series exposes the following sysfs tunables to allow the system-adminstrator to choose the state of a CPU:
To query the available hotplug states, one needs to read the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state, on needs to read/write the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/current_states
The patchset ensures that the writes to the "current_state" sysfs file are serialized against the writes to the "online" file.
This patchset also contains the offline state driver implemented for pSeries. For pSeries, we define three available_hotplug_states. They are:
online: The processor is online.
deallocate: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an rtas_stop_self() call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.
deactivate: This cedes the vCPU to the hypervisor which in turn can put the vCPU time to the best use.
NOTE: This option DOES NOT result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
Awaiting your feedback.
---
Gautham R Shenoy (2):
cpu: Implement cpu-offline-state driver for pSeries.
cpu: Offline state Framework.
arch/powerpc/platforms/pseries/Makefile | 2 arch/powerpc/platforms/pseries/hotplug-cpu.c | 76 +++++++++- arch/powerpc/platforms/pseries/offline_driver.c | 161 +++++++++++++++++++++ arch/powerpc/platforms/pseries/offline_driver.h | 20 +++ arch/powerpc/platforms/pseries/smp.c | 17 ++ drivers/base/cpu.c | 176 ++++++++++++++++++++++- include/linux/cpu.h | 30 ++++ 7 files changed, 465 insertions(+), 17 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/offline_driver.c create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h
Signature · 2 lines
Thanks and Regards gautham.
PATCH v2 1/2 cpu: Offline state Framework.
Provide an interface by which the system administrator can decide what state should the CPU go to when it is offlined.
To query the hotplug states, on needs to perform a read on the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state for a particular CPU, one needs to use the sysfs interface:
/sys/devices/system/cpu/cpu<number>/current_state
This patch implements the architecture independent bits of the cpu-offline-state framework.
Architectures which want to expose the multiple offline-states to the userspace are expected to write a driver which can register with this framework.
Such a driver should:
- Implement the callbacks defined in the structure struct cpu_offline_driver which can be called into by this framework when the corresponding sysfs interfaces are read or written into.
- Ensure that the following operation puts the CPU in the same state as it did in the absence of the driver.
echo 0 > /sys/devices/system/cpu/cpu<number>/online
This framework also serializes the writes to the "current_state" with respect to with the writes to the "online" sysfs tunable.
--- drivers/base/cpu.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++--- include/linux/cpu.h | 30 +++++++++ 2 files changed, 197 insertions(+), 9 deletions(-)
drivers/base/cpu.c +167 -9
@@ -20,7 +20,161 @@ EXPORT_SYMBOL(cpu_sysdev_class); static DEFINE_PER_CPU(struct sys_device *, cpu_sys_devices); +struct sys_device *get_cpu_sysdev (unsigned cpu) +{ +if (cpu < nr_cpu_ids &&cpu_possible (cpu)) +return per_cpu (cpu_sys_devices, cpu); +else +return NULL ; +} +EXPORT_SYMBOL_GPL (get_cpu_sysdev); + +#ifdef CONFIG_HOTPLUG_CPU + +struct cpu_offline_driver *cpu_offline_driver; +static DEFINE_MUTEX (cpu_offline_driver_lock); + +ssize_t show_available_states (struct sys_device *dev, +struct sysdev_attribute *attr,char *buf) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); +int cpu_num = cpu->sysdev.id; +ssize_t ret; + +mutex_lock (&cpu_offline_driver_lock); +if (!cpu_offline_driver) { + ret = -EEXIST ; + goto out_unlock; + } + + ret = cpu_offline_driver->read_available_states (cpu_num, buf); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock); + +return ret; + +} + +ssize_t show_current_state (struct sys_device *dev, +struct sysdev_attribute *attr,char *buf) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); +int cpu_num = cpu->sysdev.id; +ssize_t ret =0 ; + +mutex_lock (&cpu_offline_driver_lock); +if (!cpu_offline_driver) { + ret = -EEXIST ; + goto out_unlock; + } + + ret = cpu_offline_driver->read_current_state (cpu_num, buf); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock); + +return ret; +} + +ssize_t store_current_state (struct sys_device *dev, +struct sysdev_attribute *attr, +const char *buf,size_t count) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); +int cpu_num = cpu->sysdev.id; +ssize_t ret = count; + +mutex_lock (&cpu_offline_driver_lock); +if (!cpu_offline_driver) { + ret = -EEXIST ; + goto out_unlock; + } + + ret = cpu_offline_driver->write_current_state (cpu_num, buf); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock); + +if (ret >=0 ) + ret = count; +return ret; +} + +static SYSDEV_ATTR(available_hotplug_states, 0444, show_available_states, + NULL); +static SYSDEV_ATTR(current_state, 0644, show_current_state, + store_current_state); + +/* Should be called with cpu_offline_driver_lock held */ +void cpu_offline_driver_add_cpu (struct sys_device *cpu_sys_dev) +{ +if (!cpu_offline_driver || !cpu_sys_dev) +return ; + +sysdev_create_file (cpu_sys_dev, &attr_available_hotplug_states); +sysdev_create_file (cpu_sys_dev, &attr_current_state); +} + +/* Should be called with cpu_offline_driver_lock held */ +void cpu_offline_driver_remove_cpu (struct sys_device *cpu_sys_dev) +{ +if (!cpu_offline_driver || !cpu_sys_dev) +return ; + +sysdev_remove_file (cpu_sys_dev, &attr_available_hotplug_states); +sysdev_remove_file (cpu_sys_dev, &attr_current_state); + +} + +int register_cpu_offline_driver (struct cpu_offline_driver *arch_cpu_driver) +{ +int ret =0 ; +int cpu; +mutex_lock (&cpu_offline_driver_lock); + +if (cpu_offline_driver !=NULL ) { + ret = -EEXIST ; + goto out_unlock; + } + +if (!(arch_cpu_driver->read_available_states && + arch_cpu_driver->read_current_state && + arch_cpu_driver->write_current_state)) { + ret = -EINVAL ; + goto out_unlock; + } + + cpu_offline_driver = arch_cpu_driver; + +for_each_possible_cpu (cpu) +cpu_offline_driver_add_cpu (get_cpu_sysdev (cpu)); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock); +return ret; +} + +void unregister_cpu_offline_driver (struct cpu_offline_driver *arch_cpu_driver) +{ +int cpu; +mutex_lock (&cpu_offline_driver_lock); + +if (!cpu_offline_driver) { +WARN_ON (1 ); +mutex_unlock (&cpu_offline_driver_lock); +return ; + } + +for_each_possible_cpu (cpu) +cpu_offline_driver_remove_cpu (get_cpu_sysdev (cpu)); + + cpu_offline_driver =NULL ; +mutex_unlock (&cpu_offline_driver_lock); +} + +static ssize_t show_online (struct sys_device *dev,struct sysdev_attribute *attr,char *buf) {
@@ -35,6 +189,7 @@ static ssize_t __ref store_online(struct sys_device *dev, struct sysdev_attribut struct cpu *cpu = container_of(dev, struct cpu, sysdev);ssize_t ret; +mutex_lock (&cpu_offline_driver_lock);switch (buf[0 ]) {case '0' : ret =cpu_down (cpu->sysdev.id);
@@ -50,6 +205,8 @@ static ssize_t __ref store_online(struct sys_device *dev, struct sysdev_attribut ret = -EINVAL ; } +mutex_unlock (&cpu_offline_driver_lock); +if (ret >=0 ) ret = count;return ret;
@@ -59,23 +216,33 @@ static SYSDEV_ATTR(online, 0644, show_online, store_online);static void __cpuinit register_cpu_control (struct cpu *cpu) {sysdev_create_file (&cpu->sysdev, &attr_online); +mutex_lock (&cpu_offline_driver_lock); +cpu_offline_driver_add_cpu (&cpu->sysdev); +mutex_unlock (&cpu_offline_driver_lock); } +void unregister_cpu (struct cpu *cpu) {int logical_cpu = cpu->sysdev.id;unregister_cpu_under_node (logical_cpu,cpu_to_node (logical_cpu)); +mutex_lock (&cpu_offline_driver_lock); +cpu_offline_driver_remove_cpu (&cpu->sysdev); +mutex_unlock (&cpu_offline_driver_lock); +sysdev_remove_file (&cpu->sysdev, &attr_online);sysdev_unregister (&cpu->sysdev);per_cpu (cpu_sys_devices, logical_cpu) =NULL ;return ; } +#else /* ... !CONFIG_HOTPLUG_CPU */ static inline void register_cpu_control (struct cpu *cpu) { } +#endif /* CONFIG_HOTPLUG_CPU */ #ifdef CONFIG_KEXEC
@@ -224,15 +391,6 @@ int __cpuinit register_cpu(struct cpu *cpu, int num)return error; } -struct sys_device *get_cpu_sysdev (unsigned cpu) -{ -if (cpu < nr_cpu_ids &&cpu_possible (cpu)) -return per_cpu (cpu_sys_devices, cpu); -else -return NULL ; -} -EXPORT_SYMBOL_GPL (get_cpu_sysdev); -int __init cpu_dev_init (void ) {int err;
include/linux/cpu.h +30 -0
@@ -51,6 +51,36 @@ struct notifier_block;#ifdef CONFIG_HOTPLUG_CPU extern int register_cpu_notifier (struct notifier_block *nb);extern void unregister_cpu_notifier (struct notifier_block *nb); + +/* +* struct cpu_offline_driver: Callbacks for cpu-offline state framework. +* +* Defines the hooks for the architecture dependent callbacks +* which can be invoked when the user queries the +* available_hotplug_states and current_state and sets the current_state. +* +* read_available_state: Called when the user queries available_hotplug_states. +* @cpu: Cpu for which available_hotplug_states are being queried. +* @buf: Buffer in which the available hotplug states are to be populated. +* +* read_current_state: Called when the user queries current_state. +* @cpu: Cpu for which current_state is being queried. +* @buf: Buffer in which the current_state value is to be returned. +* +* write_current_state: Called when the user wants to set the current_state. +* @cpu: Cpu for which the current state is being set. +* @buf: Buffer containing the string corresponding to the current_state +* that needs to be set. +*/ +struct cpu_offline_driver { +ssize_t (*read_available_states)(unsigned int cpu,char *buf); +ssize_t (*read_current_state)(unsigned int cpu,char *buf); +ssize_t (*write_current_state)(unsigned int cpu,const char *buf); +}; + +extern int register_cpu_offline_driver (struct cpu_offline_driver *driver); +extern void unregister_cpu_offline_driver (struct cpu_offline_driver *driver); +#else #ifndef MODULE
Re: [PATCH v2 1/2] cpu: Offline state Framework.
Quoted 22 lines · On Fri, 28 Aug 2009 15:30:16 +0530 Gautham R Shenoy <[email protected]> wrote:
Provide an interface by which the system administrator can decide what state should the CPU go to when it is offlined.
To query the hotplug states, on needs to perform a read on the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_statesTo query or set the current state for a particular CPU, one needs to use the sysfs interface:
/sys/devices/system/cpu/cpu<number>/current_stateThis patch implements the architecture independent bits of the cpu-offline-state framework.
Architectures which want to expose the multiple offline-states to the userspace are expected to write a driver which can register with this framework.
Such a driver should:
- Implement the callbacks defined in the structure structcpu_offline_driverwhich can be called into by this framework when the corresponding sysfs interfaces are read or written into.
Quoted 7 lines
- Ensure that the following operation puts the CPU in the same state as it did in the absence of the driver.
echo 0 > /sys/devices/system/cpu/cpu<number>/onlineThis framework also serializes the writes to the "
current_state" with respect to with the writes to the "online" sysfs tunable.
It would be nice to document this new userspace interface somewhere.
Quoted 14 lines
drivers/base/cpu.c +14
-0+struct cpu_offline_driver *cpu_offline_driver; +static DEFINE_MUTEX (cpu_offline_driver_lock); + +ssize_t show_available_states (struct sys_device *dev, +struct sysdev_attribute *attr,char *buf) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); +int cpu_num = cpu->sysdev.id; +ssize_t ret; + +mutex_lock (&cpu_offline_driver_lock); +if (!cpu_offline_driver) { + ret = -EEXIST ; + goto out_unlock;
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 58
Quoted 10 lines
drivers/base/cpu.c +10
-0+ } + + ret = cpu_offline_driver->read_available_states (cpu_num, buf); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock); + +return ret; + +}
The patch adds boatloads of global symbols which do not have names which are appropriate for global symbols.
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 61
Quoted 2 lines
drivers/base/cpu.c +2
-0+ssize_t show_current_state (struct sys_device *dev, +struct sysdev_attribute *attr,char *buf)
Like that.
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 83
Quoted 3 lines
drivers/base/cpu.c +3
-0+ssize_t store_current_state (struct sys_device *dev, +struct sysdev_attribute *attr, +const char *buf,size_t count)
And that.
Quoted 16 lines
drivers/base/cpu.c +16
-0+ +static SYSDEV_ATTR(available_hotplug_states, 0444, show_available_states, + NULL); +static SYSDEV_ATTR(current_state, 0644, show_current_state, + store_current_state); + +/* Should be called with cpu_offline_driver_lock held */ +void cpu_offline_driver_add_cpu (struct sys_device *cpu_sys_dev) +{ +if (!cpu_offline_driver || !cpu_sys_dev) +return ; + +sysdev_create_file (cpu_sys_dev, &attr_available_hotplug_states); +sysdev_create_file (cpu_sys_dev, &attr_current_state); +} +
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 129
Quoted 10 lines
drivers/base/cpu.c +10
-0+/* Should be called with cpu_offline_driver_lock held */ +void cpu_offline_driver_remove_cpu (struct sys_device *cpu_sys_dev) +{ +if (!cpu_offline_driver || !cpu_sys_dev) +return ; + +sysdev_remove_file (cpu_sys_dev, &attr_available_hotplug_states); +sysdev_remove_file (cpu_sys_dev, &attr_current_state); + +}
Please don't just ignore possible error returns.
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 136
Quoted 6 lines
drivers/base/cpu.c +6
-0+int register_cpu_offline_driver (struct cpu_offline_driver *arch_cpu_driver) +{ +int ret =0 ; +int cpu; +mutex_lock (&cpu_offline_driver_lock); +
The blank line goes after end-of-locals and before start-of-code.
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 146
Quoted 10 lines
drivers/base/cpu.c +10
-0+if (cpu_offline_driver !=NULL ) { + ret = -EEXIST ; + goto out_unlock; + } + +if (!(arch_cpu_driver->read_available_states && + arch_cpu_driver->read_current_state && + arch_cpu_driver->write_current_state)) { + ret = -EINVAL ; + goto out_unlock;
This seems pretty pointless. Just let the code oops - the developer will notice fairly quickly.
Quoted 9 lines
drivers/base/cpu.c +9
-0+ } + + cpu_offline_driver = arch_cpu_driver; + +for_each_possible_cpu (cpu) +cpu_offline_driver_add_cpu (get_cpu_sysdev (cpu)); + +out_unlock: +mutex_unlock (&cpu_offline_driver_lock);
drivers/base/cpu.c EXPORT_SYMBOL(cpu_sysdev_class); · line 165
Quoted 10 lines
drivers/base/cpu.c +10
-0+return ret; +} + +void unregister_cpu_offline_driver (struct cpu_offline_driver *arch_cpu_driver) +{ +int cpu; +mutex_lock (&cpu_offline_driver_lock); + +if (!cpu_offline_driver) { +WARN_ON (1 );
if (WARN_ON(!cpu_offline_driver)) {Quoted 12 lines
drivers/base/cpu.c +12
-0+mutex_unlock (&cpu_offline_driver_lock); +return ; + } + +for_each_possible_cpu (cpu) +cpu_offline_driver_remove_cpu (get_cpu_sysdev (cpu)); + + cpu_offline_driver =NULL ; +mutex_unlock (&cpu_offline_driver_lock); +} + +
PATCH v2 2/2 cpu: Implement cpu-offline-state driver for pSeries.
In reply to Gautham R ShenoyThis patch implements the callbacks to handle the reads/writes into the sysfs interfaces
/sys/devices/system/cpu/cpu<number>/available_hotplug_states and /sys/devices/system/cpu/cpu<number>/current_state
Currently, the patch defines two states which the processor can go to when it is offlined. They are
- deallocate: This is the the default behaviour when the cpu is offlined even in the absense of this driver.
The CPU would call make an rtas_stop_self() call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR. This would result in a configuration change to the LPAR which is visible to the outside world.
- deactivate: This cedes the vCPU to the hypervisor which in turn can put the vCPU time to the best use. This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
--- arch/powerpc/platforms/pseries/Makefile | 2 arch/powerpc/platforms/pseries/hotplug-cpu.c | 76 ++++++++++- arch/powerpc/platforms/pseries/offline_driver.c | 161 +++++++++++++++++++++++ arch/powerpc/platforms/pseries/offline_driver.h | 20 +++ arch/powerpc/platforms/pseries/smp.c | 17 ++ 5 files changed, 268 insertions(+), 8 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/offline_driver.c create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h
arch/powerpc/platforms/pseries/Makefile +1 -1
@@ -17,7 +17,7 @@ obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_PCI) += pci.o pci_dlpar.o obj-$(CONFIG_PSERIES_MSI) += msi.o -obj-$(CONFIG_HOTPLUG_CPU) += hotplug-cpu.o +obj-$(CONFIG_HOTPLUG_CPU) += hotplug-cpu.o offline_driver.o obj-$(CONFIG_MEMORY_HOTPLUG) += hotplug-memory.o obj-$(CONFIG_HVC_CONSOLE) += hvconsole.o
arch/powerpc/platforms/pseries/hotplug-cpu.c +69 -7
99 lines not on this page · raw message
arch/powerpc/platforms/pseries/offline_driver.c new file +161 -0
161 lines not on this page · raw message
arch/powerpc/platforms/pseries/offline_driver.h new file +20 -0
@@ -0,0 +1,20 @@ +#ifndef _OFFLINE_DRIVER_H_ +#define _OFFLINE_DRIVER_H_ + +#define CPU_STATES_LEN 16 + +/* Cpu offline states go here */ +enum cpu_state_vals { +CPU_DEALLOCATE , +CPU_DEACTIVATE , +CPU_STATE_ONLINE , +CPU_MAX_OFFLINE_STATES +}; + +extern enum cpu_state_vals get_cpu_current_state (int cpu); +extern void set_cpu_current_state (int cpu,enum cpu_state_vals state); +extern enum cpu_state_vals get_preferred_offline_state (int cpu); +extern void set_preferred_offline_state (int cpu,enum cpu_state_vals state); +extern int start_secondary (void ); +extern void set_default_offline_state (int cpu); +#endif
arch/powerpc/platforms/pseries/smp.c +17 -0
@@ -48,6 +48,7 @@#include "plpar_wrappers.h" #include "pseries.h" #include "xics.h" +#include "offline_driver.h" /*
@@ -86,6 +87,9 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)/* Fixup atomic count: it exited inside IRQ handler. */ task_thread_info (paca[lcpu].__current)->preempt_count =0 ; +if (get_cpu_current_state (lcpu) ==CPU_DEACTIVATE ) + goto out; +/* * If the RTAS start-cpu token does not exist then presume the * cpu is already spinning.
@@ -100,6 +104,7 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)return 0 ; } +out:return 1 ; }
@@ -113,12 +118,15 @@ static void __devinit smp_xics_setup_cpu(int cpu)vpa_init (cpu);cpu_clear (cpu, of_spin_map); +set_cpu_current_state (cpu,CPU_STATE_ONLINE ); + set_default_offline_state(cpu); }#endif /* CONFIG_XICS */ static void __devinit smp_pSeries_kick_cpu (int nr) { +long rc;BUG_ON (nr <0 || nr >=NR_CPUS );if (!smp_startup_cpu (nr))
@@ -130,6 +138,15 @@ static void __devinit smp_pSeries_kick_cpu(int nr)* the processor will continue on to secondary_start */ paca[nr].cpu_start =1 ; + +set_preferred_offline_state (nr,CPU_STATE_ONLINE ); + +if (get_cpu_current_state (nr) ==CPU_DEACTIVATE ) { + rc =plpar_hcall_norets (H_PROD , nr); +if (rc !=H_SUCCESS ) +panic ("Error: Prod to wake up processor %d Ret= %ld\n" , + nr, rc); + } }static int smp_pSeries_cpu_bootable (unsigned int nr)
Quoted 32 lines · Gautham R Shenoy
Hi,
This is the version 2 of the patch series to provide a cpu-offline framework that enables the administrators choose the state the offline CPU must be put into when multiple such states are exposed by the underlying architecture.
Version 1 of the Patch can be found here:
http://lkml.org/lkml/2009/8/6/236The patch-series exposes the following sysfs tunables to allow the system-adminstrator to choose the state of a CPU:
To query the available hotplug states, one needs to read the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state, on needs to read/write the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/current_statesThe patchset ensures that the writes to the "
current_state" sysfs file are serialized against the writes to the "online" file.This patchset also contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
deallocate: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.
Quoted 7 lines
deactivate: This cedes the vCPU to the hypervisor which in turn can put the vCPU time to the best use.
NOTE: This option DOES NOT result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.Awaiting your feedback.
I'm still thinking this is a bad idea.
The OS should only know about online/offline.
Use the hypervisor interface to deal with the cpu once its offline.
That is, I think this interface you propose is a layering violation.
Quoted 41 lines · Peter Zijlstra
Quoted 32 lines · Gautham R Shenoy
Hi,
This is the version 2 of the patch series to provide a cpu-offline framework that enables the administrators choose the state the offline CPU must be put into when multiple such states are exposed by the underlying architecture.
Version 1 of the Patch can be found here:
http://lkml.org/lkml/2009/8/6/236The patch-series exposes the following sysfs tunables to allow the system-adminstrator to choose the state of a CPU:
To query the available hotplug states, one needs to read the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state, on needs to read/write the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/current_statesThe patchset ensures that the writes to the "
current_state" sysfs file are serialized against the writes to the "online" file.This patchset also contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
deallocate: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.Quoted 7 lines
deactivate: This cedes the vCPU to the hypervisor which in turn can put the vCPU time to the best use.
NOTE: This option DOES NOT result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.Awaiting your feedback.
Quoted 7 lines
I'm still thinking this is a bad idea.
The OS should only know about online/offline.
Use the hypervisor interface to deal with the cpu once its offline.
That is, I think this interface you propose is a layering violation.
Agreed. Plus having interface like 'go to this state during offliine' then 'go offline' is strange/stupid. For hypervisor case, you might want to change 'state' of cpu that is already offline.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 9 lines · Peter Zijlstra
I'm still thinking this is a bad idea.
The OS should only know about online/offline.
Use the hypervisor interface to deal with the cpu once its offline.
That is, I think this interface you propose is a layering violation.
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
For example, the HW can put them in some low power state where they can be re-plugged quickly, or can shutdown entire power planes completely, possibly allowing physical hotplug, but that takes much longer to bring them back into the pool.
In any case, regarding the pseries case, this is how our hypervisor works and I don't think we can change it, other than by always going into the "cede" function and having some weird separate interface in the arch to then whack them into some different state.
Ben.
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 5 lines · Peter Zijlstra
I'm still thinking this is a bad idea.
The OS should only know about online/offline.
Quoted 8 lines
Quoted 4 lines
Use the hypervisor interface to deal with the cpu once its offline.
That is, I think this interface you propose is a layering violation.
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
cpu-hotplug is an utter slow path, anybody saying latency and hotplug in the same sentence doesn't seem to grasp either or both concepts.
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
cpu-hotplug is an utter slow path, anybody saying latency and hotplug in the same sentence doesn't seem to grasp either or both concepts.
Let's forget about latency then. Let's imagine I want to set a CPU offline to save power, vs. setting it offline -and- opening the back door of the machine to actually physically replace it :-)
In any case, I don't see the added feature as being problematic, and not such a "layering violation" as you seem to imply it is. It's a convenient way to atomically take the CPU out -and- convey some information about the "intent" to the hypervisor, and I really fail to see why you have so strong objections about it.
Ben.
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
Quoted 6 lines
Quoted 2 lines
cpu-hotplug is an utter slow path, anybody saying latency and hotplug in the same sentence doesn't seem to grasp either or both concepts.
Let's forget about latency then. Let's imagine I want to set a CPU offline to save power, vs. setting it offline -and- opening the back door of the machine to actually physically replace it :-)
If the hardware is capable of physical hotplug, then surely powering the socket down saves most power and is the preferred mode?
Quoted 5 lines
In any case, I don't see the added feature as being problematic, and not such a "layering violation" as you seem to imply it is. It's a convenient way to atomically take the CPU out -and- convey some information about the "intent" to the hypervisor, and I really fail to see why you have so strong objections about it.
Ignorance on my part probably :-)
I'm simply not seeing a use case for it, except for the virt case, which I think we should bug the virt interface with and not the cpu-hotplug interface.
Quoted 12 lines · On Thu, 24 Sep 2009 13:33:07 +0200 Peter Zijlstra <[email protected]>…
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
cpu-hotplug is an utter slow path, anybody saying latency and hotplug in the same sentence doesn't seem to grasp either or both concepts.
Quoted 6 lines
Quoted 3 lines
Let's forget about latency then. Let's imagine I want to set a CPU offline to save power, vs. setting it offline -and- opening the back door of the machine to actually physically replace it :-)
If the hardware is capable of physical hotplug, then surely powering the socket down saves most power and is the preferred mode?
btw just to take away a perception that generally powering down sockets help; it does not help for all cpus. Some cpus are so efficient in idle that the incremental gain one would get by "offlining" a core is just not worth it
(in fact, in x86, it's the same thing)
I obviously can't speak for p-series cpus, just wanted to point out that there is no universal truth about "offlining saves power".
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 22 lines · Arjan van de Ven
Quoted 12 lines · On Thu, 24 Sep 2009 13:33:07 +0200 Peter Zijlstra <[email protected]>…
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines
I don't quite follow your logic here. This is useful for more than just hypervisors. For example, take the HV out of the picture for a moment and imagine that the HW has the ability to offline CPU in various power levels, with varying latencies to bring them back.
cpu-hotplug is an utter slow path, anybody saying latency and hotplug in the same sentence doesn't seem to grasp either or both concepts.
Quoted 6 lines
Quoted 3 lines
Let's forget about latency then. Let's imagine I want to set a CPU offline to save power, vs. setting it offline -and- opening the back door of the machine to actually physically replace it :-)
If the hardware is capable of physical hotplug, then surely powering the socket down saves most power and is the preferred mode?
Quoted 8 lines
btw just to take away a perception that generally powering down sockets help; it does not help for all cpus. Some cpus are so efficient in idle that the incremental gain one would get by "offlining" a core is just not worth it
(in fact, in x86, it's the same thing)I obviously can't speak for p-series cpus, just wanted to point out that there is no universal truth about "offlining saves power".
Hi Arjan,
As you have said, on some cpus the extra effort of offlining does not save us any extra power, and the state will be same as idle. The assertion that offlining saves power is still valid, it could be same as idle or better depending on the architecture and implementation.
On x86 we still need the code (Venki posted) to take cpus to C6 on offline to save power or else offlining consumes more power than idle due to C1/hlt state. This framework can help here as well if we have any apprehension on making lowest sleep state as default on x86 and want the administrator to decide.
--Vaidy
Quoted 10 lines · On Fri, 25 Sep 2009 12:55:49 +0530 Vaidyanathan Srinivasan…
Quoted 2 lines
I obviously can't speak for p-series cpus, just wanted to point out that there is no universal truth about "offlining saves power".
Hi Arjan,
As you have said, on some cpus the extra effort of offlining does not save us any extra power, and the state will be same as idle. The assertion that offlining saves power is still valid, it could be same as idle or better depending on the architecture and implementation.
Quoted 5 lines
On x86 we still need the code (Venki posted) to take cpus to C6 on offline to save power or else offlining consumes more power than idle due to C1/hlt state. This framework can help here as well if we have any apprehension on making lowest sleep state as default on x86 and want the administrator to decide.
even with Venki's patch, all our measurements indicate that taking cores away is damage on x86. Don't let that stop what you do for powerpc, but for x86 it's not a win. Linux is good at keeping cores in C6 long enough that the downside of offlining is bigger...
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org