cpu: pseries: Cpu offline states framework
Subsystems (4)
v3 cover cpu: pseries: Cpu offline states framework
Hi,
**** RFC not for inclusion ****
This is the version 3 of the patch series to provide a cpu-offline framework that enables the administrators choose the state of a CPU when it is offlined, when multiple such states are exposed by the underlying architecture.
Changes from Version 2:(can be found here: http://lkml.org/lkml/2009/8/28/102)
- Addressed Andrew Morton's review comments regarding names of global variables, handling of error conditions and documentation of the interfaces.
- Implemented a patch to provide helper functions to set the cede latency specifier value in the VPA indicating latency expectation of the guest OS when the vcpu is ceded from a subsequent H_CEDE hypercall. Hypervisor may use this for better energy savings.
- Renamed of the cpu-hotplug states. "deallocate" is renamed as "offline" and "deactivate" is renamed as "inactive".
The patch-series exposes the following sysfs tunables to allow the system-adminstrator to choose the state of a CPU:
To query the available hotplug states, one needs to read the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state, on needs to read/write the sysfs tunable:
/sys/devices/system/cpu/cpu<number>/current_hotplug_state
The patchset ensures that the writes to the "current_hotplug_state" sysfs file are serialized against the writes to the "online" file.
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three available_hotplug_states. They are:
online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an rtas_stop_self() call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.
inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
Any feedback on the patchset will be immensely valuable.
---
Arun R Bharadwaj (1):
pSeries: cede latency specifier helper function.
Gautham R Shenoy (2):
cpu: Implement cpu-offline-state callbacks for pSeries.
cpu: Offline state Framework.
Documentation/cpu-hotplug.txt | 22 +++ arch/powerpc/include/asm/lppaca.h | 9 + arch/powerpc/platforms/pseries/Makefile | 2 arch/powerpc/platforms/pseries/hotplug-cpu.c | 88 ++++++++++- arch/powerpc/platforms/pseries/offline_driver.c | 148 +++++++++++++++++++ arch/powerpc/platforms/pseries/offline_driver.h | 20 +++ arch/powerpc/platforms/pseries/plpar_wrappers.h | 17 ++ arch/powerpc/platforms/pseries/smp.c | 17 ++ arch/powerpc/xmon/xmon.c | 3 drivers/base/cpu.c | 181 ++++++++++++++++++++++- include/linux/cpu.h | 10 + 11 files changed, 498 insertions(+), 19 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/offline_driver.c create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h
Signature · 2 lines
Thanks and Regards gautham.
PATCH v3 1/3 pSeries: cede latency specifier helper function.
From: Arun R Bharadwaj <[email protected]>
This patch provides helper functions to set the cede latency specifier value in the VPA indicating the latency expectation of the guest OS to inform the hypervisor's choice of the platform dependent energy saving mode chosen for the processor when unused during the subsequent H_CEDE hypercall.
--- arch/powerpc/include/asm/lppaca.h | 9 ++++++++- arch/powerpc/platforms/pseries/plpar_wrappers.h | 17 +++++++++++++++++ arch/powerpc/xmon/xmon.c | 3 ++- 3 files changed, 27 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/lppaca.h +8 -1
@@ -100,7 +100,14 @@ struct lppaca {// Used to pass parms from the OS to PLIC for SetAsrAndRfid u64 saved_gpr3;// Saved GPR3 x20-x27 u64 saved_gpr4;// Saved GPR4 x28-x2F -u64 saved_gpr5;// Saved GPR5 x30-x37 +union { +u64 saved_gpr5;// Saved GPR5 x30-x37 +struct { +u8 cede_latency_hint;// x30 +u8 reserved[7 ];// x31-x36 + } fields; + } gpr5_dword; +u8 dtl_enable_mask;// Dispatch Trace Log mask x38-x38 u8 donate_dedicated_cpu;// Donate dedicated CPU cycles x39-x39
arch/powerpc/platforms/pseries/plpar_wrappers.h +17 -0
@@ -9,11 +9,28 @@ static inline long poll_pending(void)return plpar_hcall_norets (H_POLL_PENDING ); } +static inline u8 get_cede_latency_hint (void ) +{ +return get_lppaca ()->gpr5_dword.fields.cede_latency_hint; +} + +static inline void set_cede_latency_hint (u8 latency_hint) +{ +get_lppaca ()->gpr5_dword.fields.cede_latency_hint = latency_hint; +} +static inline long cede_processor (void ) {return plpar_hcall_norets (H_CEDE ); } +static inline long extended_cede_processor (u8 latency_hint) +{ +set_cede_latency_hint (latency_hint); +cede_processor (); + +} +static inline long vpa_call (unsigned long flags,unsigned long cpu,unsigned long vpa) {
arch/powerpc/xmon/xmon.c +2 -1
@@ -1613,7 +1613,8 @@ static void super_regs(void) ptrLpPaca->saved_srr0, ptrLpPaca->saved_srr1);printf (" Saved Gpr3=%.16lx Saved Gpr4=%.16lx \n" , ptrLpPaca->saved_gpr3, ptrLpPaca->saved_gpr4); - printf(" Saved Gpr5=%.16lx \n", ptrLpPaca->saved_gpr5); +printf (" Saved Gpr5=%.16lx \n" , + ptrLpPaca->gpr5_dword.saved_gpr5); }#endif
Re: [PATCH v3 1/3] pSeries: cede latency specifier helper function.
Quoted 5 lines · Gautham R Shenoy
// Used to pass parms from the OS to PLIC for SetAsrAndRfid
u64saved_gpr3; // Saved GPR3 x20-x27
u64saved_gpr4; // Saved GPR4 x28-x2Farch/powerpc/include/asm/lppaca.h +1
-1-u64 saved_gpr5;// Saved GPR5 x30-x37 +union {
arch/powerpc/include/asm/lppaca.h struct lppaca { · line 113
Quoted 10 lines
arch/powerpc/include/asm/lppaca.h +7
-0+u64 saved_gpr5;// Saved GPR5 x30-x37 +struct { +u8 cede_latency_hint;// x30 +u8 reserved[7 ];// x31-x36 + } fields; + } gpr5_dword; +u8 dtl_enable_mask;// Dispatch Trace Log mask x38-x38 u8 donate_dedicated_cpu;// Donate dedicated CPU cycles x39-x39
Could you drop the C99 style comments (use /* */ instead)? If you run checkpatch on this it will error on these lines..
Daniel
PATCH v3 2/3 cpu: Offline state Framework.
In reply to Gautham R ShenoyProvide an interface by which the system administrator can decide what state should the CPU go to when it is offlined.
To query the hotplug states, on needs to perform a read on:
/sys/devices/system/cpu/cpu<number>/available_hotplug_states
To query or set the current state for a particular CPU, one needs to use the sysfs interface
/sys/devices/system/cpu/cpu<number>/current_hotplug_state
This patch implements the architecture independent bits of the cpu-offline-state framework.
The architecture specific bits are expected to register the actual code which implements the callbacks when the above mentioned sysfs interfaces are read or written into. Thus the values provided by reading available_offline_states vary with the architecture.
The patch provides serialization for writes to the "current_hotplug_state" with respect to with the writes to the "online" sysfs tunable.
--- Documentation/cpu-hotplug.txt | 22 +++++ drivers/base/cpu.c | 181 +++++++++++++++++++++++++++++++++++++++-- include/linux/cpu.h | 10 ++ 3 files changed, 204 insertions(+), 9 deletions(-)
Documentation/cpu-hotplug.txt +22 -0
@@ -115,6 +115,28 @@ Just remember the critical section cannot call any function that can sleep or schedule this process away. The preempt_disable() will work as long as stop_machine_run() is used to take a cpu down. +CPU-offline states +-------------------------------------- +On architectures which allow the more than one valid state when +the CPU goes offline, the system administrator can decide +the state the CPU needs to go to when it is offlined. + +If the architecture has implemented a cpu-offline driver exposing these +multiple offline states, the system administrator can use the following sysfs +interfaces to query the available hotplug states and also query and set the +current hotplug state for a given cpu: + +To query the hotplug states, on needs to perform a read on: +/sys/devices/system/cpu/cpu<number>/available_hotplug_states + +To query or set the current state for a particular CPU, +one needs to use the sysfs interface + +/sys/devices/system/cpu/cpu<number>/current_hotplug_state + +Writes to the "online" sysfs files are serialized against the writes to the +"current_hotplug_state" file. + CPU Hotplug - Frequently Asked Questions. Q: How to enable my kernel to support CPU hotplug?
drivers/base/cpu.c +172 -9
229 lines not on this page · raw message
include/linux/cpu.h +10 -0
@@ -51,6 +51,16 @@ struct notifier_block;#ifdef CONFIG_HOTPLUG_CPU extern int register_cpu_notifier (struct notifier_block *nb);extern void unregister_cpu_notifier (struct notifier_block *nb); + +struct cpu_offline_driver { +ssize_t (*read_available_states)(unsigned int cpu,char *buf); +ssize_t (*read_current_state)(unsigned int cpu,char *buf); +ssize_t (*write_current_state)(unsigned int cpu,const char *buf); +}; + +extern int register_cpu_offline_driver (struct cpu_offline_driver *driver); +extern void unregister_cpu_offline_driver (struct cpu_offline_driver *driver); +#else #ifndef MODULE
Re: [PATCH v3 2/3] cpu: Offline state Framework.
Quoted 12 lines · On Tue, 15 Sep 2009 17:37:06 +0530 Gautham R Shenoy wrote:
Signed-off-by Gautham R Shenoy <[email protected]>--- Documentation/cpu-hotplug.txt | 22 +++++ drivers/base/cpu.c | 181 +++++++++++++++++++++++++++++++++++++++-- include/linux/cpu.h | 10 ++ 3 files changed, 204 insertions(+), 9 deletions(-)Documentation/cpu-hotplug.txt +0
-0
Documentation/cpu-hotplug.txt Just remember the critical section cannot call any · line 120
Quoted 6 lines
Documentation/cpu-hotplug.txt +3
-0@@ -115,6 +115,28 @@ Just remember the critical section cannot call any function that can sleep or schedule this process away. The preempt_disable() will work as long as stop_machine_run() is used to take a cpu down. +CPU-offline states +-------------------------------------- +On architectures which allow the more than one valid state when
^drop "the"
Documentation/cpu-hotplug.txt Just remember the critical section cannot call any · line 122
Quoted 2 lines
Documentation/cpu-hotplug.txt +2
-0+the CPU goes offline, the system administrator can decide +the state the CPU needs to go to when it is offlined.
s/needs to/should/
Documentation/cpu-hotplug.txt Just remember the critical section cannot call any · line 129
Quoted 7 lines
Documentation/cpu-hotplug.txt +7
-0+ +If the architecture has implemented a cpu-offline driver exposing these +multiple offline states, the system administrator can use the following sysfs +interfaces to query the available hotplug states and also query and set the +current hotplug state for a given cpu: + +To query the hotplug states, on needs to perform a read on:
one
Documentation/cpu-hotplug.txt Just remember the critical section cannot call any · line 142
Quoted 13 lines
Documentation/cpu-hotplug.txt +10
-0+/sys/devices/system/cpu/cpu<number>/available_hotplug_states + +To query or set the current state for a particular CPU, +one needs to use the sysfs interface + +/sys/devices/system/cpu/cpu<number>/current_hotplug_state + +Writes to the "online" sysfs files are serialized against the writes to the +"current_hotplug_state" file. + CPU Hotplug - Frequently Asked Questions. Q: How to enable my kernel to support CPU hotplug?
---
~Randy
PATCH v3 3/3 cpu: Implement cpu-offline-state callbacks for pSeries.
In reply to Gautham R ShenoyThis patch implements the callbacks to handle the reads/writes into the sysfs interfaces
/sys/devices/system/cpu/cpu<number>/available_hotplug_states and /sys/devices/system/cpu/cpu<number>/current_hotplug_state
Currently, the patch defines two states which the processor can go to when it is offlined. They are
- offline: The current behaviour when the cpu is offlined. The CPU would call make an rtas_stop_self() call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
- inactive: This is expected to cede the processor to the hypervisor with a latency hint specifier value. Hypervisor may use this hint to provide better energy savings. In this state, the control of the vCPU will continue to be with the LPAR.
--- arch/powerpc/platforms/pseries/Makefile | 2 arch/powerpc/platforms/pseries/hotplug-cpu.c | 88 +++++++++++++- arch/powerpc/platforms/pseries/offline_driver.c | 148 +++++++++++++++++++++++ arch/powerpc/platforms/pseries/offline_driver.h | 20 +++ arch/powerpc/platforms/pseries/smp.c | 17 +++ 5 files changed, 267 insertions(+), 8 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/offline_driver.c create mode 100644 arch/powerpc/platforms/pseries/offline_driver.h
arch/powerpc/platforms/pseries/Makefile +1 -1
@@ -17,7 +17,7 @@ obj-$(CONFIG_KEXEC) += kexec.o obj-$(CONFIG_PCI) += pci.o pci_dlpar.o obj-$(CONFIG_PSERIES_MSI) += msi.o -obj-$(CONFIG_HOTPLUG_CPU) += hotplug-cpu.o +obj-$(CONFIG_HOTPLUG_CPU) += hotplug-cpu.o offline_driver.o obj-$(CONFIG_MEMORY_HOTPLUG) += hotplug-memory.o obj-$(CONFIG_HVC_CONSOLE) += hvconsole.o
arch/powerpc/platforms/pseries/hotplug-cpu.c +81 -7
111 lines not on this page · raw message
arch/powerpc/platforms/pseries/offline_driver.c new file +148 -0
148 lines not on this page · raw message
arch/powerpc/platforms/pseries/offline_driver.h new file +20 -0
@@ -0,0 +1,20 @@ +#ifndef _OFFLINE_DRIVER_H_ +#define _OFFLINE_DRIVER_H_ + +#define CPU_STATES_LEN 16 + +/* Cpu offline states go here */ +enum cpu_state_vals { +CPU_STATE_OFFLINE , +CPU_STATE_INACTIVE , +CPU_STATE_ONLINE , +CPU_MAX_HOTPLUG_STATES +}; + +extern enum cpu_state_vals get_cpu_current_state (int cpu); +extern void set_cpu_current_state (int cpu,enum cpu_state_vals state); +extern enum cpu_state_vals get_preferred_offline_state (int cpu); +extern void set_preferred_offline_state (int cpu,enum cpu_state_vals state); +extern int start_secondary (void ); +extern void set_default_offline_state (int cpu); +#endif
arch/powerpc/platforms/pseries/smp.c +17 -0
@@ -48,6 +48,7 @@#include "plpar_wrappers.h" #include "pseries.h" #include "xics.h" +#include "offline_driver.h" /*
@@ -86,6 +87,9 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)/* Fixup atomic count: it exited inside IRQ handler. */ task_thread_info (paca[lcpu].__current)->preempt_count =0 ; +if (get_cpu_current_state (lcpu) !=CPU_STATE_OFFLINE ) + goto out; +/* * If the RTAS start-cpu token does not exist then presume the * cpu is already spinning.
@@ -100,6 +104,7 @@ static inline int __devinit smp_startup_cpu(unsigned int lcpu)return 0 ; } +out:return 1 ; }
@@ -113,12 +118,15 @@ static void __devinit smp_xics_setup_cpu(int cpu)vpa_init (cpu);cpu_clear (cpu, of_spin_map); +set_cpu_current_state (cpu,CPU_STATE_ONLINE ); + set_default_offline_state(cpu); }#endif /* CONFIG_XICS */ static void __devinit smp_pSeries_kick_cpu (int nr) { +long rc;BUG_ON (nr <0 || nr >=NR_CPUS );if (!smp_startup_cpu (nr))
@@ -130,6 +138,15 @@ static void __devinit smp_pSeries_kick_cpu(int nr)* the processor will continue on to secondary_start */ paca[nr].cpu_start =1 ; + +set_preferred_offline_state (nr,CPU_STATE_ONLINE ); + +if (get_cpu_current_state (nr) !=CPU_STATE_OFFLINE ) { + rc =plpar_hcall_norets (H_PROD , nr); +if (rc !=H_SUCCESS ) +panic ("Error: Prod to wake up processor %d Ret= %ld\n" , + nr, rc); + } }static int smp_pSeries_cpu_bootable (unsigned int nr)
Quoted 12 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.
Quoted 7 lines
inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
Quoted 4 lines
Quoted 1 line
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
Yeah it probably is a layering violation, but when has that stopped us before :)
Is it anticipated that this will be useful on platforms other than pseries?
cheers
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
Quoted 7 lines
Quoted 1 line
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
From a design standpoint where we stand today is
1. A cede indicates that the CPU is no longer needed and can be reassigned (remember we do dedicated CPU partitions in power)
2. What this patch is trying to do is to say "We don't need the CPU, but please don't reassign, put it to sleep"
We work the same way with the hypervisor, that applications work with the OS today, by providing madvise and other hints.
Signature · 1 line
Balbir
Quoted 27 lines · Balbir Singh
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.Quoted 5 lines
Quoted 1 line
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
Quoted 6 lines
From a design standpoint where we stand today is
1. A cede indicates that the CPU is no longer needed and can be reassigned (remember we do dedicated CPU partitions in power)
2. What this patch is trying to do is to say "We don't need the CPU, but please don't reassign, put it to sleep"
FWIW, this sounds exactly like the same we have already on s390. But back then I didn't consider adding a common code infrastructure would make sense :)
Besides the "online" attribute we have an additional "configure" attribute to which can only be written if the cpu is offline. Writing a "0" to it would mean that you currently won't need the cpu anymore and the hypervisor is free to reassign the cpu to a different LPAR.
Writing a "1" to it means you want to use it. If there are enough resources you will get it. If not.. bad luck.
Quoted 4 lines · Heiko Carstens
FWIW, this sounds exactly like the same we have already on s390. But back then I didn't consider adding a common code infrastructure would make sense :)
Quoted 8 lines
Besides the "online" attribute we have an additional "configure" attribute to which can only be written if the cpu is offline. Writing a "0" to it would mean that you currently won't need the cpu anymore and the hypervisor is free to reassign the cpu to a different LPAR.
Writing a "1" to it means you want to use it. If there are enough resources you will get it. If not.. bad luck.
Maybe we should use a common API for that then. And the right way to do so is via a generically located attribute.
Ben.
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
Quoted 8 lines
Quoted 5 lines
offline: This is the the default behaviour when the cpu is offlined
inactive: This cedes the vCPU to the hypervisor with a cede latency
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
The problem is that all hypervisor managers cannot figure out what sort of latency guest OS can tolerate under the situation. They wouldn't know from what context guest OS has ceded the vcpu. It has to have some sort of hint, which is what the guest OS provides.
Thanks
Dipankar
Quoted 13 lines · Dipankar Sarma
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined
inactive: This cedes the vCPU to the hypervisor with a cede latency
Any feedback on the patchset will be immensely valuable.
Quoted 7 lines
Quoted 2 lines
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
The problem is that all hypervisor managers cannot figure out what sort of latency guest OS can tolerate under the situation. They wouldn't know from what context guest OS has ceded the vcpu. It has to have some sort of hint, which is what the guest OS provides.
I'm missing something here, hot-unplug is a slow path and should not ever be latency critical..?
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Dipankar Sarma
Quoted 14 lines · Peter Zijlstra
Quoted 10 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined
inactive: This cedes the vCPU to the hypervisor with a cede latency
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
Quoted 7 lines
Quoted 4 lines
The problem is that all hypervisor managers cannot figure out what sort of latency guest OS can tolerate under the situation. They wouldn't know from what context guest OS has ceded the vcpu. It has to have some sort of hint, which is what the guest OS provides.
I'm missing something here, hot-unplug is a slow path and should not ever be latency critical..?
You aren't, I did :)
No, for this specific case, latency isn't an issue. The issue is - how do we cede unused vcpus to hypervisor for better energy management ? Yes, it can be done by a hypervisor manager telling the kernel to offline and make a bunch of vcpus "inactive". It does have to choose offline (release vcpu) vs. inactive (cede but guranteed if needed). The problem is that long ago we exported a lot of hotplug stuff to userspace through the sysfs interface and we cannot do something inside the kernel without keeping the sysfs stuff consistent. This seems like a sane way to do that without undoing all the virtual cpu hotplug infrastructure in different supporting archs.
Thanks
Dipankar
Quoted 10 lines · Dipankar Sarma
No, for this specific case, latency isn't an issue. The issue is - how do we cede unused vcpus to hypervisor for better energy management ? Yes, it can be done by a hypervisor manager telling the kernel to offline and make a bunch of vcpus "inactive". It does have to choose offline (release vcpu) vs. inactive (cede but guranteed if needed). The problem is that long ago we exported a lot of hotplug stuff to userspace through the sysfs interface and we cannot do something inside the kernel without keeping the sysfs stuff consistent. This seems like a sane way to do that without undoing all the virtual cpu hotplug infrastructure in different supporting archs.
I'm still not getting it..
Suppose we have some guest, it booted with 4 cpus.
We then offline 2 of them.
Apparently this LPAR binds guest cpus to physical cpus?
So we use a hypervisor interface to reclaim these 2 offlined cpus and re-assign them to some other guest.
So far so good, right?
Now if you were to try and online the cpus in the guest, it'd fail because the cpus aren't backed anymore, and the hot-plug simply times-out and fails.
And we're still good, right?
Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Dipankar Sarma
No, for this specific case, latency isn't an issue. The issue is - how do we cede unused vcpus to hypervisor for better energy management ? Yes, it can be done by a hypervisor manager telling the kernel to offline and make a bunch of vcpus "inactive". It does have to choose offline (release vcpu) vs. inactive (cede but guranteed if needed). The problem is that long ago we exported a lot of hotplug stuff to userspace through the sysfs interface and we cannot do something inside the kernel without keeping the sysfs stuff consistent. This seems like a sane way to do that without undoing all the virtual cpu hotplug infrastructure in different supporting archs.
I'm still not getting it..
Suppose we have some guest, it booted with 4 cpus.
We then offline 2 of them.
Apparently this LPAR binds guest cpus to physical cpus?
So we use a hypervisor interface to reclaim these 2 offlined cpus and re-assign them to some other guest.
Quoted 7 lines
So far so good, right?
Now if you were to try and online the cpus in the guest, it'd fail because the cpus aren't backed anymore, and the hot-plug simply times-out and fails.
And we're still good, right?
The requirement differ here. If we had offlined 2 vCPUs for the purpose of system reconfiguration, the expected behavior with offline interface will work right. However the proposed cede interface is needed when we want them to temporarily go away but still come back when we do an online. We want the online to always succeed since the backing physical resources are not relinquished. The proposed interface facilitates offline without relinquishing the physical resources assigned to LPARs.
--Vaidy
Quoted 33 lines · Vaidyanathan Srinivasan
Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Dipankar Sarma
No, for this specific case, latency isn't an issue. The issue is - how do we cede unused vcpus to hypervisor for better energy management ? Yes, it can be done by a hypervisor manager telling the kernel to offline and make a bunch of vcpus "inactive". It does have to choose offline (release vcpu) vs. inactive (cede but guranteed if needed). The problem is that long ago we exported a lot of hotplug stuff to userspace through the sysfs interface and we cannot do something inside the kernel without keeping the sysfs stuff consistent. This seems like a sane way to do that without undoing all the virtual cpu hotplug infrastructure in different supporting archs.
I'm still not getting it..
Suppose we have some guest, it booted with 4 cpus.
We then offline 2 of them.
Apparently this LPAR binds guest cpus to physical cpus?
So we use a hypervisor interface to reclaim these 2 offlined cpus and re-assign them to some other guest.Quoted 7 lines
So far so good, right?
Now if you were to try and online the cpus in the guest, it'd fail because the cpus aren't backed anymore, and the hot-plug simply times-out and fails.
And we're still good, right?
Quoted 8 lines
The requirement differ here. If we had offlined 2 vCPUs for the purpose of system reconfiguration, the expected behavior with offline interface will work right. However the proposed cede interface is needed when we want them to temporarily go away but still come back when we do an online. We want the online to always succeed since the backing physical resources are not relinquished. The proposed interface facilitates offline without relinquishing the physical resources assigned to LPARs.
Then make that the platform default and leave the lpar management to whatever pokes at the lpar?
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Vaidyanathan Srinivasan
Quoted 5 lines · Peter Zijlstra
Now if you were to try and online the cpus in the guest, it'd fail because the cpus aren't backed anymore, and the hot-plug simply times-out and fails.
And we're still good, right?
Quoted 8 lines
The requirement differ here. If we had offlined 2 vCPUs for the purpose of system reconfiguration, the expected behavior with offline interface will work right. However the proposed cede interface is needed when we want them to temporarily go away but still come back when we do an online. We want the online to always succeed since the backing physical resources are not relinquished. The proposed interface facilitates offline without relinquishing the physical resources assigned to LPARs.
Quoted 2 lines
Then make that the platform default and leave the lpar management to whatever pokes at the lpar?
That could have worked - however lpar management already uses the same sysfs interface to poke. The current semantics make the lpar vcpu deconfig state the platform default assuming that it will be used for lpar management. The only clean way to do this without breaking lpar management stuff is to add another state - "inactive" and retain backward compatibility.
Thanks
Dipankar
Quoted 14 lines · Dipankar Sarma
You aren't, I did :)
No, for this specific case, latency isn't an issue. The issue is - how do we cede unused vcpus to hypervisor for better energy management ?
Yes, it can be done by a hypervisor manager telling the kernel to offline and make a bunch of vcpus "inactive". It does have to choose offline (release vcpu) vs. inactive (cede but guranteed if needed). The problem is that long ago we exported a lot of hotplug stuff to userspace through the sysfs interface and we cannot do something inside the kernel without keeping the sysfs stuff consistent. This seems like a sane way to do that without undoing all the virtual cpu hotplug infrastructure in different supporting archs.
Well, I did bring the latency into the picture. To some extent it -is- a latency issue. Though we aren't talking milliseconds here... if the CPU's been reallocated to another partition we are talking seconds or minutes or more until we can get it back :-)
In any case, this sounds to me like a perfectly valid feature to have, which s390 already does via arch specific hooks, so I fail to see why it wouldn't be legitimate to have a common attribute for it.
I don't buy into the layering violation. It's too often a straw man and an excuse for a lack of a proper reason.
Cheers,
Ben.
Quoted 3 lines · Peter Zijlstra
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
That's not how our hypervisor works.
If you ask through the management interface, to remove a CPU from a partition, the HV will communicate with a daemon inside the partition that will then unplug the CPU via the right call.
I don't really understand your objections to be honest. And I fail to see why it would be a layering violation to have the ability for the OS to indicate in what state it wishes to relinguish a CPU to the hypervisor, which more or less defines what is the expected latency for getting it back later on.
Ben.
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Peter Zijlstra
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
That's not how our hypervisor works.
Then fix it?
Quoted 9 lines
If you ask through the management interface, to remove a CPU from a partition, the HV will communicate with a daemon inside the partition that will then unplug the CPU via the right call.
I don't really understand your objections to be honest. And I fail to see why it would be a layering violation to have the ability for the OS to indicate in what state it wishes to relinguish a CPU to the hypervisor, which more or less defines what is the expected latency for getting it back later on.
OK, so the main objection is the abuse of CPU hotplug as resource management feature.
CPU hotplug is terribly invasive and expensive to the kernel, doing hotplug on a minute basis is just plain crazy.
If you want a CPU in a keep it near and don't hand it back to the HV state, why not use cpusets to isolate it and simply not run tasks on it?
cpusets don't use stopmachine and are much nicer to the rest of the kernel over-all.
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Peter Zijlstra
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
That's not how our hypervisor works.
Then fix it?
Are you serious ? :-)
Quoted 8 lines
CPU hotplug is terribly invasive and expensive to the kernel, doing hotplug on a minute basis is just plain crazy.
If you want a CPU in a keep it near and don't hand it back to the HV state, why not use cpusets to isolate it and simply not run tasks on it?
cpusets don't use stopmachine and are much nicer to the rest of the kernel over-all.
Gautham, what is the different in term of power saving between having it idle for long periods of time (which could do H_CEDE and with NO_HZ, probably wouln't need to wake up that often) and having it unplugged in a H_CEDE loop ?
Ben.
Quoted 18 lines · Benjamin Herrenschmidt
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Peter Zijlstra
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
That's not how our hypervisor works.
Then fix it?
Are you serious ? :-)
Quoted 6 lines
CPU hotplug is terribly invasive and expensive to the kernel, doing hotplug on a minute basis is just plain crazy.
If you want a CPU in a keep it near and don't hand it back to the HV state, why not use cpusets to isolate it and simply not run tasks on it?
Quoted 7 lines
Quoted 2 lines
cpusets don't use stopmachine and are much nicer to the rest of the kernel over-all.
Gautham, what is the different in term of power saving between having it idle for long periods of time (which could do
H_CEDEand withNO_HZ, probably wouln't need to wake up that often) and having it unplugged in aH_CEDEloop ?
Hi Ben,
A cede latency specifier value indicating latency expectation of the guest OS can be established in the VPA to inform the hypervisor during the H_CEDE call. Currently, we do call H_CEDE during NO_HZ for efficient idle. However, higher cede latency values may not be suitable for idle CPUs in the kernel and instead more energy savings may result from exploiting this feature through CPU hotplug interface.
--Vaidy
Quoted 22 lines · Peter Zijlstra
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Peter Zijlstra
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
That's not how our hypervisor works.
Then fix it?
Quoted 9 lines
If you ask through the management interface, to remove a CPU from a partition, the HV will communicate with a daemon inside the partition that will then unplug the CPU via the right call.
I don't really understand your objections to be honest. And I fail to see why it would be a layering violation to have the ability for the OS to indicate in what state it wishes to relinguish a CPU to the hypervisor, which more or less defines what is the expected latency for getting it back later on.
OK, so the main objection is the abuse of CPU hotplug as resource management feature.
Quoted 8 lines
CPU hotplug is terribly invasive and expensive to the kernel, doing hotplug on a minute basis is just plain crazy.
If you want a CPU in a keep it near and don't hand it back to the HV state, why not use cpusets to isolate it and simply not run tasks on it?
cpusets don't use stopmachine and are much nicer to the rest of the kernel over-all.
Hi Peter,
This interface is not expected to be used every minute or so to impact the operation of the rest of the system. Cpuhotplug is currently used as a resource management feature in virtualised system using dlpar operations. I do understand that cpu hotplug is invasive at run time and that amount of complexity is required to carefully isolate the cpu from any involvement in the running kernel.
Building another interface to isolate the cpus to the same extent as cpu hotplug does today would be redundant and is going to be equally invasive. Alternatives like cpuset for isolation migrates only tasks.
--Vaidy
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Gautham R Shenoy
This patchset contains the offline state driver implemented for pSeries. For pSeries, we define three
available_hotplug_states. They are:online: The processor is online.
offline: This is the the default behaviour when the cpu is offlined even in the absense of this driver. The CPU would call make an
rtas_stop_self()call and hand over the CPU back to the resource pool, thereby effectively deallocating that vCPU from the LPAR.
NOTE: This would result in a configuration change to the LPAR which is visible to the outside world.inactive: This cedes the vCPU to the hypervisor with a cede latency specifier value 2.
NOTE: This option does not result in a configuration change and the vCPU would be still entitled to the LPAR to which it earlier belong to.
Quoted 4 lines
Quoted 1 line
Any feedback on the patchset will be immensely valuable.
I still think its a layering violation... its the hypervisor manager that should be bothered in what state an off-lined cpu is in.
Agreed. Proposed interface is ugly.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html