Update hotplug for pseries systems
cover Update hotplug for pseries systems
In order to support device hotplug (cpu, memory, and pci) in both the PowerVM and the PowerKVM environments the handling of hotplug events will need to be updated. This patch set adresses this by creating a common entry point for handling hotplug events in the kernel that can be used in both PowerVM and PowerKVM environments. To accomplish this several changes need to be made.
For PowerVM systems hotplug (DLPAR) events are requested by users from the HMC which then communicates the request to the partitions via the RSCT framework. The RSCT framework then invokes the drmgr command to handle the hotplug event. The drmgr command performs some of the work in user space and makes calls into the kernel to handle the remaining work.
For PowerKVM systems hotplug events are communicated to the guest via the ras epow interrupt by qemu. We could have the rtas event sent up to user space through rtas_errd that would then invoke drmgr. This is not the most ideal solution and it would be nicer to have hotplug handled completely in the kernel.
To do this, hotplug events will now be communicated to the kernel in the form of rtas hotplug events. For PowerKVM systems this is done by qemu using the ras epow interrupt. For PowerVM systems the drmgr command will be updated to create a rtas hotplug event and send it to the kernel via a new /proc/powerpc/dlpar interface. Both of these entry points for hotplug rtas events then call a common routine for handling rtas hotplug events.
Additionally we will need to be able to handle all of the work to do resource hotplug in the kernel. This patch set addresses this for cpu and memory hotplug, pci hotplug will be done later.
Once all of the work is done to move hotplug handling into the kernel we should also be able to get rid of the /proc/powerpc/ofdt interface.
Of course these updates do depend on updating the drmgr command. If you care to look the updates for this is here;
https://github.com/nfont/powerpc-utils/tree/mem_rtas_hp
Patch 1/4:
o Create a common rtas hotplug event handling routine
o Create the /proc/powerpc/dlpar interface for PowerVM systems
o Implement memory hotplug handling in the kernel.
Patch 2/4:
o Move the cpu hotplug code from pseries/dlpar.c to pseries/hotplug-cpu.c
Patch 3/4:
o Update cpu hotplug handling to allow for invocation from rtas event notifications.
Patch 4/4:
o Update the ras epow interrupt handler to recognize hotplug rtas events
This code (except for patch 4/4 which I cannot test right now) has been tested on PowerVM systems. There are some error paths that still need to be tested and of course testing on PowerKVM when cpu and memory hotplug is enabled for Power.
Thoughts?
-Nathan
--- include/asm/rtas.h | 27 ++ kernel/rtas.c | 7 platforms/pseries/dlpar.c | 253 ++++++-------------------- platforms/pseries/hotplug-cpu.c | 358 +++++++++++++++++++++++++++++++++++++ platforms/pseries/hotplug-memory.c | 351 ++++++++++++++++++++++++++++++------ platforms/pseries/pseries.h | 6 platforms/pseries/ras.c | 12 + platforms/pseries/reconfig.c | 6 8 files changed, 780 insertions(+), 240 deletions(-)
RFC PATCH 1/4 Create interface for rtas hotplug events and move mem hotplug to the kernel
In order to support hotplug of memory, cpu and pci devices in the PowerVM and the PowerKVM environments we will need to provide a single entry point. To do this requires updating the way in which we handle hotplug requests in the PowerVM environment. The idea is to have all of the hotplug in the kernel so that a hotplug rtas event can used to initiate the hotplug add/remove of a device.
The current method for handling a hotplug request in a PowerVM partition is to have the HMC notify the partition of the request through the RSCT framework which then invokes the drmgr command to hotplug add/remove the requested devices. The drmgr command does part of this in user-space and part in the kernel via sysfs and /proc interfaces.
This patch creates the entry point for initiating a hotplug request for pseries with a rtas hotplug event. For PowerVM systems the drmgr command will now create and write a hotplug rtas event to /proc/powerpc/dlpar which will then pass the hotplug rtas event to the entry point. For PowerKVM systems QEMU will generate an epow interrupt to the guest, which then calls rtas-check-execption to get the hotplug rtas event and pass it to the entry point. NOTE that the updates to handle hotplug events from epow interrupts is not in this intial patch.
This patch also adds funtionality so that we can do memory hotplug in the kernel. Using the updates to drmgr found below you can initiate memory hotplug events using the new interface.
https://github.com/nfont/powerpc-utils/tree/mem_rtas_hp
--- arch/powerpc/include/asm/rtas.h | 26 ++ arch/powerpc/kernel/rtas.c | 7 + arch/powerpc/platforms/pseries/dlpar.c | 65 ++++- arch/powerpc/platforms/pseries/hotplug-memory.c | 351 ++++++++++++++++++++---- arch/powerpc/platforms/pseries/pseries.h | 4 + arch/powerpc/platforms/pseries/reconfig.c | 6 + 6 files changed, 403 insertions(+), 56 deletions(-)
arch/powerpc/include/asm/rtas.h +26 -0
@@ -254,6 +254,31 @@ inline uint32_t rtas_ext_event_company_id(struct rtas_ext_event_log_v6 *ext_log)return be32_to_cpu (ext_log->company_id); } +/* RTAS pseries hotplug elog section */ +struct pseries_hp_elog { +uint8_t resource; +uint8_t action:8 ; +uint8_t id_type:8 ; +uint8_t reserved; +union { +__be32 drc_index; +__be32 drc_count; +char drc_name[1 ]; + }_drc_u; +}; + +#define HP_ELOG_RESOURCE_CPU 1 +#define HP_ELOG_RESOURCE_MEM 2 +#define HP_ELOG_RESOURCE_SLOT 3 +#define HP_ELOG_RESOURCE_PHB 4 + +#define HP_ELOG_ACTION_ADD 1 +#define HP_ELOG_ACTION_REMOVE 2 + +#define HP_ELOG_ID_DRC_NAME 1 +#define HP_ELOG_ID_DRC_INDEX 2 +#define HP_ELOG_ID_DRC_COUNT 3 +/* pSeries event log format */ /* Two bytes ASCII section IDs */
@@ -273,6 +298,7 @@ inline uint32_t rtas_ext_event_company_id(struct rtas_ext_event_log_v6 *ext_log)#define PSERIES_ELOG_SECT_ID_MANUFACT_INFO (('M' << 8) | 'I')#define PSERIES_ELOG_SECT_ID_CALL_HOME (('C' << 8) | 'H')#define PSERIES_ELOG_SECT_ID_USER_DEF (('U' << 8) | 'D') +#define PSERIES_ELOG_SECT_ID_HP (('H' << 8) | 'P')/* Vendor specific Platform Event Log Format, Version 6, section header */ struct pseries_errorlog {
arch/powerpc/kernel/rtas.c +7 -0
@@ -997,6 +997,13 @@ struct pseries_errorlog *get_pseries_errorlog(struct rtas_error_log *log,uint8_t log_format =rtas_ext_event_log_format (ext_log);uint32_t company_id =rtas_ext_event_company_id (ext_log); +printk (KERN_EMERG "Validation: %x : %lx\n%x : %x\n%x : %x\n" , + log->extended_log_length,sizeof (struct rtas_ext_event_log_v6 ), +rtas_ext_event_log_format (ext_log), +RTAS_V6EXT_LOG_FORMAT_EVENT_LOG , +rtas_ext_event_company_id (ext_log), +RTAS_V6EXT_COMPANY_ID_IBM ); +/* Check that we understand the format */ if (ext_log_length < sizeof(struct rtas_ext_event_log_v6) || log_format != RTAS_V6EXT_LOG_FORMAT_EVENT_LOG ||
arch/powerpc/platforms/pseries/dlpar.c +62 -3
84 lines not on this page · raw message
arch/powerpc/platforms/pseries/hotplug-memory.c +298 -53
401 lines not on this page · raw message
arch/powerpc/platforms/pseries/pseries.h +4 -0
@@ -11,6 +11,7 @@#define _PSERIES_PSERIES_H#include <linux/interrupt.h> +#include <asm/rtas.h> struct device_node ;
@@ -59,6 +60,9 @@ extern void dlpar_free_cc_property(struct property *);extern struct device_node *dlpar_configure_connector (u32 ,struct device_node *);extern int dlpar_attach_node (struct device_node *);extern int dlpar_detach_node (struct device_node *); +extern int dlpar_acquire_drc (u32 ); +extern int dlpar_release_drc (u32 ); +extern int dlpar_memory (struct pseries_hp_elog *);/* PCI root bridge prepare function override for pseries */ struct pci_host_bridge ;
arch/powerpc/platforms/pseries/reconfig.c +6 -0
@@ -370,6 +370,12 @@ static int do_update_property(char *buf, size_t bufsize)if (!strlen (name))return -ENODEV ; +/* updating the ibm,dynamic-memory property is no longer +* supported through this interface. +*/ +if (!strcmp (name,"ibm,dynamic-memory" )) +return -EINVAL ; + newprop =new_property (name, length, value,NULL );if (!newprop)return -ENOMEM ;
Signature · 1 line
2.0.0.rc3.2.g998f840
RFC PATCH 2/4 Migrate cpu hotplug code to pseries/hotplug-cpu.c
In reply to Nathan FontenotThis patch moves the cpu hotplug handling code from pseries/dlpar.c to pseries/hotplug-cpu.c. Additionally it factors out the work to add/remove a single cpu into its own routine.
--- arch/powerpc/platforms/pseries/dlpar.c | 182 ------------------------- arch/powerpc/platforms/pseries/hotplug-cpu.c | 194 +++++++++++++++++++++++++++ 2 files changed, 194 insertions(+), 182 deletions(-)
arch/powerpc/platforms/pseries/dlpar.c +0 -182
@@ -359,182 +359,6 @@ int dlpar_release_drc(u32 drc_index)return 0 ; } -#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE - -static int dlpar_online_cpu (struct device_node *dn) -{ -int rc =0 ; -unsigned int cpu; -int len, nthreads, i; -const u32 *intserv; - - intserv =of_get_property (dn,"ibm,ppc-interrupt-server#s" , &len); -if (!intserv) -return -EINVAL ; - - nthreads = len /sizeof (u32); - -cpu_maps_update_begin (); -for (i =0 ; i < nthreads; i++) { -for_each_present_cpu (cpu) { -if (get_hard_smp_processor_id (cpu) != intserv[i]) -continue ; -BUG_ON (get_cpu_current_state (cpu) - !=CPU_STATE_OFFLINE ); -cpu_maps_update_done (); - rc =cpu_up (cpu); -if (rc) - goto out; -cpu_maps_update_begin (); - -break ; - } -if (cpu ==num_possible_cpus ()) -printk (KERN_WARNING "Could not find cpu to online " -"with physical id 0x%x\n" , intserv[i]); - } -cpu_maps_update_done (); - -out: -return rc; - -} - -static ssize_t dlpar_cpu_probe (const char *buf,size_t count) -{ -struct device_node *dn, *parent; -unsigned long drc_index; -int rc; - - rc =strict_strtoul (buf,0 , &drc_index); -if (rc) -return -EINVAL ; - - parent =of_find_node_by_path ("/cpus" ); -if (!parent) -return -ENODEV ; - - dn =dlpar_configure_connector (drc_index, parent); -if (!dn) -return -EINVAL ; - -of_node_put (parent); - - rc =dlpar_acquire_drc (drc_index); -if (rc) { -dlpar_free_cc_nodes (dn); -return -EINVAL ; - } - - rc =dlpar_attach_node (dn); -if (rc) { -dlpar_release_drc (drc_index); -dlpar_free_cc_nodes (dn); -return rc; - } - - rc =dlpar_online_cpu (dn); -if (rc) -return rc; - -return count; -} - -static int dlpar_offline_cpu (struct device_node *dn) -{ -int rc =0 ; -unsigned int cpu; -int len, nthreads, i; -const u32 *intserv; - - intserv =of_get_property (dn,"ibm,ppc-interrupt-server#s" , &len); -if (!intserv) -return -EINVAL ; - - nthreads = len /sizeof (u32); - -cpu_maps_update_begin (); -for (i =0 ; i < nthreads; i++) { -for_each_present_cpu (cpu) { -if (get_hard_smp_processor_id (cpu) != intserv[i]) -continue ; - -if (get_cpu_current_state (cpu) ==CPU_STATE_OFFLINE ) -break ; - -if (get_cpu_current_state (cpu) ==CPU_STATE_ONLINE ) { -set_preferred_offline_state (cpu,CPU_STATE_OFFLINE ); -cpu_maps_update_done (); - rc =cpu_down (cpu); -if (rc) - goto out; -cpu_maps_update_begin (); -break ; - - } - -/* -* The cpu is in CPU_STATE_INACTIVE. -* Upgrade it's state to CPU_STATE_OFFLINE. -*/ -set_preferred_offline_state (cpu,CPU_STATE_OFFLINE ); -BUG_ON (plpar_hcall_norets (H_PROD , intserv[i]) - !=H_SUCCESS ); -__cpu_die (cpu); -break ; - } -if (cpu ==num_possible_cpus ()) -printk (KERN_WARNING "Could not find cpu to offline " -"with physical id 0x%x\n" , intserv[i]); - } -cpu_maps_update_done (); - -out: -return rc; - -} - -static ssize_t dlpar_cpu_release (const char *buf,size_t count) -{ -struct device_node *dn; -const u32 *drc_index; -int rc; - - dn =of_find_node_by_path (buf); -if (!dn) -return -EINVAL ; - - drc_index =of_get_property (dn,"ibm,my-drc-index" ,NULL ); -if (!drc_index) { -of_node_put (dn); -return -EINVAL ; - } - - rc =dlpar_offline_cpu (dn); -if (rc) { -of_node_put (dn); -return -EINVAL ; - } - - rc =dlpar_release_drc (*drc_index); -if (rc) { -of_node_put (dn); -return rc; - } - - rc =dlpar_detach_node (dn); -if (rc) { -dlpar_acquire_drc (*drc_index); -return rc; - } - -of_node_put (dn); - -return count; -} - -#endif /* CONFIG_ARCH_CPU_PROBE_RELEASE */ -static int handle_dlpar_errorlog (struct rtas_error_log *error_log) {struct pseries_errorlog *pseries_log;
@@ -589,12 +413,6 @@ static int __init pseries_dlpar_init(void)if (proc_ent)proc_set_size (proc_ent,0 ); -#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE - ppc_md.cpu_probe = dlpar_cpu_probe; - ppc_md.cpu_release = dlpar_cpu_release; -#endif /* CONFIG_ARCH_CPU_PROBE_RELEASE */ - -return 0 ; }machine_device_initcall (pseries, pseries_dlpar_init);
arch/powerpc/platforms/pseries/hotplug-cpu.c +194 -0
212 lines not on this page · raw message
Signature · 1 line
2.0.0.rc3.2.g998f840
RFC PATCH 3/4 Handle cpu hotplug from rtas hotplug events
In reply to Nathan FontenotThis patch updates the cpu hotplug handling code so that we can perform cpu hotplug using the new rtas hotplug event interface while still maintaining the ability to use the probe/release sysfs interface for adding and removing cpus.
At a later point we could deprecate the use of the probe/release sysfs files and remove those code bits.
--- arch/powerpc/platforms/pseries/dlpar.c | 4 + arch/powerpc/platforms/pseries/hotplug-cpu.c | 164 +++++++++++++++++++++++++++ arch/powerpc/platforms/pseries/pseries.h | 1 + 3 files changed, 169 insertions(+)
arch/powerpc/platforms/pseries/dlpar.c +4 -0
@@ -275,6 +275,7 @@ int dlpar_attach_node(struct device_node *dn)if (!dn->parent)return -ENOMEM ; +of_node_init (dn); rc =of_attach_node (dn);if (rc) { printk(KERN_ERR "Failed to add device node %s\n",
@@ -374,6 +375,9 @@ static int handle_dlpar_errorlog(struct rtas_error_log *error_log)case HP_ELOG_RESOURCE_MEM : rc =dlpar_memory (hp_elog);break ; +case HP_ELOG_RESOURCE_CPU : + rc =dlpar_cpus (hp_elog); +break ; }return rc;
arch/powerpc/platforms/pseries/hotplug-cpu.c +164 -0
176 lines not on this page · raw message
arch/powerpc/platforms/pseries/pseries.h +1 -0
@@ -63,6 +63,7 @@ extern int dlpar_detach_node(struct device_node *);extern int dlpar_acquire_drc (u32 );extern int dlpar_release_drc (u32 );extern int dlpar_memory (struct pseries_hp_elog *); +extern int dlpar_cpus (struct pseries_hp_elog *);/* PCI root bridge prepare function override for pseries */ struct pci_host_bridge ;
Signature · 1 line
2.0.0.rc3.2.g998f840
RFC PATCH 4/4 Hook into ras epow interrupt handler for hotplug
In reply to Nathan FontenotThis patch hooks into the ras EPOW interrupt handler so that we can communicate hotplug rtas events to a PowerKVM guest from qemu.
The ras epow interrupt wil lnow check for hotplgu rtas events and invoke the common handling routine accordingly.
--- arch/powerpc/include/asm/rtas.h | 1 + arch/powerpc/platforms/pseries/dlpar.c | 2 +- arch/powerpc/platforms/pseries/pseries.h | 1 + arch/powerpc/platforms/pseries/ras.c | 12 +++++++++++- 4 files changed, 14 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/rtas.h +1 -0
@@ -124,6 +124,7 @@ struct rtas_suspend_me_data {#define RTAS_TYPE_INFO 0xE2#define RTAS_TYPE_DEALLOC 0xE3#define RTAS_TYPE_DUMP 0xE4 +#define RTAS_TYPE_HOTPLUG 0xE5/* I don't add PowerMGM events right now, this is a different topic */ #define RTAS_TYPE_PMGM_POWER_SW_ON 0x60#define RTAS_TYPE_PMGM_POWER_SW_OFF 0x61
arch/powerpc/platforms/pseries/dlpar.c +1 -1
@@ -360,7 +360,7 @@ int dlpar_release_drc(u32 drc_index)return 0 ; } -static int handle_dlpar_errorlog (struct rtas_error_log *error_log) +int handle_dlpar_errorlog (struct rtas_error_log *error_log) {struct pseries_errorlog *pseries_log;struct pseries_hp_elog *hp_elog;
arch/powerpc/platforms/pseries/pseries.h +1 -0
@@ -64,6 +64,7 @@ extern int dlpar_acquire_drc(u32);extern int dlpar_release_drc (u32 );extern int dlpar_memory (struct pseries_hp_elog *);extern int dlpar_cpus (struct pseries_hp_elog *); +extern int handle_dlpar_errorlog (struct rtas_error_log *);/* PCI root bridge prepare function override for pseries */ struct pci_host_bridge ;
arch/powerpc/platforms/pseries/ras.c +11 -1
@@ -183,6 +183,7 @@ void rtas_parse_epow_errlog(struct rtas_error_log *log)/* Handle environmental and power warning (EPOW) interrupts. */ static irqreturn_t ras_epow_interrupt (int irq,void *dev_id) { +struct rtas_error_log *elog;int status;int state;int critical;
@@ -205,7 +206,16 @@ static irqreturn_t ras_epow_interrupt(int irq, void *dev_id)log_error (ras_log_buf,ERR_TYPE_RTAS_LOG ,0 ); -rtas_parse_epow_errlog ((struct rtas_error_log *)ras_log_buf); + elog = (struct rtas_error_log *)ras_log_buf; + +switch (rtas_error_type (elog)) { +case RTAS_TYPE_EPOW : +rtas_parse_epow_errlog (elog); +break ; +case RTAS_TYPE_HOTPLUG : +handle_dlpar_errorlog (elog); +break ; + }spin_unlock (&ras_log_buf_lock);return IRQ_HANDLED ;
Signature · 1 line
2.0.0.rc3.2.g998f840