Update hotplug for pseries systems

RFC PATCH New Nathan Fontenot 1 participant 5 messages Jun 17, 2014 PowerPC

Share & tools

Nathan Fontenot

author

2014-06-17 15:42 UTC #0 lore raw

cover Update hotplug for pseries systems

In order to support device hotplug (cpu, memory, and pci) in both the PowerVM and the PowerKVM environments the handling of hotplug events will need to be updated. This patch set adresses this by creating a common entry point for handling hotplug events in the kernel that can be used in both PowerVM and PowerKVM environments. To accomplish this several changes need to be made.

For PowerVM systems hotplug (DLPAR) events are requested by users from the HMC which then communicates the request to the partitions via the RSCT framework. The RSCT framework then invokes the drmgr command to handle the hotplug event. The drmgr command performs some of the work in user space and makes calls into the kernel to handle the remaining work.

For PowerKVM systems hotplug events are communicated to the guest via the ras epow interrupt by qemu. We could have the rtas event sent up to user space through rtas_errd that would then invoke drmgr. This is not the most ideal solution and it would be nicer to have hotplug handled completely in the kernel.

To do this, hotplug events will now be communicated to the kernel in the form of rtas hotplug events. For PowerKVM systems this is done by qemu using the ras epow interrupt. For PowerVM systems the drmgr command will be updated to create a rtas hotplug event and send it to the kernel via a new /proc/powerpc/dlpar interface. Both of these entry points for hotplug rtas events then call a common routine for handling rtas hotplug events.

Additionally we will need to be able to handle all of the work to do resource hotplug in the kernel. This patch set addresses this for cpu and memory hotplug, pci hotplug will be done later.

Once all of the work is done to move hotplug handling into the kernel we should also be able to get rid of the /proc/powerpc/ofdt interface.

Of course these updates do depend on updating the drmgr command. If you care to look the updates for this is here;

https://github.com/nfont/powerpc-utils/tree/mem_rtas_hp

Patch 1/4:
o Create a common rtas hotplug event handling routine
o Create the /proc/powerpc/dlpar interface for PowerVM systems
o Implement memory hotplug handling in the kernel.

Patch 2/4:
o Move the cpu hotplug code from pseries/dlpar.c to pseries/hotplug-cpu.c

Patch 3/4:
o Update cpu hotplug handling to allow for invocation from rtas event notifications.

Patch 4/4:
o Update the ras epow interrupt handler to recognize hotplug rtas events

This code (except for patch 4/4 which I cannot test right now) has been tested on PowerVM systems. There are some error paths that still need to be tested and of course testing on PowerKVM when cpu and memory hotplug is enabled for Power.

Thoughts?

-Nathan

---
 include/asm/rtas.h                 |   27 ++
 kernel/rtas.c                      |    7 
 platforms/pseries/dlpar.c          |  253 ++++++--------------------
 platforms/pseries/hotplug-cpu.c    |  358 +++++++++++++++++++++++++++++++++++++
 platforms/pseries/hotplug-memory.c |  351 ++++++++++++++++++++++++++++++------ 
 platforms/pseries/pseries.h        |    6                                      
 platforms/pseries/ras.c            |   12 +                                    
 platforms/pseries/reconfig.c       |    6 
 8 files changed, 780 insertions(+), 240 deletions(-)

Nathan Fontenot

author

2014-06-17 15:45 UTC #1 lore raw

RFC PATCH 1/4 Create interface for rtas hotplug events and move mem hotplug to the kernel

In order to support hotplug of memory, cpu and pci devices in the PowerVM and the PowerKVM environments we will need to provide a single entry point. To do this requires updating the way in which we handle hotplug requests in the PowerVM environment. The idea is to have all of the hotplug in the kernel so that a hotplug rtas event can used to initiate the hotplug add/remove of a device.

The current method for handling a hotplug request in a PowerVM partition is to have the HMC notify the partition of the request through the RSCT framework which then invokes the drmgr command to hotplug add/remove the requested devices. The drmgr command does part of this in user-space and part in the kernel via sysfs and /proc interfaces.

This patch creates the entry point for initiating a hotplug request for pseries with a rtas hotplug event. For PowerVM systems the drmgr command will now create and write a hotplug rtas event to /proc/powerpc/dlpar which will then pass the hotplug rtas event to the entry point. For PowerKVM systems QEMU will generate an epow interrupt to the guest, which then calls rtas-check-execption to get the hotplug rtas event and pass it to the entry point. NOTE that the updates to handle hotplug events from epow interrupts is not in this intial patch.

This patch also adds funtionality so that we can do memory hotplug in the kernel. Using the updates to drmgr found below you can initiate memory hotplug events using the new interface.

https://github.com/nfont/powerpc-utils/tree/mem_rtas_hp

---
 arch/powerpc/include/asm/rtas.h                 |  26 ++
 arch/powerpc/kernel/rtas.c                      |   7 +
 arch/powerpc/platforms/pseries/dlpar.c          |  65 ++++-
 arch/powerpc/platforms/pseries/hotplug-memory.c | 351 ++++++++++++++++++++----
 arch/powerpc/platforms/pseries/pseries.h        |   4 +
 arch/powerpc/platforms/pseries/reconfig.c       |   6 +
 6 files changed, 403 insertions(+), 56 deletions(-)

arch/powerpc/include/asm/rtas.h +26 -0

index b390f55..26491ae 100644
@@ -254,6 +254,31 @@ inline uint32_t rtas_ext_event_company_id(struct rtas_ext_event_log_v6 *ext_log)
 	return be32_to_cpu(ext_log->company_id);
 }
 
+/* RTAS pseries hotplug elog section */ 
+struct pseries_hp_elog {
+	uint8_t		resource;
+	uint8_t		action:8;
+        uint8_t		id_type:8;
+        uint8_t		reserved;
+        union {
+		__be32	drc_index;
+		__be32	drc_count;
+		char	drc_name[1];
+        }_drc_u;
+};
+
+#define HP_ELOG_RESOURCE_CPU	1
+#define HP_ELOG_RESOURCE_MEM	2
+#define HP_ELOG_RESOURCE_SLOT	3
+#define HP_ELOG_RESOURCE_PHB	4
+
+#define HP_ELOG_ACTION_ADD	1
+#define HP_ELOG_ACTION_REMOVE	2
+
+#define HP_ELOG_ID_DRC_NAME	1
+#define HP_ELOG_ID_DRC_INDEX	2
+#define HP_ELOG_ID_DRC_COUNT	3
+
 /* pSeries event log format */
 
 /* Two bytes ASCII section IDs */

@@ -273,6 +298,7 @@ inline uint32_t rtas_ext_event_company_id(struct rtas_ext_event_log_v6 *ext_log)
 #define PSERIES_ELOG_SECT_ID_MANUFACT_INFO	(('M' << 8) | 'I')
 #define PSERIES_ELOG_SECT_ID_CALL_HOME		(('C' << 8) | 'H')
 #define PSERIES_ELOG_SECT_ID_USER_DEF		(('U' << 8) | 'D')
+#define PSERIES_ELOG_SECT_ID_HP			(('H' << 8) | 'P')
 
 /* Vendor specific Platform Event Log Format, Version 6, section header */
 struct pseries_errorlog {

arch/powerpc/kernel/rtas.c +7 -0

index 8cd5ed0..b738b1b 100644
@@ -997,6 +997,13 @@ struct pseries_errorlog *get_pseries_errorlog(struct rtas_error_log *log,
 	uint8_t log_format = rtas_ext_event_log_format(ext_log);
 	uint32_t company_id = rtas_ext_event_company_id(ext_log);
 
+	printk(KERN_EMERG "Validation: %x : %lx\n%x : %x\n%x : %x\n",
+		log->extended_log_length, sizeof(struct rtas_ext_event_log_v6),
+		rtas_ext_event_log_format(ext_log),
+		RTAS_V6EXT_LOG_FORMAT_EVENT_LOG,
+		rtas_ext_event_company_id(ext_log),
+		RTAS_V6EXT_COMPANY_ID_IBM);
+		
 	/* Check that we understand the format */
 	if (ext_log_length < sizeof(struct rtas_ext_event_log_v6) ||
 	    log_format != RTAS_V6EXT_LOG_FORMAT_EVENT_LOG ||

arch/powerpc/platforms/pseries/dlpar.c +62 -3

84 lines not on this page · raw message

arch/powerpc/platforms/pseries/hotplug-memory.c +298 ~~-53~~

401 lines not on this page · raw message

arch/powerpc/platforms/pseries/pseries.h +4 -0

index 9921953..89c25769 100644
@@ -11,6 +11,7 @@
 #define _PSERIES_PSERIES_H
 
 #include <linux/interrupt.h>
+#include <asm/rtas.h>
 
 struct device_node;

@@ -59,6 +60,9 @@ extern void dlpar_free_cc_property(struct property *);
 extern struct device_node *dlpar_configure_connector(u32, struct device_node *);
 extern int dlpar_attach_node(struct device_node *);
 extern int dlpar_detach_node(struct device_node *);
+extern int dlpar_acquire_drc(u32);
+extern int dlpar_release_drc(u32);
+extern int dlpar_memory(struct pseries_hp_elog *);
 
 /* PCI root bridge prepare function override for pseries */
 struct pci_host_bridge;

arch/powerpc/platforms/pseries/reconfig.c +6 -0

index 0435bb6..8a1f3cf4 100644
@@ -370,6 +370,12 @@ static int do_update_property(char *buf, size_t bufsize)
 	if (!strlen(name))
 		return -ENODEV;
 
+	/* updating the ibm,dynamic-memory property is no longer
+	 * supported through this interface.
+	 */
+	if (!strcmp(name, "ibm,dynamic-memory"))
+		return -EINVAL;
+
 	newprop = new_property(name, length, value, NULL);
 	if (!newprop)
 		return -ENOMEM;

Signature · 1 line

2.0.0.rc3.2.g998f840

Nathan Fontenot

author

2014-06-17 15:46 UTC #2 lore raw

RFC PATCH 2/4 Migrate cpu hotplug code to pseries/hotplug-cpu.c

In reply to Nathan Fontenot

This patch moves the cpu hotplug handling code from pseries/dlpar.c to pseries/hotplug-cpu.c. Additionally it factors out the work to add/remove a single cpu into its own routine.

---
 arch/powerpc/platforms/pseries/dlpar.c       | 182 -------------------------
 arch/powerpc/platforms/pseries/hotplug-cpu.c | 194 +++++++++++++++++++++++++++
 2 files changed, 194 insertions(+), 182 deletions(-)

arch/powerpc/platforms/pseries/dlpar.c +0 ~~-182~~

index dfca23b..16c85b9 100644
@@ -359,182 +359,6 @@ int dlpar_release_drc(u32 drc_index)
 	return 0;
 }
 
-#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE
-
-static int dlpar_online_cpu(struct device_node *dn)
-{
-	int rc = 0;
-	unsigned int cpu;
-	int len, nthreads, i;
-	const u32 *intserv;
-
-	intserv = of_get_property(dn, "ibm,ppc-interrupt-server#s", &len);
-	if (!intserv)
-		return -EINVAL;
-
-	nthreads = len / sizeof(u32);
-
-	cpu_maps_update_begin();
-	for (i = 0; i < nthreads; i++) {
-		for_each_present_cpu(cpu) {
-			if (get_hard_smp_processor_id(cpu) != intserv[i])
-				continue;
-			BUG_ON(get_cpu_current_state(cpu)
-					!= CPU_STATE_OFFLINE);
-			cpu_maps_update_done();
-			rc = cpu_up(cpu);
-			if (rc)
-				goto out;
-			cpu_maps_update_begin();
-
-			break;
-		}
-		if (cpu == num_possible_cpus())
-			printk(KERN_WARNING "Could not find cpu to online "
-			       "with physical id 0x%x\n", intserv[i]);
-	}
-	cpu_maps_update_done();
-
-out:
-	return rc;
-
-}
-
-static ssize_t dlpar_cpu_probe(const char *buf, size_t count)
-{
-	struct device_node *dn, *parent;
-	unsigned long drc_index;
-	int rc;
-
-	rc = strict_strtoul(buf, 0, &drc_index);
-	if (rc)
-		return -EINVAL;
-
-	parent = of_find_node_by_path("/cpus");
-	if (!parent)
-		return -ENODEV;
-
-	dn = dlpar_configure_connector(drc_index, parent);
-	if (!dn)
-		return -EINVAL;
-
-	of_node_put(parent);
-
-	rc = dlpar_acquire_drc(drc_index);
-	if (rc) {
-		dlpar_free_cc_nodes(dn);
-		return -EINVAL;
-	}
-
-	rc = dlpar_attach_node(dn);
-	if (rc) {
-		dlpar_release_drc(drc_index);
-		dlpar_free_cc_nodes(dn);
-		return rc;
-	}
-
-	rc = dlpar_online_cpu(dn);
-	if (rc)
-		return rc;
-
-	return count;
-}
-
-static int dlpar_offline_cpu(struct device_node *dn)
-{
-	int rc = 0;
-	unsigned int cpu;
-	int len, nthreads, i;
-	const u32 *intserv;
-
-	intserv = of_get_property(dn, "ibm,ppc-interrupt-server#s", &len);
-	if (!intserv)
-		return -EINVAL;
-
-	nthreads = len / sizeof(u32);
-
-	cpu_maps_update_begin();
-	for (i = 0; i < nthreads; i++) {
-		for_each_present_cpu(cpu) {
-			if (get_hard_smp_processor_id(cpu) != intserv[i])
-				continue;
-
-			if (get_cpu_current_state(cpu) == CPU_STATE_OFFLINE)
-				break;
-
-			if (get_cpu_current_state(cpu) == CPU_STATE_ONLINE) {
-				set_preferred_offline_state(cpu, CPU_STATE_OFFLINE);
-				cpu_maps_update_done();
-				rc = cpu_down(cpu);
-				if (rc)
-					goto out;
-				cpu_maps_update_begin();
-				break;
-
-			}
-
-			/*
-			 * The cpu is in CPU_STATE_INACTIVE.
-			 * Upgrade it's state to CPU_STATE_OFFLINE.
-			 */
-			set_preferred_offline_state(cpu, CPU_STATE_OFFLINE);
-			BUG_ON(plpar_hcall_norets(H_PROD, intserv[i])
-								!= H_SUCCESS);
-			__cpu_die(cpu);
-			break;
-		}
-		if (cpu == num_possible_cpus())
-			printk(KERN_WARNING "Could not find cpu to offline "
-			       "with physical id 0x%x\n", intserv[i]);
-	}
-	cpu_maps_update_done();
-
-out:
-	return rc;
-
-}
-
-static ssize_t dlpar_cpu_release(const char *buf, size_t count)
-{
-	struct device_node *dn;
-	const u32 *drc_index;
-	int rc;
-
-	dn = of_find_node_by_path(buf);
-	if (!dn)
-		return -EINVAL;
-
-	drc_index = of_get_property(dn, "ibm,my-drc-index", NULL);
-	if (!drc_index) {
-		of_node_put(dn);
-		return -EINVAL;
-	}
-
-	rc = dlpar_offline_cpu(dn);
-	if (rc) {
-		of_node_put(dn);
-		return -EINVAL;
-	}
-
-	rc = dlpar_release_drc(*drc_index);
-	if (rc) {
-		of_node_put(dn);
-		return rc;
-	}
-
-	rc = dlpar_detach_node(dn);
-	if (rc) {
-		dlpar_acquire_drc(*drc_index);
-		return rc;
-	}
-
-	of_node_put(dn);
-
-	return count;
-}
-
-#endif /* CONFIG_ARCH_CPU_PROBE_RELEASE */
-
 static int handle_dlpar_errorlog(struct rtas_error_log *error_log)
 {
 	struct pseries_errorlog *pseries_log;

@@ -589,12 +413,6 @@ static int __init pseries_dlpar_init(void)
 	if (proc_ent)
 		proc_set_size(proc_ent, 0);
 
-#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE
-	ppc_md.cpu_probe = dlpar_cpu_probe;
-	ppc_md.cpu_release = dlpar_cpu_release;
-#endif /* CONFIG_ARCH_CPU_PROBE_RELEASE */
-
-
 	return 0;
 }
 machine_device_initcall(pseries, pseries_dlpar_init);

arch/powerpc/platforms/pseries/hotplug-cpu.c +194 -0

212 lines not on this page · raw message

Signature · 1 line

2.0.0.rc3.2.g998f840

Nathan Fontenot

author

2014-06-17 15:47 UTC #3 lore raw

RFC PATCH 3/4 Handle cpu hotplug from rtas hotplug events

In reply to Nathan Fontenot

This patch updates the cpu hotplug handling code so that we can perform cpu hotplug using the new rtas hotplug event interface while still maintaining the ability to use the probe/release sysfs interface for adding and removing cpus.

At a later point we could deprecate the use of the probe/release sysfs files and remove those code bits.

---
 arch/powerpc/platforms/pseries/dlpar.c       |   4 +
 arch/powerpc/platforms/pseries/hotplug-cpu.c | 164 +++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/pseries.h     |   1 +
 3 files changed, 169 insertions(+)

arch/powerpc/platforms/pseries/dlpar.c +4 -0

index 16c85b9..53f4fe6 100644
@@ -275,6 +275,7 @@ int dlpar_attach_node(struct device_node *dn)
 	if (!dn->parent)
 		return -ENOMEM;
 
+	of_node_init(dn);
 	rc = of_attach_node(dn);
 	if (rc) {
 		printk(KERN_ERR "Failed to add device node %s\n",

@@ -374,6 +375,9 @@ static int handle_dlpar_errorlog(struct rtas_error_log *error_log)
 	case HP_ELOG_RESOURCE_MEM:
 		rc = dlpar_memory(hp_elog);
 		break;
+	case HP_ELOG_RESOURCE_CPU:
+		rc = dlpar_cpus(hp_elog);
+		break;
 	}
 
 	return rc;

arch/powerpc/platforms/pseries/hotplug-cpu.c +164 -0

176 lines not on this page · raw message

arch/powerpc/platforms/pseries/pseries.h +1 -0

index 89c25769..1706215 100644
@@ -63,6 +63,7 @@ extern int dlpar_detach_node(struct device_node *);
 extern int dlpar_acquire_drc(u32);
 extern int dlpar_release_drc(u32);
 extern int dlpar_memory(struct pseries_hp_elog *);
+extern int dlpar_cpus(struct pseries_hp_elog *);
 
 /* PCI root bridge prepare function override for pseries */
 struct pci_host_bridge;

Signature · 1 line

2.0.0.rc3.2.g998f840

Nathan Fontenot

author

2014-06-17 15:47 UTC #4 lore raw

RFC PATCH 4/4 Hook into ras epow interrupt handler for hotplug

In reply to Nathan Fontenot

This patch hooks into the ras EPOW interrupt handler so that we can communicate hotplug rtas events to a PowerKVM guest from qemu.

The ras epow interrupt wil lnow check for hotplgu rtas events and invoke the common handling routine accordingly.

---
 arch/powerpc/include/asm/rtas.h          |  1 +
 arch/powerpc/platforms/pseries/dlpar.c   |  2 +-
 arch/powerpc/platforms/pseries/pseries.h |  1 +
 arch/powerpc/platforms/pseries/ras.c     | 12 +++++++++++-
 4 files changed, 14 insertions(+), 2 deletions(-)

arch/powerpc/include/asm/rtas.h +1 -0

index 26491ae..7313e6f 100644
@@ -124,6 +124,7 @@ struct rtas_suspend_me_data {
 #define RTAS_TYPE_INFO			0xE2
 #define RTAS_TYPE_DEALLOC		0xE3
 #define RTAS_TYPE_DUMP			0xE4
+#define RTAS_TYPE_HOTPLUG		0xE5
 /* I don't add PowerMGM events right now, this is a different topic */ 
 #define RTAS_TYPE_PMGM_POWER_SW_ON	0x60
 #define RTAS_TYPE_PMGM_POWER_SW_OFF	0x61

arch/powerpc/platforms/pseries/dlpar.c +1 -1

index 53f4fe6..6d3de6e 100644
@@ -360,7 +360,7 @@ int dlpar_release_drc(u32 drc_index)
 	return 0;
 }
 
-static int handle_dlpar_errorlog(struct rtas_error_log *error_log)
+int handle_dlpar_errorlog(struct rtas_error_log *error_log)
 {
 	struct pseries_errorlog *pseries_log;
 	struct pseries_hp_elog *hp_elog;

arch/powerpc/platforms/pseries/pseries.h +1 -0

index 1706215..9b9bd82 100644
@@ -64,6 +64,7 @@ extern int dlpar_acquire_drc(u32);
 extern int dlpar_release_drc(u32);
 extern int dlpar_memory(struct pseries_hp_elog *);
 extern int dlpar_cpus(struct pseries_hp_elog *);
+extern int handle_dlpar_errorlog(struct rtas_error_log *);
 
 /* PCI root bridge prepare function override for pseries */
 struct pci_host_bridge;

arch/powerpc/platforms/pseries/ras.c +11 -1

index 9c5778e..3cc6a49 100644
@@ -183,6 +183,7 @@ void rtas_parse_epow_errlog(struct rtas_error_log *log)
 /* Handle environmental and power warning (EPOW) interrupts. */
 static irqreturn_t ras_epow_interrupt(int irq, void *dev_id)
 {
+	struct rtas_error_log *elog;
 	int status;
 	int state;
 	int critical;

@@ -205,7 +206,16 @@ static irqreturn_t ras_epow_interrupt(int irq, void *dev_id)
 
 	log_error(ras_log_buf, ERR_TYPE_RTAS_LOG, 0);
 
-	rtas_parse_epow_errlog((struct rtas_error_log *)ras_log_buf);
+	elog = (struct rtas_error_log *)ras_log_buf;
+
+	switch(rtas_error_type(elog)) {
+	case RTAS_TYPE_EPOW:
+		rtas_parse_epow_errlog(elog);
+		break;
+	case RTAS_TYPE_HOTPLUG:
+		handle_dlpar_errorlog(elog);
+		break;
+	}
 
 	spin_unlock(&ras_log_buf_lock);
 	return IRQ_HANDLED;

Signature · 1 line

2.0.0.rc3.2.g998f840