Skip to contentSkip to conversation

kernel handling of dynamic logical partitioning

PATCH Superseded Nathan Fontenot 2 participants 14 messages Sep 18, 2009 – Oct 16, 2009

Subsystems (3)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover kernel handling of dynamic logical partitioning

The Dynamic Logical Partitioning (DLPAR) capabilities of the powerpc pseries platform allows for the addition and removal of resources (i.e. cpus, memory, pci devices) from a partition. The removal of a resource involves removing the resource's node from the device tree and then returning the resource to firmware via the rtas set-indicator call. To add a resource, it is first obtained from firmware via the rtas set-indicator call and then a new device tree node is created using the ibm,configure-connector rtas call and added to the device tree.

The following set of patches implements the needed infrastructure to have the kernel handle the DLPAR addition and removal of memory and cpus (other DLPAR'able items to follow in future patches). The framework for this is to create a set of probe/release sysfs files in pseries that will add or remove the cpu or memory to the system.

The majority of the code is powerpc/pseries specific except for PATCH 3/5, so I am cc'ing lkml.

Patches include in this set:
1/5 - DLPAR infracstructure for powerpc/pseries platform. 2/5 - Move the of_drconf_cell struct to prom.h 3/5 - Export the memory sysdev class
4/5 - Memory DLPAR handling
5/5 - CPU DLPAR handling

This also includes the following suggested fixes from the first version of the patches sent out.
- Correct initialization of workarea_lock and reconfig notifier chain.
- Add cc_workarea struct to help understand the work buffer layout.
- Fix memory leak in cpu_probe_store()
- Use strict_strtoul
- Correct checkpatch failures.

-Nathan Fontenot

#1 lore raw

PATCH v2 1/5 dynamic logical partitioning infrastructure

This patch provides the kernel DLPAR infrastructure in a new filed named
dlpar.c. The functionality provided is for acquiring and releasing a resource from firmware and the parsing of information returned from the ibm,configure-connector rtas call. Additionally this exports the pSeries reconfiguration notifier chain so that it can be invoked when device tree updates are made.

---

arch/powerpc/platforms/pseries/dlpar.c new file +416 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -0,0 +1,416 @@
+/*
+ * dlpar.c - support for dynamic reconfiguration (including PCI
+ * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ *
+ * Copyright (C) 2009 Nathan Fontenot
+ * Copyright (C) 2009 IBM Corporation
+ *
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ */
+
+#include <linux/kernel.h>
+#include <linux/kref.h>
+#include <linux/notifier.h>
+#include <linux/proc_fs.h>
+#include <linux/spinlock.h>
+
+#include <asm/prom.h>
+#include <asm/machdep.h>
+#include <asm/uaccess.h>
+#include <asm/rtas.h>
+#include <asm/pSeries_reconfig.h>
+
+#define CFG_CONN_WORK_SIZE	4096
+static char workarea[CFG_CONN_WORK_SIZE];
+static DEFINE_SPINLOCK(workarea_lock);
+
+struct cc_workarea {
+	u32	drc_index;
+	u32	zero;
+	u32	name_offset;
+	u32	prop_length;
+	u32	prop_offset;
+};
+
+static struct property *parse_cc_property(char *workarea)
+{
+	struct property *prop;
+	struct cc_workarea *ccwa;
+	char *name;
+	char *value;
+
+	prop = kzalloc(sizeof(*prop), GFP_KERNEL);
+	if (!prop)
+		return NULL;
+
+	ccwa = (struct cc_workarea *)workarea;
+	name = workarea + ccwa->name_offset;
+	prop->name = kzalloc(strlen(name) + 1, GFP_KERNEL);
+	if (!prop->name) {
+		kfree(prop);
+		return NULL;
+	}
+
+	strcpy(prop->name, name);
+
+	prop->length = ccwa->prop_length;
+	value = workarea + ccwa->prop_offset;
+	prop->value = kzalloc(prop->length, GFP_KERNEL);
+	if (!prop->value) {
+		kfree(prop->name);
+		kfree(prop);
+		return NULL;
+	}
+
+	memcpy(prop->value, value, prop->length);
+	return prop;
+}
+
+static void free_property(struct property *prop)
+{
+	kfree(prop->name);
+	kfree(prop->value);
+	kfree(prop);
+}
+
+static struct device_node *parse_cc_node(char *work_area)
+{
+	struct device_node *dn;
+	struct cc_workarea *ccwa;
+	char *name;
+
+	dn = kzalloc(sizeof(*dn), GFP_KERNEL);
+	if (!dn)
+		return NULL;
+
+	ccwa = (struct cc_workarea *)work_area;
+	name = work_area + ccwa->name_offset;
+	dn->full_name = kzalloc(strlen(name) + 1, GFP_KERNEL);
+	if (!dn->full_name) {
+		kfree(dn);
+		return NULL;
+	}
+
+	strcpy(dn->full_name, name);
+	return dn;
+}
+
+static void free_one_cc_node(struct device_node *dn)
+{
+	struct property *prop;
+
+	while (dn->properties) {
+		prop = dn->properties;
+		dn->properties = prop->next;
+		free_property(prop);
+	}
+
+	kfree(dn->full_name);
+	kfree(dn);
+}
+
+static void free_cc_nodes(struct device_node *dn)
+{
+	if (dn->child)
+		free_cc_nodes(dn->child);
+
+	if (dn->sibling)
+		free_cc_nodes(dn->sibling);
+
+	free_one_cc_node(dn);
+}
+
+#define NEXT_SIBLING    1
+#define NEXT_CHILD      2
+#define NEXT_PROPERTY   3
+#define PREV_PARENT     4
+#define MORE_MEMORY     5
+#define CALL_AGAIN	-2
+#define ERR_CFG_USE     -9003
+
+struct device_node *configure_connector(u32 drc_index)
+{
+	struct device_node *dn;
+	struct device_node *first_dn = NULL;
+	struct device_node *last_dn = NULL;
+	struct property *property;
+	struct property *last_property = NULL;
+	struct cc_workarea *ccwa;
+	int cc_token;
+	int rc;
+
+	cc_token = rtas_token("ibm,configure-connector");
+	if (cc_token == RTAS_UNKNOWN_SERVICE)
+		return NULL;
+
+	spin_lock(&workarea_lock);
+
+	ccwa = (struct cc_workarea *)&workarea[0];
+	ccwa->drc_index = drc_index;
+	ccwa->zero = 0;
+
+	rc = rtas_call(cc_token, 2, 1, NULL, workarea, NULL);
+	while (rc) {
+		switch (rc) {
+		case NEXT_SIBLING:
+			dn = parse_cc_node(workarea);
+			if (!dn)
+				goto cc_error;
+
+			dn->parent = last_dn->parent;
+			last_dn->sibling = dn;
+			last_dn = dn;
+			break;
+
+		case NEXT_CHILD:
+			dn = parse_cc_node(workarea);
+			if (!dn)
+				goto cc_error;
+
+			if (!first_dn)
+				first_dn = dn;
+			else {
+				dn->parent = last_dn;
+				if (last_dn)
+					last_dn->child = dn;
+			}
+
+			last_dn = dn;
+			break;
+
+		case NEXT_PROPERTY:
+			property = parse_cc_property(workarea);
+			if (!property)
+				goto cc_error;
+
+			if (!last_dn->properties)
+				last_dn->properties = property;
+			else
+				last_property->next = property;
+
+			last_property = property;
+			break;
+
+		case PREV_PARENT:
+			last_dn = last_dn->parent;
+			break;
+
+		case CALL_AGAIN:
+			break;
+
+		case MORE_MEMORY:
+		case ERR_CFG_USE:
+		default:
+			printk(KERN_ERR "Unexpected Error (%d) "
+			       "returned from configure-connector\n", rc);
+			goto cc_error;
+		}
+
+		rc = rtas_call(cc_token, 2, 1, NULL, workarea, NULL);
+	}
+
+	spin_unlock(&workarea_lock);
+	return first_dn;
+
+cc_error:
+	spin_unlock(&workarea_lock);
+
+	if (first_dn)
+		free_cc_nodes(first_dn);
+
+	return NULL;
+}
+
+static struct device_node *derive_parent(const char *path)
+{
+	struct device_node *parent;
+	char parent_path[128];
+	int parent_path_len;
+
+	parent_path_len = strrchr(path, '/') - path + 1;
+	strlcpy(parent_path, path, parent_path_len);
+
+	parent = of_find_node_by_path(parent_path);
+
+	return parent;
+}
+
+static int add_one_node(struct device_node *dn)
+{
+	struct proc_dir_entry *ent;
+	int rc;
+
+	of_node_set_flag(dn, OF_DYNAMIC);
+	kref_init(&dn->kref);
+	dn->parent = derive_parent(dn->full_name);
+
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+					  PSERIES_RECONFIG_ADD, dn);
+	if (rc == NOTIFY_BAD) {
+		printk(KERN_ERR "Failed to add device node %s\n",
+		       dn->full_name);
+		return -ENOMEM; /* For now, safe to assume kmalloc failure */
+	}
+
+	of_attach_node(dn);
+
+#ifdef CONFIG_PROC_DEVICETREE
+	ent = proc_mkdir(strrchr(dn->full_name, '/') + 1, dn->parent->pde);
+	if (ent)
+		proc_device_tree_add_node(dn, ent);
+#endif
+
+	of_node_put(dn->parent);
+	return 0;
+}
+
+int add_device_tree_nodes(struct device_node *dn)
+{
+	struct device_node *child = dn->child;
+	struct device_node *sibling = dn->sibling;
+	int rc;
+
+	dn->child = NULL;
+	dn->sibling = NULL;
+	dn->parent = NULL;
+
+	rc = add_one_node(dn);
+	if (rc)
+		return rc;
+
+	if (child) {
+		rc = add_device_tree_nodes(child);
+		if (rc)
+			return rc;
+	}
+
+	if (sibling)
+		rc = add_device_tree_nodes(sibling);
+
+	return rc;
+}
+
+static int remove_one_node(struct device_node *dn)
+{
+	struct device_node *parent = dn->parent;
+	struct property *prop = dn->properties;
+
+#ifdef CONFIG_PROC_DEVICETREE
+	while (prop) {
+		remove_proc_entry(prop->name, dn->pde);
+		prop = prop->next;
+	}
+
+	if (dn->pde)
+		remove_proc_entry(dn->pde->name, parent->pde);
+#endif
+
+	blocking_notifier_call_chain(&pSeries_reconfig_chain,
+			    PSERIES_RECONFIG_REMOVE, dn);
+	of_detach_node(dn);
+
+	of_node_put(parent);
+	of_node_put(dn); /* Must decrement the refcount */
+
+	return 0;
+}
+
+static int _remove_device_tree_nodes(struct device_node *dn)
+{
+	int rc;
+
+	if (dn->child) {
+		rc = _remove_device_tree_nodes(dn->child);
+		if (rc)
+			return rc;
+	}
+
+	if (dn->sibling) {
+		rc = _remove_device_tree_nodes(dn->sibling);
+		if (rc)
+			return rc;
+	}
+
+	rc = remove_one_node(dn);
+	return rc;
+}
+
+int remove_device_tree_nodes(struct device_node *dn)
+{
+	int rc;
+
+	if (dn->child) {
+		rc = _remove_device_tree_nodes(dn->child);
+		if (rc)
+			return rc;
+	}
+
+	rc = remove_one_node(dn);
+	return rc;
+}
+
+#define DR_ENTITY_SENSE		9003
+#define DR_ENTITY_PRESENT	1
+#define DR_ENTITY_UNUSABLE	2
+#define ALLOCATION_STATE	9003
+#define ALLOC_UNUSABLE		0
+#define ALLOC_USABLE		1
+#define ISOLATION_STATE		9001
+#define ISOLATE			0
+#define UNISOLATE		1
+
+int acquire_drc(u32 drc_index)
+{
+	int dr_status, rc;
+
+	rc = rtas_call(rtas_token("get-sensor-state"), 2, 2, &dr_status,
+		       DR_ENTITY_SENSE, drc_index);
+	if (rc || dr_status != DR_ENTITY_UNUSABLE)
+		return -1;
+
+	rc = rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_USABLE);
+	if (rc)
+		return rc;
+
+	rc = rtas_set_indicator(ISOLATION_STATE, drc_index, UNISOLATE);
+	if (rc) {
+		rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_UNUSABLE);
+		return rc;
+	}
+
+	return 0;
+}
+
+int release_drc(u32 drc_index)
+{
+	int dr_status, rc;
+
+	rc = rtas_call(rtas_token("get-sensor-state"), 2, 2, &dr_status,
+		       DR_ENTITY_SENSE, drc_index);
+	if (rc || dr_status != DR_ENTITY_PRESENT)
+		return -1;
+
+	rc = rtas_set_indicator(ISOLATION_STATE, drc_index, ISOLATE);
+	if (rc)
+		return rc;
+
+	rc = rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_UNUSABLE);
+	if (rc) {
+		rtas_set_indicator(ISOLATION_STATE, drc_index, UNISOLATE);
+		return rc;
+	}
+
+	return 0;
+}
+
+static int pseries_dlpar_init(void)
+{
+	if (!machine_is(pseries))
+		return 0;
+
+	return 0;
+}
+device_initcall(pseries_dlpar_init);
arch/powerpc/platforms/pseries/Makefile +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/Makefile
===================================================================
@@ -8,7 +8,7 @@
 
obj-y			:= lpar.o hvCall.o nvram.o reconfig.o \
			   setup.o iommu.o ras.o rtasd.o \

- firmware.o power.o
+ firmware.o power.o dlpar.o
obj-$(CONFIG_SMP) += smp.o
obj-$(CONFIG_XICS) += xics.o
obj-$(CONFIG_SCANLOG) += scanlog.o

arch/powerpc/include/asm/pSeries_reconfig.h +0 -0
Index: powerpc/arch/powerpc/include/asm/pSeries_reconfig.h
===================================================================
@@ -17,6 +17,7 @@

#ifdef CONFIG_PPC_PSERIES
extern int pSeries_reconfig_notifier_register(struct notifier_block *); extern void pSeries_reconfig_notifier_unregister(struct notifier_block *); +extern struct blocking_notifier_head pSeries_reconfig_chain; #else /* !CONFIG_PPC_PSERIES */
static inline int pSeries_reconfig_notifier_register(struct notifier_block *nb) {

arch/powerpc/platforms/pseries/reconfig.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/reconfig.c
===================================================================
@@ -95,7 +95,7 @@
	return parent;
}

-static BLOCKING_NOTIFIER_HEAD(pSeries_reconfig_chain); +BLOCKING_NOTIFIER_HEAD(pSeries_reconfig_chain);

int pSeries_reconfig_notifier_register(struct notifier_block *nb) {

#3 lore raw

PATCH v2 2/5 move of_drconf_cell definition to prom.h

In reply to Nathan Fontenot

Move the definition of the of_drconf_cell struct from numa.c to prom.h. This is needed so that we can parse the ibm,dynamic-memory device-tree property when DLPAR adding and removing memory.

---

arch/powerpc/include/asm/prom.h +0 -0
Index: powerpc/arch/powerpc/include/asm/prom.h
===================================================================
@@ -349,6 +349,18 @@
 */

extern void __iomem *of_iomap(struct device_node *device, int index);

+struct of_drconf_cell {
+ u64 base_addr;
+ u32 drc_index;
+ u32 reserved;
+ u32 aa_index;
+ u32 flags;
+};
+
+#define DRCONF_MEM_ASSIGNED 0x00000008
+#define DRCONF_MEM_AI_INVALID 0x00000040
+#define DRCONF_MEM_RESERVED 0x00000080
+
/*
* NB: This is here while we transition from using asm/prom.h
* to linux/of.h

arch/powerpc/mm/numa.c +0 -0
Index: powerpc/arch/powerpc/mm/numa.c
===================================================================
@@ -296,18 +296,6 @@
	return result;
}

-struct of_drconf_cell {
- u64 base_addr;
- u32 drc_index;
- u32 reserved;
- u32 aa_index;
- u32 flags;
-};
-
-#define DRCONF_MEM_ASSIGNED 0x00000008
-#define DRCONF_MEM_AI_INVALID 0x00000040
-#define DRCONF_MEM_RESERVED 0x00000080
-
/*
* Read the next lmb list entry from the ibm,dynamic-memory property
* and return the information in the provided of_drconf_cell structure.

#4 lore raw

PATCH v2 3/5 Export memory_sysdev_class

In reply to Nathan Fontenot

Export the memory_sysdev_class structure. This is needed so we can create a 'release' file in sysfs in addition to the existing 'probe' file in order to support DLPAR removal of memory on the powerpc/pseries platform. The new 'release' file will be powerpc/pseries only.

---

drivers/base/memory.c +0 -0
Index: powerpc/drivers/base/memory.c
===================================================================
@@ -28,9 +28,10 @@
 

#define MEMORY_CLASS_NAME "memory"

-static struct sysdev_class memory_sysdev_class = { +struct sysdev_class memory_sysdev_class = {
.name = MEMORY_CLASS_NAME,
};
+EXPORT_SYMBOL(memory_sysdev_class);

static const char *memory_uevent_name(struct kset *kset, struct kobject *kobj) {

include/linux/memory_hotplug.h +0 -0
Index: powerpc/include/linux/memory_hotplug.h
===================================================================
@@ -12,6 +12,8 @@
 

#ifdef CONFIG_MEMORY_HOTPLUG

+extern struct sysdev_class memory_sysdev_class;
+
/*
* Types for free bootmem.
* The normal smallest mapcount is -1. Here is smaller value than it.

#5 lore raw

PATCH v2 4/5 kernel handling of memory DLPAR

In reply to Nathan Fontenot

This adds the capability to DLPAR add and remove memory from the kernel. The patch extends the powerpc handling of memory_add_physaddr_to_nid(), which is called from the sysfs memory 'probe' file to first ensure that the memory has been added to the system. This is done by creating a platform specific callout from the routine. The pseries implementation of this handles the DLPAR work to add the memory to the system and update the device tree.

The patch also creates a pseries only 'release' sys file, /sys/devices/system/memory/release. This file handles the DLPAR release of memory back to firmware and updating of the device-tree.

---

arch/powerpc/platforms/pseries/dlpar.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -16,6 +16,10 @@

#include <linux/notifier.h>
#include <linux/proc_fs.h>
#include <linux/spinlock.h>
+#include <linux/memory_hotplug.h>
+#include <linux/sysdev.h>
+#include <linux/sysfs.h>
+

#include <asm/prom.h>
#include <asm/machdep.h>
@@ -406,11 +410,163 @@

	return 0;
}

+static struct property *clone_property(struct property *old_prop) +{
+ struct property *new_prop;
+
+ new_prop = kzalloc((sizeof *new_prop), GFP_KERNEL);
+ if (!new_prop)
+ return NULL;
+
+ new_prop->name = kzalloc(strlen(old_prop->name) + 1, GFP_KERNEL);
+ new_prop->value = kzalloc(old_prop->length + 1, GFP_KERNEL);
+ if (!new_prop->name || !new_prop->value) {
+ free_property(new_prop);
+ return NULL;
+ }
+
+ strcpy(new_prop->name, old_prop->name);
+ memcpy(new_prop->value, old_prop->value, old_prop->length);
+ new_prop->length = old_prop->length;
+
+ return new_prop;
+}
+
+int platform_probe_memory(u64 phys_addr)
+{
+ struct device_node *dn;
+ struct property *new_prop, *old_prop;
+ struct property *lmb_sz_prop;
+ struct of_drconf_cell *drmem;
+ u64 lmb_size;
+ int num_entries, i, rc;
+
+ if (!phys_addr)
+ return -EINVAL;
+
+ dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!dn)
+ return -EINVAL;
+
+ lmb_sz_prop = of_find_property(dn, "ibm,lmb-size", NULL);
+ lmb_size = *(u64 *)lmb_sz_prop->value;
+
+ old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+
+ num_entries = *(u32 *)old_prop->value;
+ drmem = (struct of_drconf_cell *)
+ ((char *)old_prop->value + sizeof(u32));
+
+ for (i = 0; i < num_entries; i++) {
+ u64 lmb_end_addr = drmem[i].base_addr + lmb_size;
+ if (phys_addr >= drmem[i].base_addr
+ && phys_addr < lmb_end_addr)
+ break;
+ }
+
+ if (i >= num_entries) {
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ if (drmem[i].flags & DRCONF_MEM_ASSIGNED) {
+ of_node_put(dn);
+ return 0;
+ }
+
+ rc = acquire_drc(drmem[i].drc_index);
+ if (rc) {
+ of_node_put(dn);
+ return -1;
+ }
+
+ new_prop = clone_property(old_prop);
+ drmem = (struct of_drconf_cell *)
+ ((char *)new_prop->value + sizeof(u32));
+
+ drmem[i].flags |= DRCONF_MEM_ASSIGNED;
+ prom_update_property(dn, new_prop, old_prop);
+
+ rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+ PSERIES_DRCONF_MEM_ADD,
+ &drmem[i].base_addr);
+ if (rc == NOTIFY_BAD) {
+ prom_update_property(dn, old_prop, new_prop);
+ release_drc(drmem[i].drc_index);
+ }
+
+ of_node_put(dn);
+ return rc == NOTIFY_BAD ? -1 : 0;
+}
+
+static ssize_t memory_release_store(struct class *class, const char *buf,
+ size_t count)
+{
+ unsigned long drc_index;
+ struct device_node *dn;
+ struct property *new_prop, *old_prop;
+ struct of_drconf_cell *drmem;
+ int num_entries;
+ int i, rc;
+
+ rc = strict_strtoul(buf, 0, &drc_index);
+ if (rc)
+ return -EINVAL;
+
+ dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!dn)
+ return 0;
+
+ old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+ new_prop = clone_property(old_prop);
+
+ num_entries = *(u32 *)new_prop->value;
+ drmem = (struct of_drconf_cell *)
+ ((char *)new_prop->value + sizeof(u32));
+
+ for (i = 0; i < num_entries; i++) {
+ if (drmem[i].drc_index == drc_index)
+ break;
+ }
+
+ if (i >= num_entries) {
+ free_property(new_prop);
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ drmem[i].flags &= ~DRCONF_MEM_ASSIGNED;
+ prom_update_property(dn, new_prop, old_prop);
+
+ rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+ PSERIES_DRCONF_MEM_REMOVE,
+ &drmem[i].base_addr);
+ if (rc != NOTIFY_BAD)
+ rc = release_drc(drc_index);
+
+ if (rc)
+ prom_update_property(dn, old_prop, new_prop);
+
+ of_node_put(dn);
+ return rc ? -1 : count;
+}
+
+static struct class_attribute class_attr_mem_release =
+ __ATTR(release, S_IWUSR, NULL, memory_release_store);
+
static int pseries_dlpar_init(void)
{
+ int rc;
+

	if (!machine_is(pseries))
		return 0;

+ rc = sysfs_create_file(&memory_sysdev_class.kset.kobj,
+ &class_attr_mem_release.attr);
+ if (rc)
+ printk(KERN_INFO "DLPAR: Could not create sysfs memory "
+ "release file\n");
+

	return 0;
}

device_initcall(pseries_dlpar_init);

arch/powerpc/mm/mem.c +0 -0
Index: powerpc/arch/powerpc/mm/mem.c
===================================================================
@@ -111,8 +111,19 @@

#ifdef CONFIG_MEMORY_HOTPLUG

#ifdef CONFIG_NUMA
+int __attribute ((weak)) platform_probe_memory(u64 start) +{
+ return 0;
+}
+
int memory_add_physaddr_to_nid(u64 start)
{
+ int rc;
+
+ rc = platform_probe_memory(start);
+ if (rc)
+ return rc;
+

	return hot_add_scn_to_nid(start);
}

#endif

#9 lore raw

PATCH v2 5/5 kernel handling of CPU DLPAR

In reply to Nathan Fontenot

This adds the capability to DLPAR add and remove CPUs from the kernel. The creates two new files /sys/devices/system/cpu/probe and /sys/devices/system/cpu/release to handle the DLPAR addition and removal of CPUs respectively.

CPU DLPAR add is accomplished by writing the drc-index of the CPU to the probe file, and removal is done by writing the device-tree path of the cpu to the release file.

---

arch/powerpc/platforms/pseries/dlpar.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -1,11 +1,11 @@

/*
- * dlpar.c - support for dynamic reconfiguration (including PCI
- * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ * dlpar.c - support for dynamic reconfiguration (including PCI,
+ * Memory, and CPU Hotplug and Dynamic Logical Partitioning on
+ * PAPR platforms).
*
* Copyright (C) 2009 Nathan Fontenot
* Copyright (C) 2009 IBM Corporation
*
- *
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License version
* 2 as published by the Free Software Foundation.
@@ -19,6 +19,7 @@
#include <linux/memory_hotplug.h>
#include <linux/sysdev.h>
#include <linux/sysfs.h>
+#include <linux/cpu.h>

#include <asm/prom.h>
@@ -551,8 +552,85 @@

	return rc ? -1 : count;
}

+static ssize_t cpu_probe_store(struct class *class, const char *buf,
+ size_t count)
+{
+ struct device_node *dn;
+ unsigned long drc_index;
+ char *cpu_name;
+ int rc;
+
+ rc = strict_strtoul(buf, 0, &drc_index);
+ if (rc)
+ return -EINVAL;
+
+ rc = acquire_drc(drc_index);
+ if (rc)
+ return rc;
+
+ dn = configure_connector(drc_index);
+ if (!dn) {
+ release_drc(drc_index);
+ return rc;
+ }
+
+ /* fixup dn name */
+ cpu_name = kzalloc(strlen(dn->full_name) + strlen("/cpus/") + 1,
+ GFP_KERNEL);
+ if (!cpu_name) {
+ free_cc_nodes(dn);
+ release_drc(drc_index);
+ return -ENOMEM;
+ }
+
+ sprintf(cpu_name, "/cpus/%s", dn->full_name);
+ kfree(dn->full_name);
+ dn->full_name = cpu_name;
+
+ rc = add_device_tree_nodes(dn);
+ if (rc)
+ release_drc(drc_index);
+
+ return rc ? rc : count;
+}
+
+static ssize_t cpu_release_store(struct class *class, const char *buf,
+ size_t count)
+{
+ struct device_node *dn;
+ u32 *drc_index;
+ int rc;
+
+ dn = of_find_node_by_path(buf);
+ if (!dn)
+ return -EINVAL;
+
+ drc_index = (u32 *)of_get_property(dn, "ibm,my-drc-index", NULL);
+ if (!drc_index) {
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ rc = release_drc(*drc_index);
+ if (rc) {
+ of_node_put(dn);
+ return rc;
+ }
+
+ rc = remove_device_tree_nodes(dn);
+ if (rc)
+ acquire_drc(*drc_index);
+
+ of_node_put(dn);
+ return rc ? rc : count;
+}
+
static struct class_attribute class_attr_mem_release =

			__ATTR(release, S_IWUSR, NULL, memory_release_store);

+static struct class_attribute class_attr_cpu_probe =
+ __ATTR(probe, S_IWUSR, NULL, cpu_probe_store);
+static struct class_attribute class_attr_cpu_release =
+ __ATTR(release, S_IWUSR, NULL, cpu_release_store);

static int pseries_dlpar_init(void)
{
@@ -567,6 +645,18 @@

		printk(KERN_INFO "DLPAR: Could not create sysfs memory "
		       "release file\n");

+ rc = sysfs_create_file(&cpu_sysdev_class.kset.kobj,
+ &class_attr_cpu_probe.attr);
+ if (rc)
+ printk(KERN_INFO "DLPAR: Could not create sysfs cpu "
+ "probe file\n");
+
+ rc = sysfs_create_file(&cpu_sysdev_class.kset.kobj,
+ &class_attr_cpu_release.attr);
+ if (rc)
+ printk(KERN_INFO "DLPAR: Could not create sysfs cpu "
+ "release file\n");
+

	return 0;
}

device_initcall(pseries_dlpar_init);

#2 lore raw

Re: [PATCH 1/5 v3] dynamic logical partitioning infrastructure

In reply to Nathan Fontenot

This patch provides the kernel DLPAR infrastructure in a new filed named
dlpar.c. The functionality provided is for acquiring and releasing a resource from firmware and the parsing of information returned from the ibm,configure-connector rtas call. Additionally, this exports the pSeries reconfiguration notifier chain so that it can be invoked when device tree updates are made.

Updated to remove an extraneous of_node_put() in the removal of a device tree node path.

arch/powerpc/platforms/pseries/dlpar.c new file +414 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -0,0 +1,414 @@
+/*
+ * dlpar.c - support for dynamic reconfiguration (including PCI
+ * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ *
+ * Copyright (C) 2009 Nathan Fontenot
+ * Copyright (C) 2009 IBM Corporation
+ *
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ */
+
+#include <linux/kernel.h>
+#include <linux/kref.h>
+#include <linux/notifier.h>
+#include <linux/proc_fs.h>
+#include <linux/spinlock.h>
+
+#include <asm/prom.h>
+#include <asm/machdep.h>
+#include <asm/uaccess.h>
+#include <asm/rtas.h>
+#include <asm/pSeries_reconfig.h>
+
+#define CFG_CONN_WORK_SIZE	4096
+static char workarea[CFG_CONN_WORK_SIZE];
+static DEFINE_SPINLOCK(workarea_lock);
+
+struct cc_workarea {
+	u32	drc_index;
+	u32	zero;
+	u32	name_offset;
+	u32	prop_length;
+	u32	prop_offset;
+};
+
+static struct property *parse_cc_property(char *workarea)
+{
+	struct property *prop;
+	struct cc_workarea *ccwa;
+	char *name;
+	char *value;
+
+	prop = kzalloc(sizeof(*prop), GFP_KERNEL);
+	if (!prop)
+		return NULL;
+
+	ccwa = (struct cc_workarea *)workarea;
+	name = workarea + ccwa->name_offset;
+	prop->name = kzalloc(strlen(name) + 1, GFP_KERNEL);
+	if (!prop->name) {
+		kfree(prop);
+		return NULL;
+	}
+
+	strcpy(prop->name, name);
+
+	prop->length = ccwa->prop_length;
+	value = workarea + ccwa->prop_offset;
+	prop->value = kzalloc(prop->length, GFP_KERNEL);
+	if (!prop->value) {
+		kfree(prop->name);
+		kfree(prop);
+		return NULL;
+	}
+
+	memcpy(prop->value, value, prop->length);
+	return prop;
+}
+
+static void free_property(struct property *prop)
+{
+	kfree(prop->name);
+	kfree(prop->value);
+	kfree(prop);
+}
+
+static struct device_node *parse_cc_node(char *work_area)
+{
+	struct device_node *dn;
+	struct cc_workarea *ccwa;
+	char *name;
+
+	dn = kzalloc(sizeof(*dn), GFP_KERNEL);
+	if (!dn)
+		return NULL;
+
+	ccwa = (struct cc_workarea *)work_area;
+	name = work_area + ccwa->name_offset;
+	dn->full_name = kzalloc(strlen(name) + 1, GFP_KERNEL);
+	if (!dn->full_name) {
+		kfree(dn);
+		return NULL;
+	}
+
+	strcpy(dn->full_name, name);
+	return dn;
+}
+
+static void free_one_cc_node(struct device_node *dn)
+{
+	struct property *prop;
+
+	while (dn->properties) {
+		prop = dn->properties;
+		dn->properties = prop->next;
+		free_property(prop);
+	}
+
+	kfree(dn->full_name);
+	kfree(dn);
+}
+
+static void free_cc_nodes(struct device_node *dn)
+{
+	if (dn->child)
+		free_cc_nodes(dn->child);
+
+	if (dn->sibling)
+		free_cc_nodes(dn->sibling);
+
+	free_one_cc_node(dn);
+}
+
+#define NEXT_SIBLING    1
+#define NEXT_CHILD      2
+#define NEXT_PROPERTY   3
+#define PREV_PARENT     4
+#define MORE_MEMORY     5
+#define CALL_AGAIN	-2
+#define ERR_CFG_USE     -9003
+
+struct device_node *configure_connector(u32 drc_index)
+{
+	struct device_node *dn;
+	struct device_node *first_dn = NULL;
+	struct device_node *last_dn = NULL;
+	struct property *property;
+	struct property *last_property = NULL;
+	struct cc_workarea *ccwa;
+	int cc_token;
+	int rc;
+
+	cc_token = rtas_token("ibm,configure-connector");
+	if (cc_token == RTAS_UNKNOWN_SERVICE)
+		return NULL;
+
+	spin_lock(&workarea_lock);
+
+	ccwa = (struct cc_workarea *)&workarea[0];
+	ccwa->drc_index = drc_index;
+	ccwa->zero = 0;
+
+	rc = rtas_call(cc_token, 2, 1, NULL, workarea, NULL);
+	while (rc) {
+		switch (rc) {
+		case NEXT_SIBLING:
+			dn = parse_cc_node(workarea);
+			if (!dn)
+				goto cc_error;
+
+			dn->parent = last_dn->parent;
+			last_dn->sibling = dn;
+			last_dn = dn;
+			break;
+
+		case NEXT_CHILD:
+			dn = parse_cc_node(workarea);
+			if (!dn)
+				goto cc_error;
+
+			if (!first_dn)
+				first_dn = dn;
+			else {
+				dn->parent = last_dn;
+				if (last_dn)
+					last_dn->child = dn;
+			}
+
+			last_dn = dn;
+			break;
+
+		case NEXT_PROPERTY:
+			property = parse_cc_property(workarea);
+			if (!property)
+				goto cc_error;
+
+			if (!last_dn->properties)
+				last_dn->properties = property;
+			else
+				last_property->next = property;
+
+			last_property = property;
+			break;
+
+		case PREV_PARENT:
+			last_dn = last_dn->parent;
+			break;
+
+		case CALL_AGAIN:
+			break;
+
+		case MORE_MEMORY:
+		case ERR_CFG_USE:
+		default:
+			printk(KERN_ERR "Unexpected Error (%d) "
+			       "returned from configure-connector\n", rc);
+			goto cc_error;
+		}
+
+		rc = rtas_call(cc_token, 2, 1, NULL, workarea, NULL);
+	}
+
+	spin_unlock(&workarea_lock);
+	return first_dn;
+
+cc_error:
+	spin_unlock(&workarea_lock);
+
+	if (first_dn)
+		free_cc_nodes(first_dn);
+
+	return NULL;
+}
+
+static struct device_node *derive_parent(const char *path)
+{
+	struct device_node *parent;
+	char parent_path[128];
+	int parent_path_len;
+
+	parent_path_len = strrchr(path, '/') - path + 1;
+	strlcpy(parent_path, path, parent_path_len);
+
+	parent = of_find_node_by_path(parent_path);
+
+	return parent;
+}
+
+static int add_one_node(struct device_node *dn)
+{
+	struct proc_dir_entry *ent;
+	int rc;
+
+	of_node_set_flag(dn, OF_DYNAMIC);
+	kref_init(&dn->kref);
+	dn->parent = derive_parent(dn->full_name);
+
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+					  PSERIES_RECONFIG_ADD, dn);
+	if (rc == NOTIFY_BAD) {
+		printk(KERN_ERR "Failed to add device node %s\n",
+		       dn->full_name);
+		return -ENOMEM; /* For now, safe to assume kmalloc failure */
+	}
+
+	of_attach_node(dn);
+
+#ifdef CONFIG_PROC_DEVICETREE
+	ent = proc_mkdir(strrchr(dn->full_name, '/') + 1, dn->parent->pde);
+	if (ent)
+		proc_device_tree_add_node(dn, ent);
+#endif
+
+	of_node_put(dn->parent);
+	return 0;
+}
+
+int add_device_tree_nodes(struct device_node *dn)
+{
+	struct device_node *child = dn->child;
+	struct device_node *sibling = dn->sibling;
+	int rc;
+
+	dn->child = NULL;
+	dn->sibling = NULL;
+	dn->parent = NULL;
+
+	rc = add_one_node(dn);
+	if (rc)
+		return rc;
+
+	if (child) {
+		rc = add_device_tree_nodes(child);
+		if (rc)
+			return rc;
+	}
+
+	if (sibling)
+		rc = add_device_tree_nodes(sibling);
+
+	return rc;
+}
+
+static int remove_one_node(struct device_node *dn)
+{
+	struct device_node *parent = dn->parent;
+	struct property *prop = dn->properties;
+
+#ifdef CONFIG_PROC_DEVICETREE
+	while (prop) {
+		remove_proc_entry(prop->name, dn->pde);
+		prop = prop->next;
+	}
+
+	if (dn->pde)
+		remove_proc_entry(dn->pde->name, parent->pde);
+#endif
+
+	blocking_notifier_call_chain(&pSeries_reconfig_chain,
+			    PSERIES_RECONFIG_REMOVE, dn);
+	of_detach_node(dn);
+	of_node_put(dn); /* Must decrement the refcount */
+
+	return 0;
+}
+
+static int _remove_device_tree_nodes(struct device_node *dn)
+{
+	int rc;
+
+	if (dn->child) {
+		rc = _remove_device_tree_nodes(dn->child);
+		if (rc)
+			return rc;
+	}
+
+	if (dn->sibling) {
+		rc = _remove_device_tree_nodes(dn->sibling);
+		if (rc)
+			return rc;
+	}
+
+	rc = remove_one_node(dn);
+	return rc;
+}
+
+int remove_device_tree_nodes(struct device_node *dn)
+{
+	int rc;
+
+	if (dn->child) {
+		rc = _remove_device_tree_nodes(dn->child);
+		if (rc)
+			return rc;
+	}
+
+	rc = remove_one_node(dn);
+	return rc;
+}
+
+#define DR_ENTITY_SENSE		9003
+#define DR_ENTITY_PRESENT	1
+#define DR_ENTITY_UNUSABLE	2
+#define ALLOCATION_STATE	9003
+#define ALLOC_UNUSABLE		0
+#define ALLOC_USABLE		1
+#define ISOLATION_STATE		9001
+#define ISOLATE			0
+#define UNISOLATE		1
+
+int acquire_drc(u32 drc_index)
+{
+	int dr_status, rc;
+
+	rc = rtas_call(rtas_token("get-sensor-state"), 2, 2, &dr_status,
+		       DR_ENTITY_SENSE, drc_index);
+	if (rc || dr_status != DR_ENTITY_UNUSABLE)
+		return -1;
+
+	rc = rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_USABLE);
+	if (rc)
+		return rc;
+
+	rc = rtas_set_indicator(ISOLATION_STATE, drc_index, UNISOLATE);
+	if (rc) {
+		rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_UNUSABLE);
+		return rc;
+	}
+
+	return 0;
+}
+
+int release_drc(u32 drc_index)
+{
+	int dr_status, rc;
+
+	rc = rtas_call(rtas_token("get-sensor-state"), 2, 2, &dr_status,
+		       DR_ENTITY_SENSE, drc_index);
+	if (rc || dr_status != DR_ENTITY_PRESENT)
+		return -1;
+
+	rc = rtas_set_indicator(ISOLATION_STATE, drc_index, ISOLATE);
+	if (rc)
+		return rc;
+
+	rc = rtas_set_indicator(ALLOCATION_STATE, drc_index, ALLOC_UNUSABLE);
+	if (rc) {
+		rtas_set_indicator(ISOLATION_STATE, drc_index, UNISOLATE);
+		return rc;
+	}
+
+	return 0;
+}
+
+static int pseries_dlpar_init(void)
+{
+	if (!machine_is(pseries))
+		return 0;
+
+	return 0;
+}
+device_initcall(pseries_dlpar_init);
arch/powerpc/platforms/pseries/Makefile +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/Makefile
===================================================================
@@ -8,7 +8,7 @@
 
obj-y			:= lpar.o hvCall.o nvram.o reconfig.o \
			   setup.o iommu.o ras.o rtasd.o \

- firmware.o power.o
+ firmware.o power.o dlpar.o
obj-$(CONFIG_SMP) += smp.o
obj-$(CONFIG_XICS) += xics.o
obj-$(CONFIG_SCANLOG) += scanlog.o

arch/powerpc/include/asm/pSeries_reconfig.h +0 -0
Index: powerpc/arch/powerpc/include/asm/pSeries_reconfig.h
===================================================================
@@ -17,6 +17,7 @@

#ifdef CONFIG_PPC_PSERIES
extern int pSeries_reconfig_notifier_register(struct notifier_block *); extern void pSeries_reconfig_notifier_unregister(struct notifier_block *); +extern struct blocking_notifier_head pSeries_reconfig_chain; #else /* !CONFIG_PPC_PSERIES */
static inline int pSeries_reconfig_notifier_register(struct notifier_block *nb) {

arch/powerpc/platforms/pseries/reconfig.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/reconfig.c
===================================================================
@@ -95,7 +95,7 @@
	return parent;
}

-static BLOCKING_NOTIFIER_HEAD(pSeries_reconfig_chain); +BLOCKING_NOTIFIER_HEAD(pSeries_reconfig_chain);

int pSeries_reconfig_notifier_register(struct notifier_block *nb) {

#6 lore raw

Re: [PATCH 4/5 v3] kernel handling of memory DLPAR

In reply to Nathan Fontenot

This adds the capability to DLPAR add and remove memory from the kernel. The patch extends the powerpc handling of memory_add_physaddr_to_nid(), which is called from the sysfs memory 'probe' file to first ensure that the memory has been added to the system. This is done by creating a platform specific callout from the routine. The pseries implementation of this handles the DLPAR work to add the memory to the system and update the device tree.

The patch also creates a pseries only 'release' sys file, /sys/devices/system/memory/release. This file handles the DLPAR release of memory back to firmware and updating of the device-tree.

Updated to add #ifdef CONFIG_MEMORY_HOTPLUG around the memory hotplug specific updates. This allows the file to be built without CONFIG_MEMORY_HOTPLUG defined.

---

arch/powerpc/platforms/pseries/dlpar.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -16,6 +16,10 @@

#include <linux/notifier.h>
#include <linux/proc_fs.h>
#include <linux/spinlock.h>
+#include <linux/memory_hotplug.h>
+#include <linux/sysdev.h>
+#include <linux/sysfs.h>
+

#include <asm/prom.h>
#include <asm/machdep.h>
@@ -404,11 +408,165 @@

	return 0;
}

+#ifdef CONFIG_MEMORY_HOTPLUG
+
+static struct property *clone_property(struct property *old_prop) +{
+ struct property *new_prop;
+
+ new_prop = kzalloc((sizeof *new_prop), GFP_KERNEL);
+ if (!new_prop)
+ return NULL;
+
+ new_prop->name = kzalloc(strlen(old_prop->name) + 1, GFP_KERNEL);
+ new_prop->value = kzalloc(old_prop->length + 1, GFP_KERNEL);
+ if (!new_prop->name || !new_prop->value) {
+ free_property(new_prop);
+ return NULL;
+ }
+
+ strcpy(new_prop->name, old_prop->name);
+ memcpy(new_prop->value, old_prop->value, old_prop->length);
+ new_prop->length = old_prop->length;
+
+ return new_prop;
+}
+
+int platform_probe_memory(u64 phys_addr)
+{
+ struct device_node *dn;
+ struct property *new_prop, *old_prop;
+ struct property *lmb_sz_prop;
+ struct of_drconf_cell *drmem;
+ u64 lmb_size;
+ int num_entries, i, rc;
+
+ if (!phys_addr)
+ return -EINVAL;
+
+ dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!dn)
+ return -EINVAL;
+
+ lmb_sz_prop = of_find_property(dn, "ibm,lmb-size", NULL);
+ lmb_size = *(u64 *)lmb_sz_prop->value;
+
+ old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+
+ num_entries = *(u32 *)old_prop->value;
+ drmem = (struct of_drconf_cell *)
+ ((char *)old_prop->value + sizeof(u32));
+
+ for (i = 0; i < num_entries; i++) {
+ u64 lmb_end_addr = drmem[i].base_addr + lmb_size;
+ if (phys_addr >= drmem[i].base_addr
+ && phys_addr < lmb_end_addr)
+ break;
+ }
+
+ if (i >= num_entries) {
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ if (drmem[i].flags & DRCONF_MEM_ASSIGNED) {
+ of_node_put(dn);
+ return 0;
+ }
+
+ rc = acquire_drc(drmem[i].drc_index);
+ if (rc) {
+ of_node_put(dn);
+ return -1;
+ }
+
+ new_prop = clone_property(old_prop);
+ drmem = (struct of_drconf_cell *)
+ ((char *)new_prop->value + sizeof(u32));
+
+ drmem[i].flags |= DRCONF_MEM_ASSIGNED;
+ prom_update_property(dn, new_prop, old_prop);
+
+ rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+ PSERIES_DRCONF_MEM_ADD,
+ &drmem[i].base_addr);
+ if (rc == NOTIFY_BAD) {
+ prom_update_property(dn, old_prop, new_prop);
+ release_drc(drmem[i].drc_index);
+ }
+
+ of_node_put(dn);
+ return rc == NOTIFY_BAD ? -1 : 0;
+}
+
+static ssize_t memory_release_store(struct class *class, const char *buf,
+ size_t count)
+{
+ unsigned long drc_index;
+ struct device_node *dn;
+ struct property *new_prop, *old_prop;
+ struct of_drconf_cell *drmem;
+ int num_entries;
+ int i, rc;
+
+ rc = strict_strtoul(buf, 0, &drc_index);
+ if (rc)
+ return -EINVAL;
+
+ dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+ if (!dn)
+ return 0;
+
+ old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+ new_prop = clone_property(old_prop);
+
+ num_entries = *(u32 *)new_prop->value;
+ drmem = (struct of_drconf_cell *)
+ ((char *)new_prop->value + sizeof(u32));
+
+ for (i = 0; i < num_entries; i++) {
+ if (drmem[i].drc_index == drc_index)
+ break;
+ }
+
+ if (i >= num_entries) {
+ free_property(new_prop);
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ drmem[i].flags &= ~DRCONF_MEM_ASSIGNED;
+ prom_update_property(dn, new_prop, old_prop);
+
+ rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+ PSERIES_DRCONF_MEM_REMOVE,
+ &drmem[i].base_addr);
+ if (rc != NOTIFY_BAD)
+ rc = release_drc(drc_index);
+
+ if (rc)
+ prom_update_property(dn, old_prop, new_prop);
+
+ of_node_put(dn);
+ return rc ? -1 : count;
+}
+
+static struct class_attribute class_attr_mem_release =
+ __ATTR(release, S_IWUSR, NULL, memory_release_store);
+#endif
+
static int pseries_dlpar_init(void)

{
	if (!machine_is(pseries))
		return 0;

+#ifdef CONFIG_MEMORY_HOTPLUG
+ if (sysfs_create_file(&memory_sysdev_class.kset.kobj,
+ &class_attr_mem_release.attr))
+ printk(KERN_INFO "DLPAR: Could not create sysfs memory "
+ "release file\n");
+#endif
+

	return 0;
}

device_initcall(pseries_dlpar_init);

arch/powerpc/mm/mem.c +0 -0
Index: powerpc/arch/powerpc/mm/mem.c
===================================================================
@@ -111,8 +111,19 @@

#ifdef CONFIG_MEMORY_HOTPLUG

#ifdef CONFIG_NUMA
+int __attribute ((weak)) platform_probe_memory(u64 start) +{
+ return 0;
+}
+
int memory_add_physaddr_to_nid(u64 start)
{
+ int rc;
+
+ rc = platform_probe_memory(start);
+ if (rc)
+ return rc;
+

	return hot_add_scn_to_nid(start);
}

#endif

#10 lore raw

Re: [PATCH 5/5 v2] kernel handling of CPU DLPAR

In reply to Nathan Fontenot

This adds the capability to DLPAR add and remove CPUs from the kernel. The creates two new files /sys/devices/system/cpu/probe and /sys/devices/system/cpu/release to handle the DLPAR addition and removal of CPUs respectively.

CPU DLPAR add is accomplished by writing the drc-index of the CPU to the probe file, and removal is done by writing the device-tree path of the cpu to the release file.

Updated to include #ifdef CONFIG_HOTPLUG_CPU around the cpu hotplug specific bits so that it will build without CONFIG_HOTPLUG_CPU defined.

---

arch/powerpc/platforms/pseries/dlpar.c +0 -0
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -1,11 +1,11 @@

/*
- * dlpar.c - support for dynamic reconfiguration (including PCI
- * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ * dlpar.c - support for dynamic reconfiguration (including PCI,
+ * Memory, and CPU Hotplug and Dynamic Logical Partitioning on
+ * PAPR platforms).
*
* Copyright (C) 2009 Nathan Fontenot
* Copyright (C) 2009 IBM Corporation
*
- *
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License version
* 2 as published by the Free Software Foundation.
@@ -19,6 +19,7 @@
#include <linux/memory_hotplug.h>
#include <linux/sysdev.h>
#include <linux/sysfs.h>
+#include <linux/cpu.h>

#include <asm/prom.h>
@@ -408,6 +409,82 @@

	return 0;
}

+#ifdef CONFIG_HOTPLUG_CPU
+static ssize_t cpu_probe_store(struct class *class, const char *buf,
+ size_t count)
+{
+ struct device_node *dn;
+ unsigned long drc_index;
+ char *cpu_name;
+ int rc;
+
+ rc = strict_strtoul(buf, 0, &drc_index);
+ if (rc)
+ return -EINVAL;
+
+ rc = acquire_drc(drc_index);
+ if (rc)
+ return rc;
+
+ dn = configure_connector(drc_index);
+ if (!dn) {
+ release_drc(drc_index);
+ return rc;
+ }
+
+ /* fixup dn name */
+ cpu_name = kzalloc(strlen(dn->full_name) + strlen("/cpus/") + 1,
+ GFP_KERNEL);
+ if (!cpu_name) {
+ free_cc_nodes(dn);
+ release_drc(drc_index);
+ return -ENOMEM;
+ }
+
+ sprintf(cpu_name, "/cpus/%s", dn->full_name);
+ kfree(dn->full_name);
+ dn->full_name = cpu_name;
+
+ rc = add_device_tree_nodes(dn);
+ if (rc)
+ release_drc(drc_index);
+
+ return rc ? rc : count;
+}
+
+static ssize_t cpu_release_store(struct class *class, const char *buf,
+ size_t count)
+{
+ struct device_node *dn;
+ u32 *drc_index;
+ int rc;
+
+ dn = of_find_node_by_path(buf);
+ if (!dn)
+ return -EINVAL;
+
+ drc_index = (u32 *)of_get_property(dn, "ibm,my-drc-index", NULL);
+ if (!drc_index) {
+ of_node_put(dn);
+ return -EINVAL;
+ }
+
+ rc = release_drc(*drc_index);
+ if (rc) {
+ of_node_put(dn);
+ return rc;
+ }
+
+ rc = remove_device_tree_nodes(dn);
+ if (rc)
+ acquire_drc(*drc_index);
+
+ of_node_put(dn);
+ return rc ? rc : count;
+}
+
+#endif /* CONFIG_HOTPLUG_CPU */
+
#ifdef CONFIG_MEMORY_HOTPLUG

static struct property *clone_property(struct property *old_prop) @@ -553,6 +630,13 @@

static struct class_attribute class_attr_mem_release =

			__ATTR(release, S_IWUSR, NULL, memory_release_store);

+#endif /* CONFIG_MEMORY_HOTPLUG */
+
+#ifdef CONFIG_HOTPLUG_CPU
+static struct class_attribute class_attr_cpu_probe =
+ __ATTR(probe, S_IWUSR, NULL, cpu_probe_store);
+static struct class_attribute class_attr_cpu_release =
+ __ATTR(release, S_IWUSR, NULL, cpu_release_store);
#endif

static int pseries_dlpar_init(void)
@@ -567,6 +651,18 @@

		       "release file\n");

#endif

+#ifdef CONFIG_HOTPLUG_CPU
+ if (sysfs_create_file(&cpu_sysdev_class.kset.kobj,
+ &class_attr_cpu_probe.attr))
+ printk(KERN_INFO "DLPAR: Could not create sysfs cpu "
+ "probe file\n");
+
+ if (sysfs_create_file(&cpu_sysdev_class.kset.kobj,
+ &class_attr_cpu_release.attr))
+ printk(KERN_INFO "DLPAR: Could not create sysfs cpu "
+ "release file\n");
+#endif
+

	return 0;
}

device_initcall(pseries_dlpar_init);

#11 lore raw

Re: [PATCH 5/5 v2] kernel handling of CPU DLPAR

Quoted 4 lines · Nathan Fontenot

This adds the capability to DLPAR add and remove CPUs from the kernel. The creates two new files /sys/devices/system/cpu/probe and /sys/devices/system/cpu/release to handle the DLPAR addition and removal of CPUs respectively.

How does this relate to the existing cpu hotplug mechanism? Or is this making the cpu exist (possible), vs marking it as online?

Is some other platform going to want to do the same? ie. should the probe/release part be in generic code?

arch/powerpc/platforms/pseries/dlpar.c line 2

Quoted 9 lines

Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================

arch/powerpc/platforms/pseries/dlpar.c +1 -2
@@ -1,11 +1,11 @@
 /*
- * dlpar.c - support for dynamic reconfiguration (including PCI
- * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ * dlpar.c - support for dynamic reconfiguration (including PCI,

We know it's dlpar.c :)

Quoted 47 lines
arch/powerpc/platforms/pseries/dlpar.c +28 -1
+ * Memory, and CPU Hotplug and Dynamic Logical Partitioning on
+ * PAPR platforms).
  *
  * Copyright (C) 2009 Nathan Fontenot
  * Copyright (C) 2009 IBM Corporation
  *
- *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License version
  * 2 as published by the Free Software Foundation.
@@ -19,6 +19,7 @@
 #include <linux/memory_hotplug.h>
 #include <linux/sysdev.h>
 #include <linux/sysfs.h>
+#include <linux/cpu.h>
 
 
 #include <asm/prom.h>
@@ -408,6 +409,82 @@
 	return 0;
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static ssize_t cpu_probe_store(struct class *class, const char *buf,
+			       size_t count)
+{
+	struct device_node *dn;
+	unsigned long drc_index;
+	char *cpu_name;
+	int rc;
+
+	rc = strict_strtoul(buf, 0, &drc_index);
+	if (rc)
+		return -EINVAL;
+
+	rc = acquire_drc(drc_index);
+	if (rc)
+		return rc;
+
+	dn = configure_connector(drc_index);
+	if (!dn) {
+		release_drc(drc_index);
+		return rc;
+	}
+
+	/* fixup dn name */
+	cpu_name = kzalloc(strlen(dn->full_name) + strlen("/cpus/") + 1,

arch/powerpc/platforms/pseries/dlpar.c line 446

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+			   GFP_KERNEL);
+	if (!cpu_name) {
+		free_cc_nodes(dn);
+		release_drc(drc_index);
+		return -ENOMEM;
+	}
+
+	sprintf(cpu_name, "/cpus/%s", dn->full_name);
+	kfree(dn->full_name);
+	dn->full_name = cpu_name;

What was all that? Firmware gives us a bogus full name? But the parent is right?

arch/powerpc/platforms/pseries/dlpar.c line 451

Quoted 5 lines
arch/powerpc/platforms/pseries/dlpar.c +5 -0
+	rc = add_device_tree_nodes(dn);
+	if (rc)
+		release_drc(drc_index);
+
+	return rc ? rc : count;

You're sure rc is < 0.

Quoted 4 lines
arch/powerpc/platforms/pseries/dlpar.c +4 -0
+}
+
+static ssize_t cpu_release_store(struct class *class, const char *buf,
+				 size_t count)

arch/powerpc/platforms/pseries/dlpar.c line 465

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+{
+	struct device_node *dn;
+	u32 *drc_index;
+	int rc;
+
+	dn = of_find_node_by_path(buf);
+	if (!dn)
+		return -EINVAL;
+
+	drc_index = (u32 *)of_get_property(dn, "ibm,my-drc-index", NULL);

No cast required.

Quoted 8 lines
arch/powerpc/platforms/pseries/dlpar.c +8 -0
+	if (!drc_index) {
+		of_node_put(dn);
+		return -EINVAL;
+	}
+
+	rc = release_drc(*drc_index);
+	if (rc) {
+		of_node_put(dn);

arch/powerpc/platforms/pseries/dlpar.c line 483

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+		return rc;
+	}
+
+	rc = remove_device_tree_nodes(dn);
+	if (rc)
+		acquire_drc(*drc_index);
+
+	of_node_put(dn);
+	return rc ? rc : count;
+}

cheers

#7 lore raw

Re: [PATCH 4/5 v3] kernel handling of memory DLPAR

In reply to Nathan Fontenot
Quoted 1 line · Nathan Fontenot

This adds the capability to DLPAR add and remove memory from the kernel. The

Hi Nathan,

Sorry to only get around to reviewing version 3, time is a commodity in short supply :)

Quoted 20 lines

Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================

arch/powerpc/platforms/pseries/dlpar.c +5 -0
@@ -16,6 +16,10 @@
 #include <linux/notifier.h>
 #include <linux/proc_fs.h>
 #include <linux/spinlock.h>
+#include <linux/memory_hotplug.h>
+#include <linux/sysdev.h>
+#include <linux/sysfs.h>
+
 
 #include <asm/prom.h>
 #include <asm/machdep.h>
@@ -404,11 +408,165 @@
 	return 0;
 }
 
+#ifdef CONFIG_MEMORY_HOTPLUG

arch/powerpc/platforms/pseries/dlpar.c line 421

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+
+static struct property *clone_property(struct property *old_prop)
+{
+	struct property *new_prop;
+
+	new_prop = kzalloc((sizeof *new_prop), GFP_KERNEL);
+	if (!new_prop)
+		return NULL;
+
+	new_prop->name = kzalloc(strlen(old_prop->name) + 1, GFP_KERNEL);

kstrdup()?

Quoted 21 lines
arch/powerpc/platforms/pseries/dlpar.c +21 -0
+	new_prop->value = kzalloc(old_prop->length + 1, GFP_KERNEL);
+	if (!new_prop->name || !new_prop->value) {
+		free_property(new_prop);
+		return NULL;
+	}
+
+	strcpy(new_prop->name, old_prop->name);
+	memcpy(new_prop->value, old_prop->value, old_prop->length);
+	new_prop->length = old_prop->length;
+
+	return new_prop;
+}
+
+int platform_probe_memory(u64 phys_addr)
+{
+	struct device_node *dn;
+	struct property *new_prop, *old_prop;
+	struct property *lmb_sz_prop;
+	struct of_drconf_cell *drmem;
+	u64 lmb_size;
+	int num_entries, i, rc;

arch/powerpc/platforms/pseries/dlpar.c line 452

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+
+	if (!phys_addr)
+		return -EINVAL;
+
+	dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (!dn)
+		return -EINVAL;
+
+	lmb_sz_prop = of_find_property(dn, "ibm,lmb-size", NULL);
+	lmb_size = *(u64 *)lmb_sz_prop->value;

of_get_property() ?

arch/powerpc/platforms/pseries/dlpar.c line 454

Quoted 2 lines
arch/powerpc/platforms/pseries/dlpar.c +2 -0
+
+	old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);

I know we should never fail to find these properties, but it would be nice to check just in case.

arch/powerpc/platforms/pseries/dlpar.c line 458

Quoted 4 lines
arch/powerpc/platforms/pseries/dlpar.c +4 -0
+
+	num_entries = *(u32 *)old_prop->value;
+	drmem = (struct of_drconf_cell *)
+				((char *)old_prop->value + sizeof(u32));

You do this dance twice (see below), a struct might make it cleaner.

Quoted 5 lines
arch/powerpc/platforms/pseries/dlpar.c +5 -0
+	for (i = 0; i < num_entries; i++) {
+		u64 lmb_end_addr = drmem[i].base_addr + lmb_size;
+		if (phys_addr >= drmem[i].base_addr
+		    && phys_addr < lmb_end_addr)
+			break;

arch/powerpc/platforms/pseries/dlpar.c line 473

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+	}
+
+	if (i >= num_entries) {
+		of_node_put(dn);
+		return -EINVAL;
+	}
+
+	if (drmem[i].flags & DRCONF_MEM_ASSIGNED) {
+		of_node_put(dn);
+		return 0;

This is the already added case?

arch/powerpc/platforms/pseries/dlpar.c line 479

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+	}
+
+	rc = acquire_drc(drmem[i].drc_index);
+	if (rc) {
+		of_node_put(dn);
+		return -1;

-1 ?

Quoted 9 lines
arch/powerpc/platforms/pseries/dlpar.c +9 -0
+	}
+
+	new_prop = clone_property(old_prop);
+	drmem = (struct of_drconf_cell *)
+				((char *)new_prop->value + sizeof(u32));
+
+	drmem[i].flags |= DRCONF_MEM_ASSIGNED;
+	prom_update_property(dn, new_prop, old_prop);
+

arch/powerpc/platforms/pseries/dlpar.c line 498

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+					  PSERIES_DRCONF_MEM_ADD,
+					  &drmem[i].base_addr);
+	if (rc == NOTIFY_BAD) {
+		prom_update_property(dn, old_prop, new_prop);
+		release_drc(drmem[i].drc_index);
+	}
+
+	of_node_put(dn);
+	return rc == NOTIFY_BAD ? -1 : 0;

-1 ?

Quoted 9 lines
arch/powerpc/platforms/pseries/dlpar.c +9 -0
+}
+
+static ssize_t memory_release_store(struct class *class, const char *buf,
+				    size_t count)
+{
+	unsigned long drc_index;
+	struct device_node *dn;
+	struct property *new_prop, *old_prop;
+	struct of_drconf_cell *drmem;

arch/powerpc/platforms/pseries/dlpar.c line 517

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+	int num_entries;
+	int i, rc;
+
+	rc = strict_strtoul(buf, 0, &drc_index);
+	if (rc)
+		return -EINVAL;
+
+	dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (!dn)
+		return 0;

0 really?

Quoted 8 lines
arch/powerpc/platforms/pseries/dlpar.c +8 -0
+
+	old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+	new_prop = clone_property(old_prop);
+
+	num_entries = *(u32 *)new_prop->value;
+	drmem = (struct of_drconf_cell *)
+				((char *)new_prop->value + sizeof(u32));
+

arch/powerpc/platforms/pseries/dlpar.c line 535

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+	for (i = 0; i < num_entries; i++) {
+		if (drmem[i].drc_index == drc_index)
+			break;
+	}
+
+	if (i >= num_entries) {
+		free_property(new_prop);
+		of_node_put(dn);
+		return -EINVAL;
+	}

Couldn't use old_prop up until here? They're identical aren't they, so you can do the clone here and you can avoid the free in the above error case.

Quoted 4 lines
arch/powerpc/platforms/pseries/dlpar.c +4 -0
+	drmem[i].flags &= ~DRCONF_MEM_ASSIGNED;
+	prom_update_property(dn, new_prop, old_prop);
+
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,

arch/powerpc/platforms/pseries/dlpar.c line 549

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+					  PSERIES_DRCONF_MEM_REMOVE,
+					  &drmem[i].base_addr);
+	if (rc != NOTIFY_BAD)
+		rc = release_drc(drc_index);
+
+	if (rc)
+		prom_update_property(dn, old_prop, new_prop);
+
+	of_node_put(dn);
+	return rc ? -1 : count;

-1, EPERM?

Quoted 26 lines
arch/powerpc/platforms/pseries/dlpar.c +13 -0
+}
+
+static struct class_attribute class_attr_mem_release =
+			__ATTR(release, S_IWUSR, NULL, memory_release_store);
+#endif
+
 static int pseries_dlpar_init(void)
 {
 	if (!machine_is(pseries))
 		return 0;
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	if (sysfs_create_file(&memory_sysdev_class.kset.kobj,
+			      &class_attr_mem_release.attr))
+		printk(KERN_INFO "DLPAR: Could not create sysfs memory "
+		       "release file\n");
+#endif
+
 	return 0;
 }
 device_initcall(pseries_dlpar_init);

Index: powerpc/arch/powerpc/mm/mem.c
===================================================================

arch/powerpc/mm/mem.c line 114

Quoted 4 lines
arch/powerpc/mm/mem.c +1 -0
@@ -111,8 +111,19 @@
 #ifdef CONFIG_MEMORY_HOTPLUG
 
 #ifdef CONFIG_NUMA
+int __attribute ((weak)) platform_probe_memory(u64 start)

__weak

Though be careful, I think this is vulnerable to a bug in some toolchains where the compiler will inline this version. See the comment around early_irq_init() in kernel/softirq.c for example.

This will need to be a ppc_md hook as soon as another platform supports memory hotplug, though that may be never :)

Quoted 5 lines
arch/powerpc/mm/mem.c +4 -0
+{
+	return 0;
+}
+
 int memory_add_physaddr_to_nid(u64 start)

arch/powerpc/mm/mem.c line 129

Quoted 10 lines
arch/powerpc/mm/mem.c +6 -0
 {
+	int rc;
+
+	rc = platform_probe_memory(start);
+	if (rc)
+		return rc;
+
 	return hot_add_scn_to_nid(start);
 }
 #endif

cheers

#8 lore raw

Re: [PATCH 4/5 v3] kernel handling of memory DLPAR

Quoted 27 lines · Michael Ellerman
Quoted 1 line · Nathan Fontenot

This adds the capability to DLPAR add and remove memory from the kernel. The

Hi Nathan,

Sorry to only get around to reviewing version 3, time is a commodity in short supply :)

Quoted 19 lines

Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================

arch/powerpc/platforms/pseries/dlpar.c +4 -0
@@ -16,6 +16,10 @@
 #include <linux/notifier.h>
 #include <linux/proc_fs.h>
 #include <linux/spinlock.h>
+#include <linux/memory_hotplug.h>
+#include <linux/sysdev.h>
+#include <linux/sysfs.h>
+
 
 #include <asm/prom.h>
 #include <asm/machdep.h>
@@ -404,11 +408,165 @@
 	return 0;
 }
 
Quoted 13 lines

arch/powerpc/platforms/pseries/dlpar.c line 421

Quoted 11 lines
arch/powerpc/platforms/pseries/dlpar.c +11 -0
+#ifdef CONFIG_MEMORY_HOTPLUG
+
+static struct property *clone_property(struct property *old_prop)
+{
+	struct property *new_prop;
+
+	new_prop = kzalloc((sizeof *new_prop), GFP_KERNEL);
+	if (!new_prop)
+		return NULL;
+
+	new_prop->name = kzalloc(strlen(old_prop->name) + 1, GFP_KERNEL);

kstrdup()?

Ahhh.. I did not know of kstrdup(). I will update to use this instead.

Quoted 26 lines
Quoted 26 lines
arch/powerpc/platforms/pseries/dlpar.c +25 -0
+	new_prop->value = kzalloc(old_prop->length + 1, GFP_KERNEL);
+	if (!new_prop->name || !new_prop->value) {
+		free_property(new_prop);
+		return NULL;
+	}
+
+	strcpy(new_prop->name, old_prop->name);
+	memcpy(new_prop->value, old_prop->value, old_prop->length);
+	new_prop->length = old_prop->length;
+
+	return new_prop;
+}
+
+int platform_probe_memory(u64 phys_addr)
+{
+	struct device_node *dn;
+	struct property *new_prop, *old_prop;
+	struct property *lmb_sz_prop;
+	struct of_drconf_cell *drmem;
+	u64 lmb_size;
+	int num_entries, i, rc;
+
+	if (!phys_addr)
+		return -EINVAL;
+
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 452

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+	dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (!dn)
+		return -EINVAL;
+
+	lmb_sz_prop = of_find_property(dn, "ibm,lmb-size", NULL);
+	lmb_size = *(u64 *)lmb_sz_prop->value;

of_get_property() ?

ok, will switch to of_find_property()

Quoted 6 lines

arch/powerpc/platforms/pseries/dlpar.c line 454

Quoted 2 lines
arch/powerpc/platforms/pseries/dlpar.c +2 -0
+
+	old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);

I know we should never fail to find these properties, but it would be nice to check just in case.

yes, will update.

Quoted 6 lines

arch/powerpc/platforms/pseries/dlpar.c line 458

Quoted 4 lines
arch/powerpc/platforms/pseries/dlpar.c +4 -0
+
+	num_entries = *(u32 *)old_prop->value;
+	drmem = (struct of_drconf_cell *)
+				((char *)old_prop->value + sizeof(u32));

You do this dance twice (see below), a struct might make it cleaner.

Agreed. I think I will make this update in a separate patch. Updating this to use a of_drconf struct would benefit this code as well as the code in powerpc/numa.c that also deals with manipulating this property.

Quoted 10 lines
Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +9 -0
+	for (i = 0; i < num_entries; i++) {
+		u64 lmb_end_addr = drmem[i].base_addr + lmb_size;
+		if (phys_addr >= drmem[i].base_addr
+		    && phys_addr < lmb_end_addr)
+			break;
+	}
+
+	if (i >= num_entries) {
+		of_node_put(dn);
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 473

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+		return -EINVAL;
+	}
+
+	if (drmem[i].flags & DRCONF_MEM_ASSIGNED) {
+		of_node_put(dn);
+		return 0;

This is the already added case?

Yes. In this case the lmb is already assigned to the system.

Quoted 9 lines

arch/powerpc/platforms/pseries/dlpar.c line 479

Quoted 7 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+	}
+
+	rc = acquire_drc(drmem[i].drc_index);
+	if (rc) {
+		of_node_put(dn);
+		return -1;

-1 ?

Yeah, bad choice.

Quoted 14 lines
Quoted 14 lines
arch/powerpc/platforms/pseries/dlpar.c +13 -0
+	}
+
+	new_prop = clone_property(old_prop);
+	drmem = (struct of_drconf_cell *)
+				((char *)new_prop->value + sizeof(u32));
+
+	drmem[i].flags |= DRCONF_MEM_ASSIGNED;
+	prom_update_property(dn, new_prop, old_prop);
+
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+					  PSERIES_DRCONF_MEM_ADD,
+					  &drmem[i].base_addr);
+	if (rc == NOTIFY_BAD) {
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 498

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+		prom_update_property(dn, old_prop, new_prop);
+		release_drc(drmem[i].drc_index);
+	}
+
+	of_node_put(dn);
+	return rc == NOTIFY_BAD ? -1 : 0;

-1 ?

Another bad return code choice.

Quoted 14 lines
Quoted 14 lines
arch/powerpc/platforms/pseries/dlpar.c +13 -0
+}
+
+static ssize_t memory_release_store(struct class *class, const char *buf,
+				    size_t count)
+{
+	unsigned long drc_index;
+	struct device_node *dn;
+	struct property *new_prop, *old_prop;
+	struct of_drconf_cell *drmem;
+	int num_entries;
+	int i, rc;
+
+	rc = strict_strtoul(buf, 0, &drc_index);
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 517

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+	if (rc)
+		return -EINVAL;
+
+	dn = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (!dn)
+		return 0;

0 really?

... and again...

Quoted 15 lines
Quoted 15 lines
arch/powerpc/platforms/pseries/dlpar.c +14 -0
+
+	old_prop = of_find_property(dn, "ibm,dynamic-memory", NULL);
+	new_prop = clone_property(old_prop);
+
+	num_entries = *(u32 *)new_prop->value;
+	drmem = (struct of_drconf_cell *)
+				((char *)new_prop->value + sizeof(u32));
+
+	for (i = 0; i < num_entries; i++) {
+		if (drmem[i].drc_index == drc_index)
+			break;
+	}
+
+	if (i >= num_entries) {
Quoted 9 lines

arch/powerpc/platforms/pseries/dlpar.c line 535

Quoted 4 lines
arch/powerpc/platforms/pseries/dlpar.c +4 -0
+		free_property(new_prop);
+		of_node_put(dn);
+		return -EINVAL;
+	}

Couldn't use old_prop up until here? They're identical aren't they, so you can do the clone here and you can avoid the free in the above error case.

Yes, this is possible. I will clean this up.

Quoted 8 lines
Quoted 8 lines
arch/powerpc/platforms/pseries/dlpar.c +8 -0
+	drmem[i].flags &= ~DRCONF_MEM_ASSIGNED;
+	prom_update_property(dn, new_prop, old_prop);
+
+	rc = blocking_notifier_call_chain(&pSeries_reconfig_chain,
+					  PSERIES_DRCONF_MEM_REMOVE,
+					  &drmem[i].base_addr);
+	if (rc != NOTIFY_BAD)
+		rc = release_drc(drc_index);
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 549

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+
+	if (rc)
+		prom_update_property(dn, old_prop, new_prop);
+
+	of_node_put(dn);
+	return rc ? -1 : count;

-1, EPERM?

EPERM.

Quoted 34 lines
Quoted 27 lines
arch/powerpc/platforms/pseries/dlpar.c +13 -0
+}
+
+static struct class_attribute class_attr_mem_release =
+			__ATTR(release, S_IWUSR, NULL, memory_release_store);
+#endif
+
 static int pseries_dlpar_init(void)
 {
 	if (!machine_is(pseries))
 		return 0;
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	if (sysfs_create_file(&memory_sysdev_class.kset.kobj,
+			      &class_attr_mem_release.attr))
+		printk(KERN_INFO "DLPAR: Could not create sysfs memory "
+		       "release file\n");
+#endif
+
 	return 0;
 }
 device_initcall(pseries_dlpar_init);

Index: powerpc/arch/powerpc/mm/mem.c
===================================================================

arch/powerpc/mm/mem.c line 114

Quoted 4 lines
arch/powerpc/mm/mem.c +1 -0
@@ -111,8 +111,19 @@
 #ifdef CONFIG_MEMORY_HOTPLUG
 
 #ifdef CONFIG_NUMA
+int __attribute ((weak)) platform_probe_memory(u64 start)

__weak

Quoted 7 lines

Though be careful, I think this is vulnerable to a bug in some toolchains where the compiler will inline this version. See the comment around early_irq_init() in kernel/softirq.c for example.

This will need to be a ppc_md hook as soon as another platform supports memory hotplug, though that may be never :)

I didn't know any other way to implement this. If using a weak symbol is ok I will leave it. I am open to suggestions if there is a better way to do this.

thanks for reviewing the patch.

-Nathan

Quoted 18 lines
Quoted 5 lines
arch/powerpc/mm/mem.c +4 -0
+{
+	return 0;
+}
+
 int memory_add_physaddr_to_nid(u64 start)

arch/powerpc/mm/mem.c line 129

Quoted 10 lines
arch/powerpc/mm/mem.c +6 -0
 {
+	int rc;
+
+	rc = platform_probe_memory(start);
+	if (rc)
+		return rc;
+
 	return hot_add_scn_to_nid(start);
 }
 #endif

cheers

#12 lore raw

Re: [PATCH 5/5 v2] kernel handling of CPU DLPAR

In reply to Michael Ellerman
Quoted 8 lines · Michael Ellerman
Quoted 4 lines · Nathan Fontenot

This adds the capability to DLPAR add and remove CPUs from the kernel. The creates two new files /sys/devices/system/cpu/probe and /sys/devices/system/cpu/release to handle the DLPAR addition and removal of CPUs respectively.

How does this relate to the existing cpu hotplug mechanism? Or is this making the cpu exist (possible), vs marking it as online?

This update makes the cpu exist, it does not mark the cpu online.

Quoted 3 lines

Is some other platform going to want to do the same? ie. should the probe/release part be in generic code?

I thought about making this generic code, perhaps a follow-on patch to move the creation of the probe/release files to generic code to see what the community thinks. I would assume there would still need to be a arch and/or platforms specific callout to do the real work.

Quoted 12 lines

arch/powerpc/platforms/pseries/dlpar.c line 2

Quoted 10 lines

Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================

arch/powerpc/platforms/pseries/dlpar.c +1 -2
@@ -1,11 +1,11 @@
 /*
- * dlpar.c - support for dynamic reconfiguration (including PCI
- * Hotplug and Dynamic Logical Partitioning on RPA platforms).
+ * dlpar.c - support for dynamic reconfiguration (including PCI,

We know it's dlpar.c :)

heh! good point. Consider it gone.

Quoted 53 lines
Quoted 53 lines
arch/powerpc/platforms/pseries/dlpar.c +33 -1
+ * Memory, and CPU Hotplug and Dynamic Logical Partitioning on
+ * PAPR platforms).
  *
  * Copyright (C) 2009 Nathan Fontenot
  * Copyright (C) 2009 IBM Corporation
  *
- *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License version
  * 2 as published by the Free Software Foundation.
@@ -19,6 +19,7 @@
 #include <linux/memory_hotplug.h>
 #include <linux/sysdev.h>
 #include <linux/sysfs.h>
+#include <linux/cpu.h>
 
 
 #include <asm/prom.h>
@@ -408,6 +409,82 @@
 	return 0;
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static ssize_t cpu_probe_store(struct class *class, const char *buf,
+			       size_t count)
+{
+	struct device_node *dn;
+	unsigned long drc_index;
+	char *cpu_name;
+	int rc;
+
+	rc = strict_strtoul(buf, 0, &drc_index);
+	if (rc)
+		return -EINVAL;
+
+	rc = acquire_drc(drc_index);
+	if (rc)
+		return rc;
+
+	dn = configure_connector(drc_index);
+	if (!dn) {
+		release_drc(drc_index);
+		return rc;
+	}
+
+	/* fixup dn name */
+	cpu_name = kzalloc(strlen(dn->full_name) + strlen("/cpus/") + 1,
+			   GFP_KERNEL);
+	if (!cpu_name) {
+		free_cc_nodes(dn);
+		release_drc(drc_index);
+		return -ENOMEM;
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 446

Quoted 5 lines
arch/powerpc/platforms/pseries/dlpar.c +5 -0
+	}
+
+	sprintf(cpu_name, "/cpus/%s", dn->full_name);
+	kfree(dn->full_name);
+	dn->full_name = cpu_name;

What was all that? Firmware gives us a bogus full name? But the parent is right?

Yeah, this has always been an oddity to me. The name that is given to us from firmware puts the cpu node in the root of the device tree as opposed to /cpus. Since cpu nodes are in /cpus, the name is fixed up to reflect this.

This name fixup has always been done by the drmgr userspace command, which is where this functionality is being imported from.

Quoted 9 lines

arch/powerpc/platforms/pseries/dlpar.c line 451

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +5 -0
+	rc = add_device_tree_nodes(dn);
+	if (rc)
+		release_drc(drc_index);
+
+	return rc ? rc : count;

You're sure rc is < 0.

Oh, good catch.

Quoted 8 lines
Quoted 8 lines
arch/powerpc/platforms/pseries/dlpar.c +8 -0
+}
+
+static ssize_t cpu_release_store(struct class *class, const char *buf,
+				 size_t count)
+{
+	struct device_node *dn;
+	u32 *drc_index;
+	int rc;
Quoted 8 lines

arch/powerpc/platforms/pseries/dlpar.c line 465

Quoted 6 lines
arch/powerpc/platforms/pseries/dlpar.c +6 -0
+
+	dn = of_find_node_by_path(buf);
+	if (!dn)
+		return -EINVAL;
+
+	drc_index = (u32 *)of_get_property(dn, "ibm,my-drc-index", NULL);

No cast required.

ok.

thanks for the review.

-Nathan

Quoted 22 lines
Quoted 9 lines
arch/powerpc/platforms/pseries/dlpar.c +8 -0
+	if (!drc_index) {
+		of_node_put(dn);
+		return -EINVAL;
+	}
+
+	rc = release_drc(*drc_index);
+	if (rc) {
+		of_node_put(dn);

arch/powerpc/platforms/pseries/dlpar.c line 483

Quoted 10 lines
arch/powerpc/platforms/pseries/dlpar.c +10 -0
+		return rc;
+	}
+
+	rc = remove_device_tree_nodes(dn);
+	if (rc)
+		acquire_drc(*drc_index);
+
+	of_node_put(dn);
+	return rc ? rc : count;
+}

cheers

#13 lore raw

Re: [PATCH 5/5 v2] kernel handling of CPU DLPAR

Quoted 13 lines · Nathan Fontenot
Quoted 8 lines · Michael Ellerman
Quoted 4 lines · Nathan Fontenot

This adds the capability to DLPAR add and remove CPUs from the kernel. The creates two new files /sys/devices/system/cpu/probe and /sys/devices/system/cpu/release to handle the DLPAR addition and removal of CPUs respectively.

How does this relate to the existing cpu hotplug mechanism? Or is this making the cpu exist (possible), vs marking it as online?

This update makes the cpu exist, it does not mark the cpu online.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Is some other platform going to want to do the same? ie. should the probe/release part be in generic code?

I thought about making this generic code, perhaps a follow-on patch to move the creation of the probe/release files to generic code to see what the community thinks. I would assume there would still need to be a arch and/or platforms specific callout to do the real work.

I'm not so worried about the code being generic, there's not much that would be generic. More the mechanism. We don't want to add probe/release to powerpc, and then have a different mechanism get added later by some other platform, or generically.

But I guess you CC'ed lkml and there's not much more to do, I don't know specifically who'd care about it.

cheers