NUMA hotplug emulator
Subsystems (3)
RFC 3/7 NUMA hotplug emulator
Userland interface to hotplug-add fake offlined nodes.
Add a sysfs entry "probe" under /sys/devices/system/node/:
- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe
- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probe
---
arch/x86/Kconfig +14 -0
@@ -1214,6 +1214,20 @@ config NUMA_EMU into virtual nodes when booted with "numa=fake=N", where N is the number of nodes. This is only useful for debugging. +config NUMA_HOTPLUG_EMU + bool "NUMA hotplug emulator" + depends on X86_64 && NUMA && HOTPLUG + ---help--- + +config NODE_HOTPLUG_EMU + bool "Node hotplug emulation" + depends on NUMA_HOTPLUG_EMU && MEMORY_HOTPLUG + ---help--- + Enable Node hotplug emulation. The machine will be setup with + hidden virtual nodes when booted with "numa=hide=N*size", where + N is the number of hidden nodes, size is the memory size per + hidden node. This is only useful for debugging. + config NODES_SHIFT int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP range 1 10
drivers/base/node.c +40 -11
72 lines not on this page · raw message
Signature · 2 lines
Thanks & Regards, Shaohui
Userland interface to hotplug-add fake offlined nodes.
Add a sysfs entry "probe" under /sys/devices/system/node/:
- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe
- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probe
---
arch/x86/Kconfig +14 -0
@@ -1214,6 +1214,20 @@ config NUMA_EMU into virtual nodes when booted with "numa=fake=N", where N is the number of nodes. This is only useful for debugging. +config NUMA_HOTPLUG_EMU + bool "NUMA hotplug emulator" + depends on X86_64 && NUMA && HOTPLUG + ---help--- + +config NODE_HOTPLUG_EMU + bool "Node hotplug emulation" + depends on NUMA_HOTPLUG_EMU && MEMORY_HOTPLUG + ---help--- + Enable Node hotplug emulation. The machine will be setup with + hidden virtual nodes when booted with "numa=hide=N*size", where + N is the number of hidden nodes, size is the memory size per + hidden node. This is only useful for debugging. + config NODES_SHIFT int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP range 1 10
drivers/base/node.c +40 -11
72 lines not on this page · raw message
Quoted 1 line · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Why include 2 copies of the patch in one email?
Quoted 7 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probe
As you are trying to add a new sysfs file, please create the matching Documentation/ABI/ file as well.
Also note that sysfs files are "one value per file", which I don't think this file follows, right?
thanks,
greg k-h
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Greg KH
Quoted 5 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe
Quoted 8 lines
Quoted 2 lines
- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.Also note that sysfs files are "one value per file", which I don't think this file follows, right?
I think in this case, it was meant to be a list of acceptable parameters rather than a set of values, kinda like /sys/power/state. Instead, I guess we could have:
/sys/devices/system/node/probeable/3 /sys/devices/system/node/probeable/43 /sys/devices/system/node/probeable/65 /sys/devices/system/node/probeable/5145
and the knowledge that you need to pick one of those to echo into /sys/devices/system/node/probe. But, it's a lot more self explanatory if you 'cat /sys/devices/system/node/probe', and then pick one of those to echo back into the file.
Seems like a decent place to violate the "rule". :)
-- Dave
Quoted 24 lines · Dave Hansen
Quoted 5 lines · Greg KH
Quoted 5 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probeQuoted 8 lines
Quoted 2 lines
- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.Also note that sysfs files are "one value per file", which I don't think this file follows, right?
I think in this case, it was meant to be a list of acceptable parameters rather than a set of values, kinda like
/sys/power/state. Instead, I guess we could have:/sys/devices/system/node/probeable/3 /sys/devices/system/node/probeable/43 /sys/devices/system/node/probeable/65 /sys/devices/system/node/probeable/5145
Quoted 6 lines
and the knowledge that you need to pick one of those to echo into
/sys/devices/system/node/probe. But, it's a lot more self explanatory if you 'cat/sys/devices/system/node/probe', and then pick one of those to echo back into the file.Seems like a decent place to violate the "rule". :)
How big would this "list" be? What will it look like exactly?
thanks,
greg k-h
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Dave Hansen
Quoted 7 lines · Greg KH
Quoted 7 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probe
Quoted 8 lines
Quoted 5 lines
As you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.Also note that sysfs files are "one value per file", which I don't think this file follows, right?
I think in this case, it was meant to be a list of acceptable parameters rather than a set of values, kinda like
/sys/power/state.
Right.
Quoted 9 lines
Instead, I guess we could have:
/sys/devices/system/node/probeable/3 /sys/devices/system/node/probeable/43 /sys/devices/system/node/probeable/65 /sys/devices/system/node/probeable/5145and the knowledge that you need to pick one of those to echo into
/sys/devices/system/node/probe.
I think this way would make things complex if we just want to show user which node could be hotadded.
Quoted 3 lines
But, it's a lot more self
explanatory if you 'cat/sys/devices/system/node/probe', and then pick one of those to echo back into the file.
agreed.
Quoted 1 line
Seems like a decent place to violate the "rule". :)
Quoted 4 lines · Greg KH
Quoted 1 line · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Why include 2 copies of the patch in one email?
I always try to attach the patch as attachment, it is the same with the mail content, I guess it should take convenience when you need to save the patch to local, it might be a bad habbit, I will be careful when I send patch next time. thanks for the reminding.
Quoted 11 lines
Quoted 8 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.
Agree, We will document it in.
Quoted 3 lines
Also note that sysfs files are "one value per file", which I don't think this file follows, right?
Agree, the cpu/probe interface should write only, and we should create another file to indicate the hidden nodes, such as cpu/hidden. We will follow this rule when we send the formal patch.
Thanks greg's comments so much, have a nice day.
Quoted 4 lines
thanks,
greg k-h
Signature · 2 lines
Thanks & Regards, Shaohui
Quoted 9 lines · Shaohui
Quoted 4 lines · Greg KH
Quoted 1 line · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Why include 2 copies of the patch in one email?
I always try to attach the patch as attachment, it is the same with the mail content, I guess it should take convenience when you need to save the patch to local, it might be a bad habbit, I will be careful when I send patch next time. thanks for the reminding.
Shaohui, git/quilt are great tools for submitting patch series.
Quoted 15 lines
Quoted 11 lines
Quoted 8 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.Agree, We will document it in.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
Also note that sysfs files are "one value per file", which I don't think this file follows, right?
Agree, the cpu/probe interface should write only, and we should create another file to indicate the hidden nodes, such as cpu/hidden. We will follow this rule when we send the formal patch.
I'd prefer to avoid new interfaces if not absolutely necessary.
Thanks,
Fengguang
Quoted 5 lines · Wu Fengguang
Quoted 4 lines · Shaohui
Quoted 3 lines · Greg KH
Quoted 1 line · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Why include 2 copies of the patch in one email?
I always try to attach the patch as attachment, it is the same with the mail content, I guess it should take convenience when you need to save the patch to local, it might be a bad habbit, I will be careful when I send patch next time. thanks for the reminding.
Shaohui, git/quilt are great tools for submitting patch series.
Thanks fengguang, I am the new fish in LKML, so I send the patches manually one by one, I will try this tool next time.
Quoted 27 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:Quoted 8 lines
Quoted 5 lines
- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.Agree, We will document it in.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
Also note that sysfs files are "one value per file", which I don't think this file follows, right?
Agree, the cpu/probe interface should write only, and we should create another file to indicate the hidden nodes, such as cpu/hidden. We will follow this rule when we send the formal patch.
I'd prefer to avoid new interfaces if not absolutely necessary.
Thanks,
Fengguang
Signature · 8 lines
Thanks & Regards, Shaohui -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Greg KH
Quoted 7 lines
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeAs you are trying to add a new sysfs file, please create the matching
Documentation/ABI/file as well.
Yes, we missed it. thank you.
Quoted 2 lines
Also note that sysfs files are "one value per file", which I don't think this file follows, right?
Sorry, we didn't make the descriptions clear enough.
It should follow "one value per file". It intends to show acceptable parameters.
For example, if we have 4 fake offlined nodes, like node 2-5, then:
$ cat /sys/devices/system/node/probe
2-5Then user hotadds node3 to system:
$ echo 3 > /sys/devices/system/node/probe
$ cat /sys/devices/system/node/probe
2,4-5-haicheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeSigned-off-by Haicheng Li <[email protected]>Signed-off-by Shaohui Zheng <[email protected]>---
drivers/base/node.c +0
-0@@ -535,6 +535,26 @@ void unregister_one_node(int nid)unregister_node (&node_devices[nid]); }
drivers/base/node.c void unregister_one_node(int nid) · line 547
Quoted 10 lines
drivers/base/node.c +10
-0+#ifdef CONFIG_NODE_HOTPLUG_EMU +static ssize_t store_nodes_probe (struct sysdev_class *class, +struct sysdev_class_attribute *attr, +const char *buf,size_t count) +{ +long nid; +int ret; + +strict_strtol (buf,0 , &nid); +if (nid <0 || nid > nr_node_ids -1 ) {
Shaohui,
In fact, no need to do such check here, hotadd_hidden_nodes() can handle such invalid parameter by itself.
drivers/base/node.c void unregister_one_node(int nid) · line 550
Quoted 3 lines
drivers/base/node.c +3
-0+printk (KERN_ERR "Invalid NUMA node id: %d (0 <= nid < %d).\n" , + nid, nr_node_ids); +return -EPERM ;
Per Andi's earlier review comments, -EPERM is odd, we'd fix it.
drivers/base/node.c void unregister_one_node(int nid) · line 556
Quoted 6 lines
drivers/base/node.c +6
-0+ } +hotadd_hidden_nodes (nid); + +return count; +} +#endif
Pls. replace it with following code:
+#ifdef CONFIG_NODE_HOTPLUG_EMU
+static ssize_t store_nodes_probe(struct sysdev_class *class,
+ struct sysdev_class_attribute *attr,
+ const char *buf, size_t count)
+{
+ long nid;
+ int ret;
+
+ ret = strict_strtol(buf, 0, &nid);
+ if (ret == -EINVAL)
+ return ret;
+
+ ret = hotadd_hidden_nodes(nid);
+ if (!ret)
+ return count;
+ else
+ return -EIO;
+}
+#endif
-haicheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines
Pls. replace it with following code:
drivers/base/node.c +9
-0+#ifdef CONFIG_NODE_HOTPLUG_EMU +static ssize_t store_nodes_probe (struct sysdev_class *class, +struct sysdev_class_attribute *attr, +const char *buf,size_t count) +{ +long nid; +int ret; + + ret =strict_strtol (buf,0 , &nid);
drivers/base/node.c void unregister_one_node(int nid) · line 556
Quoted 10 lines
drivers/base/node.c +10
-0+if (ret == -EINVAL ) +return ret; + + ret =hotadd_hidden_nodes (nid); +if (!ret) +return count; +else + return -EIO; +} +#endif
How about this?
err = strict_strtol(buf, 0, &nid);
if (err < 0)
return err;
err = hotadd_hidden_nodes(nid);
if (err < 0)
return err;
return count;Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 34 lines · Wu Fengguang
Quoted 11 lines
Pls. replace it with following code:
drivers/base/node.c +9
-0+#ifdef CONFIG_NODE_HOTPLUG_EMU +static ssize_t store_nodes_probe (struct sysdev_class *class, +struct sysdev_class_attribute *attr, +const char *buf,size_t count) +{ +long nid; +int ret; + + ret =strict_strtol (buf,0 , &nid);drivers/base/node.c void unregister_one_node(int nid) · line 556
Quoted 10 lines
drivers/base/node.c +10
-0+if (ret == -EINVAL ) +return ret; + + ret =hotadd_hidden_nodes (nid); +if (!ret) +return count; +else + return -EIO; +} +#endif How about this?
err = strict_strtol(buf, 0, &nid); if (err < 0) return err; err = hotadd_hidden_nodes(nid); if (err < 0) return err; return count;
Quoted 2 lines
Thanks,
Fengguang
Both seems good.
Haicheng's code is prefered, since it was already reviewed.
Signature · 8 lines
Thanks & Regards, Shaohui -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · Wu Fengguang
Quoted 11 lines
Pls. replace it with following code:
drivers/base/node.c +9
-0+#ifdef CONFIG_NODE_HOTPLUG_EMU +static ssize_t store_nodes_probe (struct sysdev_class *class, +struct sysdev_class_attribute *attr, +const char *buf,size_t count) +{ +long nid; +int ret; + + ret =strict_strtol (buf,0 , &nid);drivers/base/node.c void unregister_one_node(int nid) · line 556
Quoted 10 lines
drivers/base/node.c +10
-0+if (ret == -EINVAL ) +return ret; + + ret =hotadd_hidden_nodes (nid); +if (!ret) +return count; +else + return -EIO; +} +#endif How about this?
Quoted 3 lines
err = strict_strtol(buf, 0, &nid);
if (err < 0)
return err;other negative value would be odd here.
Quoted 3 lines
err = hotadd_hidden_nodes(nid);
if (err < 0)
return err;hotadd_hidden_nodes could return -EEXIST, which is also odd here, right?
Quoted 4 lines
return count;Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 25 lines · Haicheng Li
Quoted 24 lines · Wu Fengguang
Quoted 11 lines
Pls. replace it with following code:
drivers/base/node.c +9
-0+#ifdef CONFIG_NODE_HOTPLUG_EMU +static ssize_t store_nodes_probe (struct sysdev_class *class, +struct sysdev_class_attribute *attr, +const char *buf,size_t count) +{ +long nid; +int ret; + + ret =strict_strtol (buf,0 , &nid);drivers/base/node.c void unregister_one_node(int nid) · line 556
Quoted 10 lines
drivers/base/node.c +10
-0+if (ret == -EINVAL ) +return ret; + + ret =hotadd_hidden_nodes (nid); +if (!ret) +return count; +else + return -EIO; +} +#endif How about this?
Quoted 5 lines
Quoted 3 lines
err = strict_strtol(buf, 0, &nid); if (err < 0) return err;other negative value would be odd here.
Yes, strict_strtoul() will be better.
Quoted 5 lines
Quoted 3 lines
err = hotadd_hidden_nodes(nid); if (err < 0) return err;
hotadd_hidden_nodescould return-EEXIST, which is also odd here, right?
Why not? write(2) says "Other errors may occur, depending on the object connected to fd."
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi,
Quoted 25 lines · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeSigned-off-by Haicheng Li <[email protected]>Signed-off-by Shaohui Zheng <[email protected]>---
arch/x86/Kconfig +4
-0@@ -1214,6 +1214,20 @@ config NUMA_EMU into virtual nodes when booted with "numa=fake=N", where N is the number of nodes. This is only useful for debugging. +config NUMA_HOTPLUG_EMU + bool "NUMA hotplug emulator" + depends on X86_64 && NUMA && HOTPLUG + ---help---
arch/x86/Kconfig config NUMA_EMU · line 1230
Quoted 10 lines
arch/x86/Kconfig +10
-0+ +config NODE_HOTPLUG_EMU + bool "Node hotplug emulation" + depends on NUMA_HOTPLUG_EMU && MEMORY_HOTPLUG + ---help--- + Enable Node hotplug emulation. The machine will be setup with + hidden virtual nodes when booted with "numa=hide=N*size", where + N is the number of hidden nodes, size is the memory size per + hidden node. This is only useful for debugging. +
The above dependencies do not work as expected. I could configure NUMA_HOTPLUG_EMU & NODE_HOTPLUG_EMU without having MEMORY_HOTPLUG turned on. By pushing the above definition below SPARSEMEM and memory hot add and remove, the dependencies could be sorted out.
Signature · 5 lines
Regards, Ankita Garg ([email protected]) Linux Technology Center IBM India Systems & Technology Labs, Bangalore, India
Quoted 35 lines · Ankita Garg
Hi,
Quoted 32 lines · Shaohui Zheng
Userland interface to hotplug-add fake offlined nodes.
Add a sysfs entry "probe" under
/sys/devices/system/node/:- to show all fake offlined nodes:
$ cat /sys/devices/system/node/probe- to hotadd a fake offlined node, e.g. nodeid is N:
$ echo N > /sys/devices/system/node/probeSigned-off-by Haicheng Li <[email protected]>Signed-off-by Shaohui Zheng <[email protected]>---
arch/x86/Kconfig +11
-0@@ -1214,6 +1214,20 @@ config NUMA_EMU into virtual nodes when booted with "numa=fake=N", where N is the number of nodes. This is only useful for debugging. +config NUMA_HOTPLUG_EMU + bool "NUMA hotplug emulator" + depends on X86_64 && NUMA && HOTPLUG + ---help--- + +config NODE_HOTPLUG_EMU + bool "Node hotplug emulation" + depends on NUMA_HOTPLUG_EMU && MEMORY_HOTPLUG + ---help--- + Enable Node hotplug emulation. The machine will be setup with + hidden virtual nodes when booted with "numa=hide=N*size", where
Quoted 8 lines
arch/x86/Kconfig config NUMA_EMU · line 1230
Quoted 3 lines
arch/x86/Kconfig +3
-0+ N is the number of hidden nodes, size is the memory size per + hidden node. This is only useful for debugging. +The above dependencies do not work as expected. I could configure
NUMA_HOTPLUG_EMU&NODE_HOTPLUG_EMUwithout havingMEMORY_HOTPLUGturned on. By pushing the above definition below SPARSEMEM and memory hot add and remove, the dependencies could be sorted out.
Ankita,
The emulation code was tested by many times, but we did not try each
combination for the new options, good catching.
We will includes your suggestion in the formal patch. thanks so much.
Quoted 7 lines
Signature · 6 lines
-- Regards, Ankita Garg ([email protected]) Linux Technology Center IBM India Systems & Technology Labs, Bangalore, India
Signature · 8 lines
Thanks & Regards, Shaohui -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>