linux/vnuma: vNUMA PV guest support introduction
Subsystems (4)
v2 cover linux/vnuma: vNUMA PV guest support introduction
This patchset introduces vNUMA for PV domU guest.
Enables PV guest to discover NUMA topology provided by Xen
and initializes NUMA topology on boot. XENMEM subop hypercall
is used to retreive information from Xen. Xen provides number
of NUMA nodes, memory regions (start and end pfn) constructed
based on e820 domU map, distance table and cpu to node map. i xen_numa_init is called to setup NUMA related structures.
To enable this mechanism, kernel should be compiled as PV guest
with CONFIG_NUMA=y and Xen should support vNUMA functionality
(patchset http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg01337.html ).
Elena Ufimtseva (2):
vNUMA topology support for PV domu guest
Enables NUMA for PV vNUMA-enabled domu guest.
arch/x86/include/asm/xen/vnuma.h | 12 +++++ arch/x86/mm/numa.c | 5 ++ arch/x86/xen/Makefile | 2 +- arch/x86/xen/setup.c | 6 ++- arch/x86/xen/vnuma.c | 94 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 27 +++++++++++ 6 files changed, 144 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/xen/vnuma.h create mode 100644 arch/x86/xen/vnuma.c
Signature · 1 line
1.7.10.4
RFC PATCH v2 1/2 linux/vnuma: vNUMA for PV domu guest
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure vnuma_topology_info where start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;
Changes since v1:
- moved the test for xen_pv_domain() into xen_numa_init;
- replaced memory block search/allocation by single memblock_alloc;
- moved xen_numa_init to vnuma.c from enlighten.c;
- moved memblock structure to public interface memory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;
TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.11 6e4664525b1db28f8c4e1130957f70a94c19213e with boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.
--- arch/x86/include/asm/xen/vnuma.h | 12 +++++ arch/x86/mm/numa.c | 5 +++ arch/x86/xen/Makefile | 2 +- arch/x86/xen/vnuma.c | 92 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 27 +++++++++++ 5 files changed, 137 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/xen/vnuma.h create mode 100644 arch/x86/xen/vnuma.c
arch/x86/include/asm/xen/vnuma.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef _ASM_X86_VNUMA_H +#define _ASM_X86_VNUMA_H + +#ifdef CONFIG_XEN +int xen_vnuma_support (void ); +int xen_numa_init (void ); +#else +int xen_vnuma_support (void ) {return 0 ; }; +int xen_numa_init (void ) {}; +#endif + +#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +5 -0
@@ -19,6 +19,7 @@#include <asm/amd_nb.h> #include "numa_internal.h" +#include "asm/xen/vnuma.h" int __initdata numa_off;nodemask_t numa_nodes_parsed__initdata ;
@@ -621,6 +622,10 @@ static int __init dummy_numa_init(void)void __init x86_numa_init (void ) {if (!numa_off) { +#ifdef CONFIG_XEN +if (!numa_init (xen_numa_init)) +return ; +#endif #ifdef CONFIG_X86_NUMAQ if (!numa_init (numaq_numa_init))return ;
arch/x86/xen/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o
arch/x86/xen/vnuma.c new file +92 -0
92 lines not on this page · raw message
include/xen/interface/memory.h +27 -0
@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { };DEFINE_GUEST_HANDLE_STRUCT (xen_remove_from_physmap); +/* vNUMA structures */ +struct vnuma_memarea { +uint64_t start, end; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_memarea); + +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */ +uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad; +/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen +* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map +*/ +GUEST_HANDLE (vnuma_memarea) vmemarea; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */
Signature · 1 line
1.7.10.4
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 31 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure
vnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;Changes since v1:
- moved the test forxen_pv_domain()intoxen_numa_init;
- replaced memory block search/allocation by singlememblock_alloc;
- movedxen_numa_inittovnuma.cfromenlighten.c;
- moved memblock structure to public interfacememory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.116e4664525b1db28f8c4e1130957f70a94c19213ewith boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.
arch/x86/mm/numa.c line 19
Quoted 1 line
arch/x86/mm/numa.c +0
-0@@ -19,6 +19,7 @@#include <asm/amd_nb.h>
#include <asm/xen/vnuma.h> here...
arch/x86/mm/numa.c line 22
Quoted 2 lines
arch/x86/mm/numa.c +1
-0#include "numa_internal.h" +#include "asm/xen/vnuma.h"
... not here.
Quoted 7 lines
arch/x86/xen/vnuma.c new file +4
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h>
arch/x86/xen/vnuma.c line 14
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ; +int xen_vnuma_support () +{ +return xen_vnuma_init; +}
I'm not sure how this and the usage in the next patch actually work. xen_vnuma_init is only set after the test of numa_off prior to calling xen_numa_init() which will set xen_vnuma_init.
Quoted 40 lines
arch/x86/xen/vnuma.c new file +40
-0+int __init xen_numa_init (void ) +{ +int rc; +unsigned int i, j, cpu, idx, pcpus; +u64 phys, physd, physc; +unsigned int *vdistance, *cpu_to_node; +unsigned long mem_size, dist_size, cpu_to_node_size; +struct vnuma_memarea *varea; + +struct vnuma_topology_info numa_topo = { + .domid =DOMID_SELF + }; + rc = -EINVAL ; +if (!xen_pv_domain ()) +return rc; + pcpus =num_possible_cpus (); + mem_size = pcpus *sizeof (struct vnuma_memarea ); + dist_size = pcpus * pcpus *sizeof (*numa_topo.vdistance); + cpu_to_node_size = pcpus *sizeof (*numa_topo.cpu_to_node); + phys =memblock_alloc (mem_size,PAGE_SIZE ); + physd =memblock_alloc (dist_size,PAGE_SIZE ); + physc =memblock_alloc (cpu_to_node_size,PAGE_SIZE ); +if (!phys || !physc || !physd) + goto vnumaout; + varea =__va (phys); + vdistance =__va (physd); + cpu_to_node =__va (physc); +set_xen_guest_handle (numa_topo.vmemarea, varea); +set_xen_guest_handle (numa_topo.vdistance, vdistance); +set_xen_guest_handle (numa_topo.cpu_to_node, cpu_to_node); + rc =HYPERVISOR_memory_op (XENMEM_get_vnuma_info, &numa_topo); +if (rc <0 ) + goto vnumaout; + rc = -EINVAL ; +if (numa_topo.nr_nodes ==0 ) { +/* will pass to dummy_numa_init */ + goto vnumaout; + } +if (numa_topo.nr_nodes >num_possible_cpus ()) { +pr_debug ("vNUMA: Node without cpu is not supported in this version.\n" );
arch/x86/xen/vnuma.c line 64
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+ goto vnumaout; + } +/* +* NUMA nodes memory ranges are in pfns, constructed and +* aligned based on e820 ram domain map +*/ +for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end)) +/* pass to numa_dummy_init */ + goto vnumaout;
If there's a failure here, numa may be partially setup. Do you need to undo any of the bits that have already setup?
Quoted 9 lines
arch/x86/xen/vnuma.c new file +9
-0+node_set (i, numa_nodes_parsed); + } +setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu]; +cpumask_set_cpu (cpu, node_to_cpumask_map[cpu_to_node[cpu]]);
arch/x86/xen/vnuma.c line 83
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+ } +for (i =0 ; i < numa_topo.nr_nodes; i++) { +for (j =0 ; j < numa_topo.nr_nodes; j++) { + idx = (j * numa_topo.nr_nodes) + i; +numa_set_distance (i, j, *(vdistance + idx)); + } + } + rc =0 ; + xen_vnuma_init =1 ; +vnumaout:
Call this label "out".
arch/x86/xen/vnuma.c line 90
Quoted 7 lines
arch/x86/xen/vnuma.c new file +7
-0+if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc;
If you return an error, x86_numa_init() will try to call setup for other NUMA system. Consider calling numa_dummy_init() directly instead and then returning success.
arch/x86/xen/vnuma.c line 91
Quoted 1 line
arch/x86/xen/vnuma.c new file +1
-0+}
Please use blank lines to space the logical bits of this function out some more.
Quoted 11 lines
arch/x86/xen/vnuma.c new file +1
-0+#endif include/xen/interface/memory.h +2
-0@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { };DEFINE_GUEST_HANDLE_STRUCT (xen_remove_from_physmap); +/* vNUMA structures */ +struct vnuma_memarea {
include/xen/interface/memory.h struct xen_remove_from_physmap { · line 277
Quoted 10 lines
include/xen/interface/memory.h +10
-0+uint64_t start, end; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_memarea); + +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */ +uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad;
Is this _pad intended to make this structure uniform across 32-bit and 64-bit guests? The following GUEST_HANDLES() have different sizes on 32- and 64-bit guests.
Quoted 6 lines
include/xen/interface/memory.h +6
-0+/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen
include/xen/interface/memory.h struct xen_remove_from_physmap { · line 293
Quoted 10 lines
include/xen/interface/memory.h +9
-0+* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map +*/ + GUEST_HANDLE(vnuma_memarea) vmemarea; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */
David
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 54 lines · David Vrabel
Quoted 31 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure
vnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;Changes since v1:
- moved the test forxen_pv_domain()intoxen_numa_init;
- replaced memory block search/allocation by singlememblock_alloc;
- movedxen_numa_inittovnuma.cfromenlighten.c;
- moved memblock structure to public interfacememory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.116e4664525b1db28f8c4e1130957f70a94c19213ewith boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.arch/x86/mm/numa.c line 19
Quoted 1 line
arch/x86/mm/numa.c +0
-0@@ -19,6 +19,7 @@#include <asm/amd_nb.h> #include <
asm/xen/vnuma.h> here...arch/x86/mm/numa.c line 22
Quoted 2 lines
arch/x86/mm/numa.c +1
-0#include "numa_internal.h" +#include "asm/xen/vnuma.h" ... not here.
Quoted 13 lines
arch/x86/xen/vnuma.c new file +10
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ;
Quoted 8 lines
arch/x86/xen/vnuma.c line 14
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+int xen_vnuma_support () +{ +return xen_vnuma_init; +}I'm not sure how this and the usage in the next patch actually work.
xen_vnuma_initis only set after the test ofnuma_offprior to callingxen_numa_init()which will setxen_vnuma_init.
David, its obscure and naming is not self explanatory.. Will fix it. But the idea was to make sure
that NUMA can be safely turned on (for domu domain and if xen_numa_init call was sucessfull).
Quoted 46 lines
Quoted 46 lines
arch/x86/xen/vnuma.c new file +45
-0+int __init xen_numa_init (void ) +{ +int rc; +unsigned int i, j, cpu, idx, pcpus; +u64 phys, physd, physc; +unsigned int *vdistance, *cpu_to_node; +unsigned long mem_size, dist_size, cpu_to_node_size; +struct vnuma_memarea *varea; + +struct vnuma_topology_info numa_topo = { + .domid =DOMID_SELF + }; + rc = -EINVAL ; +if (!xen_pv_domain ()) +return rc; + pcpus =num_possible_cpus (); + mem_size = pcpus *sizeof (struct vnuma_memarea ); + dist_size = pcpus * pcpus *sizeof (*numa_topo.vdistance); + cpu_to_node_size = pcpus *sizeof (*numa_topo.cpu_to_node); + phys =memblock_alloc (mem_size,PAGE_SIZE ); + physd =memblock_alloc (dist_size,PAGE_SIZE ); + physc =memblock_alloc (cpu_to_node_size,PAGE_SIZE ); +if (!phys || !physc || !physd) + goto vnumaout; + varea =__va (phys); + vdistance =__va (physd); + cpu_to_node =__va (physc); +set_xen_guest_handle (numa_topo.vmemarea, varea); +set_xen_guest_handle (numa_topo.vdistance, vdistance); +set_xen_guest_handle (numa_topo.cpu_to_node, cpu_to_node); + rc =HYPERVISOR_memory_op (XENMEM_get_vnuma_info, &numa_topo); +if (rc <0 ) + goto vnumaout; + rc = -EINVAL ; +if (numa_topo.nr_nodes ==0 ) { +/* will pass to dummy_numa_init */ + goto vnumaout; + } +if (numa_topo.nr_nodes >num_possible_cpus ()) { +pr_debug ("vNUMA: Node without cpu is not supported in this version.\n" ); + goto vnumaout; + } +/* +* NUMA nodes memory ranges are in pfns, constructed and +* aligned based on e820 ram domain map
Quoted 8 lines
arch/x86/xen/vnuma.c line 64
Quoted 5 lines
arch/x86/xen/vnuma.c new file +5
-0+*/ +for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end)) +/* pass to numa_dummy_init */ + goto vnumaout;If there's a failure here, numa may be partially setup. Do you need to undo any of the bits that have already setup?
Konrad asked me the same and I was under impression it is safe. But that was based on assumptions
what I would rather avoid making. I will add bits to unset numa in case of failure.
Quoted 31 lines
Quoted 10 lines
arch/x86/xen/vnuma.c new file +9
-0+node_set (i, numa_nodes_parsed); + } +setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu]; +cpumask_set_cpu (cpu, node_to_cpumask_map[cpu_to_node[cpu]]);arch/x86/xen/vnuma.c line 83
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+ } +for (i =0 ; i < numa_topo.nr_nodes; i++) { +for (j =0 ; j < numa_topo.nr_nodes; j++) { + idx = (j * numa_topo.nr_nodes) + i; +numa_set_distance (i, j, *(vdistance + idx)); + } + } + rc =0 ; + xen_vnuma_init =1 ; +vnumaout:Call this label "out".
arch/x86/xen/vnuma.c line 90
Quoted 7 lines
arch/x86/xen/vnuma.c new file +7
-0+if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc;
Quoted 3 lines
If you return an error,
x86_numa_init()will try to call setup for other NUMA system. Consider callingnuma_dummy_init()directly instead and then returning success.
David, isnt it what x86_numa_init() supposed to do? try every *numa_init until one succeed?
Will adding excplicit call to dummy numa from xen_init_numa brake this logic?
Quoted 42 lines
arch/x86/xen/vnuma.c line 91
Quoted 2 lines
arch/x86/xen/vnuma.c new file +1
-0+}Please use blank lines to space the logical bits of this function out some more.
Quoted 11 lines
arch/x86/xen/vnuma.c new file +1
-0+#endif include/xen/interface/memory.h +2
-0@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { };DEFINE_GUEST_HANDLE_STRUCT (xen_remove_from_physmap); +/* vNUMA structures */ +struct vnuma_memarea {include/xen/interface/memory.h struct xen_remove_from_physmap { · line 277
Quoted 10 lines
include/xen/interface/memory.h +10
-0+uint64_t start, end; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_memarea); + +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */ +uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad;Is this
_padintended to make this structure uniform across 32-bit and 64-bit guests? The followingGUEST_HANDLES()have different sizes on 32- and 64-bit guests.Quoted 10 lines
include/xen/interface/memory.h +10
-0+/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen +* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map +*/ +GUEST_HANDLE (vnuma_memarea) vmemarea;
Quoted 8 lines
include/xen/interface/memory.h struct xen_remove_from_physmap { · line 293
Quoted 6 lines
include/xen/interface/memory.h +5
-0+}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */ David
Thank you, will work on rest of code.
Signature · 1 line
Elena
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 24 lines · Elena Ufimtseva
Quoted 13 lines · David Vrabel
Quoted 13 lines
arch/x86/xen/vnuma.c new file +10
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ;Quoted 8 lines
arch/x86/xen/vnuma.c line 14
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+int xen_vnuma_support () +{ +return xen_vnuma_init; +}I'm not sure how this and the usage in the next patch actually work.
xen_vnuma_initis only set after the test ofnuma_offprior to callingxen_numa_init()which will setxen_vnuma_init.
Quoted 5 lines
David, its obscure and naming is not self explanatory.. Will fix it. But the idea was to make sure
that NUMA can be safely turned on (for domu domain and ifxen_numa_initcall was sucessfull).
I think what David meant was to actually issue one/the hypercall, perhaps with factitious and known to be wrong parameters, and check the return value.
If that is EINVAL (or anything different than ENOSYS), then it means that the hypercall is supported (i.e., we're running on a version of the Xen hypervisor that has it implemented), and we can go ahead.
If that is ENOSYS, then it means there is no such hypercall, in which case, the sooner we give up, the better. :-)
David, is this what you meant? If yes, Elena, does that make sense to you? Do you think you can make the code look like this?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
arch/x86/xen/vnuma.c new file +2
-0+int __init xen_numa_init (void ) +{
[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
arch/x86/xen/vnuma.c new file +7
-0+if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc;
Quoted 8 lines
Quoted 3 lines
If you return an error,
x86_numa_init()will try to call setup for other NUMA system. Consider callingnuma_dummy_init()directly instead and then returning success.David, isnt it what
x86_numa_init()supposed to do? try every*numa_inituntil one succeed?
Will adding excplicit call to dummy numa fromxen_init_numabrake this logic?
I was about to replay exactly the same (as Elena) but, on a second thought, I think David has a point here.
After all, what's the point in calling stuff line acpi_numa_init(), etc., we already know they're going to fail! So, yes, I think his idea worth a try...
Also, bear in mind what Konrad said about a call to one of the functions in x86_numa_init() blowing up if run as Dom0 on some AMD chips. If we return 'success' here, that will never happen: we either setup a proper vNUMA topology or go straight to dummy_*, no more room for weird stuff to happen... It actually sounds pretty cool, doesn't it? :-P
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 38 lines · Dario Faggioli
Quoted 24 lines · Elena Ufimtseva
Quoted 13 lines · David Vrabel
Quoted 13 lines
arch/x86/xen/vnuma.c new file +10
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ;Quoted 8 lines
arch/x86/xen/vnuma.c line 14
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+int xen_vnuma_support () +{ +return xen_vnuma_init; +}I'm not sure how this and the usage in the next patch actually work.
xen_vnuma_initis only set after the test ofnuma_offprior to callingxen_numa_init()which will setxen_vnuma_init.Quoted 5 lines
David, its obscure and naming is not self explanatory.. Will fix it. But the idea was to make sure
that NUMA can be safely turned on (for domu domain and ifxen_numa_initcall was sucessfull).I think what David meant was to actually issue one/the hypercall, perhaps with factitious and known to be wrong parameters, and check the return value.
If that is EINVAL (or anything different than ENOSYS), then it means that the hypercall is supported (i.e., we're running on a version of the Xen hypervisor that has it implemented), and we can go ahead.
Quoted 5 lines
If that is ENOSYS, then it means there is no such hypercall, in which case, the sooner we give up, the better. :-)
David, is this what you meant? If yes, Elena, does that make sense to you? Do you think you can make the code look like this?
Sure, in my todo list )
Quoted 27 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
arch/x86/xen/vnuma.c new file +2
-0+int __init xen_numa_init (void ) +{[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
arch/x86/xen/vnuma.c new file +7
-0+if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc;Quoted 8 lines
Quoted 3 lines
If you return an error,
x86_numa_init()will try to call setup for other NUMA system. Consider callingnuma_dummy_init()directly instead and then returning success.David, isnt it what
x86_numa_init()supposed to do? try every*numa_inituntil one succeed?
Will adding excplicit call to dummy numa fromxen_init_numabrake this logic?I was about to replay exactly the same (as Elena) but, on a second thought, I think David has a point here.
After all, what's the point in calling stuff line
acpi_numa_init(), etc., we already know they're going to fail! So, yes, I think his idea worth a try...
Quoted 5 lines
Also, bear in mind what Konrad said about a call to one of the functions in
x86_numa_init()blowing up if run as Dom0 on some AMD chips. If we return 'success' here, that will never happen: we either setup a proper vNUMA topology or go straight to dummy_*, no more room for weird stuff to happen... It actually sounds pretty cool, doesn't it? :-P
I agree and it does it make sense. I did not know how far I can go with modifying
original linux logic.
Quoted 10 lines
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
In reply to Elena UfimtsevaQuoted 64 lines · Elena Ufimtseva
Quoted 54 lines · David Vrabel
Quoted 31 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure
vnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;Changes since v1:
- moved the test forxen_pv_domain()intoxen_numa_init;
- replaced memory block search/allocation by singlememblock_alloc;
- movedxen_numa_inittovnuma.cfromenlighten.c;
- moved memblock structure to public interfacememory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.116e4664525b1db28f8c4e1130957f70a94c19213ewith boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.arch/x86/mm/numa.c line 19
Quoted 1 line
arch/x86/mm/numa.c +0
-0@@ -19,6 +19,7 @@#include <asm/amd_nb.h> #include <
asm/xen/vnuma.h> here...arch/x86/mm/numa.c line 22
Quoted 2 lines
arch/x86/mm/numa.c +1
-0#include "numa_internal.h" +#include "asm/xen/vnuma.h" ... not here.
Quoted 13 lines
arch/x86/xen/vnuma.c new file +10
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ;Quoted 8 lines
arch/x86/xen/vnuma.c line 14
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+int xen_vnuma_support () +{ +return xen_vnuma_init; +}I'm not sure how this and the usage in the next patch actually work.
xen_vnuma_initis only set after the test ofnuma_offprior to callingxen_numa_init()which will setxen_vnuma_init.
Quoted 4 lines
David, its obscure and naming is not self explanatory.. Will fix it. But the idea was to make sure
that NUMA can be safely turned on (for domu domain and ifxen_numa_initcall was sucessfull).
I understand what it's for, I just don't see how it works.
The code path looks like (I think):
xen_vnuma_init = 0;
if (!xen_vnuma_init)
numa_off = 1
if (!numa_off)
xen_numa_init()
However, if you go with the idea of calling dummy init in xen_num_init()'s error path you don't need this.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end)) +/* pass to numa_dummy_init */ + goto vnumaout;
Quoted 7 lines
Quoted 2 lines
If there's a failure here, numa may be partially setup. Do you need to undo any of the bits that have already setup?
Konrad asked me the same and I was under impression it is safe. But that was based on assumptions
what I would rather avoid making. I will add bits to unset numa in case of failure.
I looked at the other uses of this and none of them undo on failure so I think it is fine as is.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
arch/x86/xen/vnuma.c new file +7
-0+if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc;
Quoted 7 lines
Quoted 3 lines
If you return an error,
x86_numa_init()will try to call setup for other NUMA system. Consider callingnuma_dummy_init()directly instead and then returning success.David, isnt it what
x86_numa_init()supposed to do? try every*numa_inituntil one succeed?
Will adding excplicit call to dummy numa fromxen_init_numabrake this logic?
Yes, but if we know we're a PV guest we do not want to try any other one, we want to fallback to the dummy init immediately.
David
Boris Ostrovsky
reviewerRe: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
In reply to Elena UfimtsevaQuoted 45 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure
vnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;Changes since v1:
- moved the test forxen_pv_domain()intoxen_numa_init;
- replaced memory block search/allocation by singlememblock_alloc;
- movedxen_numa_inittovnuma.cfromenlighten.c;
- moved memblock structure to public interfacememory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.116e4664525b1db28f8c4e1130957f70a94c19213ewith boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.Signed-off-by Elena Ufimtseva <[email protected]>--- arch/x86/include/asm/xen/vnuma.h | 12 +++++ arch/x86/mm/numa.c | 5 +++ arch/x86/xen/Makefile | 2 +- arch/x86/xen/vnuma.c | 92 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 27 +++++++++++ 5 files changed, 137 insertions(+), 1 deletion(-)create mode 100644
arch/x86/include/asm/xen/vnuma.hcreate mode 100644arch/x86/xen/vnuma.carch/x86/include/asm/xen/vnuma.h new file +0
-0
arch/x86/include/asm/xen/vnuma.h line 9
Quoted 9 lines
arch/x86/include/asm/xen/vnuma.h new file +9
-0@@ -0,0 +1,12 @@ +#ifndef _ASM_X86_VNUMA_H +#define _ASM_X86_VNUMA_H + +#ifdef CONFIG_XEN +int xen_vnuma_support (void ); +int xen_numa_init (void ); +#else +int xen_vnuma_support (void ) {return 0 ; }; +int xen_numa_init (void ) {};
This should return -EINVAL. Or perhaps you can add
#ifdef CONFIG_XEN
#include "asm/xen/vnuma.h"
#endif
in numa.c and not bother with ifdef here.
Quoted 55 lines
arch/x86/include/asm/xen/vnuma.h new file +3
-0+#endif + +#endif /* _ASM_X86_VNUMA_H */ arch/x86/mm/numa.c +5
-0@@ -19,6 +19,7 @@#include <asm/amd_nb.h> #include "numa_internal.h" +#include "asm/xen/vnuma.h" int __initdata numa_off;nodemask_t numa_nodes_parsed__initdata ;@@ -621,6 +622,10 @@ static int __init dummy_numa_init(void)void __init x86_numa_init (void ) {if (!numa_off) { +#ifdef CONFIG_XEN +if (!numa_init (xen_numa_init)) +return ; +#endif #ifdef CONFIG_X86_NUMAQ if (!numa_init (numaq_numa_init))return ;arch/x86/xen/Makefile +1
-1@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.oarch/x86/xen/vnuma.c new file +10
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ;
arch/x86/xen/vnuma.c line 20
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+int xen_vnuma_support () +{ +return xen_vnuma_init; +} +int __init xen_numa_init (void ) +{ +int rc; +unsigned int i, j, cpu, idx, pcpus; +u64 phys, physd, physc; +unsigned int *vdistance, *cpu_to_node;
cpu_to_node may not be a particularly good name as there is a macro with the same name in topology.h
arch/x86/xen/vnuma.c line 29
Quoted 9 lines
arch/x86/xen/vnuma.c new file +9
-0+unsigned long mem_size, dist_size, cpu_to_node_size; +struct vnuma_memarea *varea; + +struct vnuma_topology_info numa_topo = { + .domid =DOMID_SELF + }; + rc = -EINVAL ; +if (!xen_pv_domain ()) +return rc;
No need to set rc here, just return -EINVAL;
And please add spaces between lines to separate logical blocks a little.
Quoted 33 lines
arch/x86/xen/vnuma.c new file +33
-0+ pcpus =num_possible_cpus (); + mem_size = pcpus *sizeof (struct vnuma_memarea ); + dist_size = pcpus * pcpus *sizeof (*numa_topo.vdistance); + cpu_to_node_size = pcpus *sizeof (*numa_topo.cpu_to_node); + phys =memblock_alloc (mem_size,PAGE_SIZE ); + physd =memblock_alloc (dist_size,PAGE_SIZE ); + physc =memblock_alloc (cpu_to_node_size,PAGE_SIZE ); +if (!phys || !physc || !physd) + goto vnumaout; + varea =__va (phys); + vdistance =__va (physd); + cpu_to_node =__va (physc); +set_xen_guest_handle (numa_topo.vmemarea, varea); +set_xen_guest_handle (numa_topo.vdistance, vdistance); +set_xen_guest_handle (numa_topo.cpu_to_node, cpu_to_node); + rc =HYPERVISOR_memory_op (XENMEM_get_vnuma_info, &numa_topo); +if (rc <0 ) + goto vnumaout; + rc = -EINVAL ; +if (numa_topo.nr_nodes ==0 ) { +/* will pass to dummy_numa_init */ + goto vnumaout; + } +if (numa_topo.nr_nodes >num_possible_cpus ()) { +pr_debug ("vNUMA: Node without cpu is not supported in this version.\n" ); + goto vnumaout; + } +/* +* NUMA nodes memory ranges are in pfns, constructed and +* aligned based on e820 ram domain map +*/ +for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end))
arch/x86/xen/vnuma.c line 72
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+/* pass to numa_dummy_init */ + goto vnumaout; +node_set (i, numa_nodes_parsed); + } +setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu];
Isn't this what set_apicid_to_node() above will do?
-boris
Quoted 56 lines
arch/x86/xen/vnuma.c new file +20
-0+ cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node[cpu]]); + } +for (i =0 ; i < numa_topo.nr_nodes; i++) { +for (j =0 ; j < numa_topo.nr_nodes; j++) { + idx = (j * numa_topo.nr_nodes) + i; +numa_set_distance (i, j, *(vdistance + idx)); + } + } + rc =0 ; + xen_vnuma_init =1 ; +vnumaout: +if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc; +} +#endif include/xen/interface/memory.h +27
-0@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { };DEFINE_GUEST_HANDLE_STRUCT (xen_remove_from_physmap); +/* vNUMA structures */ +struct vnuma_memarea { +uint64_t start, end; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_memarea); + +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */ +uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad; +/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen +* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map +*/ +GUEST_HANDLE (vnuma_memarea) vmemarea; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 128 lines · Boris Ostrovsky
Quoted 48 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall. Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file. Memory ranges are represented by structure
vnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
In case the received structure has errors, will fail to dummy numa init.
Requires XEN with applied patches from vnuma patchset;Changes since v1:
- moved the test forxen_pv_domain()intoxen_numa_init;
- replaced memory block search/allocation by singlememblock_alloc;
- movedxen_numa_inittovnuma.cfromenlighten.c;
- moved memblock structure to public interfacememory.h;
- specified signedness of vnuma topology structure members;
- removed excessive debug output;TODO:
- consider common interface for Dom0, HVM and PV guests to provide vNUMA topology;
- dynamic numa balancing at the time of this patch (kernel 3.116e4664525b1db28f8c4e1130957f70a94c19213ewith boot parameter numa_balancing=true that is such by default) crashes numa-enabled guest. Investigate further.Signed-off-by Elena Ufimtseva <[email protected]>--- arch/x86/include/asm/xen/vnuma.h | 12 +++++ arch/x86/mm/numa.c | 5 +++ arch/x86/xen/Makefile | 2 +- arch/x86/xen/vnuma.c | 92 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 27 +++++++++++ 5 files changed, 137 insertions(+), 1 deletion(-)create mode 100644
arch/x86/include/asm/xen/vnuma.hcreate mode 100644arch/x86/xen/vnuma.carch/x86/include/asm/xen/vnuma.h new file +0
-0arch/x86/include/asm/xen/vnuma.h line 9
Quoted 9 lines
arch/x86/include/asm/xen/vnuma.h new file +9
-0@@ -0,0 +1,12 @@ +#ifndef _ASM_X86_VNUMA_H +#define _ASM_X86_VNUMA_H + +#ifdef CONFIG_XEN +int xen_vnuma_support (void ); +int xen_numa_init (void ); +#else +int xen_vnuma_support (void ) {return 0 ; }; +int xen_numa_init (void ) {};This should return
-EINVAL. Or perhaps you can add#ifdef
CONFIG_XEN
#include "asm/xen/vnuma.h"
#endifin
numa.cand not bother with ifdef here.Quoted 57 lines
arch/x86/include/asm/xen/vnuma.h new file +3
-0+#endif + +#endif /* _ASM_X86_VNUMA_H */ arch/x86/mm/numa.c +5
-0@@ -19,6 +19,7 @@#include <asm/amd_nb.h> #include "numa_internal.h" +#include "asm/xen/vnuma.h" int __initdata numa_off;nodemask_t numa_nodes_parsed__initdata ;@@ -621,6 +622,10 @@ static int __init dummy_numa_init(void)void __init x86_numa_init (void ) {if (!numa_off) { +#ifdef CONFIG_XEN +if (!numa_init (xen_numa_init)) +return ; +#endif #ifdef CONFIG_X86_NUMAQ if (!numa_init (numaq_numa_init))return ;arch/x86/xen/Makefile +1
-1@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o diff --git a/arch/x86/xen/vnuma.c b/arch/x86/xen/vnuma.cnew file mode 100644
index 0000000..3c6c73farch/x86/xen/vnuma.c new file +16
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ; +int xen_vnuma_support () +{ +return xen_vnuma_init; +} +int __init xen_numa_init (void ) +{
Quoted 8 lines
arch/x86/xen/vnuma.c line 20
Quoted 4 lines
arch/x86/xen/vnuma.c new file +4
-0+int rc; +unsigned int i, j, cpu, idx, pcpus; +u64 phys, physd, physc; +unsigned int *vdistance, *cpu_to_node;
cpu_to_nodemay not be a particularly good name as there is a macro with the same name intopology.h
Sure, will fix.
Quoted 13 lines
arch/x86/xen/vnuma.c line 29
Quoted 11 lines
arch/x86/xen/vnuma.c new file +9
-0+unsigned long mem_size, dist_size, cpu_to_node_size; +struct vnuma_memarea *varea; + +struct vnuma_topology_info numa_topo = { + .domid =DOMID_SELF + }; + rc = -EINVAL ; +if (!xen_pv_domain ()) +return rc;
Quoted 3 lines
No need to set rc here, just return
-EINVAL;And please add spaces between lines to separate logical blocks a little.\
Ok.
Quoted 48 lines
Quoted 36 lines
arch/x86/xen/vnuma.c new file +25
-0+ pcpus =num_possible_cpus (); + mem_size = pcpus *sizeof (struct vnuma_memarea ); + dist_size = pcpus * pcpus *sizeof (*numa_topo.vdistance); + cpu_to_node_size = pcpus *sizeof (*numa_topo.cpu_to_node); + phys =memblock_alloc (mem_size,PAGE_SIZE ); + physd =memblock_alloc (dist_size,PAGE_SIZE ); + physc =memblock_alloc (cpu_to_node_size,PAGE_SIZE ); +if (!phys || !physc || !physd) + goto vnumaout; + varea =__va (phys); + vdistance =__va (physd); + cpu_to_node =__va (physc); +set_xen_guest_handle (numa_topo.vmemarea, varea); +set_xen_guest_handle (numa_topo.vdistance, vdistance); +set_xen_guest_handle (numa_topo.cpu_to_node, cpu_to_node); + rc =HYPERVISOR_memory_op (XENMEM_get_vnuma_info, &numa_topo); +if (rc <0 ) + goto vnumaout; + rc = -EINVAL ; +if (numa_topo.nr_nodes ==0 ) { +/* will pass to dummy_numa_init */ + goto vnumaout; + } +if (numa_topo.nr_nodes >num_possible_cpus ()) { + pr_debug("vNUMA: Node without cpu is not supported in thisversion.\n");
arch/x86/xen/vnuma.c new file +8
-0+ goto vnumaout; + } +/* +* NUMA nodes memory ranges are in pfns, constructed and +* aligned based on e820 ram domain map +*/ +for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end))arch/x86/xen/vnuma.c line 72
Quoted 10 lines
arch/x86/xen/vnuma.c new file +10
-0+/* pass to numa_dummy_init */ + goto vnumaout; +node_set (i, numa_nodes_parsed); + } +setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu];
Quoted 1 line
Isn't this what
set_apicid_to_node()above will do?
Yes, exactly the same ) will fix.
Quoted 63 lines
-boris
Quoted 57 lines
+ cpumask_set_cpu(cpu,
node_to_cpumask_map[cpu_to_node[cpu]]);arch/x86/xen/vnuma.c new file +19
-0+ } +for (i =0 ; i < numa_topo.nr_nodes; i++) { +for (j =0 ; j < numa_topo.nr_nodes; j++) { + idx = (j * numa_topo.nr_nodes) + i; +numa_set_distance (i, j, *(vdistance + idx)); + } + } + rc =0 ; + xen_vnuma_init =1 ; +vnumaout: +if (phys) +memblock_free (__pa (phys), mem_size); +if (physd) +memblock_free (__pa (physd), dist_size); +if (physc) +memblock_free (__pa (physc), cpu_to_node_size); +return rc; +} +#endif include/xen/interface/memory.h +26
-0@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { };DEFINE_GUEST_HANDLE_STRUCT (xen_remove_from_physmap); +/* vNUMA structures */ +struct vnuma_memarea { +uint64_t start, end; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_memarea); + +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */ +uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad; +/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen +* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map +*/ +GUEST_HANDLE (vnuma_memarea) vmemarea; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */
Signature · 1 line
Elena
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 4 lines · Elena Ufimtseva
On Tue, Sep 17, 2013 at 10:21 AM, Boris Ostrovsky
Quoted 2 lines
Quoted 2 lines
arch/x86/xen/vnuma.c new file +2
-0+int __init xen_numa_init (void ) +{
[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
arch/x86/xen/vnuma.c new file +6
-0+setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu];
Quoted 4 lines
Quoted 1 line
Isn't this what
set_apicid_to_node()above will do?Yes, exactly the same ) will fix.
I seem to recall that something strange was happening if we do not do this in this way (i.e., calling the same stuff twice, or something like that), isn't it so Elena?
If it is, please, explain that in a comment. However, it may well be possible that my recollection is wrong.. In which case, sorry for the noise.
Anyway, in general, and both for this series and for the Xen one, I think the code could use a little bit more of commenting (along with the breaking up in paragraphs, as many have already pointed out).
I know, I know, too much comments is also bad... Actually, finding the right balance between too few and too much is as much important as it is difficult! :-P
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
Quoted 14 lines · Dario Faggioli
Quoted 4 lines · Elena Ufimtseva
On Tue, Sep 17, 2013 at 10:21 AM, Boris Ostrovsky
Quoted 2 lines
Quoted 2 lines
arch/x86/xen/vnuma.c new file +2
-0+int __init xen_numa_init (void ) +{[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
arch/x86/xen/vnuma.c new file +6
-0+setup_nr_node_ids (); +/* Setting the cpu, apicid to node */ +for_each_cpu (cpu, cpu_possible_mask) { +set_apicid_to_node (cpu, cpu_to_node[cpu]); +numa_set_node (cpu, cpu_to_node[cpu]); + __apicid_to_node[cpu] = cpu_to_node[cpu];
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Isn't this what
set_apicid_to_node()above will do?Yes, exactly the same ) will fix.
I seem to recall that something strange was happening if we do not do this in this way (i.e., calling the same stuff twice, or something like that), isn't it so Elena?
I think its in the past as proper ordering made sense and that second call was removed :)
Quoted 11 lines
If it is, please, explain that in a comment. However, it may well be possible that my recollection is wrong.. In which case, sorry for the noise.
Anyway, in general, and both for this series and for the Xen one, I think the code could use a little bit more of commenting (along with the breaking up in paragraphs, as many have already pointed out).
I know, I know, too much comments is also bad... Actually, finding the right balance between too few and too much is as much important as it is difficult! :-P
I see that ) I will learn.
Quoted 9 lines
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Re: [PATCH RFC v2 1/2] linux/vnuma: vNUMA for PV domu guest
In reply to Elena UfimtsevaQuoted 3 lines · Elena Ufimtseva
Requests NUMA topology info from Xen by issuing subop hypercall.
Not such a big deal, but I think you can call it just an hypercall, and perhaps put it the name (or something like "XENMEM_get_vnuma_info" via "HYPERVISOR_memory_op()" ). What I'm after is that 'subop hcall' may be too much of a Xen-ism.
Quoted 4 lines
Initializes NUMA nodes, sets number of CPUs, distance table and NUMA nodes memory ranges during boot. vNUMA topology defined by user in VM config file.
Here (I mean in the changelog of this patch) I'd just say that the vNUMA topology is hosted in Xen. You can add a quick note about from where it come from but, (1), it pretty much always comes from the config file right now, but that may change in the future (when we introduce Dom0 vNUMA); (2) I don't think Linux cares much from where Xen has gotten the information. :-P
Quoted 6 lines
Memory
ranges are represented by structurevnuma_topology_infowhere start and end of memory area are defined in guests pfns numbers, constructed and aligned accordingly to e820 domain map.
Fine, and what does that mean for Linux? The fact that all is already aligned and respectful of the e820, I mena? I actually have a question about this, which I ask below, but whatever the answer is, I think a summary of that should go up here.
Oh, and BTW, as it is in the code, some blank lines separating a bit more the various paragraphs would be nice in the changelogs too.
Quoted 27 lines
Signed-off-by Elena Ufimtseva <[email protected]>---
arch/x86/include/asm/xen/vnuma.h +0
-0arch/x86/xen/vnuma.c new file +16
-0@@ -0,0 +1,92 @@ +#include <linux/err.h> +#include <linux/memblock.h> +#include <xen/interface/xen.h> +#include <xen/interface/memory.h> +#include <asm/xen/interface.h> +#include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> +#ifdef CONFIG_NUMA +/* Xen PV NUMA topology initialization */ +static unsigned int xen_vnuma_init =0 ; +int xen_vnuma_support () +{ +return xen_vnuma_init; +} +int __init xen_numa_init (void ) +{
[snip]
arch/x86/xen/vnuma.c line 66
Quoted 11 lines
arch/x86/xen/vnuma.c new file +10
-0+/* +* NUMA nodes memory ranges are in pfns, constructed and +* aligned based on e820 ram domain map +*/ +for (i =0 ; i < numa_topo.nr_nodes; i++) { +if (numa_add_memblk (i, varea[i].start, varea[i].end)) +/* pass to numa_dummy_init */ + goto vnumaout; +node_set (i, numa_nodes_parsed); + }
So, forgive me if my e820-fu is not black-belt level, here you go through nr_nodes elements of the varea[] array, and setup nr_nodes memblks, i.e., one for each node.
I take this like there only can be one memblk per node, which, foor instance, means no holes, is that accurate?
If yes, the "constructed and aligned based on e820" means basically just alignment, right?
More important, are we sure this is always ok, i.e., that we don't and won't ever need to take holes into account?
Quoted 10 lines
include/xen/interface/memory.h +4
-0@@ -263,4 +263,31 @@ struct xen_remove_from_physmap { +struct vnuma_topology_info { +/* OUT */ +domid_t domid; +/* IN */
include/xen/interface/memory.h struct xen_remove_from_physmap { · line 277
Quoted 10 lines
include/xen/interface/memory.h +10
-0+uint16_t nr_nodes;/* number of virtual numa nodes */ +uint32_t _pad; +/* distance table */ +GUEST_HANDLE (uint) vdistance; +/* cpu mapping to vnodes */ +GUEST_HANDLE (uint) cpu_to_node; +/* +* array of numa memory areas constructed by Xen +* where start and end are pfn numbers of the area +* Xen takes into account domains e820 map
^domain's ?
include/xen/interface/memory.h struct xen_remove_from_physmap { · line 293
Quoted 8 lines
include/xen/interface/memory.h +7
-0+*/ + GUEST_HANDLE(vnuma_memarea) vmemarea; +}; +DEFINE_GUEST_HANDLE_STRUCT (vnuma_topology_info); + +#define XENMEM_get_vnuma_info 25 +#endif /* __XEN_PUBLIC_MEMORY_H__ */
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
RFC PATCH v2 2/2 linux/vnuma: Enables NUMA for domu PV guest
In reply to Elena UfimtsevaAfter the NUMA topology was received from Xen,
enable NUMA during boot. Should have CONFIG_NUMA
enabled in kernel.
Changes since v1:
- added additional checks for PV guest and hypercall support before enablinf NUMA;
--- arch/x86/xen/setup.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/xen/setup.c +5 -1
@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> #include <xen/xen.h> #include <xen/page.h>
@@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ()) + numa_off =0 ; +else + numa_off =1 ;#endif }
Signature · 1 line
1.7.10.4
Re: [PATCH RFC v2 2/2] linux/vnuma: Enables NUMA for domu PV guest
Quoted 7 lines · Elena Ufimtseva
After the NUMA topology was received from Xen,
enable NUMA during boot. Should haveCONFIG_NUMA
enabled in kernel.Changes since v1:
- added additional checks for PV guest and hypercall support before enablinf NUMA;
As I said in response to the other patch, I don't think this does the right thing.
I think xen_vnuma_support() needs to try the vnuma hypercall and check it is successful.
Quoted 11 lines
arch/x86/xen/setup.c +1
-0@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> #include <xen/xen.h> #include <xen/page.h>
arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 587
Quoted 5 lines
arch/x86/xen/setup.c +1
-1@@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ())
I don't think there's a need to special case the initial domain here is there?
arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 592
Quoted 5 lines
arch/x86/xen/setup.c +3
-0+ numa_off =0 ; + else + numa_off =1 ;#endif }
David
Re: [PATCH RFC v2 2/2] linux/vnuma: Enables NUMA for domu PV guest
Quoted 9 lines · David Vrabel
Quoted 7 lines · Elena Ufimtseva
After the NUMA topology was received from Xen,
enable NUMA during boot. Should haveCONFIG_NUMA
enabled in kernel.Changes since v1:
- added additional checks for PV guest and hypercall support before enablinf NUMA;
Quoted 6 lines
As I said in response to the other patch, I don't think this does the right thing.
I think
xen_vnuma_support()needs to try the vnuma hypercall and check it is successful.
That can surely be done, I think. Elena?
Quoted 8 lines
Quoted 8 lines
arch/x86/xen/setup.c +1
-0@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h>
Quoted 12 lines
arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 587
Quoted 8 lines
arch/x86/xen/setup.c +1
-1#include <xen/xen.h> #include <xen/page.h> @@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ())I don't think there's a need to special case the initial domain here is there?
This is actually something that Konrad asked, since, apparently, there are AMD machines that just blows up, as Dom0, if this is on.
Therefore, since, right now, the status of Elena's work is "just DomUs", we figured this could be fine for now.
Of course, the final goal (whether for Elena or for someone else to pick it up) is to be able to enable vNUMA for Dom0 too, at which point we definitely will have to figure out a way to kill this special casing safely...
What do you think?
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH RFC v2 2/2] linux/vnuma: Enables NUMA for domu PV guest
Quoted 10 lines · Dario Faggioli
Quoted 9 lines · David Vrabel
Quoted 7 lines · Elena Ufimtseva
After the NUMA topology was received from Xen,
enable NUMA during boot. Should haveCONFIG_NUMA
enabled in kernel.Changes since v1:
- added additional checks for PV guest and hypercall support before enablinf NUMA;
Quoted 7 lines
Quoted 6 lines
As I said in response to the other patch, I don't think this does the right thing.
I think
xen_vnuma_support()needs to try the vnuma hypercall and check it is successful.That can surely be done, I think. Elena?
Yes, sure. I will change it.
Quoted 43 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/xen/setup.c +1
-0@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> Quoted 12 lines
arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 587
Quoted 8 lines
arch/x86/xen/setup.c +1
-1#include <xen/xen.h> #include <xen/page.h> @@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ())I don't think there's a need to special case the initial domain here is there?
This is actually something that Konrad asked, since, apparently, there are AMD machines that just blows up, as Dom0, if this is on.
Therefore, since, right now, the status of Elena's work is "just DomUs", we figured this could be fine for now.
Of course, the final goal (whether for Elena or for someone else to pick it up) is to be able to enable vNUMA for Dom0 too, at which point we definitely will have to figure out a way to kill this special casing safely...
What do you think?
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Re: [PATCH RFC v2 2/2] linux/vnuma: Enables NUMA for domu PV guest
In reply to Elena UfimtsevaQuoted 8 lines · Elena Ufimtseva
arch/x86/xen/setup.c +0
-0#include <xen/xen.h> #include <xen/page.h>
arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 592
Quoted 10 lines
arch/x86/xen/setup.c +4
-1@@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ()) + numa_off =0 ; +else + numa_off =1 ;#endif }
I can't be positive about this, but I think that, if we go for David's suggestions of:
- testing for the hypercall being actually supported,
- calling dummy_numa_init() directly from within xen_numa_init (in case something go wrong), instead of going through all the alternatives from x86_numa_init()
then we can even get rid of this numa_off=0|1 all together (which is also something David was already suggesting, AFAICR).
It would be nice to know what the issue was at the time they had to introduce this, and whether we can get to do some testing on any of the boxes where it was exploding.
I'll investigate more, in the meanwhile, does anyone had any clue?
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH RFC v2 2/2] linux/vnuma: Enables NUMA for domu PV guest
Quoted 26 lines · Dario Faggioli
Quoted 8 lines · Elena Ufimtseva
arch/x86/xen/setup.c +0
-0#include <xen/xen.h> #include <xen/page.h> arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 592
Quoted 10 lines
arch/x86/xen/setup.c +4
-1@@ -583,6 +584,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_support ()) + numa_off =0 ; +else + numa_off =1 ;#endif }I can't be positive about this, but I think that, if we go for David's suggestions of:
- testing for the hypercall being actually supported,
- callingdummy_numa_init()directly from withinxen_numa_init(in case something go wrong), instead of going through all the alternatives fromx86_numa_init()
Quoted 2 lines
then we can even get rid of this numa_off=0|1 all together (which is also something David was already suggesting, AFAICR).
You could do:
if (xen_initial_domain()) && !xen_vnuma_support()) numa_off = 1;
And just do that. The PV guest (the plain one) can then go on and just do the dummy one.
Quoted 4 lines
It would be nice to know what the issue was at the time they had to introduce this, and whether we can get to do some testing on any of the boxes where it was exploding.
konrad@phenom:~/linux$ git annotate arch/x86/xen/setup.c|grep numa_off 8d54db795 (Konrad Rzeszutek Wilk 2012-08-17 10:22:37 -0400 601) numa_off = 1; konrad@phenom:~/linux$ git show 8d54db795 commit 8d54db795dfb1049d45dc34f0dddbc5347ec5642 Author: Konrad Rzeszutek Wilk <[email protected]> Date: Fri Aug 17 10:22:37 2012 -0400
xen/boot: Disable NUMA for PV guests.
....
this scanning is still enabled (K8 and Fam10h, not Bulldozer class)
..
Pid: 0, comm: swapper Not tainted 3.3.6 #1 AMD Dinar/Dinar
Quoted 18 lines
I'll investigate more, in the meanwhile, does anyone had any clue?
Dario
Signature · 12 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list [email protected] http://lists.xen.org/xen-devel
Quoted 2 lines · Elena Ufimtseva
This patchset introduces vNUMA for PV domU guest.
I'm picky and I know it, but I think domU and guest are synonyms... Having just one of them should be enough.
Also, the subject line "linux/vnuma: vNUMA...". It is indeed a good practice to indicate to what component and subsystem the patches applies to. However, in this case, I don't think havin "linux/" there adds much, since you'll be sending these mainly in LKML (although, yes, they'll go on xen-devel too, but I honestly think we can manage).
Regarding the "/vnuma" part, well, vnuma isn't really a subsystem. Actually, it does not even exist before this series, so again, I won't put it there.
Actually, from a Linux developer/maintainer point of view, these patches are about Xen, so something like "xen:..." or "x86/xen:" is probably better
Quoted 4 lines
Enables PV guest to discover NUMA topology provided by Xen and initializes NUMA topology on boot. XENMEM subop hypercall is used to retreive information from Xen.
The fact that this happens during the regular x86 NUMA initialization phase (i.e., in x86_numa_init()) is worth mentioning here.
Quoted 8 lines
Xen provides number
of NUMA nodes, memory regions (start and end pfn) constructed
based on e820 domU map, distance table and cpu to node map. ixen_numa_initis called to setup NUMA related structures.
To enable this mechanism, kernel should be compiled as PV guest
withCONFIG_NUMA=yand Xen should support vNUMA functionality
(patchset http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg01337.html ).
This is fine. Perhaps I'd add something about future plans, which are to extend this to work for Dom0 too.
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 36 lines · Dario Faggioli
Quoted 2 lines · Elena Ufimtseva
This patchset introduces vNUMA for PV domU guest.
I'm picky and I know it, but I think domU and guest are synonyms... Having just one of them should be enough.
Also, the subject line "linux/vnuma: vNUMA...". It is indeed a good practice to indicate to what component and subsystem the patches applies to. However, in this case, I don't think havin "linux/" there adds much, since you'll be sending these mainly in LKML (although, yes, they'll go on xen-devel too, but I honestly think we can manage).
Regarding the "/vnuma" part, well, vnuma isn't really a subsystem. Actually, it does not even exist before this series, so again, I won't put it there.
Actually, from a Linux developer/maintainer point of view, these patches are about Xen, so something like "xen:..." or "x86/xen:" is probably better
Quoted 4 lines
Enables PV guest to discover NUMA topology provided by Xen and initializes NUMA topology on boot. XENMEM subop hypercall is used to retreive information from Xen.
The fact that this happens during the regular x86 NUMA initialization phase (i.e., in
x86_numa_init()) is worth mentioning here.Quoted 8 lines
Xen provides number
of NUMA nodes, memory regions (start and end pfn) constructed
based on e820 domU map, distance table and cpu to node map. ixen_numa_initis called to setup NUMA related structures.
To enable this mechanism, kernel should be compiled as PV guest
withCONFIG_NUMA=yand Xen should support vNUMA functionality
(patchset http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg01337.html ).
Quoted 3 lines
This is fine. Perhaps I'd add something about future plans, which are to extend this to work for Dom0 too.
Sure, will change these.
Quoted 8 lines
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena