xen: vnuma introduction for pv guest
Subsystems (4)
v2 cover xen: vnuma introduction for pv guest
Xen vnuma introduction.
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.
Example of vnuma enabled pv domain dmesg:
[ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009ffff] [ 0.000000] node 0: [mem 0x00100000-0xffffffff] [ 0.000000] node 1: [mem 0x100000000-0x1ffffffff] [ 0.000000] node 2: [mem 0x200000000-0x2ffffffff] [ 0.000000] node 3: [mem 0x300000000-0x3ffffffff] [ 0.000000] On node 0 totalpages: 1048479 [ 0.000000] DMA zone: 56 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3999 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 14280 pages used for memmap [ 0.000000] DMA32 zone: 1044480 pages, LIFO batch:31 [ 0.000000] On node 1 totalpages: 1048576 [ 0.000000] Normal zone: 14336 pages used for memmap [ 0.000000] Normal zone: 1048576 pages, LIFO batch:31 [ 0.000000] On node 2 totalpages: 1048576 [ 0.000000] Normal zone: 14336 pages used for memmap [ 0.000000] Normal zone: 1048576 pages, LIFO batch:31 [ 0.000000] On node 3 totalpages: 1048576 [ 0.000000] Normal zone: 14336 pages used for memmap [ 0.000000] Normal zone: 1048576 pages, LIFO batch:31 [ 0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] No local APIC present [ 0.000000] APIC: disable apic facility [ 0.000000] APIC: switched to apic NOOP [ 0.000000] nr_irqs_gsi: 16 [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] [ 0.000000] e820: cannot find a gap in the 32bit address range [ 0.000000] e820: PCI devices with unassigned 32bit BARs may break! [ 0.000000] e820: [mem 0x400100000-0x4004fffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on Xen [ 0.000000] Xen version: 4.4-unstable (preserve-AD) [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:4 [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff8800ffc00000 s85376 r8192 d21120 u2097152 [ 0.000000] pcpu-alloc: s85376 r8192 d21120 u2097152 alloc=1*2097152
numactl output:
root@heatpipe:~# numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 node 0 size: 4031 MB node 0 free: 3997 MB node 1 cpus: 1 node 1 size: 4039 MB node 1 free: 4022 MB node 2 cpus: 2 node 2 size: 4039 MB node 2 free: 4023 MB node 3 cpus: 3 node 3 size: 3975 MB node 3 free: 3963 MB node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10
Current patchset is available at https://git.gitorious.org/xenvnuma/linuxvnuma.git:v3
Xen patchset is available at: https://git.gitorious.org/xenvnuma/xenvnuma.git:v3
TODO
* dom0, pvh and hvm vnuma support; * multiple memory ranges per node support; * benchmarking;
Elena Ufimtseva (2):
xen: vnuma support for PV guests running as domU
xen: enable vnuma for PV guest
arch/x86/include/asm/xen/vnuma.h | 12 ++++ arch/x86/mm/numa.c | 3 + arch/x86/xen/Makefile | 2 +- arch/x86/xen/setup.c | 6 +- arch/x86/xen/vnuma.c | 127 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 44 +++++++++++++ 6 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/xen/vnuma.h create mode 100644 arch/x86/xen/vnuma.c
Signature · 1 line
1.7.10.4
PATCH v2 1/2 xen: vnuma support for PV guests running as domU
Issues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.
--- arch/x86/include/asm/xen/vnuma.h | 12 ++++ arch/x86/mm/numa.c | 3 + arch/x86/xen/Makefile | 2 +- arch/x86/xen/vnuma.c | 127 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 44 +++++++++++++ 5 files changed, 187 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/xen/vnuma.h create mode 100644 arch/x86/xen/vnuma.c
arch/x86/include/asm/xen/vnuma.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef _ASM_X86_VNUMA_H +#define _ASM_X86_VNUMA_H + +#ifdef CONFIG_XEN +bool xen_vnuma_supported (void ); +int xen_numa_init (void ); +#else +static inline bool xen_vnuma_supported (void ) {return false; }; +static inline int xen_numa_init (void ) {return -1 ; }; +#endif + +#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
@@ -17,6 +17,7 @@#include <asm/dma.h> #include <asm/acpi.h> #include <asm/amd_nb.h> +#include "asm/xen/vnuma.h" #include "numa_internal.h"
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)void __init x86_numa_init (void ) {if (!numa_off) { +if (!numa_init (xen_numa_init)) +return ;#ifdef CONFIG_X86_NUMAQ if (!numa_init (numaq_numa_init))return ;
arch/x86/xen/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o
arch/x86/xen/vnuma.c new file +127 -0
127 lines not on this page · raw message
include/xen/interface/memory.h +44 -0
48 lines not on this page · raw message
Signature · 1 line
1.7.10.4
PATCH v2 1/2 xen: vnuma support for PV guests running as domU
In reply to Elena UfimtsevaIssues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.
--- arch/x86/include/asm/xen/vnuma.h | 12 ++++ arch/x86/mm/numa.c | 3 + arch/x86/xen/Makefile | 2 +- arch/x86/xen/vnuma.c | 127 ++++++++++++++++++++++++++++++++++++++ include/xen/interface/memory.h | 44 +++++++++++++ 5 files changed, 187 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/xen/vnuma.h create mode 100644 arch/x86/xen/vnuma.c
arch/x86/include/asm/xen/vnuma.h new file +12 -0
@@ -0,0 +1,12 @@ +#ifndef _ASM_X86_VNUMA_H +#define _ASM_X86_VNUMA_H + +#ifdef CONFIG_XEN +bool xen_vnuma_supported (void ); +int xen_numa_init (void ); +#else +static inline bool xen_vnuma_supported (void ) {return false; }; +static inline int xen_numa_init (void ) {return -1 ; }; +#endif + +#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
@@ -17,6 +17,7 @@#include <asm/dma.h> #include <asm/acpi.h> #include <asm/amd_nb.h> +#include "asm/xen/vnuma.h" #include "numa_internal.h"
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)void __init x86_numa_init (void ) {if (!numa_off) { +if (!numa_init (xen_numa_init)) +return ;#ifdef CONFIG_X86_NUMAQ if (!numa_init (numaq_numa_init))return ;
arch/x86/xen/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o
arch/x86/xen/vnuma.c new file +127 -0
127 lines not on this page · raw message
include/xen/interface/memory.h +44 -0
48 lines not on this page · raw message
Signature · 1 line
1.7.10.4
H. Peter Anvin
reviewerRe: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
arch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()
This isn't C++...
http://lwn.net/Articles/487493/
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
https://lkml.org/lkml/2012/3/16/510
-hpa
H. Peter Anvin
reviewerRe: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to Elena Ufimtsevaarch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()
This isn't C++...
http://lwn.net/Articles/487493/
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
https://lkml.org/lkml/2012/3/16/510
-hpa
Re: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
Quoted 6 lines · H. Peter Anvin
arch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()This isn't C++...
Quoted 8 lines
http://lwn.net/Articles/487493/
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
Thank you Peter, good to know. Will resend these.
Quoted 4 lines
-hpa
Signature · 1 line
Elena
Re: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to H. Peter AnvinQuoted 6 lines · H. Peter Anvin
arch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()This isn't C++...
Quoted 8 lines
http://lwn.net/Articles/487493/
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
Thank you Peter, good to know. Will resend these.
Quoted 4 lines
-hpa
Signature · 1 line
Elena
Re: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to H. Peter AnvinQuoted 7 lines · H. Peter Anvin
arch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()This isn't C++...
http://lwn.net/Articles/487493/
Quoted 6 lines
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
I gave notes to the patch and no follow up was done.
https://lkml.org/lkml/2012/3/16/514
Re: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to H. Peter AnvinQuoted 7 lines · H. Peter Anvin
arch/x86/xen/vnuma.c line 12
Quoted 2 lines · Elena Ufimtseva
arch/x86/xen/vnuma.c new file +2
-0+/* Checks if hypercall is supported */ +bool xen_vnuma_supported ()This isn't C++...
http://lwn.net/Articles/487493/
Quoted 6 lines
There are several more things in this patchset that get flagged by checkpatch, but apparently this rather common (and rather serious) problem is still not being detected, even through a patch was submitted almost two years ago:
I gave notes to the patch and no follow up was done.
https://lkml.org/lkml/2012/3/16/514
Re: [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to Elena UfimtsevaQuoted 7 lines · Elena Ufimtseva
Signed-off-by Elena Ufimtseva <[email protected]>arch/x86/xen/Makefile +0
-0
arch/x86/xen/Makefile CFLAGS_mmu.o := $(nostackp) · line 18
Quoted 7 lines
arch/x86/xen/Makefile +1
-1@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o
I think David said something about this during last round (going fetchin'-cuttin'-pastin' it):
"
obj-$(CONFIG_NUMA) += vnuma.o
Then you can remove the #ifdef CONFIG_NUMA from xen/vnuma.c
"
arch/x86/xen/vnuma.c line 18
Quoted 4 lines
arch/x86/xen/vnuma.c +2
-0+/* +* Called from numa_init if numa_off = 0;
^ if numa_off = 1 ?
arch/x86/xen/vnuma.c line 77
Quoted 11 lines
+ * we set
numa_off= 0 ifxen_vnuma_supported()
+ * returns true and its a domU;
+ */arch/x86/xen/vnuma.c +7
-0+int __init xen_numa_init (void ) +{ +if (nr_nodes >num_possible_cpus ()) { +pr_debug ("vNUMA: Node without cpu is not supported in this version.\n" ); + goto out; + } +
This is a super-minor thing, but I wouldn't say "in this version". It makes people think that there will be a later version where that will be supported, which we don't know. :-)
arch/x86/xen/vnuma.c line 115
Quoted 6 lines
arch/x86/xen/vnuma.c +5
-0+/* +* Set a dummy node and return success. This prevents calling any +* hardware-specific initializers which do not work in a PV guest. +* Taken from dummy_numa_init code. +*/
This is a lot better... Thanks! :-)
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 1/2] xen: vnuma support for PV guests running as domU
In reply to Elena UfimtsevaQuoted 7 lines · Elena Ufimtseva
Signed-off-by Elena Ufimtseva <[email protected]>arch/x86/xen/Makefile +0
-0
arch/x86/xen/Makefile CFLAGS_mmu.o := $(nostackp) · line 18
Quoted 7 lines
arch/x86/xen/Makefile +1
-1@@ -13,7 +13,7 @@ CFLAGS_mmu.o := $(nostackp) obj-y := enlighten.o setup.o multicalls.o mmu.o irq.o \ time.o xen-asm.o xen-asm_$(BITS).o \ grant-table.o suspend.o platform-pci-unplug.o \ - p2m.o + p2m.o vnuma.o obj-$(CONFIG_EVENT_TRACING) += trace.o
I think David said something about this during last round (going fetchin'-cuttin'-pastin' it):
"
obj-$(CONFIG_NUMA) += vnuma.o
Then you can remove the #ifdef CONFIG_NUMA from xen/vnuma.c
"
arch/x86/xen/vnuma.c line 18
Quoted 4 lines
arch/x86/xen/vnuma.c +2
-0+/* +* Called from numa_init if numa_off = 0;
^ if numa_off = 1 ?
arch/x86/xen/vnuma.c line 77
Quoted 11 lines
+ * we set
numa_off= 0 ifxen_vnuma_supported()
+ * returns true and its a domU;
+ */arch/x86/xen/vnuma.c +7
-0+int __init xen_numa_init (void ) +{ +if (nr_nodes >num_possible_cpus ()) { +pr_debug ("vNUMA: Node without cpu is not supported in this version.\n" ); + goto out; + } +
This is a super-minor thing, but I wouldn't say "in this version". It makes people think that there will be a later version where that will be supported, which we don't know. :-)
arch/x86/xen/vnuma.c line 115
Quoted 6 lines
arch/x86/xen/vnuma.c +5
-0+/* +* Set a dummy node and return success. This prevents calling any +* hardware-specific initializers which do not work in a PV guest. +* Taken from dummy_numa_init code. +*/
This is a lot better... Thanks! :-)
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
PATCH v2 2/2 xen: enable vnuma for PV guest
In reply to Elena UfimtsevaEnables numa if vnuma topology hypercall is supported and it is domU.
--- arch/x86/xen/setup.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/xen/setup.c +5 -1
@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> #include <xen/xen.h> #include <xen/page.h>
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_supported ()) + numa_off =0 ; +else + numa_off =1 ;#endif }
Signature · 1 line
1.7.10.4
PATCH v2 2/2 xen: enable vnuma for PV guest
In reply to Elena UfimtsevaEnables numa if vnuma topology hypercall is supported and it is domU.
--- arch/x86/xen/setup.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/xen/setup.c +5 -1
@@ -20,6 +20,7 @@#include <asm/numa.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h> +#include <asm/xen/vnuma.h> #include <xen/xen.h> #include <xen/page.h>
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)WARN_ON (xen_set_default_idle ());fiddle_vdso ();#ifdef CONFIG_NUMA - numa_off =1 ; +if (!xen_initial_domain () &&xen_vnuma_supported ()) + numa_off =0 ; +else + numa_off =1 ;#endif }
Signature · 1 line
1.7.10.4
Quoted 11 lines · Elena Ufimtseva
Xen vnuma introduction.
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.
Yeey!
One question - I know you had questions about the PROT_GLOBAL | ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.
And that the hypercall to set such PTE entry disallows the PROT_GLOBAL (it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see the PROT_GLOBAL (as it has been filtered out). Which means that the AutoNUMA code won't kick in.
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
Quoted 11 lines · Elena Ufimtseva
Xen vnuma introduction.
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.
Yeey!
One question - I know you had questions about the PROT_GLOBAL | ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.
And that the hypercall to set such PTE entry disallows the PROT_GLOBAL (it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see the PROT_GLOBAL (as it has been filtered out). Which means that the AutoNUMA code won't kick in.
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.
Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.
Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.
Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.
Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Quoted 42 lines · Dario Faggioli
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
Quoted 8 lines
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
Yes, you were right Dario :) See at the end. pv guests do not crash, but they have user space memory corruption.
Ok, so I will try to understand what again had happened during this weekend.
Meanwhile posting patches for Xen.
Quoted 8 lines
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.
Konrad,
as of PROT_GLOBAL flag, I will double check once more to exclude errors from my side.
Last time I was able to have numa_balancing working without any modifications from hypervisor side.
But again, I want to double check this, some experiments might have appear being good :)
Quoted 9 lines
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Quoted 77 lines · Elena Ufimtseva
Quoted 42 lines · Dario Faggioli
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
Quoted 8 lines
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
Yes, you were right Dario :) See at the end. pv guests do not crash, but they have user space memory corruption.
Ok, so I will try to understand what again had happened during this weekend.
Meanwhile posting patches for Xen.Quoted 8 lines
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.
Quoted 19 lines
Konrad,
as ofPROT_GLOBALflag, I will double check once more to exclude errors from my side.
Last time I was able to havenuma_balancingworking without any modifications from hypervisor side.
But again, I want to double check this, some experiments might have appear being good :)Quoted 9 lines
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on change_prot_numa way :
[ 6638.021439] pfn 45e602, highest_memmap_pfn - 14ddd7 [ 6638.021444] BUG: Bad page map in process dd pte:800000045e602166 pmd:abf1a067 [ 6638.021449] addr:00007f4fda2d8000 vm_flags:00100073 anon_vma:ffff8800abf77b90 mapping: (null) index:7f4fda2d8 [ 6638.021457] CPU: 1 PID: 1033 Comm: dd Tainted: G B W 3.13.0-rc2+ #10 [ 6638.021462] 0000000000000000 00007f4fda2d8000 ffffffff813ca5b1 ffff88010d68deb8 [ 6638.021471] ffffffff810f2c88 00000000abf1a067 800000045e602166 0000000000000000 [ 6638.021482] 000000000045e602 ffff88010d68deb8 00007f4fda2d8000 800000045e602166 [ 6638.021492] Call Trace: [ 6638.021497] [<ffffffff813ca5b1>] ? dump_stack+0x41/0x51 [ 6638.021503] [<ffffffff810f2c88>] ? print_bad_pte+0x19d/0x1c9 [ 6638.021509] [<ffffffff810f3aef>] ? vm_normal_page+0x94/0xb3 [ 6638.021519] [<ffffffff810fb788>] ? change_protection+0x35c/0x5a8 [ 6638.021527] [<ffffffff81107965>] ? change_prot_numa+0x13/0x24 [ 6638.021533] [<ffffffff81071697>] ? task_numa_work+0x1fb/0x299 [ 6638.021539] [<ffffffff8105ef54>] ? task_work_run+0x7b/0x8f [ 6638.021545] [<ffffffff8100e658>] ? do_notify_resume+0x53/0x68 [ 6638.021552] [<ffffffff813d4432>] ? int_signal+0x12/0x17 [ 6638.021560] pfn 45d732, highest_memmap_pfn - 14ddd7 [ 6638.021565] BUG: Bad page map in process dd pte:800000045d732166 pmd:10d684067 [ 6638.021572] addr:00007fff7c143000 vm_flags:00100173 anon_vma:ffff8800abf77960 mapping: (null) index:7fffffffc [ 6638.021582] CPU: 1 PID: 1033 Comm: dd Tainted: G B W 3.13.0-rc2+ #10 [ 6638.021587] 0000000000000000 00007fff7c143000 ffffffff813ca5b1 ffff8800abf339b0 [ 6638.021595] ffffffff810f2c88 000000010d684067 800000045d732166 0000000000000000 [ 6638.021603] 000000000045d732 ffff8800abf339b0 00007fff7c143000
800000045d732166
The code has changed since last problem, I will work on this to see where it comes from.
Elena
Quoted 4 lines
--
Elena
Signature · 1 line
Elena
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.
You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :
Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing the PAGE_PRESENT bit from actually present pages, and adding a new special one (PAGE_NUMA or something like that);
- when a page fault is triggered and the PAGE_NUMA flag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
Thanks and Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Quoted 29 lines · Dario Faggioli
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
Quoted 12 lines
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing thePAGE_PRESENTbit from actually present pages, and adding a new special one (PAGE_NUMAor something like that);
- when a page fault is triggered and thePAGE_NUMAflag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.
Hello Dario, Konrad.
- Yes, there is a kernel worker that runs on each node and scans some pages stats and
marks them as _PROT_NONE and resets _PAGE_PRESENT.
The page fault at this moment is triggered and control is being returned back to the linux pv kernel
to process with handle_mm_fault and page numa fault handler if discovered if that was a numa pmd/pte with
present flag cleared.
About the stats, I will have to collect some sensible information.
Quoted 11 lines
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?
The real healp needed from hypervisor is to allow _PAGE_NUMA flags on pte/pmd entries.
I have done so in hypervisor by utilizing same _PAGE_NUMA bit and including into the allowed bit mask.
As this bit is the same as PAGE_GLOBAL in hypervisor, that may induce some other errors. So far I have not seen any
and I will double check on this.
Quoted 11 lines
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
here is the best part :)
After a fresh look at the numa autobalancing, applying recent patches, talking some to riel who works now on mm numa autobalancing and running some tests including dd, ltp, kernel compiling and my own tests, autobalancing now is working
correctly with vnuma. Now I can see sucessfully migrated pages in /proc/vmstat:
numa_pte_updates 39numa_huge_pte_updates 0numa_hint_faults 36numa_hint_faults_local 23numa_pages_migrated 4pgmigrate_success 4pgmigrate_fail 0
I will be running some tests with transparent huge pages as the migration of such will be failing.
Probably it is possible to find all the patches related to numa autobalancing and figure out possible reasons
of why previously balancing was not working. Giving the amount of work kernel folks spent recently to fix
issues with numa and the significance of the changes itself, I might need few more attempts to understand it.
I am going to test THP and if that works will follow up with patches.
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.
Quoted 9 lines
Thanks and Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Quoted 112 lines · Elena Ufimtseva
Quoted 29 lines · Dario Faggioli
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
Quoted 12 lines
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing thePAGE_PRESENTbit from actually present pages, and adding a new special one (PAGE_NUMAor something like that);
- when a page fault is triggered and thePAGE_NUMAflag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.Hello Dario, Konrad.
- Yes, there is a kernel worker that runs on each node and scans some pages stats and
marks them as_PROT_NONEand resets_PAGE_PRESENT.
The page fault at this moment is triggered and control is being returned back to the linux pv kernel
to process withhandle_mm_faultand page numa fault handler if discovered if that was a numa pmd/pte with
present flag cleared.
About the stats, I will have to collect some sensible information.Quoted 11 lines
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?The real healp needed from hypervisor is to allow
_PAGE_NUMAflags on pte/pmd entries.
I have done so in hypervisor by utilizing same_PAGE_NUMAbit and including into the allowed bit mask.
As this bit is the same asPAGE_GLOBALin hypervisor, that may induce some other errors. So far I have not seen any
and I will double check on this.Quoted 11 lines
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
here is the best part :)
After a fresh look at the numa autobalancing, applying recent patches, talking some to riel who works now on mm numa autobalancing and running some tests including dd, ltp, kernel compiling and my own tests, autobalancing now is working
correctly with vnuma. Now I can see sucessfully migrated pages in/proc/vmstat:
numa_pte_updates39numa_huge_pte_updates0numa_hint_faults36numa_hint_faults_local23numa_pages_migrated4pgmigrate_success4pgmigrate_fail0I will be running some tests with transparent huge pages as the migration of such will be failing.
Probably it is possible to find all the patches related to numa autobalancing and figure out possible reasons
of why previously balancing was not working. Giving the amount of work kernel folks spent recently to fix
issues with numa and the significance of the changes itself, I might need few more attempts to understand it.
Quoted 6 lines
I am going to test THP and if that works will follow up with patches.
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.
And of course, more details will follow... :)
Quoted 15 lines
Quoted 10 lines
Thanks and Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)--
Elena
Signature · 1 line
Elena
Quoted 9 lines · Elena Ufimtseva
Quoted 4 lines · Elena Ufimtseva
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.And of course, more details will follow... :)
Yeah, well, during early investigation, I also used basically just SpecJBB. See here:
http://blog.xen.org/index.php/2012/04/26/numa-and-xen-part-1-introduction/
http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/
For benchmarking our NUMA aware scheduling solution, in Xen, I used SpecJBB, sysbench and LMbench:
http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/
The trickiest part is have the benchmark(s) of choice running concurrently is a bunch of VMs on the same host. I used some hand crafted scripts at the time, very specific to my own environment.
I'm working on abstracting that and putting together something more easy to be shared and used outside my attic. :-P
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Quoted 9 lines · Elena Ufimtseva
Quoted 4 lines · Elena Ufimtseva
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.And of course, more details will follow... :)
Yeah, well, during early investigation, I also used basically just SpecJBB. See here:
http://blog.xen.org/index.php/2012/04/26/numa-and-xen-part-1-introduction/
http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/
For benchmarking our NUMA aware scheduling solution, in Xen, I used SpecJBB, sysbench and LMbench:
http://blog.xen.org/index.php/2012/05/16/numa-and-xen-part-ii-scheduling-and-placement/
The trickiest part is have the benchmark(s) of choice running concurrently is a bunch of VMs on the same host. I used some hand crafted scripts at the time, very specific to my own environment.
I'm working on abstracting that and putting together something more easy to be shared and used outside my attic. :-P
Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 112 lines · Elena Ufimtseva
Quoted 29 lines · Dario Faggioli
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
Quoted 12 lines
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing thePAGE_PRESENTbit from actually present pages, and adding a new special one (PAGE_NUMAor something like that);
- when a page fault is triggered and thePAGE_NUMAflag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.Hello Dario, Konrad.
- Yes, there is a kernel worker that runs on each node and scans some pages stats and
marks them as_PROT_NONEand resets_PAGE_PRESENT.
The page fault at this moment is triggered and control is being returned back to the linux pv kernel
to process withhandle_mm_faultand page numa fault handler if discovered if that was a numa pmd/pte with
present flag cleared.
About the stats, I will have to collect some sensible information.Quoted 11 lines
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?The real healp needed from hypervisor is to allow
_PAGE_NUMAflags on pte/pmd entries.
I have done so in hypervisor by utilizing same_PAGE_NUMAbit and including into the allowed bit mask.
As this bit is the same asPAGE_GLOBALin hypervisor, that may induce some other errors. So far I have not seen any
and I will double check on this.Quoted 11 lines
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
here is the best part :)
After a fresh look at the numa autobalancing, applying recent patches, talking some to riel who works now on mm numa autobalancing and running some tests including dd, ltp, kernel compiling and my own tests, autobalancing now is working
correctly with vnuma. Now I can see sucessfully migrated pages in/proc/vmstat:
numa_pte_updates39numa_huge_pte_updates0numa_hint_faults36numa_hint_faults_local23numa_pages_migrated4pgmigrate_success4pgmigrate_fail0I will be running some tests with transparent huge pages as the migration of such will be failing.
Probably it is possible to find all the patches related to numa autobalancing and figure out possible reasons
of why previously balancing was not working. Giving the amount of work kernel folks spent recently to fix
issues with numa and the significance of the changes itself, I might need few more attempts to understand it.
Quoted 6 lines
I am going to test THP and if that works will follow up with patches.
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.
And of course, more details will follow... :)
Quoted 15 lines
Quoted 10 lines
Thanks and Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)--
Elena
Signature · 1 line
Elena
Quoted 29 lines · Dario Faggioli
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
Quoted 12 lines
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing thePAGE_PRESENTbit from actually present pages, and adding a new special one (PAGE_NUMAor something like that);
- when a page fault is triggered and thePAGE_NUMAflag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.
Hello Dario, Konrad.
- Yes, there is a kernel worker that runs on each node and scans some pages stats and
marks them as _PROT_NONE and resets _PAGE_PRESENT.
The page fault at this moment is triggered and control is being returned back to the linux pv kernel
to process with handle_mm_fault and page numa fault handler if discovered if that was a numa pmd/pte with
present flag cleared.
About the stats, I will have to collect some sensible information.
Quoted 11 lines
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?
The real healp needed from hypervisor is to allow _PAGE_NUMA flags on pte/pmd entries.
I have done so in hypervisor by utilizing same _PAGE_NUMA bit and including into the allowed bit mask.
As this bit is the same as PAGE_GLOBAL in hypervisor, that may induce some other errors. So far I have not seen any
and I will double check on this.
Quoted 11 lines
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
here is the best part :)
After a fresh look at the numa autobalancing, applying recent patches, talking some to riel who works now on mm numa autobalancing and running some tests including dd, ltp, kernel compiling and my own tests, autobalancing now is working
correctly with vnuma. Now I can see sucessfully migrated pages in /proc/vmstat:
numa_pte_updates 39numa_huge_pte_updates 0numa_hint_faults 36numa_hint_faults_local 23numa_pages_migrated 4pgmigrate_success 4pgmigrate_fail 0
I will be running some tests with transparent huge pages as the migration of such will be failing.
Probably it is possible to find all the patches related to numa autobalancing and figure out possible reasons
of why previously balancing was not working. Giving the amount of work kernel folks spent recently to fix
issues with numa and the significance of the changes itself, I might need few more attempts to understand it.
I am going to test THP and if that works will follow up with patches.
Dario, what tools did you use to test NUMA on xen? Maybe there is something I can use as well?
Here http://lwn.net/Articles/558593/ Mel Gorman uses specjbb and jvm, I though I can run something similar.
Quoted 9 lines
Thanks and Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena
Quoted 4 lines · Elena Ufimtseva
Quoted 3 lines · Elena Ufimtseva
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
No worries... You should see *my* backlog. :-P
Quoted 10 lines
Quoted 6 lines
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.
You mean the Linux side? Perhaps stick somewhere a reference to the git tree/branch where it lives, but, before re-sending, let's wait for it to be as issue free as we can tell?
Quoted 3 lines
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on
change_prot_numaway :
Ok, so, I think I need to step back a bit from the actual stack trace and look at the big picture. Please, Elena or anyone, correct me if I'm saying something wrong about how Linux's autonuma works and interacts with Xen.
The way it worked when I last looked at it was sort of like this:
- there was a kthread scanning all the pages, removing the PAGE_PRESENT bit from actually present pages, and adding a new special one (PAGE_NUMA or something like that);
- when a page fault is triggered and the PAGE_NUMA flag is found, it figures out the page is actually there, so no swap or anything. However, it tracks from what node the access to that page came from, matches it with the node where the page actually is and collect some statistics about that;
- at some point (and here I don't remember the exact logic, since it changed quite a few times) pages ranking badly in the stats above are moved from one node to another.
Is this description still accurate? If yes, here's what I would (double) check, when running this in a PV guest on top of Xen:
1. the NUMA hinting page fault, are we getting and handling them correctly in the PV guest? Are the stats in the guest kernel being updated in a sensible way, i.e., do they make sense and properly relate to the virtual topology of the guest?
At some point we thought it would have been necessary to intercept these faults and make sure the above is true with some help from the hypervisor... Is this the case? Why? Why not?
2. what happens when autonuma tries to move pages from one node to another? For us, that would mean in moving from one virtual node to another... Is there a need to do anything at all? I mean, is this, from our perspective, just copying the content of an MFN from node X into another MFN on node Y, or do we need to update some of our vnuma tracking data structures in Xen?
If we have this figured out already, then I think we just chase bugs and repost the series. If not, well, I think we should. :-D
Thanks and Regards,
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 77 lines · Elena Ufimtseva
Quoted 42 lines · Dario Faggioli
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
Quoted 8 lines
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
Yes, you were right Dario :) See at the end. pv guests do not crash, but they have user space memory corruption.
Ok, so I will try to understand what again had happened during this weekend.
Meanwhile posting patches for Xen.Quoted 8 lines
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.
Quoted 19 lines
Konrad,
as ofPROT_GLOBALflag, I will double check once more to exclude errors from my side.
Last time I was able to havenuma_balancingworking without any modifications from hypervisor side.
But again, I want to double check this, some experiments might have appear being good :)Quoted 9 lines
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
As of now I have patch v4 for reviewing. Not sure if it will be beneficial to post it for review
or look closer at the current problem.
The issue I am seeing right now is defferent from what was happening before. The corruption happens when on change_prot_numa way :
[ 6638.021439] pfn 45e602, highest_memmap_pfn - 14ddd7 [ 6638.021444] BUG: Bad page map in process dd pte:800000045e602166 pmd:abf1a067 [ 6638.021449] addr:00007f4fda2d8000 vm_flags:00100073 anon_vma:ffff8800abf77b90 mapping: (null) index:7f4fda2d8 [ 6638.021457] CPU: 1 PID: 1033 Comm: dd Tainted: G B W 3.13.0-rc2+ #10 [ 6638.021462] 0000000000000000 00007f4fda2d8000 ffffffff813ca5b1 ffff88010d68deb8 [ 6638.021471] ffffffff810f2c88 00000000abf1a067 800000045e602166 0000000000000000 [ 6638.021482] 000000000045e602 ffff88010d68deb8 00007f4fda2d8000 800000045e602166 [ 6638.021492] Call Trace: [ 6638.021497] [<ffffffff813ca5b1>] ? dump_stack+0x41/0x51 [ 6638.021503] [<ffffffff810f2c88>] ? print_bad_pte+0x19d/0x1c9 [ 6638.021509] [<ffffffff810f3aef>] ? vm_normal_page+0x94/0xb3 [ 6638.021519] [<ffffffff810fb788>] ? change_protection+0x35c/0x5a8 [ 6638.021527] [<ffffffff81107965>] ? change_prot_numa+0x13/0x24 [ 6638.021533] [<ffffffff81071697>] ? task_numa_work+0x1fb/0x299 [ 6638.021539] [<ffffffff8105ef54>] ? task_work_run+0x7b/0x8f [ 6638.021545] [<ffffffff8100e658>] ? do_notify_resume+0x53/0x68 [ 6638.021552] [<ffffffff813d4432>] ? int_signal+0x12/0x17 [ 6638.021560] pfn 45d732, highest_memmap_pfn - 14ddd7 [ 6638.021565] BUG: Bad page map in process dd pte:800000045d732166 pmd:10d684067 [ 6638.021572] addr:00007fff7c143000 vm_flags:00100173 anon_vma:ffff8800abf77960 mapping: (null) index:7fffffffc [ 6638.021582] CPU: 1 PID: 1033 Comm: dd Tainted: G B W 3.13.0-rc2+ #10 [ 6638.021587] 0000000000000000 00007fff7c143000 ffffffff813ca5b1 ffff8800abf339b0 [ 6638.021595] ffffffff810f2c88 000000010d684067 800000045d732166 0000000000000000 [ 6638.021603] 000000000045d732 ffff8800abf339b0 00007fff7c143000
800000045d732166
The code has changed since last problem, I will work on this to see where it comes from.
Elena
Quoted 4 lines
--
Elena
Signature · 1 line
Elena
Quoted 42 lines · Dario Faggioli
Quoted 11 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Elena Ufimtseva
The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask.
vNUMA topology is constructed by Xen toolstack. Xen patchset is available at https://git.gitorious.org/xenvnuma/xenvnuma.git:v3.Quoted 2 lines
Yeey!
:-)
Quoted 11 lines
One question - I know you had questions about the
PROT_GLOBAL| ~PAGE_PRESENT being set on PTEs that are going to be harvested for AutoNUMA balancing.And that the hypercall to set such PTE entry disallows the
PROT_GLOBAL(it stripts it off)? That means that when the Linux page system kicks in (as it has ~PAGE_PRESENT) the Linux pagehandler won't see thePROT_GLOBAL(as it has been filtered out). Which means that the AutoNUMA code won't kick in.Quoted 4 lines
(see http://article.gmane.org/gmane.comp.emulators.xen.devel/174317)
Was that problem ever answered?
I think the issue is a twofold one.
If I remember correctly (Elena, please, correct me if I'm wrong) Elena was seeing _crashes_ with both vNUMA and AutoNUMA enabled for the guest. That's what pushed her to investigate the issue, and led to what you're summing up above.
However, it appears the crash was due to something completely unrelated to Xen and vNUMA, was affecting baremetal too, and got fixed, which means the crash is now gone.
Quoted 8 lines
It remains to be seen (I think) whether that also means that AutoNUMA works. In fact, chatting about this in Edinburgh, Elena managed to convince me pretty badly that we should --as part of the vNUMA support-- do something about this, in order to make it work. At that time I thought we should be doing something to avoid the system to go ka-boom, but as I said, even now that it does not crash anymore, she was so persuasive that I now find it quite hard to believe that we really don't need to do anything. :-P
Yes, you were right Dario :) See at the end. pv guests do not crash, but they have user space memory corruption.
Ok, so I will try to understand what again had happened during this weekend.
Meanwhile posting patches for Xen.
Quoted 8 lines
I guess, as soon as we get the chance, we should see if this actually works, i.e., in addition to seeing the proper topology and not crashing, verify that AutoNUMA in the guest is actually doing is job.
What do you think? Again, Elena, please chime in and explain how things are, if I got something wrong. :-)
Oh guys, I feel really bad about not replying to these emails... Somehow these replies all got deleted.. wierd.
Ok, about that automatic balancing. At the moment of the last patch automatic numa balancing seem to
work, but after rebasing on the top of 3.12-rc2 I see similar issues. I will try to figure out what commits broke and will contact Ingo Molnar and Mel Gorman.
Konrad,
as of PROT_GLOBAL flag, I will double check once more to exclude errors from my side.
Last time I was able to have numa_balancing working without any modifications from hypervisor side.
But again, I want to double check this, some experiments might have appear being good :)
Quoted 9 lines
Regards,
Dario--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Signature · 1 line
Elena