Skip to contentSkip to conversation

xen: vnuma introduction for pv guest

PATCH RESEND In review Elena Ufimtseva 4 participants 27 messages Nov 18, 2013 – Nov 23, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover xen: vnuma introduction for pv guest

Xen vnuma introduction.

The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask. vNUMA topology is constructed by Xen toolstack.

Example of vnuma enabled pv domain dmesg:

[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0xffffffff]
[    0.000000]   node   1: [mem 0x100000000-0x1ffffffff]
[    0.000000]   node   2: [mem 0x200000000-0x2ffffffff]
[    0.000000]   node   3: [mem 0x300000000-0x3ffffffff]
[    0.000000] On node 0 totalpages: 1048479
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14280 pages used for memmap
[    0.000000]   DMA32 zone: 1044480 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 1048576
[    0.000000]   Normal zone: 14336 pages used for memmap
[    0.000000]   Normal zone: 1048576 pages, LIFO batch:31
[    0.000000] On node 2 totalpages: 1048576
[    0.000000]   Normal zone: 14336 pages used for memmap
[    0.000000]   Normal zone: 1048576 pages, LIFO batch:31
[    0.000000] On node 3 totalpages: 1048576
[    0.000000]   Normal zone: 14336 pages used for memmap
[    0.000000]   Normal zone: 1048576 pages, LIFO batch:31
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.000000] e820: cannot find a gap in the 32bit address range
[    0.000000] e820: PCI devices with unassigned 32bit BARs may break!
[    0.000000] e820: [mem 0x400100000-0x4004fffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.4-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:4
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff8800ffc00000 s85376 r8192 d21120 u2097152
[    0.000000] pcpu-alloc: s85376 r8192 d21120 u2097152 alloc=1*2097152

numactl output:

root@heatpipe:~# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0
node 0 size: 4031 MB
node 0 free: 3997 MB
node 1 cpus: 1
node 1 size: 4039 MB
node 1 free: 4022 MB
node 2 cpus: 2
node 2 size: 4039 MB
node 2 free: 4023 MB
node 3 cpus: 3
node 3 size: 3975 MB
node 3 free: 3963 MB
node distances:
node   0   1   2   3
  0:  10  20  20  20
  1:  20  10  20  20
  2:  20  20  10  20
  3:  20  20  20  10

Current patchset is available at https://git.gitorious.org/xenvnuma/linuxvnuma.git:v4
[email protected]:xenvnuma/linuxvnuma.git:v4
Xen patchset is available at: https://git.gitorious.org/xenvnuma/xenvnuma.git:v3

TODO

*       dom0, pvh and hvm vnuma support;
*       multiple memory ranges per node support;
*       benchmarking;

Elena Ufimtseva (2):
xen: vnuma support for PV guests running as domU
Subject: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

 arch/x86/include/asm/xen/vnuma.h |   12 ++++
 arch/x86/mm/numa.c               |    3 +
 arch/x86/xen/Makefile            |    2 +-
 arch/x86/xen/setup.c             |    6 +-
 arch/x86/xen/vnuma.c             |  127 ++++++++++++++++++++++++++++++++++++++
 include/xen/interface/memory.h   |   43 +++++++++++++
 6 files changed, 191 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/xen/vnuma.h
 create mode 100644 arch/x86/xen/vnuma.c
Signature · 1 line
1.7.10.4
#1 lore raw

PATCH RESEND v2 1/2 xen: vnuma support for PV guests running as domU

Issues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.

---
 arch/x86/include/asm/xen/vnuma.h |   12 ++++
 arch/x86/mm/numa.c               |    3 +
 arch/x86/xen/Makefile            |    2 +-
 arch/x86/xen/vnuma.c             |  127 ++++++++++++++++++++++++++++++++++++++
 include/xen/interface/memory.h   |   43 +++++++++++++
 5 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/xen/vnuma.h
 create mode 100644 arch/x86/xen/vnuma.c
arch/x86/include/asm/xen/vnuma.h new file +12 -0
new file mode 100644
index 0000000..aee4e92
@@ -0,0 +1,12 @@
+#ifndef _ASM_X86_VNUMA_H
+#define _ASM_X86_VNUMA_H
+
+#ifdef CONFIG_XEN
+bool xen_vnuma_supported(void);
+int xen_numa_init(void);
+#else
+static inline bool xen_vnuma_supported(void) { return false; };
+static inline int xen_numa_init(void) { return -1; };
+#endif
+
+#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
index 24aec58..99efa1b 100644
@@ -17,6 +17,7 @@
 #include <asm/dma.h>
 #include <asm/acpi.h>
 #include <asm/amd_nb.h>
+#include "asm/xen/vnuma.h"
 
 #include "numa_internal.h"
 
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)
 void __init x86_numa_init(void)
 {
 	if (!numa_off) {
+		if (!numa_init(xen_numa_init))
+			return;
 #ifdef CONFIG_X86_NUMAQ
 		if (!numa_init(numaq_numa_init))
 			return;
arch/x86/xen/Makefile +1 -1
index 96ab2c0..de9deab 100644
@@ -13,7 +13,7 @@ CFLAGS_mmu.o			:= $(nostackp)
 obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
 			grant-table.o suspend.o platform-pci-unplug.o \
-			p2m.o
+			p2m.o vnuma.o
 
 obj-$(CONFIG_EVENT_TRACING) += trace.o
 
arch/x86/xen/vnuma.c new file +127 -0

127 lines not on this page · raw message

include/xen/interface/memory.h +43 -0

47 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#2 lore raw

PATCH RESEND v2 1/2 xen: vnuma support for PV guests running as domU

In reply to Elena Ufimtseva

Issues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.

---
 arch/x86/include/asm/xen/vnuma.h |   12 ++++
 arch/x86/mm/numa.c               |    3 +
 arch/x86/xen/Makefile            |    2 +-
 arch/x86/xen/vnuma.c             |  127 ++++++++++++++++++++++++++++++++++++++
 include/xen/interface/memory.h   |   43 +++++++++++++
 5 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/xen/vnuma.h
 create mode 100644 arch/x86/xen/vnuma.c
arch/x86/include/asm/xen/vnuma.h new file +12 -0
new file mode 100644
index 0000000..aee4e92
@@ -0,0 +1,12 @@
+#ifndef _ASM_X86_VNUMA_H
+#define _ASM_X86_VNUMA_H
+
+#ifdef CONFIG_XEN
+bool xen_vnuma_supported(void);
+int xen_numa_init(void);
+#else
+static inline bool xen_vnuma_supported(void) { return false; };
+static inline int xen_numa_init(void) { return -1; };
+#endif
+
+#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
index 24aec58..99efa1b 100644
@@ -17,6 +17,7 @@
 #include <asm/dma.h>
 #include <asm/acpi.h>
 #include <asm/amd_nb.h>
+#include "asm/xen/vnuma.h"
 
 #include "numa_internal.h"
 
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)
 void __init x86_numa_init(void)
 {
 	if (!numa_off) {
+		if (!numa_init(xen_numa_init))
+			return;
 #ifdef CONFIG_X86_NUMAQ
 		if (!numa_init(numaq_numa_init))
 			return;
arch/x86/xen/Makefile +1 -1
index 96ab2c0..de9deab 100644
@@ -13,7 +13,7 @@ CFLAGS_mmu.o			:= $(nostackp)
 obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
 			grant-table.o suspend.o platform-pci-unplug.o \
-			p2m.o
+			p2m.o vnuma.o
 
 obj-$(CONFIG_EVENT_TRACING) += trace.o
 
arch/x86/xen/vnuma.c new file +127 -0

127 lines not on this page · raw message

include/xen/interface/memory.h +43 -0

47 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#3 lore raw

Re: [PATCH RESEND v2 1/2] xen: vnuma support for PV guests running as domU

Quoted 4 lines · Elena Ufimtseva

Issues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.

[...]

arch/x86/include/asm/xen/vnuma.h line 9

Quoted 12 lines
arch/x86/include/asm/xen/vnuma.h new file +9 -0
@@ -0,0 +1,12 @@
+#ifndef _ASM_X86_VNUMA_H
+#define _ASM_X86_VNUMA_H
+
+#ifdef CONFIG_XEN
+bool xen_vnuma_supported(void);
+int xen_numa_init(void);
+#else
+static inline bool xen_vnuma_supported(void) { return false; };
+static inline int xen_numa_init(void) { return -1; };

Return a valid -ve errno here. e.g., -ENOSYS.

Quoted 29 lines
arch/x86/include/asm/xen/vnuma.h new file +3 -0
+#endif
+
+#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
index 24aec58..99efa1b 100644
@@ -17,6 +17,7 @@
 #include <asm/dma.h>
 #include <asm/acpi.h>
 #include <asm/amd_nb.h>
+#include "asm/xen/vnuma.h"
 
 #include "numa_internal.h"
 
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)
 void __init x86_numa_init(void)
 {
 	if (!numa_off) {
+		if (!numa_init(xen_numa_init))
+			return;
 #ifdef CONFIG_X86_NUMAQ
 		if (!numa_init(numaq_numa_init))
 			return;
arch/x86/xen/Makefile +0 -0
index 96ab2c0..de9deab 100644

arch/x86/xen/Makefile CFLAGS_mmu.o := $(nostackp) · line 16

Quoted 5 lines
arch/x86/xen/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_mmu.o			:= $(nostackp)
 obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
 			grant-table.o suspend.o platform-pci-unplug.o \
-			p2m.o
+			p2m.o vnuma.o

obj-$(CONFIG_NUMA) += vnuma.o

I commented on this already, please take more care to address all comments before reposting.

Quoted 8 lines
arch/x86/xen/vnuma.c new file +5 -0
@@ -0,0 +1,127 @@
+#include <linux/err.h>
+#include <linux/memblock.h>
+#include <xen/interface/xen.h>
+#include <xen/interface/memory.h>
+#include <asm/xen/interface.h>

arch/x86/xen/vnuma.c line 15

Quoted 10 lines
arch/x86/xen/vnuma.c new file +10 -0
+#include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
+
+#ifdef CONFIG_NUMA
+
+/* Checks if hypercall is supported */
+bool xen_vnuma_supported(void)
+{
+	return HYPERVISOR_memory_op(XENMEM_get_vnuma_info, NULL)
+					== -ENOSYS ? false : true;

return HYPERVISOR_memory_op() != -ENOSYS;

But as I note in the other email, I think this function isn't needed.

include/xen/interface/memory.h struct xen_remove_from_physmap { · line 270

Quoted 11 lines
include/xen/interface/memory.h +5 -0
@@ -263,4 +263,47 @@ struct xen_remove_from_physmap {
 };
 DEFINE_GUEST_HANDLE_STRUCT(xen_remove_from_physmap);
 
+/* vNUMA structures */
+struct vmemrange {
+	uint64_t start, end;
+	/* reserved */
+	uint64_t _padm;

If this field is reserved for future use, please name it _reserved and ensure that the hypervisor checks that it is 0.

include/xen/interface/memory.h struct xen_remove_from_physmap { · line 283

Quoted 10 lines
include/xen/interface/memory.h +10 -0
+struct vnuma_topology_info {
+	/* OUT */
+	domid_t domid;
+	uint32_t __pad;
+	/* IN */
+	/* number of virtual numa nodes */
+	union {
+		GUEST_HANDLE(uint) nr_nodes;
+		uint64_t    _padn;
+	};

I don't think anonymous unions are allowed in the ABI. They're a c99 extension I think.

David

#4 lore raw

Re: [PATCH RESEND v2 1/2] xen: vnuma support for PV guests running as domU

In reply to Elena Ufimtseva
Quoted 4 lines · Elena Ufimtseva

Issues Xen hypercall subop XENMEM_get_vnumainfo and sets the NUMA topology, otherwise sets dummy NUMA node and prevents numa_init from calling other numa initializators as they dont work with pv guests.

[...]

arch/x86/include/asm/xen/vnuma.h line 9

Quoted 12 lines
arch/x86/include/asm/xen/vnuma.h new file +9 -0
@@ -0,0 +1,12 @@
+#ifndef _ASM_X86_VNUMA_H
+#define _ASM_X86_VNUMA_H
+
+#ifdef CONFIG_XEN
+bool xen_vnuma_supported(void);
+int xen_numa_init(void);
+#else
+static inline bool xen_vnuma_supported(void) { return false; };
+static inline int xen_numa_init(void) { return -1; };

Return a valid -ve errno here. e.g., -ENOSYS.

Quoted 29 lines
arch/x86/include/asm/xen/vnuma.h new file +3 -0
+#endif
+
+#endif /* _ASM_X86_VNUMA_H */
arch/x86/mm/numa.c +3 -0
index 24aec58..99efa1b 100644
@@ -17,6 +17,7 @@
 #include <asm/dma.h>
 #include <asm/acpi.h>
 #include <asm/amd_nb.h>
+#include "asm/xen/vnuma.h"
 
 #include "numa_internal.h"
 
@@ -632,6 +633,8 @@ static int __init dummy_numa_init(void)
 void __init x86_numa_init(void)
 {
 	if (!numa_off) {
+		if (!numa_init(xen_numa_init))
+			return;
 #ifdef CONFIG_X86_NUMAQ
 		if (!numa_init(numaq_numa_init))
 			return;
arch/x86/xen/Makefile +0 -0
index 96ab2c0..de9deab 100644

arch/x86/xen/Makefile CFLAGS_mmu.o := $(nostackp) · line 16

Quoted 5 lines
arch/x86/xen/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_mmu.o			:= $(nostackp)
 obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
 			grant-table.o suspend.o platform-pci-unplug.o \
-			p2m.o
+			p2m.o vnuma.o

obj-$(CONFIG_NUMA) += vnuma.o

I commented on this already, please take more care to address all comments before reposting.

Quoted 8 lines
arch/x86/xen/vnuma.c new file +5 -0
@@ -0,0 +1,127 @@
+#include <linux/err.h>
+#include <linux/memblock.h>
+#include <xen/interface/xen.h>
+#include <xen/interface/memory.h>
+#include <asm/xen/interface.h>

arch/x86/xen/vnuma.c line 15

Quoted 10 lines
arch/x86/xen/vnuma.c new file +10 -0
+#include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
+
+#ifdef CONFIG_NUMA
+
+/* Checks if hypercall is supported */
+bool xen_vnuma_supported(void)
+{
+	return HYPERVISOR_memory_op(XENMEM_get_vnuma_info, NULL)
+					== -ENOSYS ? false : true;

return HYPERVISOR_memory_op() != -ENOSYS;

But as I note in the other email, I think this function isn't needed.

include/xen/interface/memory.h struct xen_remove_from_physmap { · line 270

Quoted 11 lines
include/xen/interface/memory.h +5 -0
@@ -263,4 +263,47 @@ struct xen_remove_from_physmap {
 };
 DEFINE_GUEST_HANDLE_STRUCT(xen_remove_from_physmap);
 
+/* vNUMA structures */
+struct vmemrange {
+	uint64_t start, end;
+	/* reserved */
+	uint64_t _padm;

If this field is reserved for future use, please name it _reserved and ensure that the hypervisor checks that it is 0.

include/xen/interface/memory.h struct xen_remove_from_physmap { · line 283

Quoted 10 lines
include/xen/interface/memory.h +10 -0
+struct vnuma_topology_info {
+	/* OUT */
+	domid_t domid;
+	uint32_t __pad;
+	/* IN */
+	/* number of virtual numa nodes */
+	union {
+		GUEST_HANDLE(uint) nr_nodes;
+		uint64_t    _padn;
+	};

I don't think anonymous unions are allowed in the ABI. They're a c99 extension I think.

David

#5 lore raw

PATCH RESEND v2 2/2 xen: enable vnuma for PV guest

In reply to Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

---
 arch/x86/xen/setup.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/xen/setup.c +5 -1
index 68c054f..0aab799 100644
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }
Signature · 1 line
1.7.10.4
#6 lore raw

PATCH RESEND v2 2/2 xen: enable vnuma for PV guest

In reply to Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

---
 arch/x86/xen/setup.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/xen/setup.c +5 -1
index 68c054f..0aab799 100644
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }
Signature · 1 line
1.7.10.4
#7 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

If you do this, you can fold this change in with the previous patch.

David

#8 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

Quoted 4 lines

If you do this, you can fold this change in with the previous patch.

David

#9 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

Quoted 4 lines

If you do this, you can fold this change in with the previous patch.

David

#10 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 37 lines · Konrad Rzeszutek Wilk
Quoted 36 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

Quoted 8 lines
Quoted 1 line

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

David

#11 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

#12 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

#13 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 51 lines · Konrad Rzeszutek Wilk
Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Quoted 8 lines

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

David

#14 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 57 lines · David Vrabel
Quoted 56 lines · Konrad Rzeszutek Wilk
Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

Quoted 6 lines
Quoted 3 lines

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Calling from within the implementation on something that is eventually done on the upper level already is not right.

Quoted 3 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

Quoted 3 lines

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

That reads to me as an Ack, but I know you like to have it stated explicitly - so could you state the proper tag please?

Quoted 2 lines

David

#15 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 57 lines · David Vrabel
Quoted 56 lines · Konrad Rzeszutek Wilk
Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

Quoted 6 lines
Quoted 3 lines

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Calling from within the implementation on something that is eventually done on the upper level already is not right.

Quoted 3 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

Quoted 3 lines

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

That reads to me as an Ack, but I know you like to have it stated explicitly - so could you state the proper tag please?

Quoted 2 lines

David

#16 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Think of it as not the dummy, but Xen setting the NUMA configuration up with only a single node.

The useful bits in dummy_numa_init() are two calls to standard functions for use by *_numa_init() calls so it just seems easier all round to just call then directly than add a dependancy on dummy_numa_init().

Quoted 4 lines

Calling from within the implementation on something that is eventually done on the upper level already is not right.

From the point of view of the caller, it does the right thing. NUMA is

setup.

Quoted 5 lines
Quoted 2 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

I don't know what "fakenuma" is refering to.

Quoted 5 lines
Quoted 2 lines

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

That reads to me as an Ack, but I know you like to have it stated explicitly - so could you state the proper tag please?

"If I thought the hypervisor ABI was finalized..." is a pretty big "if" so I have deliberately /not/ given an ack or a reviewed tag but I've tried to be clear than I think the Linux side is now good enough (except for any changes for any updates to the hypervisor ABI).

David

#17 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

Quoted 11 lines · David Vrabel
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Think of it as not the dummy, but Xen setting the NUMA configuration up with only a single node.

Or with multiple nodes if numa=fake is used.

Quoted 4 lines

The useful bits in dummy_numa_init() are two calls to standard functions for use by *_numa_init() calls so it just seems easier all round to just call then directly than add a dependancy on dummy_numa_init().

My worry is more of the numa_init not being completly executed.

Quoted 7 lines
Quoted 5 lines

Calling from within the implementation on something that is eventually done on the upper level already is not right.

From the point of view of the caller, it does the right thing. NUMA is

setup.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

I don't know what "fakenuma" is refering to.

numa=fake or see arch/x86/mm/numa_emulation.c

Which gets executed if you end up calling: numa_init(X) and 'X' returns a zero or positive value.

Which it won't as the first thing it the Xen PV NUMA code does is to see if the hypercall is supported. If not, it bails out - so we never get to do the 'numa=fake' if a user wanted to.

#18 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 11 lines · David Vrabel
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Think of it as not the dummy, but Xen setting the NUMA configuration up with only a single node.

Or with multiple nodes if numa=fake is used.

Quoted 4 lines

The useful bits in dummy_numa_init() are two calls to standard functions for use by *_numa_init() calls so it just seems easier all round to just call then directly than add a dependancy on dummy_numa_init().

My worry is more of the numa_init not being completly executed.

Quoted 7 lines
Quoted 5 lines

Calling from within the implementation on something that is eventually done on the upper level already is not right.

From the point of view of the caller, it does the right thing. NUMA is

setup.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

I don't know what "fakenuma" is refering to.

numa=fake or see arch/x86/mm/numa_emulation.c

Which gets executed if you end up calling: numa_init(X) and 'X' returns a zero or positive value.

Which it won't as the first thing it the Xen PV NUMA code does is to see if the hypercall is supported. If not, it bails out - so we never get to do the 'numa=fake' if a user wanted to.

#19 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 2 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

That seems the wrong place to do it. The top layer calls in each of the numa implementations and then falls back to the dummy.

Think of it as not the dummy, but Xen setting the NUMA configuration up with only a single node.

The useful bits in dummy_numa_init() are two calls to standard functions for use by *_numa_init() calls so it just seems easier all round to just call then directly than add a dependancy on dummy_numa_init().

Quoted 4 lines

Calling from within the implementation on something that is eventually done on the upper level already is not right.

From the point of view of the caller, it does the right thing. NUMA is

setup.

Quoted 5 lines
Quoted 2 lines

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

Doesn't it do the same thing? And also what about if you the user provides fakenuma?

I don't know what "fakenuma" is refering to.

Quoted 5 lines
Quoted 2 lines

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

That reads to me as an Ack, but I know you like to have it stated explicitly - so could you state the proper tag please?

"If I thought the hypervisor ABI was finalized..." is a pretty big "if" so I have deliberately /not/ given an ack or a reviewed tag but I've tried to be clear than I think the Linux side is now good enough (except for any changes for any updates to the hypervisor ABI).

David

#20 lore raw

Re: [Xen-devel] [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 9 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

So, David, in this regard, what do you think about these (and the following messages in the thread?

http://bugs.xenproject.org/xen/mid/%[email protected]%3E
http://bugs.xenproject.org/xen/mid/%3C1384871712.19880.90.camel@Abyss%3E
http://bugs.xenproject.org/xen/mid/%[email protected]%3E
http://bugs.xenproject.org/xen/mid/%3C1384875772.15360.6.camel@Solace%3E
http://bugs.xenproject.org/xen/mid/%[email protected]%3E

I think Jan has a point about how we are right now retrieving and treating the number of vnodes...

Thanks and Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
#21 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to David Vrabel
Quoted 9 lines · David Vrabel

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

So, David, in this regard, what do you think about these (and the following messages in the thread?

http://bugs.xenproject.org/xen/mid/%[email protected]%3E
http://bugs.xenproject.org/xen/mid/%3C1384871712.19880.90.camel@Abyss%3E
http://bugs.xenproject.org/xen/mid/%[email protected]%3E
http://bugs.xenproject.org/xen/mid/%3C1384875772.15360.6.camel@Solace%3E
http://bugs.xenproject.org/xen/mid/%[email protected]%3E

I think Jan has a point about how we are right now retrieving and treating the number of vnodes...

Thanks and Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#22 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to Konrad Rzeszutek Wilk
Quoted 51 lines · Konrad Rzeszutek Wilk
Quoted 42 lines · David Vrabel
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 31 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

Quoted 6 lines

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

Quoted 7 lines
Quoted 4 lines
                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

Quoted 8 lines

Duh! I totally missed 'return' at the end of the check!

However, even with that (so the return), that means this part won't be called:

649 numa_init(dummy_numa_init);

Which means there won't be any dummy numa setup?

The relevant bits in dummy_numa_init are in the error path of xen_numa_init().

I do think this approach (using the provided API to setup the single (dummy) node), is preferable to calling dummy_numa_init().

If I thought the hypervisor ABI was finalized, I'd be happy with this series as-is -- the remaining issues are superficial.

David

#23 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to Konrad Rzeszutek Wilk
Quoted 37 lines · Konrad Rzeszutek Wilk
Quoted 36 lines · David Vrabel
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

Quoted 8 lines
Quoted 1 line

In both error cases, the dummy numa node is setup as required.

Incorrect. It will end up calling:

                 if (!numa_init(amd_numa_init))                                  

which will crash dom0 (see 8d54db795 "xen/boot: Disable NUMA for PV guests.") as that amd_numa_init is called before the dummy node init.

No it won't. Any error path after the check for a PV guest will add the dummy node and return success, skipping any of the hardware-specific setup.

David

#24 lore raw

Re: [PATCH RESEND v2 2/2] xen: enable vnuma for PV guest

In reply to Elena Ufimtseva
Quoted 1 line · Elena Ufimtseva

Enables numa if vnuma topology hypercall is supported and it is domU.

[...]

Quoted 11 lines
arch/x86/xen/setup.c +1 -0
@@ -20,6 +20,7 @@
 #include <asm/numa.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <asm/xen/vnuma.h>
 
 #include <xen/xen.h>
 #include <xen/page.h>

arch/x86/xen/setup.c void __init xen_arch_setup(void) · line 607

Quoted 10 lines
arch/x86/xen/setup.c +4 -1
@@ -598,6 +599,9 @@ void __init xen_arch_setup(void)
 	WARN_ON(xen_set_default_idle());
 	fiddle_vdso();
 #ifdef CONFIG_NUMA
-	numa_off = 1;
+	if (!xen_initial_domain() && xen_vnuma_supported())
+		numa_off = 0;
+	else
+		numa_off = 1;
 #endif
 }

I think this whole #ifdef CONFIG_NUMA can be removed and hence xen_vnuma_supported() can be removed as well.

For any PV guest we can call the xen_numa_init() and it will do the right thing.

For dom0, the hypercall will either: return something sensible (if in the future Xen sets something up), or it will error.

If Xen does not have vnuma support, the hypercall will error.

In both error cases, the dummy numa node is setup as required.

If you do this, you can fold this change in with the previous patch.

David

#25 lore raw
In reply to Elena Ufimtseva
Quoted 10 lines · Elena Ufimtseva

Xen vnuma introduction.

The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask. vNUMA topology is constructed by Xen toolstack.

Quoted 2 lines

Example of vnuma enabled pv domain dmesg:

Mmm... So, why the resend? When you do things like this, you usually tell the reason, if only, to let people know what happened and which series they should actually look at and review. :-)

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
#26 lore raw
In reply to Elena Ufimtseva
Quoted 10 lines · Elena Ufimtseva

Xen vnuma introduction.

The patchset introduces vnuma to paravirtualized Xen guests runnning as domU.
Xen subop hypercall is used to retreive vnuma topology information. Bases on the retreived topology from Xen, NUMA number of nodes, memory ranges, distance table and cpumask is being set. If initialization is incorrect, sets 'dummy' node and unsets nodemask. vNUMA topology is constructed by Xen toolstack.

Quoted 2 lines

Example of vnuma enabled pv domain dmesg:

Mmm... So, why the resend? When you do things like this, you usually tell the reason, if only, to let people know what happened and which series they should actually look at and review. :-)

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel