Skip to contentSkip to conversation

Device tree based NUMA support for Arm - Part#2

PATCH Superseded Wei Chen 3 participants 50 messages Jul 8, 2022 – Jul 25, 2022

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Wei Chen

author
#0 lore raw

v2 cover Device tree based NUMA support for Arm - Part#2

(The Arm device tree based NUMA support patch set contains 35 patches. In order to make stuff easier for reviewers, I split them into 3 parts:
1. Preparation. I have re-sorted the patch series. And moved independent patches to the head of the series - merged in [1]
2. Move generically usable code from x86 to common - this series.
3. Add new code to support Arm.

This series only contains the second part patches. As the whole NUMA series has been reviewed for 1 round in [2], so this series would be v2)

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

[1] https://lists.xenproject.org/archives/html/xen-devel/2022-06/msg00499.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg01903.html

---
v1 -> v2:
1. Refine the commit messages of several patches.
2. Merge v1 patch#9,10 into one patch. Introduce the new functions in the same patch that this patch will be used first time.
3. Fold if ( end > mem_hotplug ) to mem_hotplug_update_boundary, in this case, we can drop mem_hotplug_boundary.
4. Remove fw_numa, use enumeration to replace numa_off and acpi_numa.
5. Correct return value of srat_disabled.
6. Introduce numa_enabled_with_firmware.
7. Refine the justification of using !node_data[nid].node_spanned_pages.
8. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
9. Adjust the conditional express for ASSERT.
10. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
11. Use conditional macro to gate MAX_NUMNODES for other architectures.
12. Use arch_get_memory_map to replace arch_get_memory_bank_range and arch_get_memory_bank_number.
13. Remove the !start || !end check, because caller guarantee these two pointers will not be NULL.
14. Add code comment for numa_update_node_memblks to explain:
Assumes all memory regions belonging to a single node
are in one chunk. Holes between them will be included
in the node.
15. Merge this single patch instead of serval patches to move x86 SRAT code to common.
16. Export node_to_pxm to keep pxm information in NUMA scan nodes error messages.
17. Change the code style to target file's Xen code-style.
18. Adjust some __init and __initdata for some functions and variables.
19. Replace CONFIG_ACPI_NUMA by CONFIG_NUMA. Replace "SRAT" texts.
20. Turn numa_scan_nodes to static.
21. Change NR_NUMA_NODES upper bound from 4095 to 255.

Wei Chen (9):
xen/x86: introduce a helper to update memory hotplug end xen/x86: Use enumerations to indicate NUMA status xen/x86: move generically usable NUMA code from x86 to common xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid xen/x86: use arch_get_memory_map to get information from E820 map xen/x86: move NUMA scan nodes codes from x86 to common xen/x86: rename bad_srat to numa_bad
xen: rename acpi_scan_nodes to numa_scan_nodes
xen: introduce a Kconfig option to configure NUMA nodes number

 xen/arch/Kconfig                 |  11 +
 xen/arch/x86/include/asm/acpi.h  |   2 -
 xen/arch/x86/include/asm/mm.h    |   6 +
 xen/arch/x86/include/asm/numa.h  |  61 +--
 xen/arch/x86/include/asm/setup.h |   1 -
 xen/arch/x86/numa.c              | 446 +-----------------
 xen/arch/x86/setup.c             |   3 +-
 xen/arch/x86/srat.c              | 310 ++-----------
 xen/common/Makefile              |   1 +
 xen/common/numa.c                | 746 +++++++++++++++++++++++++++++++
 xen/include/xen/numa.h           |  92 +++-
 11 files changed, 902 insertions(+), 777 deletions(-)
 create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/9 xen/x86: introduce a helper to update memory hotplug end

x86 provides a mem_hotplug variable to maintain the memory hotplug end address. We want to move some codes from x86 to common, so that it can be reused by other architectures. But not all architectures have supported memory hotplug. So in this patch, we introduce this helper to replace mem_hotplug direct access in these codes. This will give the ability of stubbing this helper to those architectures without memory hotplug support.

---
v1 -> v2:
1. Refine the commit message.
2. Merge v1 patch#9,10 into one patch. Introduce the new functions in the same patch that this patch will be used first time.
3. Fold if ( end > mem_hotplug ) to mem_hotplug_update_boundary, in this case, we can drop mem_hotplug_boundary.

---
 xen/arch/x86/include/asm/mm.h | 6 ++++++
 xen/arch/x86/srat.c           | 3 +--
 2 files changed, 7 insertions(+), 2 deletions(-)
xen/arch/x86/include/asm/mm.h +6 -0
index 07b59c982b..b3dfbdb52b 100644
@@ -476,6 +476,12 @@ static inline int get_page_and_type(struct page_info *page,
 
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    if ( end > mem_hotplug )
+        mem_hotplug = end;
+}
+
 /******************************************************************************
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.
xen/arch/x86/srat.c +1 -2
index b62a152911..f53431f5e8 100644
@@ -418,8 +418,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Quoted 7 lines · Wei Chen

x86 provides a mem_hotplug variable to maintain the memory hotplug end address. We want to move some codes from x86 to common, so that it can be reused by other architectures. But not all architectures have supported memory hotplug. So in this patch, we introduce this helper to replace mem_hotplug direct access in these codes. This will give the ability of stubbing this helper to those architectures without memory hotplug support.

What remains unclear to me is why Arm can't simply gain the necessary variable. Sooner or later you'll want to support hotplug there anyway, I suppose. Mechanically the change is fine. Albeit maybe "top" instead of "boundary", and maybe also pass "node" even if x86 doesn't need it?

Jan

Quoted 44 lines

---
v1 -> v2:
1. Refine the commit message.
2. Merge v1 patch#9,10 into one patch. Introduce the new functions in the same patch that this patch will be used first time.
3. Fold if ( end > mem_hotplug ) to mem_hotplug_update_boundary, in this case, we can drop mem_hotplug_boundary.

---
 xen/arch/x86/include/asm/mm.h | 6 ++++++
 xen/arch/x86/srat.c           | 3 +--
 2 files changed, 7 insertions(+), 2 deletions(-)
xen/arch/x86/include/asm/mm.h +6 -0
index 07b59c982b..b3dfbdb52b 100644
@@ -476,6 +476,12 @@ static inline int get_page_and_type(struct page_info *page,
 
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    if ( end > mem_hotplug )
+        mem_hotplug = end;
+}
+
 /******************************************************************************
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.
xen/arch/x86/srat.c +1 -2
index b62a152911..f53431f5e8 100644
@@ -418,8 +418,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
author
#3 lore raw

RE: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Hi Jan,

Quoted 19 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 19:54
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Quoted 7 lines · Wei Chen

x86 provides a mem_hotplug variable to maintain the memory hotplug end address. We want to move some codes from x86 to common, so that it can be reused by other architectures. But not all architectures have supported memory hotplug. So in this patch, we introduce this helper to replace mem_hotplug direct access in these codes. This will give the ability of stubbing this helper to those architectures without memory hotplug support.

Quoted 2 lines

What remains unclear to me is why Arm can't simply gain the necessary variable. Sooner or later you'll want to support hotplug there anyway,

I am not sure my limited memory hotplug knowledge can answer your question or not. As memory hotplug depends on hardware and firmware support. Now for Arm, we only have ACPI firmware that can notify OS through GED event (not very sure). But ACPI and device tree couldn't be enabled at the same time from OS booting. In device tree based NUMA, we will enable device tree, ACPI service will not be initialized, that means we can not get notification of memory hotplug events from ACPI firmware.

Of course, adding GED device nodes to the device tree, and getting these events through GED interrupts should not be a big technical problem. But there may be other reasons, and no one is planning to add memory hotplug support to the device tree (perhaps need an ACPI-like specification or a firmware abstraction interface). So if we want to implement Xen Arm memory hotplug, it should also start from ACPI. So currently even if we gain the variable in Arm, it will not be used. Device tree does not have property to indicate a memory block can be hotplug or not.

Quoted 3 lines

I suppose. Mechanically the change is fine. Albeit maybe "top" instead of "boundary", and maybe also pass "node" even if x86 doesn't need it?

Sorry, I am not very clear about these comments:
It makes sense to use mem_hotplug_update_top instead
of mem_hotplug_update_boundary. But how can I understand pass "node"? did you mean mem_hotplug_update_top(node, end)? But mem_hotplug is a global top for memory hotplug ranges. I don't think pass "node" to this function can be useful.

Cheers,
Wei Chen

Quoted 51 lines

Jan

Quoted 8 lines

---
v1 -> v2:
1. Refine the commit message.
2. Merge v1 patch#9,10 into one patch. Introduce the new functions in the same patch that this patch will be used first time.
3. Fold if ( end > mem_hotplug ) to mem_hotplug_update_boundary, in this case, we can drop mem_hotplug_boundary.

Review

Quoted 6 lines
---
 xen/arch/x86/include/asm/mm.h | 6 ++++++
 xen/arch/x86/srat.c           | 3 +--
 2 files changed, 7 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/x86/include/asm/mm.h

xen/arch/x86/include/asm/mm.h static inline int get_page_and_type(struct

Quoted 4 lines

index 07b59c982b..b3dfbdb52b 100644

xen/arch/x86/include/asm/mm.h +0 -0
@@ -476,6 +476,12 @@ static inline int get_page_and_type(struct

page_info *page,

xen/arch/x86/include/asm/mm.h static inline int get_page_and_type(struct page_info *page,

Quoted 10 lines
xen/arch/x86/include/asm/mm.h +6 -0
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    if ( end > mem_hotplug )
+        mem_hotplug = end;
+}
+
 

/*************************************************************************
*****

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 7 lines
xen/arch/x86/include/asm/mm.h +0 -0
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.
xen/arch/x86/srat.c +0 -0
index b62a152911..f53431f5e8 100644
@@ -418,8 +418,7 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 9 lines
xen/arch/x86/srat.c +1 -2
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
#4 lore raw

Re: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Quoted 23 lines · Wei Chen
Quoted 12 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 19:54

Quoted 7 lines · Wei Chen

x86 provides a mem_hotplug variable to maintain the memory hotplug end address. We want to move some codes from x86 to common, so that it can be reused by other architectures. But not all architectures have supported memory hotplug. So in this patch, we introduce this helper to replace mem_hotplug direct access in these codes. This will give the ability of stubbing this helper to those architectures without memory hotplug support.

Quoted 2 lines

What remains unclear to me is why Arm can't simply gain the necessary variable. Sooner or later you'll want to support hotplug there anyway,

I am not sure my limited memory hotplug knowledge can answer your question or not. As memory hotplug depends on hardware and firmware support. Now for Arm, we only have ACPI firmware that can notify OS through GED event (not very sure). But ACPI and device tree couldn't be enabled at the same time from OS booting. In device tree based NUMA, we will enable device tree, ACPI service will not be initialized, that means we can not get notification of memory hotplug events from ACPI firmware.

Quoted 8 lines

Of course, adding GED device nodes to the device tree, and getting these events through GED interrupts should not be a big technical problem. But there may be other reasons, and no one is planning to add memory hotplug support to the device tree (perhaps need an ACPI-like specification or a firmware abstraction interface). So if we want to implement Xen Arm memory hotplug, it should also start from ACPI. So currently even if we gain the variable in Arm, it will not be used. Device tree does not have property to indicate a memory block can be hotplug or not.

But ACPI is possible to be enabled for Arm64, and hence hotplug could be made work that way. Therefore I view the variable as potentially useful on Arm as well, and hence introducing the variable might be less trouble than introducing the per-arch helper.

Quoted 10 lines
Quoted 3 lines

I suppose. Mechanically the change is fine. Albeit maybe "top" instead of "boundary", and maybe also pass "node" even if x86 doesn't need it?

Sorry, I am not very clear about these comments:
It makes sense to use mem_hotplug_update_top instead
of mem_hotplug_update_boundary. But how can I understand pass "node"? did you mean mem_hotplug_update_top(node, end)? But mem_hotplug is a global top for memory hotplug ranges. I don't think pass "node" to this function can be useful.

Please separate "current implementation" from "abstract model". In the latter it would seem quite reasonable to me to have per-node upper bounds (or even per-node ranges). Therefore adding node (and, on top of what I did suggest before, region start) to the parameters of the new per-arch hook would seem a good move to me, even if at present only / at most the "end" parameter would actually be used.

Jan

author
#5 lore raw

RE: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Hi Jan,

Quoted 43 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月13日 16:46
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Quoted 12 lines · Wei Chen
Quoted 12 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 19:54

Quoted 7 lines · Wei Chen

x86 provides a mem_hotplug variable to maintain the memory hotplug end address. We want to move some codes from x86 to common, so that it can be reused by other architectures. But not all architectures have supported memory hotplug. So in this patch, we introduce this helper to replace mem_hotplug direct access in these codes. This will give the ability of stubbing this helper to those architectures without memory hotplug support.

Quoted 4 lines
Quoted 2 lines

What remains unclear to me is why Arm can't simply gain the necessary variable. Sooner or later you'll want to support hotplug there anyway,

I am not sure my limited memory hotplug knowledge can answer your

question

Quoted 3 lines

or not. As memory hotplug depends on hardware and firmware support. Now for Arm, we only have ACPI firmware that can notify OS through GED event (not very sure). But ACPI and device tree couldn't be enabled at the

same

Quoted 9 lines

time from OS booting. In device tree based NUMA, we will enable device tree, ACPI service will not be initialized, that means we can not get notification of memory hotplug events from ACPI firmware.

Of course, adding GED device nodes to the device tree, and getting these events through GED interrupts should not be a big technical problem. But there may be other reasons, and no one is planning to add memory hotplug support to the device tree (perhaps need an ACPI-like specification or a firmware abstraction interface). So if we want to implement Xen Arm

memory

Quoted 1 line

hotplug, it should also start from ACPI. So currently even if we gain

Quoted 9 lines

the

Quoted 2 lines

variable in Arm, it will not be used. Device tree does not have property to indicate a memory block can be hotplug or not.

But ACPI is possible to be enabled for Arm64, and hence hotplug could be made work that way. Therefore I view the variable as potentially useful on Arm as well, and hence introducing the variable might be less trouble than introducing the per-arch helper.

Ok, I will try to expose mem_hotplug to common/mm.c. As both Arm and x86 code can access it, we can move related NUMA code to common Without this helper. And mem_hotplug will always be zero for Arm until Arm support memory hotplug. This should be harmless for Arm.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

I suppose. Mechanically the change is fine. Albeit maybe "top" instead of "boundary", and maybe also pass "node" even if x86 doesn't need it?

Sorry, I am not very clear about these comments:
It makes sense to use mem_hotplug_update_top instead
of mem_hotplug_update_boundary. But how can I understand pass "node"? did you mean mem_hotplug_update_top(node, end)? But mem_hotplug is a global top for memory hotplug ranges. I don't think pass "node" to this function can be useful.

Quoted 7 lines

Please separate "current implementation" from "abstract model". In the latter it would seem quite reasonable to me to have per-node upper bounds (or even per-node ranges). Therefore adding node (and, on top of what I did suggest before, region start) to the parameters of the new per-arch hook would seem a good move to me, even if at present only / at most the "end" parameter would actually be used.

As we will export mem_hotplug to common, I think these changes are not needed any more?

Cheers,
Wei Chen

Quoted 1 line

Jan

#6 lore raw

Re: [PATCH v2 1/9] xen/x86: introduce a helper to update memory hotplug end

Quoted 26 lines · Wei Chen
Quoted 18 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月13日 16:46

Quoted 7 lines · Wei Chen
Quoted 6 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 19:54

Mechanically the change is fine. Albeit maybe "top" instead of "boundary", and maybe also pass "node" even if x86 doesn't need it?

Quoted 6 lines

Sorry, I am not very clear about these comments:
It makes sense to use mem_hotplug_update_top instead
of mem_hotplug_update_boundary. But how can I understand pass "node"? did you mean mem_hotplug_update_top(node, end)? But mem_hotplug is a global top for memory hotplug ranges. I don't think pass "node" to this function can be useful.

Quoted 7 lines

Please separate "current implementation" from "abstract model". In the latter it would seem quite reasonable to me to have per-node upper bounds (or even per-node ranges). Therefore adding node (and, on top of what I did suggest before, region start) to the parameters of the new per-arch hook would seem a good move to me, even if at present only / at most the "end" parameter would actually be used.

Quoted 2 lines

As we will export mem_hotplug to common, I think these changes are not needed any more?

Indeed. I merely wanted to address your question nevertheless, or in case there was still a reason to avoid making the global variable common.

Jan

author
#7 lore raw

PATCH v2 2/9 xen/x86: Use enumerations to indicate NUMA status

In reply to Wei Chen

In current code, x86 is using two variables, numa_off and acpi_numa, to indicate the NUMA status. This is because NUMA is not coupled with ACPI, and ACPI still can work without NUMA on x86. With these two variables' combinations, x86 can have several NUMA status:
NUMA swith on,
NUMA swith off,
NUMA swith on with NUMA emulation,
NUMA swith on with no-ACPI,
NUMA swith on with ACPI.

In this case, we introduce an enumeration numa_mode in this patch to indicate above NUMA status, except NUMA on with emulation. Because NUMA emulation has another variable, numa_fake, to indicate the number of nodes for emulation. We can't use the enumeration to replace it at the same time. But it still can be indicated by numa_on and numa_fake as what it has been indicated.

Based on the enumeration we introduce numa_enabled_with_firmware for callers to check NUMA status is enabled + ACPI. Using this helper is because some NUMA implementation will use other firmware, this helper will be easy to them to check enabled + others.

As we have touched srat_disabled, we have corrected its return value from int to bool.

---
v1 -> v2:
1. Remove fw_numa.
2. Use enumeration to replace numa_off and acpi_numa.
3. Correct return value of srat_disabled.
4. Introduce numa_enabled_with_firmware.

---
 xen/arch/x86/include/asm/acpi.h |  1 -
 xen/arch/x86/include/asm/numa.h | 16 +++++++++++++---
 xen/arch/x86/numa.c             | 28 +++++++++++++++-------------
 xen/arch/x86/setup.c            |  3 ++-
 xen/arch/x86/srat.c             | 13 +++++++------
 5 files changed, 37 insertions(+), 24 deletions(-)
xen/arch/x86/include/asm/acpi.h +0 -1
index 9a9cc4c240..ab0d56dd70 100644
@@ -101,7 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
 extern int acpi_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
xen/arch/x86/include/asm/numa.h +13 -3
index c32ccffde3..ee8262d969 100644
@@ -28,12 +28,22 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 #define VIRTUAL_BUG_ON(x) 
 
+/* Enumerations for NUMA status. */
+enum numa_mode {
+	numa_on = 0,
+	numa_off,
+	/* NUMA turns on, but ACPI table is bad or disabled. */
+	numa_no_acpi,
+	/* NUMA turns on, and ACPI table works well. */
+	numa_acpi,
+};
+
 extern void numa_add_cpu(int cpu);
 extern void numa_init_array(void);
-extern bool numa_off;
-
+extern bool numa_enabled_with_firmware(void);
+extern enum numa_mode numa_status;
 
-extern int srat_disabled(void);
+extern bool srat_disabled(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
xen/arch/x86/numa.c +15 -13
index 627ae8aa95..0777a7518d 100644
@@ -47,12 +47,16 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-bool numa_off;
-s8 acpi_numa = 0;
+enum numa_mode numa_status;
 
-int srat_disabled(void)
+bool srat_disabled(void)
 {
-    return numa_off || acpi_numa < 0;
+    return numa_status == numa_off || numa_status == numa_no_acpi;
+}
+
+bool __init numa_enabled_with_firmware(void)
+{
+    return numa_status == numa_acpi;
 }
 
 /*
@@ -254,12 +258,13 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
 #endif
 
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes(start, end) )
+    if ( numa_status != numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration found");
+           numa_status == numa_off ? "NUMA turned off"
+                                   : "No NUMA configuration found");
 
     printk(KERN_INFO "Faking a node at %"PRIpaddr"-%"PRIpaddr"\n",
            start, end);
@@ -292,13 +297,13 @@ void numa_set_node(int cpu, nodeid_t node)
 static int __init cf_check numa_setup(const char *opt)
 {
     if ( !strncmp(opt,"off",3) )
-        numa_off = true;
+        numa_status = numa_off;
     else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
+        numa_status = numa_on;
 #ifdef CONFIG_NUMA_EMU
     else if ( !strncmp(opt, "fake=", 5) )
     {
-        numa_off = false;
+        numa_status = numa_on;
         numa_fake = simple_strtoul(opt+5,NULL,0);
         if ( numa_fake >= MAX_NUMNODES )
             numa_fake = MAX_NUMNODES;
@@ -306,10 +311,7 @@ static int __init cf_check numa_setup(const char *opt)
 #endif
 #ifdef CONFIG_ACPI_NUMA
     else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        acpi_numa = -1;
-    }
+        numa_status = numa_no_acpi;
 #endif
     else
         return -EINVAL;
xen/arch/x86/setup.c +2 -1
index f08b07b8de..4841af5926 100644
@@ -329,7 +329,8 @@ void srat_detect_node(int cpu)
     node_set_online(node);
     numa_set_node(cpu, node);
 
-    if ( opt_cpu_info && acpi_numa > 0 )
+    /* Print CPU info when NUMA is enabled with ACPI. */
+    if ( opt_cpu_info && numa_enabled_with_firmware() )
         printk("CPU %d APIC %d -> Node %d\n", cpu, apicid, node);
 }
 
xen/arch/x86/srat.c +7 -6
index f53431f5e8..422e4c73e3 100644
@@ -185,7 +185,7 @@ static __init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
-	acpi_numa = -1;
+	numa_status = numa_no_acpi;
 	for (i = 0; i < MAX_LOCAL_APIC; i++)
 		apicid_to_node[i] = NUMA_NO_NODE;
 	for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
@@ -260,7 +260,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 
 	apicid_to_node[pa->apic_id] = node;
 	node_set(node, processor_nodes_parsed);
-	acpi_numa = 1;
+	numa_status = numa_acpi;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %08x -> Node %u\n",
@@ -295,7 +295,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 	}
 	apicid_to_node[pa->apic_id] = node;
 	node_set(node, processor_nodes_parsed);
-	acpi_numa = 1;
+	numa_status = numa_acpi;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %02x -> Node %u\n",
@@ -484,7 +484,7 @@ static int __init cf_check srat_parse_region(
 	    (ma->flags & ACPI_SRAT_MEM_NON_VOLATILE))
 		return 0;
 
-	if (numa_off)
+	if (numa_status == numa_off)
 		printk(KERN_INFO "SRAT: %013"PRIx64"-%013"PRIx64"\n",
 		       ma->base_address, ma->base_address + ma->length - 1);
 
@@ -499,7 +499,7 @@ void __init srat_parse_regions(paddr_t addr)
 	u64 mask;
 	unsigned int i;
 
-	if (acpi_disabled || acpi_numa < 0 ||
+	if (acpi_disabled || numa_status == numa_no_acpi ||
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
@@ -528,7 +528,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
-	if (acpi_numa <= 0)
+	/* Only when numa_on with good firmware, we can do numa scan nodes. */
+	if (!numa_enabled_with_firmware())
 		return -1;
 
 	if (!nodes_cover_memory()) {
Signature · 1 line
2.25.1
#8 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 9 lines · Wei Chen

In current code, x86 is using two variables, numa_off and acpi_numa, to indicate the NUMA status. This is because NUMA is not coupled with ACPI, and ACPI still can work without NUMA on x86. With these two variables' combinations, x86 can have several NUMA status:
NUMA swith on,
NUMA swith off,
NUMA swith on with NUMA emulation,
NUMA swith on with no-ACPI,
NUMA swith on with ACPI.

Hmm, with both this and the actual change I'm not able to convince myself that you've expressed the prior combinations correctly. May I suggest that you make table representing the 6 (I think) combinations of original states with their mapping to the new enumerators? (It doesn't need to be 6 different enumerators, but all 6 existing states need a [proper] representation in the new model.)

As an aside - I think you mean "switched" in all five of these lines.

xen/arch/x86/include/asm/numa.h extern nodeid_t pxm_to_node(unsigned int pxm); · line 34

Quoted 10 lines
xen/arch/x86/include/asm/numa.h +4 -0
@@ -28,12 +28,22 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 #define VIRTUAL_BUG_ON(x) 
 
+/* Enumerations for NUMA status. */
+enum numa_mode {
+	numa_on = 0,
+	numa_off,

May I suggest to switch these two around, such that "off" becomes the meaning of 0, potentially allowing ! to be used in a boolean- like fashion here or there? And please omit the "= 0" part - it's only non-zero first values which actually need spelling out.

xen/arch/x86/include/asm/numa.h extern nodeid_t pxm_to_node(unsigned int pxm); · line 38

Quoted 4 lines
xen/arch/x86/include/asm/numa.h +4 -0
+	/* NUMA turns on, but ACPI table is bad or disabled. */
+	numa_no_acpi,
+	/* NUMA turns on, and ACPI table works well. */
+	numa_acpi,

As to the names of these: In the description you already say that you want to re-use the code for non-ACPI cases. Wouldn't you better avoid "acpi" in the names then (rather than perhaps renaming these another time later on)?

I'd also like to understand what useful state "numa_no_acpi" is. I realize this was a state expressable by the two original variables, but does it make sense?

xen/arch/x86/include/asm/numa.h int __init acpi_scan_nodes(paddr_t start, paddr_t end) · line 533

Quoted 8 lines
xen/arch/x86/include/asm/numa.h +2 -1
@@ -528,7 +528,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
-	if (acpi_numa <= 0)
+	/* Only when numa_on with good firmware, we can do numa scan nodes. */
+	if (!numa_enabled_with_firmware())
 		return -1;

Nit: Perhaps drop "do numa" from the comment?

Jan

author
#9 lore raw

RE: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 21 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 20:49
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 9 lines · Wei Chen

In current code, x86 is using two variables, numa_off and acpi_numa, to indicate the NUMA status. This is because NUMA is not coupled with ACPI, and ACPI still can work without NUMA on x86. With these two variables' combinations, x86 can have several NUMA status:
NUMA swith on,
NUMA swith off,
NUMA swith on with NUMA emulation,
NUMA swith on with no-ACPI,
NUMA swith on with ACPI.

Quoted 8 lines

Hmm, with both this and the actual change I'm not able to convince myself that you've expressed the prior combinations correctly. May I suggest that you make table representing the 6 (I think) combinations of original states with their mapping to the new enumerators? (It doesn't need to be 6 different enumerators, but all 6 existing states need a [proper] representation in the new model.)

Sorry for replying later, I paid sometime to check the code again, and drew a table like below, I ignore two columns when numa_off=true and acpi_numa=1/-1. Because when numa_off = true, ACPI srat table will not be parsed:

+-----------+---------+---------------+-----------+------------+----------+
| original  |  col1   |  col2         |  col3     |  col4      |  col5    |
+-----------+---------+---------------+-----------+------------+----------+
|numa_off   |true     |false          |false      |false       |false     |
|acpi_numa  |0        |0              |1          |-1          |x         |
|numa_fake  |x        |x              |x          |x           |fake_nodes|
|enum state |numa_off |numa_fw_nodata |numa_fw_ok |numa_fw_bad |numa_emu  |
+-----------+---------+---------------+-----------+------------+----------+

Notes:
The following scenarios will make acpi_numa=0:
1. Xen disable ACPI through acpi_disabled = 1.
2. ACPI table doesn't have SRAT, or SRAT doesn't contain CPU and memory NUMA affinity information.
3. Emulate numa through "numa=fake" command line parameter. But I found that when numa_fake is enabled, current code will not prevent ACPI srat parsers to parse NUMA information. So acpi_numa still may be changed later. Is there any further reasons that we still need to parse NUMA info from SRAT table when numa=fake? Or can we set acpi_numa = -1 in nume_setup when "numa=fake" to make srat_disabled can prevent ACPI SRAT parsing.

If meet the following conditions, then acpi_numa = 1:
1. Xen enable ACPI through acpi_disabled = 0.
2. ACPI SRAT parser can parse CPU and Memory NUMA affinities successfully from SRAT table.
3. Pass the memmory blocks' sanity check and hash computing in acpi_scan_nodes.

The following scenarios will make acpi_numa=-1:
1. ACPI table is disabled by "numa=noacpi" command like parameter.
2. Xen enable ACPI through acpi_disabled = 0 but ACPI SRAT parser can not parse CPU or Memory NUMA affinities successfully from SRAT table.
3. The memmory blocks' sanity check or hash computing in acpi_scan_nodes is failed.

Quoted 3 lines

As an aside - I think you mean "switched" in all five of these lines.

Yes, I will fix them in next version.

Quoted 6 lines
Quoted 6 lines
xen/arch/x86/include/asm/numa.h +0 -0
@@ -28,12 +28,22 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 #define VIRTUAL_BUG_ON(x)
 
Quoted 10 lines

xen/arch/x86/include/asm/numa.h extern nodeid_t pxm_to_node(unsigned int pxm); · line 34

Quoted 4 lines
xen/arch/x86/include/asm/numa.h +4 -0
+/* Enumerations for NUMA status. */
+enum numa_mode {
+	numa_on = 0,
+	numa_off,

May I suggest to switch these two around, such that "off" becomes the meaning of 0, potentially allowing ! to be used in a boolean- like fashion here or there? And please omit the "= 0" part - it's only non-zero first values which actually need spelling out.

Ok.

Quoted 10 lines

xen/arch/x86/include/asm/numa.h extern nodeid_t pxm_to_node(unsigned int pxm); · line 38

Quoted 4 lines
xen/arch/x86/include/asm/numa.h +4 -0
+	/* NUMA turns on, but ACPI table is bad or disabled. */
+	numa_no_acpi,
+	/* NUMA turns on, and ACPI table works well. */
+	numa_acpi,

As to the names of these: In the description you already say that you want to re-use the code for non-ACPI cases. Wouldn't you better avoid "acpi" in the names then (rather than perhaps renaming these another time later on)?

Quoted 4 lines

I'd also like to understand what useful state "numa_no_acpi" is. I realize this was a state expressable by the two original variables, but does it make sense?

I have updated the new names in above table. And "numa_no_acpi" is mapping to " numa_fw_bad" enum state.

Quoted 5 lines

xen/arch/x86/include/asm/numa.h int __init acpi_scan_nodes(paddr_t start, paddr_t

Quoted 1 line
xen/arch/x86/include/asm/numa.h +0 -0
@@ -528,7 +528,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t

end)

Quoted 3 lines
xen/arch/x86/include/asm/numa.h +0 -0
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
Quoted 8 lines

xen/arch/x86/include/asm/numa.h int __init acpi_scan_nodes(paddr_t start, paddr_t · line 531

Quoted 2 lines
xen/arch/x86/include/asm/numa.h +1 -1
-	if (acpi_numa <= 0)
+	/* Only when numa_on with good firmware, we can do numa scan nodes.

*/

xen/arch/x86/include/asm/numa.h int __init acpi_scan_nodes(paddr_t start, paddr_t · line 533

Quoted 2 lines
xen/arch/x86/include/asm/numa.h +1 -0
+	if (!numa_enabled_with_firmware())
 		return -1;

Nit: Perhaps drop "do numa" from the comment?

Ok, I will do it in next version.

Cheers,
Wei Chen

Quoted 1 line

Jan

#10 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 32 lines · Wei Chen

Hi Jan,

Quoted 21 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 20:49
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 9 lines · Wei Chen

In current code, x86 is using two variables, numa_off and acpi_numa, to indicate the NUMA status. This is because NUMA is not coupled with ACPI, and ACPI still can work without NUMA on x86. With these two variables' combinations, x86 can have several NUMA status:
NUMA swith on,
NUMA swith off,
NUMA swith on with NUMA emulation,
NUMA swith on with no-ACPI,
NUMA swith on with ACPI.

Quoted 8 lines

Hmm, with both this and the actual change I'm not able to convince myself that you've expressed the prior combinations correctly. May I suggest that you make table representing the 6 (I think) combinations of original states with their mapping to the new enumerators? (It doesn't need to be 6 different enumerators, but all 6 existing states need a [proper] representation in the new model.)

Quoted 4 lines

Sorry for replying later, I paid sometime to check the code again, and drew a table like below, I ignore two columns when numa_off=true and acpi_numa=1/-1. Because when numa_off = true, ACPI srat table will not be parsed:

While I agree with this fact, the problem is that there are two independent command line options driving the then single variable. IOW numa_off and acpi_numa both turned on may still need a representation. In fact I'm not convinced we can eliminate the original variables. What we ought to be able to do is consolidate their values into the one single new variable you add, before ever evaluating anything. _Then_ I think I agree with the table.

Jan

Quoted 35 lines
+-----------+---------+---------------+-----------+------------+----------+
| original  |  col1   |  col2         |  col3     |  col4      |  col5    |
+-----------+---------+---------------+-----------+------------+----------+
|numa_off   |true     |false          |false      |false       |false     |
|acpi_numa  |0        |0              |1          |-1          |x         |
|numa_fake  |x        |x              |x          |x           |fake_nodes|
|enum state |numa_off |numa_fw_nodata |numa_fw_ok |numa_fw_bad |numa_emu  |
+-----------+---------+---------------+-----------+------------+----------+

Notes:
The following scenarios will make acpi_numa=0:
1. Xen disable ACPI through acpi_disabled = 1.
2. ACPI table doesn't have SRAT, or SRAT doesn't contain CPU and memory NUMA affinity information.
3. Emulate numa through "numa=fake" command line parameter. But I found that when numa_fake is enabled, current code will not prevent ACPI srat parsers to parse NUMA information. So acpi_numa still may be changed later. Is there any further reasons that we still need to parse NUMA info from SRAT table when numa=fake? Or can we set acpi_numa = -1 in nume_setup when "numa=fake" to make srat_disabled can prevent ACPI SRAT parsing.

If meet the following conditions, then acpi_numa = 1:
1. Xen enable ACPI through acpi_disabled = 0.
2. ACPI SRAT parser can parse CPU and Memory NUMA affinities successfully from SRAT table.
3. Pass the memmory blocks' sanity check and hash computing in acpi_scan_nodes.

The following scenarios will make acpi_numa=-1:
1. ACPI table is disabled by "numa=noacpi" command like parameter.
2. Xen enable ACPI through acpi_disabled = 0 but ACPI SRAT parser can not parse CPU or Memory NUMA affinities successfully from SRAT table.
3. The memmory blocks' sanity check or hash computing in acpi_scan_nodes is failed.

author
#11 lore raw

RE: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 26 lines

-----Original Message-----

Quoted 6 lines
Quoted 1 line

Sorry for replying later, I paid sometime to check the code again, and drew a table like below, I ignore two columns when numa_off=true and acpi_numa=1/-1. Because when numa_off = true, ACPI srat table will not be parsed:

While I agree with this fact, the problem is that there are two independent command line options driving the then single variable. IOW numa_off and acpi_numa both turned on may still need a representation. In fact I'm not convinced we can eliminate the original variables. What we ought to be able to do is consolidate their values into the one single new variable you add, before ever evaluating anything. _Then_ I think I agree with the table.

Jan

Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

| original | col1 | col2 | col3 | col4 | col5

|

Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

|numa_off |true |false |false |false |false

|

Quoted 9 lines
Quoted 1 line

|acpi_numa |0 |0 |1 |-1 |x

|

Quoted 1 line

|numa_fake |x |x |x |x

|fake_nodes|
> |enum state |numa_off |numa_fw_nodata |numa_fw_ok |numa_fw_bad |numa_emu
|
Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

How about update the table like this:

+------------+----------+----------------+----------------+------------+
|  original  |          |                |                |            |
+------------+----------+----------------+----------------+------------+
| numa_off   | true     | true           | true           | true       |
| acpi_numa  | 0        | 1              | -1             | x          |
| numa_fake  | x        | x              | x              | fake_nodes |
| enum state | numa_off | numa_off       | numa_off       | numa_off   |
+------------+----------+----------------+----------------+------------+
+------------+----------------+------------+-------------+------------+
|  original  |                |            |             |            |
+------------+----------------+------------+-------------+------------+
| numa_off   | false          | false      | false       | false      |
| acpi_numa  | 0              | 1          | -1          | x          |
| numa_fake  | x              | x          | x           | fake_nodes |
| enum state | numa_fw_nodata | numa_fw_ok | numa_fw_bad | numa_emu   |
+------------+----------------+------------+-------------+------------+

Cheers,
Wei Chen

Quoted 2 lines
Quoted 2 lines

Notes:
The following scenarios will make acpi_numa=0:

sanity check or hash computing in acpi_scan_nodes

Quoted 1 line
Quoted 1 line

is failed.

#12 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 48 lines · Wei Chen

Hi Jan,

Quoted 26 lines

-----Original Message-----

Quoted 6 lines
Quoted 1 line

Sorry for replying later, I paid sometime to check the code again, and drew a table like below, I ignore two columns when numa_off=true and acpi_numa=1/-1. Because when numa_off = true, ACPI srat table will not be parsed:

While I agree with this fact, the problem is that there are two independent command line options driving the then single variable. IOW numa_off and acpi_numa both turned on may still need a representation. In fact I'm not convinced we can eliminate the original variables. What we ought to be able to do is consolidate their values into the one single new variable you add, before ever evaluating anything. _Then_ I think I agree with the table.

Jan

Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

| original | col1 | col2 | col3 | col4 | col5

|

Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

|numa_off |true |false |false |false |false

|

Quoted 9 lines
Quoted 1 line

|acpi_numa |0 |0 |1 |-1 |x

|

Quoted 1 line

|numa_fake |x |x |x |x

|fake_nodes|
> |enum state |numa_off |numa_fw_nodata |numa_fw_ok |numa_fw_bad |numa_emu
|
Quoted 1 line

+-----------+---------+---------------+-----------+------------+--------

--+

Quoted 1 line

How about update the table like this:

+------------+----------+----------------+----------------+------------+
|  original  |          |                |                |            |
+------------+----------+----------------+----------------+------------+
| numa_off   | true     | true           | true           | true       |
| acpi_numa  | 0        | 1              | -1             | x          |
| numa_fake  | x        | x              | x              | fake_nodes |
| enum state | numa_off | numa_off       | numa_off       | numa_off   |
+------------+----------+----------------+----------------+------------+
Quoted 8 lines
+------------+----------------+------------+-------------+------------+
|  original  |                |            |             |            |
+------------+----------------+------------+-------------+------------+
| numa_off   | false          | false      | false       | false      |
| acpi_numa  | 0              | 1          | -1          | x          |
| numa_fake  | x              | x          | x           | fake_nodes |
| enum state | numa_fw_nodata | numa_fw_ok | numa_fw_bad | numa_emu   |
+------------+----------------+------------+-------------+------------+

Well, this makes the table complete, but it doesn't explain how you mean to fold the settings of the two command line options into one variable.

Jan

author
#13 lore raw

RE: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 30 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 17:58
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 11 lines

How about update the table like this:

+------------+----------+----------------+----------------+------------+
|  original  |          |                |                |            |
+------------+----------+----------------+----------------+------------+
| numa_off   | true     | true           | true           | true       |
| acpi_numa  | 0        | 1              | -1             | x          |
| numa_fake  | x        | x              | x              | fake_nodes |
| enum state | numa_off | numa_off       | numa_off       | numa_off   |
+------------+----------+----------------+----------------+------------+
Quoted 8 lines
+------------+----------------+------------+-------------+------------+
|  original  |                |            |             |            |
+------------+----------------+------------+-------------+------------+
| numa_off   | false          | false      | false       | false      |
| acpi_numa  | 0              | 1          | -1          | x          |
| numa_fake  | x              | x          | x           | fake_nodes |
| enum state | numa_fw_nodata | numa_fw_ok | numa_fw_bad | numa_emu   |
+------------+----------------+------------+-------------+------------+
Quoted 3 lines

Well, this makes the table complete, but it doesn't explain how you mean to fold the settings of the two command line options into one variable.

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Thanks,
Wei Chen

Quoted 1 line

Jan

#14 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 32 lines · Wei Chen

Hi Jan,

Quoted 30 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 17:58
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 19 lines

How about update the table like this:

+------------+----------+----------------+----------------+------------+
|  original  |          |                |                |            |
+------------+----------+----------------+----------------+------------+
| numa_off   | true     | true           | true           | true       |
| acpi_numa  | 0        | 1              | -1             | x          |
| numa_fake  | x        | x              | x              | fake_nodes |
| enum state | numa_off | numa_off       | numa_off       | numa_off   |
+------------+----------+----------------+----------------+------------+
+------------+----------------+------------+-------------+------------+
|  original  |                |            |             |            |
+------------+----------------+------------+-------------+------------+
| numa_off   | false          | false      | false       | false      |
| acpi_numa  | 0              | 1          | -1          | x          |
| numa_fake  | x              | x          | x           | fake_nodes |
| enum state | numa_fw_nodata | numa_fw_ok | numa_fw_bad | numa_emu   |
+------------+----------------+------------+-------------+------------+
Quoted 8 lines
Quoted 3 lines

Well, this makes the table complete, but it doesn't explain how you mean to fold the settings of the two command line options into one variable.

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't convince myself that the new mapping fully represents all originally possible combinations (nor can I convince myself that in the existing code everything is working as intended).

Jan

author
#15 lore raw

RE: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 20 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 18:37
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 2 lines
Quoted 2 lines

Well, this makes the table complete, but it doesn't explain how you

mean

Quoted 7 lines
Quoted 2 lines

to fold the settings of the two command line options into one variable.

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Quoted 4 lines

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't

Can you provide some examples? This way I can better understand your concerns.

Cheers,
Wei Chen

Quoted 5 lines

convince myself that the new mapping fully represents all originally possible combinations (nor can I convince myself that in the existing code everything is working as intended).

Jan

#16 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 18:37
status

Quoted 2 lines
Quoted 2 lines

Well, this makes the table complete, but it doesn't explain how you

mean

Quoted 7 lines
Quoted 2 lines

to fold the settings of the two command line options into one variable.

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Quoted 7 lines
Quoted 4 lines

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't

Can you provide some examples? This way I can better understand your concerns.

Take bad_srat(): you convert "acpi_numa = -1" to setting numa_no_acpi. Yet imo (matching the present model) numa_off shouldn't be affected. While your change is fine in practice for (current) x86, it is wrong in the abstract model (which is relevant when making things common).

Jan

author
#17 lore raw

RE: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 31 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 18:58
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; xen- [email protected]
Subject: Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Quoted 7 lines · Wei Chen
Quoted 7 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 18:37
status

Quoted 2 lines
Quoted 2 lines

Well, this makes the table complete, but it doesn't explain how you

mean

Quoted 1 line
Quoted 1 line

to fold the settings of the two command line options into one

variable.

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Quoted 1 line

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't

Quoted 8 lines
Quoted 2 lines

Can you provide some examples? This way I can better understand your concerns.

Take bad_srat(): you convert "acpi_numa = -1" to setting numa_no_acpi. Yet imo (matching the present model) numa_off shouldn't be affected. While your change is fine in practice for (current) x86, it is wrong in the abstract model (which is relevant when making things common).

Thanks, I understand your concerns now. In this case, I agree with your suggestion in previous e-mail:

Quoted 6 lines

int numa_disabled(void)

{
   return numa_off || arch_numa_disabled();
}

with arch_numa_disabled() evaluating acpi_numa on x86.

I would update this patch like above sample in next version. And in this way, I think We don't need the new enumerations and mapping table.

Cheers,
Wei Chen

Quoted 1 line

Jan

#18 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

In reply to Jan Beulich
Quoted 33 lines · Jan Beulich
Quoted 23 lines · Wei Chen
Quoted 23 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月14日 17:58
To: Wei Chen <[email protected]>

Quoted 19 lines

How about update the table like this:

+------------+----------+----------------+----------------+------------+
|  original  |          |                |                |            |
+------------+----------+----------------+----------------+------------+
| numa_off   | true     | true           | true           | true       |
| acpi_numa  | 0        | 1              | -1             | x          |
| numa_fake  | x        | x              | x              | fake_nodes |
| enum state | numa_off | numa_off       | numa_off       | numa_off   |
+------------+----------+----------------+----------------+------------+
+------------+----------------+------------+-------------+------------+
|  original  |                |            |             |            |
+------------+----------------+------------+-------------+------------+
| numa_off   | false          | false      | false       | false      |
| acpi_numa  | 0              | 1          | -1          | x          |
| numa_fake  | x              | x          | x           | fake_nodes |
| enum state | numa_fw_nodata | numa_fw_ok | numa_fw_bad | numa_emu   |
+------------+----------------+------------+-------------+------------+
Quoted 8 lines
Quoted 3 lines

Well, this makes the table complete, but it doesn't explain how you mean to fold the settings of the two command line options into one variable.

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Quoted 7 lines

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't convince myself that the new mapping fully represents all originally possible combinations (nor can I convince myself that in the existing code everything is working as intended).

Maybe the solution is to make numa_off common but keep acpi_numa arch-specific? Then e.g. the replacement of srat_disabled() could be

int numa_disabled(void)

{
    return numa_off || arch_numa_disabled();
}

with arch_numa_disabled() evaluating acpi_numa on x86.

Jan

author
#19 lore raw

Re: [PATCH v2 2/9] xen/x86: Use enumerations to indicate NUMA status

Hi Jan,

Quoted 18 lines · Jan Beulich
Quoted 6 lines
Quoted 5 lines

No matter how many separate "numa=" parameters have been parsed from Command line, the values of these original variables are determined after parsing the command line. So the determined status can be mapped to the new one variable from above table.

Quoted 7 lines

Hmm, I was partly wrong - the initial values of both variables are indeed set from just the single "numa=" parsing. But later on they "evolve" independently, and multiple "numa=" on the command line can also have "interesting" effects. I'm afraid I still can't convince myself that the new mapping fully represents all originally possible combinations (nor can I convince myself that in the existing code everything is working as intended).

Maybe the solution is to make numa_off common but keep acpi_numa arch-specific? Then e.g. the replacement of srat_disabled() could be

Quoted 7 lines

int numa_disabled(void)

{
     return numa_off || arch_numa_disabled();
}

with arch_numa_disabled() evaluating acpi_numa on x86.

While I am working on the new version, I think this may be not enough, we may need another helper like arch_handle_numa_param to prevent direct accessing of acpi_numa in numa_setup.

Cheers,
Wei Chen

Quoted 1 line

Jan

author
#20 lore raw

PATCH v2 3/9 xen/x86: move generically usable NUMA code from x86 to common

In reply to Wei Chen

There are some codes in x86/numa.c can be shared by architectures to implement NUMA support. Just like some variables and functions to check and store NUMA memory map. And some variables and functions to do NUMA initialization.

In this patch, we move them to common/numa.c and xen/numa.h and use the CONFIG_NUMA to gate them for non-NUMA supported architectures. As the target header file is Xen-style, so we trim some spaces and replace tabs for the codes that has been moved to xen/numa.h at the same time.

As we have moved phy_to_nid to xen/numa.h, we don't need a separate MAX_NUMNODES in asm/numa.h. Now all architectures can use the same MAX_NUMNODES in xen/numa.h. The conditional macro can be removed.

---
v1 -> v2:
1. New patch in v2.

---
 xen/arch/x86/include/asm/numa.h  |  64 -----
 xen/arch/x86/include/asm/setup.h |   1 -
 xen/arch/x86/numa.c              | 431 +-----------------------------
 xen/common/Makefile              |   1 +
 xen/common/numa.c                | 439 +++++++++++++++++++++++++++++++
 xen/include/xen/numa.h           |  74 ++++++
 6 files changed, 515 insertions(+), 495 deletions(-)
 create mode 100644 xen/common/numa.c
xen/arch/x86/include/asm/numa.h +0 -64
index ee8262d969..d8960743d4 100644
@@ -9,81 +9,17 @@ typedef u8 nodeid_t;
 
 extern int srat_rev;
 
-extern nodeid_t      cpu_to_node[NR_CPUS];
-extern cpumask_t     node_to_cpumask[];
-
-#define cpu_to_node(cpu)		(cpu_to_node[cpu])
-#define parent_node(node)		(node)
-#define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
-#define node_to_cpumask(node)    (node_to_cpumask[node])
-
-struct node { 
-	paddr_t start, end;
-};
-
-extern int compute_hash_shift(struct node *nodes, int numnodes,
-			      nodeid_t *nodeids);
 extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-#define VIRTUAL_BUG_ON(x) 
-
-/* Enumerations for NUMA status. */
-enum numa_mode {
-	numa_on = 0,
-	numa_off,
-	/* NUMA turns on, but ACPI table is bad or disabled. */
-	numa_no_acpi,
-	/* NUMA turns on, and ACPI table works well. */
-	numa_acpi,
-};
-
-extern void numa_add_cpu(int cpu);
-extern void numa_init_array(void);
-extern bool numa_enabled_with_firmware(void);
-extern enum numa_mode numa_status;
 
 extern bool srat_disabled(void);
-extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
-static inline void clear_node_cpumask(int cpu)
-{
-	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-}
-
-/* Simple perfect hash to map pdx to node numbers */
-extern int memnode_shift; 
-extern unsigned long memnodemapsize;
-extern u8 *memnodemap;
-
-struct node_data {
-    unsigned long node_start_pfn;
-    unsigned long node_spanned_pages;
-};
-
-extern struct node_data node_data[];
-
-static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
-{ 
-	nodeid_t nid;
-	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
-	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift]; 
-	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]); 
-	return nid; 
-} 
-
-#define NODE_DATA(nid)		(&(node_data[nid]))
-
-#define node_start_pfn(nid)	(NODE_DATA(nid)->node_start_pfn)
-#define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
-#define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
-				 NODE_DATA(nid)->node_spanned_pages)
 #define arch_want_default_dmazone() (num_online_nodes() > 1)
 
 extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
xen/arch/x86/include/asm/setup.h +0 -1
index 21037b7f31..ae470ea12f 100644
@@ -20,7 +20,6 @@ void early_time_init(void);
 
 void set_nr_cpu_ids(unsigned int max_cpus);
 
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 void arch_init_memory(void);
 void subarch_init_memory(void);
 
xen/arch/x86/numa.c +1 -430

462 lines not on this page · raw message

xen/common/Makefile +1 -0
index 3baf83d527..9a3a12b12d 100644
@@ -26,6 +26,7 @@ obj-$(CONFIG_MEM_ACCESS) += mem_access.o
 obj-y += memory.o
 obj-y += multicall.o
 obj-y += notifier.o
+obj-$(CONFIG_NUMA) += numa.o
 obj-y += page_alloc.o
 obj-$(CONFIG_HAS_PDX) += pdx.o
 obj-$(CONFIG_PERF_COUNTERS) += perfc.o
xen/common/numa.c new file +439 -0

439 lines not on this page · raw message

xen/include/xen/numa.h +74 -0

78 lines not on this page · raw message

Signature · 1 line
2.25.1
#21 lore raw

Re: [PATCH v2 3/9] xen/x86: move generically usable NUMA code from x86 to common

Quoted 8 lines · Wei Chen
xen/common/numa.c new file +5 -0
@@ -0,0 +1,439 @@
+/*
+ * Generic VM initialization for NUMA setups.
+ * Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Adapted for Xen: Ryan Harper <[email protected]>
+ */

xen/common/numa.c line 15

Quoted 10 lines
xen/common/numa.c new file +10 -0
+
+#include <xen/init.h>
+#include <xen/keyhandler.h>
+#include <xen/mm.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/param.h>
+#include <xen/sched.h>
+#include <xen/softirq.h>
+#include <asm/acpi.h>

Isn't the goal for the now common code to not be dependent upon ACPI?

xen/common/numa.c line 23

Quoted 7 lines
xen/common/numa.c new file +7 -0
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+int memnode_shift;
+static typeof(*memnodemap) _memnodemap[64];
+unsigned long memnodemapsize;
+u8 *memnodemap;

For code moved, please switch to (in this case) uint8_t. I'm on the edge of asking to go further and
- also use __read_mostly for some / all of these,
- make memnode_shift unsigned int (sadly this looks to require more adjustments, even if negative shift counts are meaningless),
- convert from plain int to unsigned int elsewhere as appropriate,
- add const where appropriate / possible.

xen/common/numa.c line 29

Quoted 5 lines
xen/common/numa.c new file +5 -0
+nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
+    [0 ... NR_CPUS-1] = NUMA_NO_NODE
+};
+
+cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;

For these two please put __read_mostly into its more conventional place (right after the type).

xen/common/numa.c line 33

Quoted 3 lines
xen/common/numa.c new file +3 -0
+nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
+
+enum numa_mode numa_status;

This should probably have been __read_mostly already in the earlier patch introducing it.

xen/common/numa.c line 176

Quoted 2 lines
xen/common/numa.c new file +2 -0
+#ifdef CONFIG_NUMA_EMU
+static int numa_fake __initdata = 0;

The __initdata again wants to move, and the initializer can be omitted.

xen/common/numa.c line 275

Quoted 4 lines
xen/common/numa.c new file +4 -0
+/* [numa=off] */
+static int __init cf_check numa_setup(const char *opt)
+{
+    if ( !strncmp(opt,"off",3) )

As you're correcting style violations elsewhere, please also insert the missing spaces here and below.

Quoted 53 lines
xen/include/xen/numa.h +47 -0
@@ -18,4 +18,78 @@
   (((d)->vcpu != NULL && (d)->vcpu[0] != NULL) \
    ? vcpu_to_node((d)->vcpu[0]) : NUMA_NO_NODE)
 
+/* The following content can be used when NUMA feature is enabled */
+#ifdef CONFIG_NUMA
+
+extern nodeid_t      cpu_to_node[NR_CPUS];
+extern cpumask_t     node_to_cpumask[];
+
+#define cpu_to_node(cpu)        (cpu_to_node[cpu])
+#define parent_node(node)       (node)
+#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
+#define node_to_cpumask(node)   (node_to_cpumask[node])
+
+struct node {
+    paddr_t start, end;
+};
+
+extern int compute_hash_shift(struct node *nodes, int numnodes,
+                              nodeid_t *nodeids);
+
+#define VIRTUAL_BUG_ON(x)
+
+/* Enumerations for NUMA status. */
+enum numa_mode {
+    numa_on = 0,
+    numa_off,
+    /* NUMA turns on, but ACPI table is bad or disabled. */
+    numa_no_acpi,
+    /* NUMA turns on, and ACPI table works well. */
+    numa_acpi,
+};
+
+extern void numa_add_cpu(int cpu);
+extern void numa_init_array(void);
+extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+extern bool numa_enabled_with_firmware(void);
+extern enum numa_mode numa_status;
+
+extern void numa_set_node(int cpu, nodeid_t node);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
+
+static inline void clear_node_cpumask(int cpu)
+{
+    cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+}
+
+/* Simple perfect hash to map pdx to node numbers */
+extern int memnode_shift;
+extern unsigned long memnodemapsize;

xen/include/xen/numa.h line 77

Quoted 10 lines
xen/include/xen/numa.h +10 -0
+extern u8 *memnodemap;
+
+struct node_data {
+    unsigned long node_start_pfn;
+    unsigned long node_spanned_pages;
+};
+
+extern struct node_data node_data[];
+
+static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)

Please use __attribute_pure__.

Jan

author
#22 lore raw

RE: [PATCH v2 3/9] xen/x86: move generically usable NUMA code from x86 to common

Hi Jan,

Quoted 24 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 21:13
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 3/9] xen/x86: move generically usable NUMA code from x86 to common

Quoted 12 lines · Wei Chen
xen/common/numa.c new file +9 -0
@@ -0,0 +1,439 @@
+/*
+ * Generic VM initialization for NUMA setups.
+ * Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Adapted for Xen: Ryan Harper <[email protected]>
+ */
+
+#include <xen/init.h>
+#include <xen/keyhandler.h>
+#include <xen/mm.h>
Quoted 9 lines

xen/common/numa.c line 15

Quoted 6 lines
xen/common/numa.c new file +6 -0
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/param.h>
+#include <xen/sched.h>
+#include <xen/softirq.h>
+#include <asm/acpi.h>

Isn't the goal for the now common code to not be dependent upon ACPI?

Ah, good catch, current common code should be ACPI decoupled. I will remove "#include <asm/acpi.h>"

Thanks,
Wei Chen

Quoted 8 lines

xen/common/numa.c line 23

Quoted 7 lines
xen/common/numa.c new file +7 -0
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+int memnode_shift;
+static typeof(*memnodemap) _memnodemap[64];
+unsigned long memnodemapsize;
+u8 *memnodemap;
Quoted 8 lines

For code moved, please switch to (in this case) uint8_t. I'm on the edge of asking to go further and
- also use __read_mostly for some / all of these,
- make memnode_shift unsigned int (sadly this looks to require more adjustments, even if negative shift counts are meaningless),
- convert from plain int to unsigned int elsewhere as appropriate,
- add const where appropriate / possible.

Ok, I will address them in next version.

Quoted 9 lines

xen/common/numa.c line 29

Quoted 5 lines
xen/common/numa.c new file +5 -0
+nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
+    [0 ... NR_CPUS-1] = NUMA_NO_NODE
+};
+
+cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;

For these two please put __read_mostly into its more conventional place (right after the type).

Ok.

Quoted 7 lines

xen/common/numa.c line 33

Quoted 3 lines
xen/common/numa.c new file +3 -0
+nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
+
+enum numa_mode numa_status;

This should probably have been __read_mostly already in the earlier patch introducing it.

Ok.

Quoted 5 lines

xen/common/numa.c line 176

Quoted 2 lines
xen/common/numa.c new file +2 -0
+#ifdef CONFIG_NUMA_EMU
+static int numa_fake __initdata = 0;

The __initdata again wants to move, and the initializer can be omitted.

Ok.

Quoted 8 lines

xen/common/numa.c line 275

Quoted 4 lines
xen/common/numa.c new file +4 -0
+/* [numa=off] */
+static int __init cf_check numa_setup(const char *opt)
+{
+    if ( !strncmp(opt,"off",3) )

As you're correcting style violations elsewhere, please also insert the missing spaces here and below.

Ack.

Quoted 59 lines
Quoted 29 lines
xen/include/xen/numa.h +23 -0
@@ -18,4 +18,78 @@
   (((d)->vcpu != NULL && (d)->vcpu[0] != NULL) \
    ? vcpu_to_node((d)->vcpu[0]) : NUMA_NO_NODE)
 
+/* The following content can be used when NUMA feature is enabled */
+#ifdef CONFIG_NUMA
+
+extern nodeid_t      cpu_to_node[NR_CPUS];
+extern cpumask_t     node_to_cpumask[];
+
+#define cpu_to_node(cpu)        (cpu_to_node[cpu])
+#define parent_node(node)       (node)
+#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
+#define node_to_cpumask(node)   (node_to_cpumask[node])
+
+struct node {
+    paddr_t start, end;
+};
+
+extern int compute_hash_shift(struct node *nodes, int numnodes,
+                              nodeid_t *nodeids);
+
+#define VIRTUAL_BUG_ON(x)
+
+/* Enumerations for NUMA status. */
+enum numa_mode {
+    numa_on = 0,

xen/include/xen/numa.h line 53

Quoted 10 lines
xen/include/xen/numa.h +10 -0
+    numa_off,
+    /* NUMA turns on, but ACPI table is bad or disabled. */
+    numa_no_acpi,
+    /* NUMA turns on, and ACPI table works well. */
+    numa_acpi,
+};
+
+extern void numa_add_cpu(int cpu);
+extern void numa_init_array(void);
+extern void numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn);

xen/include/xen/numa.h

Quoted 5 lines
xen/include/xen/numa.h +5 -0
+extern bool numa_enabled_with_firmware(void);
+extern enum numa_mode numa_status;
+
+extern void numa_set_node(int cpu, nodeid_t node);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t

end);

Quoted 13 lines
xen/include/xen/numa.h +13 -0
+
+static inline void clear_node_cpumask(int cpu)
+{
+    cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+}
+
+/* Simple perfect hash to map pdx to node numbers */
+extern int memnode_shift;
+extern unsigned long memnodemapsize;
+extern u8 *memnodemap;
+
+struct node_data {
+    unsigned long node_start_pfn;
Quoted 9 lines

xen/include/xen/numa.h line 77

Quoted 6 lines
xen/include/xen/numa.h +6 -0
+    unsigned long node_spanned_pages;
+};
+
+extern struct node_data node_data[];
+
+static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)

Please use __attribute_pure__.

Ack.

Quoted 1 line

Jan

#23 lore raw

Re: [PATCH v2 3/9] xen/x86: move generically usable NUMA code from x86 to common

In reply to Jan Beulich
Quoted 22 lines · Jan Beulich
Quoted 8 lines · Wei Chen
xen/common/numa.c new file +5 -0
@@ -0,0 +1,439 @@
+/*
+ * Generic VM initialization for NUMA setups.
+ * Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Adapted for Xen: Ryan Harper <[email protected]>
+ */

xen/common/numa.c line 15

Quoted 10 lines
xen/common/numa.c new file +10 -0
+
+#include <xen/init.h>
+#include <xen/keyhandler.h>
+#include <xen/mm.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/param.h>
+#include <xen/sched.h>
+#include <xen/softirq.h>
+#include <asm/acpi.h>

Isn't the goal for the now common code to not be dependent upon ACPI?

Quoted 11 lines

xen/common/numa.c line 23

Quoted 7 lines
xen/common/numa.c new file +7 -0
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+int memnode_shift;
+static typeof(*memnodemap) _memnodemap[64];
+unsigned long memnodemapsize;
+u8 *memnodemap;

For code moved, please switch to (in this case) uint8_t. I'm on the edge of asking to go further and
- also use __read_mostly for some / all of these,

Actually where possible please prefer __ro_after_init over __read_mostly. While this isn't properly enabled on Arm yet, it should at least not cause problems, as the result section is covered by .data.* in the linker script.

Jan

author
#24 lore raw

PATCH v2 4/9 xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

In reply to Wei Chen

VIRTUAL_BUG_ON is an empty macro used in phys_to_nid. This results in two lines of error-checking code in phys_to_nid that is not actually working and causing two compilation errors:
1. error: "MAX_NUMNODES" undeclared (first use in this function). This is because in the common header file, "MAX_NUMNODES" is defined after the common header file includes the ARCH header file, where phys_to_nid has attempted to use "MAX_NUMNODES". This error was resolved after we moved the phys_to_nid from x86 ARCH header file to common header file.
2. error: wrong type argument to unary exclamation mark. This is because, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT instead of VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages. Although NUMA allows one node can only have CPUs but without any memory. And node with 0 bytes of memory might have an entry in memnodemap[] theoretically. But that doesn't mean phys_to_nid can find any valid address from a node with 0 bytes memory.

---
v1 -> v2:
1. Move from part#1 to part#2. (Comment from NUMA part1 series)
2. Refine the justification of using !node_data[nid].node_spanned_pages. (From NUMA part1 series)
3. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
4. Adjust the conditional express for ASSERT.
5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
6. Use conditional macro to gate MAX_NUMNODES for other architectures.

---
 xen/include/xen/numa.h | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
xen/include/xen/numa.h +2 -4
index db0e524a0e..695ad51df0 100644
@@ -36,8 +36,6 @@ struct node {
 extern int compute_hash_shift(struct node *nodes, int numnodes,
                               nodeid_t *nodeids);
 
-#define VIRTUAL_BUG_ON(x)
-
 /* Enumerations for NUMA status. */
 enum numa_mode {
     numa_on = 0,
@@ -77,9 +75,9 @@ extern struct node_data node_data[];
 static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 {
     nodeid_t nid;
-    VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
+    ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
     nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
-    VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+    ASSERT(nid < MAX_NUMNODES && node_data[nid].node_spanned_pages);
     return nid;
 }
 
Signature · 1 line
2.25.1
#25 lore raw

Re: [PATCH v2 4/9] xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

Quoted 25 lines · Wei Chen

VIRTUAL_BUG_ON is an empty macro used in phys_to_nid. This results in two lines of error-checking code in phys_to_nid that is not actually working and causing two compilation errors:
1. error: "MAX_NUMNODES" undeclared (first use in this function). This is because in the common header file, "MAX_NUMNODES" is defined after the common header file includes the ARCH header file, where phys_to_nid has attempted to use "MAX_NUMNODES". This error was resolved after we moved the phys_to_nid from x86 ARCH header file to common header file.
2. error: wrong type argument to unary exclamation mark. This is because, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT instead of VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages. Although NUMA allows one node can only have CPUs but without any memory. And node with 0 bytes of memory might have an entry in memnodemap[] theoretically. But that doesn't mean phys_to_nid can find any valid address from a node with 0 bytes memory.

Quoted 9 lines

---
v1 -> v2:
1. Move from part#1 to part#2. (Comment from NUMA part1 series)
2. Refine the justification of using !node_data[nid].node_spanned_pages. (From NUMA part1 series)
3. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
4. Adjust the conditional express for ASSERT.
5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
6. Use conditional macro to gate MAX_NUMNODES for other architectures.

The change looks okay, but can you please clarify what these last two points describe? They don't seem to match any change ...

Quoted 13 lines
xen/include/xen/numa.h +0 -2
@@ -36,8 +36,6 @@ struct node {
 extern int compute_hash_shift(struct node *nodes, int numnodes,
                               nodeid_t *nodeids);
 
-#define VIRTUAL_BUG_ON(x)
-
 /* Enumerations for NUMA status. */
 enum numa_mode {
     numa_on = 0,
@@ -77,9 +75,9 @@ extern struct node_data node_data[];
 static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)

xen/include/xen/numa.h extern struct node_data node_data[]; · line 83

Quoted 10 lines
xen/include/xen/numa.h +2 -2
 {
     nodeid_t nid;
-    VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
+    ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
     nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
-    VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+    ASSERT(nid < MAX_NUMNODES && node_data[nid].node_spanned_pages);
     return nid;
 }
 

... in the entire patch.

Jan

author
#26 lore raw

RE: [PATCH v2 4/9] xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

Hi Jan,

Quoted 47 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月11日 14:32
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Jiamei Xie <[email protected]>; [email protected]
Subject: Re: [PATCH v2 4/9] xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

Quoted 25 lines · Wei Chen

VIRTUAL_BUG_ON is an empty macro used in phys_to_nid. This results in two lines of error-checking code in phys_to_nid that is not actually working and causing two compilation errors:
1. error: "MAX_NUMNODES" undeclared (first use in this function). This is because in the common header file, "MAX_NUMNODES" is defined after the common header file includes the ARCH header file, where phys_to_nid has attempted to use "MAX_NUMNODES". This error was resolved after we moved the phys_to_nid from x86 ARCH header file to common header file.
2. error: wrong type argument to unary exclamation mark. This is because, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT instead of VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages. Although NUMA allows one node can only have CPUs but without any memory. And node with 0 bytes of memory might have an entry in memnodemap[] theoretically. But that doesn't mean phys_to_nid can find any valid address from a node with 0 bytes memory.

Quoted 9 lines

---
v1 -> v2:
1. Move from part#1 to part#2. (Comment from NUMA part1 series)
2. Refine the justification of using !node_data[nid].node_spanned_pages. (From NUMA part1 series)
3. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
4. Adjust the conditional express for ASSERT.
5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
6. Use conditional macro to gate MAX_NUMNODES for other architectures.

Quoted 3 lines

The change looks okay, but can you please clarify what these last two points describe? They don't seem to match any change ...

I moved this patch form Part#1 to Part#2, but forgot to remove these two change logs. In Part#1, we do those changes, but after we moved this patch to Part#2, those changes are unnecessary. I will remove these two lines of change logs. Sorry for the confusion!

Cheers,
Wei Chen

Quoted 28 lines
Quoted 12 lines
xen/include/xen/numa.h +0 -2
@@ -36,8 +36,6 @@ struct node {
 extern int compute_hash_shift(struct node *nodes, int numnodes,
                               nodeid_t *nodeids);
 
-#define VIRTUAL_BUG_ON(x)
-
 /* Enumerations for NUMA status. */
 enum numa_mode {
     numa_on = 0,

xen/include/xen/numa.h extern struct node_data node_data[]; · line 80

Quoted 4 lines
xen/include/xen/numa.h +0 -1
@@ -77,9 +75,9 @@ extern struct node_data node_data[];
 static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 {
     nodeid_t nid;
-    VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >=

memnodemapsize);

xen/include/xen/numa.h extern struct node_data node_data[]; · line 83

Quoted 7 lines
xen/include/xen/numa.h +2 -1
+    ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
     nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
-    VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+    ASSERT(nid < MAX_NUMNODES && node_data[nid].node_spanned_pages);
     return nid;
 }
 

... in the entire patch.

Jan

Acked-by
#27 lore raw

Re: [PATCH v2 4/9] xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

Quoted 49 lines · Wei Chen

Hi Jan,

Quoted 47 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月11日 14:32
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Jiamei Xie <[email protected]>; [email protected]
Subject: Re: [PATCH v2 4/9] xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid

Quoted 34 lines · Wei Chen

VIRTUAL_BUG_ON is an empty macro used in phys_to_nid. This results in two lines of error-checking code in phys_to_nid that is not actually working and causing two compilation errors:
1. error: "MAX_NUMNODES" undeclared (first use in this function). This is because in the common header file, "MAX_NUMNODES" is defined after the common header file includes the ARCH header file, where phys_to_nid has attempted to use "MAX_NUMNODES". This error was resolved after we moved the phys_to_nid from x86 ARCH header file to common header file.
2. error: wrong type argument to unary exclamation mark. This is because, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT instead of VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages. Although NUMA allows one node can only have CPUs but without any memory. And node with 0 bytes of memory might have an entry in memnodemap[] theoretically. But that doesn't mean phys_to_nid can find any valid address from a node with 0 bytes memory.

---
v1 -> v2:
1. Move from part#1 to part#2. (Comment from NUMA part1 series)
2. Refine the justification of using !node_data[nid].node_spanned_pages. (From NUMA part1 series)
3. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
4. Adjust the conditional express for ASSERT.
5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
6. Use conditional macro to gate MAX_NUMNODES for other architectures.

Quoted 8 lines
Quoted 3 lines

The change looks okay, but can you please clarify what these last two points describe? They don't seem to match any change ...

I moved this patch form Part#1 to Part#2, but forgot to remove these two change logs. In Part#1, we do those changes, but after we moved this patch to Part#2, those changes are unnecessary. I will remove these two lines of change logs. Sorry for the confusion!

With this clarified

Jan

author
#28 lore raw

PATCH v2 5/9 xen/x86: use arch_get_memory_map to get information from E820 map

In reply to Wei Chen

The sanity check performed by the nodes_cover_memory function is also necessary for other architectures that do not yet support NUMA. But now, the code of nodes_cover_memory is tied to the x86 E820. In this case, we introduce arch_get_memory_map to decouple architecture specific memory map from this function. This means, other architectures like Arm can also use it to check its bootmem info.

Depending on arch_get_memory_map, we make nodes_cover_memory become architecture independent. We also use neutral words to replace SRAT and E820 in the print message of this function. This will make the message more common.

As arch_get_memory_map use unsigned int for index, we also adjust the index in nodes_cover_memory from int to unsigned int.

---
v1 -> v2:
1. Use arch_get_memory_map to replace arch_get_memory_bank_range and arch_get_memory_bank_number.
2. Remove the !start || !end check, because caller guarantee these two pointers will not be NULL.

---
 xen/arch/x86/numa.c    | 23 +++++++++++++++++++++++
 xen/arch/x86/srat.c    | 18 +++++++++++-------
 xen/include/xen/numa.h |  3 +++
 3 files changed, 37 insertions(+), 7 deletions(-)
xen/arch/x86/numa.c +23 -0
index 193314dbd9..fb235c07ec 100644
@@ -9,6 +9,7 @@
 #include <xen/nodemask.h>
 #include <xen/numa.h>
 #include <asm/acpi.h>
+#include <asm/e820.h>
 
 #ifndef Dprintk
 #define Dprintk(x...)
@@ -82,3 +83,25 @@ unsigned int __init arch_get_dma_bitsize(void)
                  flsl(node_start_pfn(node) + node_spanned_pages(node) / 4 - 1)
                  + PAGE_SHIFT, 32);
 }
+
+/*
+ * This function provides the ability for caller to get one RAM entry
+ * from architectural memory map by index.
+ *
+ * This function will return zero if it can return a proper RAM entry.
+ * otherwise it will return -ENODEV for out of scope index, or return
+ * -EINVAL for non-RAM type memory entry.
+ */
+int __init arch_get_memory_map(unsigned int idx, paddr_t *start, paddr_t *end)
+{
+    if ( idx >= e820.nr_map )
+        return -ENODEV;
+
+    if ( e820.map[idx].type != E820_RAM )
+        return -EINVAL;
+
+    *start = e820.map[idx].addr;
+    *end = e820.map[idx].addr + e820.map[idx].size;
+
+    return 0;
+}
xen/arch/x86/srat.c +11 -7
index 422e4c73e3..5bc9096a15 100644
@@ -427,18 +427,22 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
    Make sure the PXMs cover all memory. */
 static int __init nodes_cover_memory(void)
 {
-	int i;
+	unsigned int i;
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; ; i++) {
 		int j, found;
 		paddr_t start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		/* Try to loop memory map from index 0 to end. */
+		found = arch_get_memory_map(i, &start, &end);
+
+		/* Index relate entry is not RAM, skip it. */
+		if (found == -EINVAL)
 			continue;
-		}
 
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
+		/* Reach the end of arch's memory map */
+		if (found == -ENODEV)
+			break;
 
 		do {
 			found = 0;
@@ -457,7 +461,7 @@ static int __init nodes_cover_memory(void)
 		} while (found && start < end);
 
 		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+			printk(KERN_ERR "NUMA: No node for memory map range: "
 				"[%"PRIpaddr", %"PRIpaddr"]\n", start, end - 1);
 			return 0;
 		}
xen/include/xen/numa.h +3 -0
index 695ad51df0..6d02204991 100644
@@ -88,6 +88,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
                                 NODE_DATA(nid)->node_spanned_pages)
 
+extern int arch_get_memory_map(unsigned int idx,
+                               paddr_t *start, paddr_t *end);
+
 #endif
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#29 lore raw

Re: [PATCH v2 5/9] xen/x86: use arch_get_memory_map to get information from E820 map

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 94

Quoted 13 lines · Wei Chen
xen/arch/x86/numa.c +9 -0
@@ -82,3 +83,25 @@ unsigned int __init arch_get_dma_bitsize(void)
                  flsl(node_start_pfn(node) + node_spanned_pages(node) / 4 - 1)
                  + PAGE_SHIFT, 32);
 }
+
+/*
+ * This function provides the ability for caller to get one RAM entry
+ * from architectural memory map by index.
+ *
+ * This function will return zero if it can return a proper RAM entry.
+ * otherwise it will return -ENODEV for out of scope index, or return
+ * -EINVAL for non-RAM type memory entry.
+ */

I think the comment also wants to spell out that the range is exclusive at the end (assuming that's suitable for Arm; else now would perhaps be the time to change it).

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 98

Quoted 4 lines
xen/arch/x86/numa.c +4 -0
+int __init arch_get_memory_map(unsigned int idx, paddr_t *start, paddr_t *end)
+{
+    if ( idx >= e820.nr_map )
+        return -ENODEV;

Perhaps better -ENOENT?

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 101

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+    if ( e820.map[idx].type != E820_RAM )
+        return -EINVAL;

I'm sorry, this should have occurred to me already when commenting on
v1: "get_memory_map" doesn't really fit this "RAM only" restriction. Maybe arch_get_ram_range()? Or maybe others have some good naming suggestion?

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 104

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+    *start = e820.map[idx].addr;
+    *end = e820.map[idx].addr + e820.map[idx].size;

Nit: Would be shorter to read if you (re)used *start.

Jan

author
#30 lore raw

RE: [PATCH v2 5/9] xen/x86: use arch_get_memory_map to get information from E820 map

Hi Jan,

Quoted 15 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 21:40
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 5/9] xen/x86: use arch_get_memory_map to get information from E820 map

Review unsigned int __init arch_get_dma_bitsize(void) · line 83

Quoted 2 lines · Wei Chen
(quoted diff) +0 -0
@@ -82,3 +83,25 @@ unsigned int __init arch_get_dma_bitsize(void)
                  flsl(node_start_pfn(node) + node_spanned_pages(node) /

4 - 1)

Quoted 16 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 94

Quoted 11 lines
xen/arch/x86/numa.c +9 -0
                  + PAGE_SHIFT, 32);
 }
+
+/*
+ * This function provides the ability for caller to get one RAM entry
+ * from architectural memory map by index.
+ *
+ * This function will return zero if it can return a proper RAM entry.
+ * otherwise it will return -ENODEV for out of scope index, or return
+ * -EINVAL for non-RAM type memory entry.
+ */

I think the comment also wants to spell out that the range is exclusive at the end (assuming that's suitable for Arm; else now would perhaps be the time to change it).

Did you mean we have to mention the range is [star, end)? If yes, I will add related comment. And...

Quoted 8 lines

Review unsigned int __init arch_get_dma_bitsize(void) · line 95

Quoted 1 line
(quoted diff) +1 -0
+int __init arch_get_memory_map(unsigned int idx, paddr_t *start,

paddr_t *end)

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 98

Quoted 3 lines
xen/arch/x86/numa.c +3 -0
+{
+    if ( idx >= e820.nr_map )
+        return -ENODEV;

Perhaps better -ENOENT?

Ack.

Quoted 7 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 101

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+    if ( e820.map[idx].type != E820_RAM )
+        return -EINVAL;

I'm sorry, this should have occurred to me already when commenting on
v1: "get_memory_map" doesn't really fit this "RAM only" restriction. Maybe arch_get_ram_range()? Or maybe others have some good naming suggestion?

arch_get_ram_range makes sense to me. I will update in next version.

Quoted 4 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 104

Quoted 3 lines
xen/arch/x86/numa.c +2 -0
+    *start = e820.map[idx].addr;
+    *end = e820.map[idx].addr + e820.map[idx].size;

...In this case, I think we don't need to adjust this line to:
*end = e820.map[idx].addr + e820.map[idx].size - 1; As we have mentioned the range is [start, end).

Quoted 2 lines

Nit: Would be shorter to read if you (re)used *start.

Ack.

Quoted 1 line

Jan

#31 lore raw

Re: [PATCH v2 5/9] xen/x86: use arch_get_memory_map to get information from E820 map

Quoted 24 lines · Wei Chen
Quoted 7 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 21:40

Review unsigned int __init arch_get_dma_bitsize(void) · line 83

Quoted 2 lines · Wei Chen
(quoted diff) +0 -0
@@ -82,3 +83,25 @@ unsigned int __init arch_get_dma_bitsize(void)
                  flsl(node_start_pfn(node) + node_spanned_pages(node) /

4 - 1)

Quoted 16 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 94

Quoted 11 lines
xen/arch/x86/numa.c +9 -0
                  + PAGE_SHIFT, 32);
 }
+
+/*
+ * This function provides the ability for caller to get one RAM entry
+ * from architectural memory map by index.
+ *
+ * This function will return zero if it can return a proper RAM entry.
+ * otherwise it will return -ENODEV for out of scope index, or return
+ * -EINVAL for non-RAM type memory entry.
+ */

I think the comment also wants to spell out that the range is exclusive at the end (assuming that's suitable for Arm; else now would perhaps be the time to change it).

Quoted 2 lines

Did you mean we have to mention the range is [star, end)? If yes, I will add related comment.

Yes (worded one way or another).

Jan

author
#32 lore raw

PATCH v2 6/9 xen/x86: move NUMA scan nodes codes from x86 to common

In reply to Wei Chen

x86 has implemented a set of codes to scan NUMA nodes. These codes will parse NUMA memory and processor information from ACPI SRAT table. But except some ACPI specific codes, most of the scan codes like memory blocks validation, node memory range updates and some sanity check can be reused by other NUMA implementation.

So in this patch, we move some variables and related functions for NUMA memory and processor to common code. At the same time, numa_set_processor_nodes_parsed has been introduced for ACPI specific code to update processor parsing results. With this helper, we can move most of NUMA memory affinity init code from ACPI. And bad_srat and node_to_pxm functions have been exported for common code to do architectural fallback and node to proximity converting. For those NUMA implementations haven't supported proximity domain to node map. A simple 1-1 mapping helper can help us to avoid the modification of some very valuable print messages.

---
v1 -> v2:
1. Add code comment for numa_update_node_memblks to explain:
Assumes all memory regions belonging to a single node are in one chunk. Holes between them will be included in the node.
2. Merge this single patch instead of serval patches to move x86 SRAT code to common.
3. Export node_to_pxm to keep pxm information in NUMA scan nodes error messages.
4. Change the code style to target file's Xen code-style.
5. Adjust some __init and __initdata for some functions and variables.

---
 xen/arch/x86/include/asm/numa.h |   3 +-
 xen/arch/x86/srat.c             | 288 ++----------------------------
 xen/common/numa.c               | 307 ++++++++++++++++++++++++++++++++
 xen/include/xen/numa.h          |   7 +
 4 files changed, 328 insertions(+), 277 deletions(-)
xen/arch/x86/include/asm/numa.h +1 -2
index d8960743d4..eeb431cdb7 100644
@@ -10,6 +10,7 @@ typedef u8 nodeid_t;
 extern int srat_rev;
 
 extern nodeid_t pxm_to_node(unsigned int pxm);
+extern unsigned node_to_pxm(nodeid_t n);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
@@ -22,8 +23,6 @@ extern void init_cpu_to_node(void);
 
 #define arch_want_default_dmazone() (num_online_nodes() > 1)
 
-extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
-
 void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
xen/arch/x86/srat.c +13 -275

348 lines not on this page · raw message

xen/common/numa.c +307 -0

319 lines not on this page · raw message

xen/include/xen/numa.h +7 -0
index 6d02204991..564add430c 100644
@@ -52,6 +52,7 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 extern bool numa_enabled_with_firmware(void);
 extern enum numa_mode numa_status;
 
+extern void bad_srat(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 
@@ -90,6 +91,12 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 
 extern int arch_get_memory_map(unsigned int idx,
                                paddr_t *start, paddr_t *end);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
+extern bool numa_memblks_available(void);
+extern int numa_update_node_memblks(nodeid_t node, unsigned pxm,
+                                    paddr_t start, paddr_t size,
+                                    bool hotplug);
+extern void numa_set_processor_nodes_parsed(nodeid_t node);
 
 #endif
 
Signature · 1 line
2.25.1
#33 lore raw

Re: [PATCH v2 6/9] xen/x86: move NUMA scan nodes codes from x86 to common

Quoted 7 lines · Wei Chen

x86 has implemented a set of codes to scan NUMA nodes. These codes will parse NUMA memory and processor information from ACPI SRAT table. But except some ACPI specific codes, most of the scan codes like memory blocks validation, node memory range updates and some sanity check can be reused by other NUMA implementation.

Quoted 8 lines

So in this patch, we move some variables and related functions for NUMA memory and processor to common code. At the same time, numa_set_processor_nodes_parsed has been introduced for ACPI specific code to update processor parsing results. With this helper, we can move most of NUMA memory affinity init code from ACPI. And bad_srat and node_to_pxm functions have been exported for common code to do architectural fallback and node to proximity converting.

I consider it wrong for generic (ACPI-independent) code to use terms like "srat" or "pxm". This wants abstracting in some way, albeit I have to admit I lack a good idea for a suggestion right now.

Quoted 3 lines

For those NUMA implementations haven't supported proximity domain to node map. A simple 1-1 mapping helper can help us to avoid the modification of some very valuable print messages.

I don't think a simple 1:1 mapping can do. Surely firmware represents nodes by some identifiers everywhere. And I don't consider it very likely that we would want to blindly re-use such numbering inside Xen.

xen/common/numa.c line 23

Quoted 13 lines
xen/common/numa.c +7 -0
@@ -14,6 +14,21 @@
 #include <xen/softirq.h>
 #include <asm/acpi.h>
 
+static nodemask_t __initdata processor_nodes_parsed;
+static nodemask_t __initdata memory_nodes_parsed;
+static struct node __initdata nodes[MAX_NUMNODES];
+
+static int num_node_memblks;
+static struct node node_memblk_range[NR_NODE_MEMBLKS];
+static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];

Some (all) of these likely want to become __read_mostly again, at this occasion.

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] = 1UL } }; · line 57

Quoted 12 lines
xen/common/numa.c +8 -0
@@ -32,6 +47,298 @@ nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
 enum numa_mode numa_status;
 
+void __init numa_set_processor_nodes_parsed(nodeid_t node)
+{
+    node_set(node, processor_nodes_parsed);
+}
+
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
+{
+    int i;

unsigned int? Then matching e.g. ...

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] = 1UL } }; · line 76

Quoted 6 lines
xen/common/numa.c +6 -0
+static
+enum conflicts __init conflicting_memblks(nodeid_t nid, paddr_t start,
+                                          paddr_t end, paddr_t nd_start,
+                                          paddr_t nd_end, unsigned int *mblkid)
+{
+    unsigned int i;

... this. But perhaps also elsewhere.

Jan

author
#34 lore raw

RE: [PATCH v2 6/9] xen/x86: move NUMA scan nodes codes from x86 to common

Hi Jan,

Quoted 28 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:21
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 6/9] xen/x86: move NUMA scan nodes codes from x86 to common

Quoted 7 lines · Wei Chen

x86 has implemented a set of codes to scan NUMA nodes. These codes will parse NUMA memory and processor information from ACPI SRAT table. But except some ACPI specific codes, most of the scan codes like memory blocks validation, node memory range updates and some sanity check can be reused by other NUMA implementation.

Quoted 8 lines

So in this patch, we move some variables and related functions for NUMA memory and processor to common code. At the same time, numa_set_processor_nodes_parsed has been introduced for ACPI specific code to update processor parsing results. With this helper, we can move most of NUMA memory affinity init code from ACPI. And bad_srat and node_to_pxm functions have been exported for common code to do architectural fallback and node to proximity converting.

Quoted 5 lines

I consider it wrong for generic (ACPI-independent) code to use terms like "srat" or "pxm". This wants abstracting in some way, albeit I have to admit I lack a good idea for a suggestion right now.

Maybe we can use fw_rsc_table or rsc_table to replace srat, because srat is one kind of NUMA resource description table of ACPI? For PXM, I had tried to keep PXM in x86 ACPI implementation. But the cost is that, we have to move some common code to architectural code, because some messages use pxm for info, and they have different meanings for each platform, we cannot simply remove them.

Quoted 9 lines
Quoted 3 lines

For those NUMA implementations haven't supported proximity domain to node map. A simple 1-1 mapping helper can help us to avoid the modification of some very valuable print messages.

I don't think a simple 1:1 mapping can do. Surely firmware represents nodes by some identifiers everywhere. And I don't consider it very likely that we would want to blindly re-use such numbering inside Xen.

If we hide pxm to x86 only now, we can avoid this 1:1 mapping between Node and PXM.

Quoted 6 lines
Quoted 6 lines
xen/common/numa.c +0 -0
@@ -14,6 +14,21 @@
 #include <xen/softirq.h>
 #include <asm/acpi.h>
 
Quoted 11 lines

xen/common/numa.c line 23

Quoted 7 lines
xen/common/numa.c +7 -0
+static nodemask_t __initdata processor_nodes_parsed;
+static nodemask_t __initdata memory_nodes_parsed;
+static struct node __initdata nodes[MAX_NUMNODES];
+
+static int num_node_memblks;
+static struct node node_memblk_range[NR_NODE_MEMBLKS];
+static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];

Some (all) of these likely want to become __read_mostly again, at this occasion.

Ok.

Quoted 5 lines

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] =

Quoted 1 line
xen/common/numa.c +0 -0
@@ -32,6 +47,298 @@ nodemask_t __read_mostly node_online_map = { { [0] =

1UL } };

Quoted 3 lines
xen/common/numa.c +0 -0
 enum numa_mode numa_status;
 
Quoted 11 lines

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] = 1UL } }; · line 57

Quoted 8 lines
xen/common/numa.c +8 -0
+void __init numa_set_processor_nodes_parsed(nodeid_t node)
+{
+    node_set(node, processor_nodes_parsed);
+}
+
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
+{
+    int i;

unsigned int? Then matching e.g. ...

Ok.

Quoted 10 lines

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };

Quoted 4 lines
xen/common/numa.c +4 -0
+static
+enum conflicts __init conflicting_memblks(nodeid_t nid, paddr_t start,
+                                          paddr_t end, paddr_t nd_start,
+                                          paddr_t nd_end, unsigned int

*mblkid)

xen/common/numa.c nodemask_t __read_mostly node_online_map = { { [0] = · line 62

Quoted 2 lines
xen/common/numa.c +2 -0
+{
+    unsigned int i;

... this. But perhaps also elsewhere.

I will check the code an fix them.

Cheers,
Wei Chen

Quoted 1 line

Jan

#35 lore raw

Re: [PATCH v2 6/9] xen/x86: move NUMA scan nodes codes from x86 to common

Quoted 30 lines · Wei Chen

Hi Jan,

Quoted 28 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:21
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 6/9] xen/x86: move NUMA scan nodes codes from x86 to common

Quoted 15 lines · Wei Chen

x86 has implemented a set of codes to scan NUMA nodes. These codes will parse NUMA memory and processor information from ACPI SRAT table. But except some ACPI specific codes, most of the scan codes like memory blocks validation, node memory range updates and some sanity check can be reused by other NUMA implementation.

So in this patch, we move some variables and related functions for NUMA memory and processor to common code. At the same time, numa_set_processor_nodes_parsed has been introduced for ACPI specific code to update processor parsing results. With this helper, we can move most of NUMA memory affinity init code from ACPI. And bad_srat and node_to_pxm functions have been exported for common code to do architectural fallback and node to proximity converting.

Quoted 8 lines
Quoted 5 lines

I consider it wrong for generic (ACPI-independent) code to use terms like "srat" or "pxm". This wants abstracting in some way, albeit I have to admit I lack a good idea for a suggestion right now.

Maybe we can use fw_rsc_table or rsc_table to replace srat, because srat is one kind of NUMA resource description table of ACPI?

Is "rsc" meant to stand for "resource"? Would be a somewhat unusual abbreviation. I could see use using e.g. numa_fw_ as a prefix, as in e.g. numa_fw_bad() (replacing bad_srat()).

Quoted 4 lines

For PXM, I had tried to keep PXM in x86 ACPI implementation. But the cost is that, we have to move some common code to architectural code, because some messages use pxm for info, and they have different meanings for each platform, we cannot simply remove them.

Well, for functions wanting to emit log messages, suitable abstractions can likely be made without needing the retain a lot of per-arch code. E.g. the arch could pass in "PXM" and format strings then would use %s together with it. Similarly the translation (if any is necessary) could likely be abstracted by, in the worst case, passing in a function pointer.

Jan

author
#36 lore raw

PATCH v2 7/9 xen/x86: rename bad_srat to numa_bad

In reply to Wei Chen

When NUMA initialization code is failed in scanning SRAT. It will call bad_srat to set disable NUMA and clear relate data. But this name is ACPI specific, we have moved generically usable NUMA codes to common, bad_srat has came with these codes to common code. But it's not reasonable for other NUMA implementations to implement a fall back function named bad_srat. So in this patch, we rename bad_srat to numa_bad.

---
v1 -> v2:
1. New in v2.

---
 xen/arch/x86/srat.c    | 18 +++++++++---------
 xen/common/numa.c      |  4 ++--
 xen/include/xen/numa.h |  2 +-
 3 files changed, 12 insertions(+), 12 deletions(-)
xen/arch/x86/srat.c +9 -9
index 9ae81afdff..4afb37bf9f 100644
@@ -93,7 +93,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-void __init bad_srat(void)
+void __init numa_bad(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
@@ -153,7 +153,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 	if (srat_disabled())
 		return;
 	if (pa->header.length < sizeof(struct acpi_srat_x2apic_cpu_affinity)) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 	if (!(pa->flags & ACPI_SRAT_CPU_ENABLED))
@@ -166,7 +166,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 	pxm = pa->proximity_domain;
 	node = setup_node(pxm);
 	if (node == NUMA_NO_NODE) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 
@@ -189,7 +189,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 	if (srat_disabled())
 		return;
 	if (pa->header.length != sizeof(struct acpi_srat_cpu_affinity)) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 	if (!(pa->flags & ACPI_SRAT_CPU_ENABLED))
@@ -202,7 +202,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 	}
 	node = setup_node(pxm);
 	if (node == NUMA_NO_NODE) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 	apicid_to_node[pa->apic_id] = node;
@@ -225,7 +225,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	if (srat_disabled())
 		return;
 	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
@@ -239,7 +239,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	if (!numa_memblks_available()) {
 		dprintk(XENLOG_WARNING,
 			"Too many numa entries, try bigger NR_NODE_MEMBLKS!\n");
-		bad_srat();
+		numa_bad();
 		return;
 	}
 
@@ -248,14 +248,14 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node == NUMA_NO_NODE) {
-		bad_srat();
+		numa_bad();
 		return;
 	}
 
 	ret = numa_update_node_memblks(node, pxm, ma->base_address, ma->length,
 				       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE);
 	if (ret)
-		bad_srat();
+		numa_bad();
 }
 
 void __init acpi_numa_arch_fixup(void) {}
xen/common/numa.c +2 -2
index e3b66c54b5..5ab061e991 100644
@@ -300,7 +300,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 
     if ( !nodes_cover_memory() )
     {
-        bad_srat();
+        numa_bad();
         return -1;
     }
 
@@ -311,7 +311,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
     {
         printk(KERN_ERR
                "SRAT: No NUMA node hash function found. Contact maintainer\n");
-        bad_srat();
+        numa_bad();
         return -1;
     }
 
xen/include/xen/numa.h +1 -1
index 564add430c..4c4632ec27 100644
@@ -52,7 +52,7 @@ extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 extern bool numa_enabled_with_firmware(void);
 extern enum numa_mode numa_status;
 
-extern void bad_srat(void);
+extern void numa_bad(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 
Signature · 1 line
2.25.1
#37 lore raw

Re: [PATCH v2 7/9] xen/x86: rename bad_srat to numa_bad

Quoted 8 lines · Wei Chen

When NUMA initialization code is failed in scanning SRAT. It will call bad_srat to set disable NUMA and clear relate data. But this name is ACPI specific, we have moved generically usable NUMA codes to common, bad_srat has came with these codes to common code. But it's not reasonable for other NUMA implementations to implement a fall back function named bad_srat. So in this patch, we rename bad_srat to numa_bad.

Quoted 4 lines

---
v1 -> v2:
1. New in v2.

I think this wants to come before moving to common code. And I'd also like you to consider using e.g. numa_fw_bad() or numa_firmware_bad() to better reflect the original purpose.

Jan

author
#38 lore raw

RE: [PATCH v2 7/9] xen/x86: rename bad_srat to numa_bad

Hi Jan,

Quoted 19 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:26
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 7/9] xen/x86: rename bad_srat to numa_bad

Quoted 8 lines · Wei Chen

When NUMA initialization code is failed in scanning SRAT. It will call bad_srat to set disable NUMA and clear relate data. But this name is ACPI specific, we have moved generically usable NUMA codes to common, bad_srat has came with these codes to common code. But it's not reasonable for other NUMA implementations to implement a fall back function named bad_srat. So in this patch, we rename bad_srat to numa_bad.

Quoted 9 lines
Quoted 4 lines

---
v1 -> v2:
1. New in v2.

I think this wants to come before moving to common code. And I'd also like you to consider using e.g. numa_fw_bad() or numa_firmware_bad() to better reflect the original purpose.

Ok, I will re-order the series and update the function names.

Thanks,
Wei Chen

Quoted 1 line

Jan

author
#39 lore raw

PATCH v2 8/9 xen: rename acpi_scan_nodes to numa_scan_nodes

In reply to Wei Chen

We have moved acpi_scan_nodes from x86 to common. Because of of our previous work, this function no longer has many ACPI characteristics, except some "SRAT" words in print messages. So we rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace "SRAT" words in print messages.

After doing this, it doesn't make sense to use CONFIG_ACPI_NUMA to gate numa_scan_nodes in numa_initmem_init. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.

We take this opportunity to make this function static, since it currently has no external callers.

---
v1 -> v2:
1. Merge two patches into this patch:
1. replace CONFIG_ACPI_NUMA by CONFIG_NUMA.
2. replace "SRAT" texts.
2. Turn numa_scan_nodes to static.

---
 xen/arch/x86/include/asm/acpi.h |  1 -
 xen/common/numa.c               | 10 +++++-----
 2 files changed, 5 insertions(+), 6 deletions(-)
xen/arch/x86/include/asm/acpi.h +0 -1
index ab0d56dd70..f6ea3f1a9a 100644
@@ -101,7 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern int acpi_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
 extern struct acpi_sleep_info acpi_sinfo;
xen/common/numa.c +5 -5
index 5ab061e991..0f62638e4c 100644
@@ -285,7 +285,7 @@ static int __init nodes_cover_memory(void)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+static int __init numa_scan_nodes(paddr_t start, paddr_t end)
 {
     int i;
     nodemask_t all_nodes_parsed;
@@ -310,7 +310,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
     if ( memnode_shift < 0 )
     {
         printk(KERN_ERR
-               "SRAT: No NUMA node hash function found. Contact maintainer\n");
+               "NUMA: No NUMA node hash function found. Contact maintainer\n");
         numa_bad();
         return -1;
     }
@@ -323,7 +323,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
         paddr_t size = nodes[i].end - nodes[i].start;
 
         if ( size == 0 )
-            printk(KERN_INFO "SRAT: node %u has no memory\n", i);
+            printk(KERN_INFO "NUMA: node %u has no memory\n", i);
 
         setup_node_bootmem(i, nodes[i].start, nodes[i].end);
     }
@@ -540,8 +540,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
         return;
 #endif
 
-#ifdef CONFIG_ACPI_NUMA
-    if ( numa_status != numa_off && !acpi_scan_nodes(start, end) )
+#ifdef CONFIG_NUMA
+    if ( numa_status != numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif
 
Signature · 1 line
2.25.1
#40 lore raw

Re: [PATCH v2 8/9] xen: rename acpi_scan_nodes to numa_scan_nodes

Quoted 5 lines · Wei Chen

We have moved acpi_scan_nodes from x86 to common. Because of of our previous work, this function no longer has many ACPI characteristics, except some "SRAT" words in print messages. So we rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace "SRAT" words in print messages.

Once again I think the rename should happen before the moving to common code.

Jan

author
#41 lore raw

RE: [PATCH v2 8/9] xen: rename acpi_scan_nodes to numa_scan_nodes

Hi Jan,

Quoted 11 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:27
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH v2 8/9] xen: rename acpi_scan_nodes to numa_scan_nodes

On 08.07.2022 16:54, Wei Chen wrote:

Quoted 9 lines
Quoted 5 lines

We have moved acpi_scan_nodes from x86 to common. Because of of our previous work, this function no longer has many ACPI characteristics, except some "SRAT" words in print messages. So we rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace "SRAT" words in print messages.

Once again I think the rename should happen before the moving to common code.

Ack again.

Cheers,
Wei Chen

Quoted 1 line

Jan

author
#42 lore raw

PATCH v2 9/9 xen: introduce a Kconfig option to configure NUMA nodes number

In reply to Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

As NODES_SHIFT is currently unused, we're taking this opportunity to remove it.

---
v1 -> v2:
1. Add NODES_SHIFT remove message in commit log
2. Change NR_NUMA_NODES upper bound from 4095 to 255.

---
 xen/arch/Kconfig                | 11 +++++++++++
 xen/arch/x86/include/asm/numa.h |  2 --
 xen/include/xen/numa.h          | 10 +++++-----
 3 files changed, 16 insertions(+), 7 deletions(-)
xen/arch/Kconfig +11 -0
index f16eb0df43..4fc16f83ac 100644
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or similar
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound of
+	  the number of NUMA nodes that the scheduler, memory allocation and
+          other NUMA-aware components can handle.
xen/arch/x86/include/asm/numa.h +0 -2
index eeb431cdb7..db76281c08 100644
@@ -3,8 +3,6 @@
 
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;
 
 extern int srat_rev;
xen/include/xen/numa.h +5 -5
index 4c4632ec27..cac92d2688 100644
@@ -3,14 +3,14 @@
 
 #include <asm/numa.h>
 
-#ifndef NODES_SHIFT
-#define NODES_SHIFT     0
-#endif
-
 #define NUMA_NO_NODE     0xFF
 #define NUMA_NO_DISTANCE 0xFF
 
-#define MAX_NUMNODES    (1 << NODES_SHIFT)
+#ifdef CONFIG_NR_NUMA_NODES
+#define MAX_NUMNODES CONFIG_NR_NUMA_NODES
+#else
+#define MAX_NUMNODES 1
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
#43 lore raw

Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 12 lines · Wei Chen
xen/arch/Kconfig +6 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or similar
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Does the value being (perhaps far) larger than NR_CPUS make sense?

Why does the range end at a not-power-of-2 value? (I was actually going to suggest to have a shift value specified here, until spotting that NODES_SHIFT isn't used anywhere else, and hence you're rightfully deleting it.)

xen/arch/Kconfig config NR_CPUS · line 30

Quoted 5 lines
xen/arch/Kconfig +5 -0
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound of
+	  the number of NUMA nodes that the scheduler, memory allocation and
+          other NUMA-aware components can handle.

Nit: indentation.

Jan

author
#44 lore raw

RE: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Jan,

Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:34
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; [email protected]
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 11 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 8 lines
xen/arch/Kconfig +6 -0
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

Quoted 5 lines

Why does the range end at a not-power-of-2 value? (I was actually going to suggest to have a shift value specified here, until spotting that NODES_SHIFT isn't used anywhere else, and hence you're rightfully deleting it.)

I think we have discussed about the 255 in v1. Because Xen is using u8 as nodeid_t, so 255 may be a upper bound.

And if use a shift value, from a user perspective, I don't like it. It needs to be converted, not intuitive enough. It also limits my input range, even though my numerical values are reasonable. Yes, if a machine has 15 node, we can ask them to input 16, but why not let the users decide? instead of being forced to enter 16 by the program?

Quoted 9 lines

xen/arch/Kconfig config NR_CPUS

Quoted 3 lines
xen/arch/Kconfig +3 -0
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound

of

xen/arch/Kconfig config NR_CPUS · line 30

Quoted 2 lines
xen/arch/Kconfig +2 -0
+	  the number of NUMA nodes that the scheduler, memory allocation and
+          other NUMA-aware components can handle.

Nit: indentation.

Ok.

Cheers,
Wei Chen

Quoted 1 line

Jan

#45 lore raw

Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 33 lines · Wei Chen

Hi Jan,

Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:34
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; [email protected]
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 11 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 8 lines
xen/arch/Kconfig +6 -0
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 7 lines
Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

The defaults are fine. But does it make sense to have 255 nodes when just 32 CPUs were selected? I'm afraid kconfig is going to get in the way, but I think I'd like the upper bound to be min(NR_CPUS, 255).

Quoted 8 lines
Quoted 5 lines

Why does the range end at a not-power-of-2 value? (I was actually going to suggest to have a shift value specified here, until spotting that NODES_SHIFT isn't used anywhere else, and hence you're rightfully deleting it.)

I think we have discussed about the 255 in v1. Because Xen is using u8 as nodeid_t, so 255 may be a upper bound.

Indeed, but that's something you could have mentioned in the commit message as justification. But you could also have opted to make the upper bound 128. Let me ask you: Are you aware of systems with more than a dozen or so nodes, that Xen can in principle run on?

Quoted 5 lines

And if use a shift value, from a user perspective, I don't like it. It needs to be converted, not intuitive enough. It also limits my input range, even though my numerical values are reasonable. Yes, if a machine has 15 node, we can ask them to input 16, but why not let the users decide? instead of being forced to enter 16 by the program?

At least x86 Linux also wants this specified as a shift value, so people may actually be (more) used to that. Plus non-pwer-of-2 values may yield more complex calculations when the compiler generates code. Think of a two-dimensional distance table, for example.

Jan

#46 lore raw

Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Jan,

Quoted 37 lines · Jan Beulich
Quoted 36 lines · Wei Chen

Hi Jan,

Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:34
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; [email protected]
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
  	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 11 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 8 lines
xen/arch/Kconfig +6 -0
  	  technologies, this the number of logical threads which Xen will
  	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Quoted 8 lines
Quoted 4 lines

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

The defaults are fine. But does it make sense to have 255 nodes when just 32 CPUs were selected? I'm afraid kconfig is going to get in the way, but I think I'd like the upper bound to be min(NR_CPUS, 255).

Looking around NUMA nodes with 0 CPUs exists. So I don't think we should tie the two.

Cheers,

Signature · 1 line
Julien Grall
author
#47 lore raw

RE: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Julien, Jan,

Quoted 60 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年7月14日 19:27
To: Jan Beulich <[email protected]>; Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; [email protected]
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Jan,

Quoted 9 lines · Jan Beulich
Quoted 8 lines · Wei Chen

Hi Jan,

Quoted 6 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:34
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Julien Grall <[email protected]>;

Stefano

Quoted 1 line
Quoted 1 line
Quoted 1 line

Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau

Monné

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

<[email protected]>; [email protected]
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to

configure

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

NUMA nodes number

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
  	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 1 line
xen/arch/Kconfig +0 -0
  	  technologies, this the number of logical threads which Xen

will

Quoted 21 lines
Quoted 20 lines
Quoted 10 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 7 lines
xen/arch/Kconfig +6 -0
  	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

Quoted 7 lines
Quoted 3 lines

The defaults are fine. But does it make sense to have 255 nodes when just 32 CPUs were selected? I'm afraid kconfig is going to get in the way, but I think I'd like the upper bound to be min(NR_CPUS, 255).

Looking around NUMA nodes with 0 CPUs exists. So I don't think we should tie the two.

Yes, some nodes can only have RAM and some nodes can only have CPUs. So is it ok to use 2-255 for node range?

Quoted 4 lines

Cheers,

--
Julien Grall

#48 lore raw

Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 41 lines · Wei Chen
Quoted 41 lines

From: Julien Grall <[email protected]>
Sent: 2022年7月14日 19:27

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Wei Chen
Quoted 10 lines

From: Jan Beulich <[email protected]>
Sent: 2022年7月12日 22:34

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
  	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 1 line
xen/arch/Kconfig +0 -0
  	  technologies, this the number of logical threads which Xen

will

Quoted 24 lines
Quoted 13 lines
Quoted 10 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 7 lines
xen/arch/Kconfig +6 -0
  	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 7 lines
Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

The defaults are fine. But does it make sense to have 255 nodes when just 32 CPUs were selected? I'm afraid kconfig is going to get in the way, but I think I'd like the upper bound to be min(NR_CPUS, 255).

Quoted 6 lines
Quoted 3 lines

Looking around NUMA nodes with 0 CPUs exists. So I don't think we should tie the two.

Yes, some nodes can only have RAM and some nodes can only have CPUs. So is it ok to use 2-255 for node range?

Personally I think we shouldn't allow unreasonably high node counts, unless proven by real hardware existing. Which goes hand in hand with me wanting the upper bound to be a power of 2 value, if for nothing else than a hint that using power-of-2 values here is preferable. Hence I'd like to propose 64 or 128 as upper bound, in this order of (my personal) preference.

Jan

author
#49 lore raw

RE: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Jan,

Quoted 55 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年7月18日 16:10
To: Wei Chen <[email protected]>
Cc: nd <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; [email protected]; Julien Grall <[email protected]>
Subject: Re: [PATCH v2 9/9] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 31 lines
Quoted 31 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Sent: 2022年7月12日 22:34

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
  	  For CPU cores which support Simultaneous Multi-Threading or

similar

Quoted 1 line
xen/arch/Kconfig +0 -0
  	  technologies, this the number of logical threads which Xen

will

Quoted 21 lines
Quoted 20 lines
Quoted 10 lines

xen/arch/Kconfig config NR_CPUS · line 25

Quoted 7 lines
xen/arch/Kconfig +6 -0
  	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 255
+	default "64"
+	depends on NUMA

Does 1 make sense? That's not going to be NUMA then, I would say.

Ok, we need at least 2 nodes to be a real NUMA.

Quoted 2 lines

Does the value being (perhaps far) larger than NR_CPUS make sense?

Arm has 128 default NR_CPUS (except some platforms) and x86 has 256. So I am not very clear about your comments about far larger? As my Understanding, one node has 2 or 4 cores are very common in a NUMA System.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The defaults are fine. But does it make sense to have 255 nodes when just 32 CPUs were selected? I'm afraid kconfig is going to get in the way, but I think I'd like the upper bound to be min(NR_CPUS, 255).

Looking around NUMA nodes with 0 CPUs exists. So I don't think we

should

Quoted 5 lines
Quoted 2 lines

tie the two.

Yes, some nodes can only have RAM and some nodes can only have CPUs. So is it ok to use 2-255 for node range?

Quoted 7 lines

Personally I think we shouldn't allow unreasonably high node counts, unless proven by real hardware existing. Which goes hand in hand with me wanting the upper bound to be a power of 2 value, if for nothing else than a hint that using power-of-2 values here is preferable. Hence I'd like to propose 64 or 128 as upper bound, in this order of (my personal) preference.

Thanks, I will use 64 in next version.

Wei Chen

Quoted 1 line

Jan