Skip to contentSkip to conversation

Add device tree based NUMA support to Arm

PATCH Superseded Wei Chen 6 participants 192 messages Sep 23, 2021 – Jan 28, 2022 Page 1 of 2

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Wei Chen

author
#0 lore raw

cover Add device tree based NUMA support to Arm

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

---
@Julien, about numa=noacpi option, I haven't remove it in this patch series. I had tried, and this option is not easy to remove. So I'm not going to think about doing that in this series

rfc -> v1:
1. Re-order the whole patch series to avoid temporary code
2. Add detection of discontinous node memory range
3. Fix typos in commit messages and code comments.
4. For variables that are used in common code, we don't covert them to external any more. Instead, we export helpers to provide variable access.
5. Revert memnodemap[0] to 0 when NUMA init failed. Change memnodemapsize from ARRAY_SIZE(_memnodemap) to 1 to reflect reality.
6. Use arch_has_default_dmazone for page_alloc.c instead of changing code inside Arm
7. Keep Kconfig options alphabetically sorted.
8. Replace #if !defined by #ifndef
9. Use paddr_t for addresses in NUMA node structures and function parameters
10. Use fw_numa to replace acpi_numa for neutrality
11. Change BIOS to Firmware in print message.
12. Promote VIRTUAL_BUG_ON to ASSERT
13. Introduce CONFIG_EFI to stub API for non-EFI architecture
14. Use EFI stub API to replace arch helper for efi_enabled
15. Use NR_MEM_BANKS for Arm's NR_NODE_MEMBLKS
16. Change matrix map default value from NUMA_REMOTE_DISTANCE to 0
17. Remove check in numa_set_node.
18. Follow the x86's method of adding CPU to NUMA
19. Use fdt prefix for all device tree NUMA parser's API
20. Check un-matched bi-direction distance in matrix map
21. Remove unless fdt type check function
22. Update doc to remove numa x86 spcific
23. Introduce Arm generic NUMA Kconfig option

Wei Chen (37):
xen/arm: Print a 64-bit number in hex from early uart
xen: introduce a Kconfig option to configure NUMA nodes number xen/x86: Initialize memnodemapsize while faking NUMA node
xen: introduce an arch helper for default dma zone status
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API xen/x86: use paddr_t for addresses in NUMA node structure xen/x86: add detection of discontinous node memory range xen/x86: introduce two helpers to access memory hotplug end xen/x86: use helpers to access/update mem_hotplug xen/x86: abstract neutral code from acpi_numa_memory_affinity_init xen/x86: decouple nodes_cover_memory from E820 map xen/x86: decouple processor_nodes_parsed from acpi numa functions xen/x86: use name fw_numa to replace acpi_numa
xen/x86: rename acpi_scan_nodes to numa_scan_nodes xen/x86: export srat_bad to external
xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
xen: move NUMA common code from x86 to common
xen/x86: promote VIRTUAL_BUG_ON to ASSERT in
xen: introduce CONFIG_EFI to stub API for non-EFI architecture xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS xen/arm: implement node distance helpers for Arm xen/arm: implement two arch helpers to get memory map info xen/arm: implement bad_srat for Arm NUMA initialization xen/arm: build NUMA cpu_to_node map in dt_smp_init_cpus xen/arm: Add boot and secondary CPU to NUMA system xen/arm: stub memory hotplug access helpers for Arm xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: keep guest still be NUMA unware
xen/arm: enable device tree based NUMA in system init xen/arm: use CONFIG_NUMA to gate node_online_map in smpboot xen/arm: Provide Kconfig options for Arm to enable NUMA
docs: update numa command line to support Arm

 docs/misc/xen-command-line.pandoc |   2 +-
 xen/arch/Kconfig                  |  11 +
 xen/arch/arm/Kconfig              |  12 +
 xen/arch/arm/Makefile             |   4 +-
 xen/arch/arm/arm64/head.S         |   9 +-
 xen/arch/arm/bootfdt.c            |   8 +-
 xen/arch/arm/domain_build.c       |   6 +
 xen/arch/arm/efi/efi-boot.h       |  25 --
 xen/arch/arm/numa.c               | 155 ++++++++++
 xen/arch/arm/numa_device_tree.c   | 274 ++++++++++++++++++
 xen/arch/arm/setup.c              |  12 +
 xen/arch/arm/smpboot.c            |  39 ++-
 xen/arch/x86/Kconfig              |   3 +-
 xen/arch/x86/numa.c               | 449 ++---------------------------
 xen/arch/x86/setup.c              |   2 +-
 xen/arch/x86/srat.c               | 232 ++-------------
 xen/common/Kconfig                |  14 +
 xen/common/Makefile               |   2 +
 xen/common/numa.c                 | 450 ++++++++++++++++++++++++++++++
 xen/common/numa_srat.c            | 264 ++++++++++++++++++
 xen/common/page_alloc.c           |   2 +-
 xen/drivers/acpi/Kconfig          |   3 +-
 xen/drivers/acpi/Makefile         |   2 +-
 xen/include/asm-arm/mm.h          |  10 +
 xen/include/asm-arm/numa.h        |  50 ++++
 xen/include/asm-x86/acpi.h        |   4 -
 xen/include/asm-x86/config.h      |   1 -
 xen/include/asm-x86/mm.h          |  10 +
 xen/include/asm-x86/numa.h        |  65 +----
 xen/include/asm-x86/setup.h       |   1 -
 xen/include/xen/efi.h             |  11 +
 xen/include/xen/numa.h            |  94 ++++++-
 32 files changed, 1470 insertions(+), 756 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c
 create mode 100644 xen/common/numa_srat.c
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/37 xen/arm: Print a 64-bit number in hex from early uart

Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -

In this patch, we increased the print loops and shift bits to make putn print 64-bit number.

---
 xen/arch/arm/arm64/head.S | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +5 -4
index aa1f88c764..d957ea377b 100644
@@ -862,17 +862,18 @@ puts:
         ret
 ENDPROC(puts)
 
-/* Print a 32-bit number in hex.  Specific to the PL011 UART.
+/* Print a 64-bit number in hex.
  * x0: Number to print.
  * x23: Early UART base address
  * Clobbers x0-x3 */
+#define PRINT_MASK 0xf000000000000000
 putn:
         adr   x1, hex
-        mov   x3, #8
+        mov   x3, #16
 1:
         early_uart_ready x23, 2
-        and   x2, x0, #0xf0000000    /* Mask off the top nybble */
-        lsr   x2, x2, #28
+        and   x2, x0, #PRINT_MASK    /* Mask off the top nybble */
+        lsr   x2, x2, #60
         ldrb  w2, [x1, x2]           /* Convert to a char */
         early_uart_transmit x23, w2
         lsl   x0, x0, #4             /* Roll it through one nybble at a time */
Signature · 1 line
2.25.1
author
#2 lore raw

PATCH 2/37 xen: introduce a Kconfig option to configure NUMA nodes number

In reply to Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

---
 xen/arch/Kconfig           | 11 +++++++++++
 xen/include/asm-x86/numa.h |  2 --
 xen/include/xen/numa.h     | 10 +++++-----
 3 files changed, 16 insertions(+), 7 deletions(-)
xen/arch/Kconfig +11 -0
index f16eb0df43..8a20da67ed 100644
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or similar
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095
+	default "64"
+	depends on NUMA
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound of
+	  the number of NUMA nodes the scheduler, memory allocation and other
+	  NUMA-aware components can handle.
xen/include/asm-x86/numa.h +0 -2
index bada2c0bb9..3cf26c2def 100644
@@ -3,8 +3,6 @@
 
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;
 
 extern int srat_rev;
xen/include/xen/numa.h +5 -5
index 7aef1a88dc..52950a3150 100644
@@ -3,14 +3,14 @@
 
 #include <asm/numa.h>
 
-#ifndef NODES_SHIFT
-#define NODES_SHIFT     0
-#endif
-
 #define NUMA_NO_NODE     0xFF
 #define NUMA_NO_DISTANCE 0xFF
 
-#define MAX_NUMNODES    (1 << NODES_SHIFT)
+#ifdef CONFIG_NR_NUMA_NODES
+#define MAX_NUMNODES CONFIG_NR_NUMA_NODES
+#else
+#define MAX_NUMNODES    1
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
Reviewed-by on v1 2/37
#3 lore raw

Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 10 lines · Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

This is OK but I think you should also mention in the commit message that you are taking the opportunity to remove NODES_SHIFT because it is currently unused.

With that:

Quoted 66 lines
---
 xen/arch/Kconfig           | 11 +++++++++++
 xen/include/asm-x86/numa.h |  2 --
 xen/include/xen/numa.h     | 10 +++++-----
 3 files changed, 16 insertions(+), 7 deletions(-)
xen/arch/Kconfig +11 -0
index f16eb0df43..8a20da67ed 100644
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or similar
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095
+	default "64"
+	depends on NUMA
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound of
+	  the number of NUMA nodes the scheduler, memory allocation and other
+	  NUMA-aware components can handle.
xen/include/asm-x86/numa.h +0 -2
index bada2c0bb9..3cf26c2def 100644
@@ -3,8 +3,6 @@
 
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;
 
 extern int srat_rev;
xen/include/xen/numa.h +5 -5
index 7aef1a88dc..52950a3150 100644
@@ -3,14 +3,14 @@
 
 #include <asm/numa.h>
 
-#ifndef NODES_SHIFT
-#define NODES_SHIFT     0
-#endif
-
 #define NUMA_NO_NODE     0xFF
 #define NUMA_NO_DISTANCE 0xFF
 
-#define MAX_NUMNODES    (1 << NODES_SHIFT)
+#ifdef CONFIG_NR_NUMA_NODES
+#define MAX_NUMNODES CONFIG_NR_NUMA_NODES
+#else
+#define MAX_NUMNODES    1
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
author
#4 lore raw

RE: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Stefano,

Quoted 21 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 7:45
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 10 lines · Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

Quoted 9 lines

This is OK but I think you should also mention in the commit message that you are taking the opportunity to remove NODES_SHIFT because it is currently unused.

With that:

Thanks, I will update it in next version.

Quoted 69 lines

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 13 lines
---
 xen/arch/Kconfig           | 11 +++++++++++
 xen/include/asm-x86/numa.h |  2 --
 xen/include/xen/numa.h     | 10 +++++-----
 3 files changed, 16 insertions(+), 7 deletions(-)
xen/arch/Kconfig +0 -0
index f16eb0df43..8a20da67ed 100644
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or

similar

xen/arch/Kconfig config NR_CPUS

Quoted 11 lines
xen/arch/Kconfig +9 -0
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095
+	default "64"
+	depends on NUMA
+	help
+	  Controls the build-time size of various arrays and bitmaps
+	  associated with multiple-nodes management. It is the upper bound

of

xen/arch/Kconfig config NR_CPUS · line 29

Quoted 1 line
xen/arch/Kconfig +1 -0
+	  the number of NUMA nodes the scheduler, memory allocation and

other

Quoted 41 lines
xen/arch/Kconfig +1 -0
+	  NUMA-aware components can handle.
xen/include/asm-x86/numa.h +0 -2
index bada2c0bb9..3cf26c2def 100644
@@ -3,8 +3,6 @@
 
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;
 
 extern int srat_rev;
xen/include/xen/numa.h +5 -6
index 7aef1a88dc..52950a3150 100644
@@ -3,14 +3,14 @@
 
 #include <asm/numa.h>
 
-#ifndef NODES_SHIFT
-#define NODES_SHIFT     0
-#endif
-
 #define NUMA_NO_NODE     0xFF
 #define NUMA_NO_DISTANCE 0xFF
 
-#define MAX_NUMNODES    (1 << NODES_SHIFT)
+#ifdef CONFIG_NR_NUMA_NODES
+#define MAX_NUMNODES CONFIG_NR_NUMA_NODES
+#else
+#define MAX_NUMNODES    1
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
--

2.25.1

#5 lore raw

Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

In reply to Wei Chen
Quoted 10 lines · Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

Do you really mean administrators here? To me command line options are for administrators, but build decisions are usually taken by build managers of distros.

xen/arch/Kconfig config NR_CPUS · line 23

Quoted 10 lines
xen/arch/Kconfig +4 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or similar
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095

How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...

xen/include/asm-x86/numa.h line 6

Quoted 9 lines
xen/include/asm-x86/numa.h +0 -2
@@ -3,8 +3,6 @@
 
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;

... this.

Jan

author
#6 lore raw

RE: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Hi Jan,

Quoted 46 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:56
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 10 lines · Wei Chen

Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.

So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.

Do you really mean administrators here? To me command line options are for administrators, but build decisions are usually taken by build managers of distros.

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or

similar

xen/arch/Kconfig config NR_CPUS · line 23

Quoted 6 lines
xen/arch/Kconfig +4 -0
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095

How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...

Quoted 4 lines
xen/include/asm-x86/numa.h +0 -0
@@ -3,8 +3,6 @@
 
Quoted 8 lines

xen/include/asm-x86/numa.h line 6

Quoted 5 lines
xen/include/asm-x86/numa.h +0 -2
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;

... this.

you're right, we use u8 as nodeid_t. 4095 for node number in this option is not reasonable. Maybe a 255 upper bound is good?

Quoted 1 line

Jan

#7 lore raw

Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number

Quoted 34 lines · Wei Chen
Quoted 25 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:56

xen/arch/Kconfig config NR_CPUS · line 17

Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0 -0
@@ -17,3 +17,14 @@ config NR_CPUS
 	  For CPU cores which support Simultaneous Multi-Threading or

similar

xen/arch/Kconfig config NR_CPUS · line 23

Quoted 6 lines
xen/arch/Kconfig +4 -0
 	  technologies, this the number of logical threads which Xen will
 	  support.
+
+config NR_NUMA_NODES
+	int "Maximum number of NUMA nodes supported"
+	range 1 4095

How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...

Quoted 4 lines
xen/include/asm-x86/numa.h +0 -0
@@ -3,8 +3,6 @@
 
Quoted 8 lines

xen/include/asm-x86/numa.h line 6

Quoted 5 lines
xen/include/asm-x86/numa.h +0 -2
 #include <xen/cpumask.h>
 
-#define NODES_SHIFT 6
-
 typedef u8 nodeid_t;

... this.

Quoted 2 lines

you're right, we use u8 as nodeid_t. 4095 for node number in this option is not reasonable. Maybe a 255 upper bound is good?

I think it is, yes, but you will want to properly check.

Jan

author
#8 lore raw

PATCH 3/37 xen/x86: Initialize memnodemapsize while faking NUMA node

In reply to Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

In this case the memory node map doesn't need to be allocated from boot pages, it will use the _memnodemap directly. But memnodemapsize hasn't been set. Xen should assert in phys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.

Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.

---
 xen/arch/x86/numa.c | 4 ++++
 1 file changed, 4 insertions(+)
xen/arch/x86/numa.c +4 -0
index f1066c59c7..ce79ee44ce 100644
@@ -270,6 +270,10 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
+    /* Dummy node only uses 1 slot in reality */
+    memnodemap[0] = 0;
+    memnodemapsize = 1;
+
     nodes_clear(node_online_map);
     node_set_online(0);
     for ( i = 0; i < nr_cpu_ids; i++ )
Signature · 1 line
2.25.1
Acked-by on v1 3/37
#9 lore raw

Re: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 9 lines · Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

In this case the memory node map doesn't need to be allocated from boot pages, it will use the _memnodemap directly. But memnodemapsize hasn't been set. Xen should assert in phys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.

Somehow and A got lost here, which I'll add back while committing.

Quoted 5 lines

Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.

author
#10 lore raw

RE: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 16 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:57
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 4 lines · Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

Quoted 8 lines
Quoted 5 lines

In this case the memory node map doesn't need to be allocated from boot pages, it will use the _memnodemap directly. But memnodemapsize hasn't been set. Xen should assert in phys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.

Somehow and A got lost here, which I'll add back while committing.

Thanks!

Quoted 7 lines
Quoted 5 lines

Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.

author
#11 lore raw

PATCH 4/37 xen: introduce an arch helper for default dma zone status

In reply to Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

---
 xen/arch/x86/numa.c        | 5 +++++
 xen/common/page_alloc.c    | 2 +-
 xen/include/asm-arm/numa.h | 5 +++++
 xen/include/asm-x86/numa.h | 1 +
 4 files changed, 12 insertions(+), 1 deletion(-)
xen/arch/x86/numa.c +5 -0
index ce79ee44ce..1fabbe8281 100644
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/common/page_alloc.c +1 -1
index 5801358b4b..80916205e5 100644
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();
 
     printk("Domain heap initialised");
xen/include/asm-arm/numa.h +5 -0
index 31a6de4e23..9d5739542d 100644
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+static inline unsigned int arch_have_default_dmazone(void)
+{
+    return 0;
+}
+
 #endif /* __ARCH_ARM_NUMA_H */
 /*
  * Local variables:
xen/include/asm-x86/numa.h +1 -0
index 3cf26c2def..8060cbf3f4 100644
@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 void srat_parse_regions(u64 addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
+unsigned int arch_have_default_dmazone(void);
 
 #endif
Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 5 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase

^ the ^Because

Quoted 2 lines

on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86

                                                              ^,
Quoted 3 lines

prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges.

^ of memory

Quoted 7 lines

The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

^ needs

Quoted 14 lines
---
 xen/arch/x86/numa.c        | 5 +++++
 xen/common/page_alloc.c    | 2 +-
 xen/include/asm-arm/numa.h | 5 +++++
 xen/include/asm-x86/numa.h | 1 +
 4 files changed, 12 insertions(+), 1 deletion(-)
xen/arch/x86/numa.c +0 -0
index ce79ee44ce..1fabbe8281 100644

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 374

Quoted 4 lines
xen/arch/x86/numa.c +1 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)

Can this function return bool?
Also, can it be a static inline?

Quoted 50 lines
xen/arch/x86/numa.c +4 -0
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/common/page_alloc.c +1 -1
index 5801358b4b..80916205e5 100644
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();
 
     printk("Domain heap initialised");
xen/include/asm-arm/numa.h +5 -0
index 31a6de4e23..9d5739542d 100644
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+static inline unsigned int arch_have_default_dmazone(void)
+{
+    return 0;
+}
+
 #endif /* __ARCH_ARM_NUMA_H */
 /*
  * Local variables:
xen/include/asm-x86/numa.h +1 -0
index 3cf26c2def..8060cbf3f4 100644
@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 void srat_parse_regions(u64 addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
+unsigned int arch_have_default_dmazone(void);
 
 #endif
Signature · 1 line
2.25.1
author
#13 lore raw

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Stefano,

Quoted 26 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 7:56
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 5 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase

^ the ^Because

Quoted 2 lines

on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86

                                                              ^,
Quoted 3 lines

prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges.

^ of memory

Quoted 9 lines
Quoted 7 lines

The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

^ needs

I will fix above typos in next version.

Quoted 17 lines
Quoted 17 lines
---
 xen/arch/x86/numa.c        | 5 +++++
 xen/common/page_alloc.c    | 2 +-
 xen/include/asm-arm/numa.h | 5 +++++
 xen/include/asm-x86/numa.h | 1 +
 4 files changed, 12 insertions(+), 1 deletion(-)
xen/arch/x86/numa.c +0 -0
index ce79ee44ce..1fabbe8281 100644
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
Quoted 5 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 374

Quoted 1 line
xen/arch/x86/numa.c +1 -0
+unsigned int arch_have_default_dmazone(void)

Can this function return bool?
Also, can it be a static inline?

Yes, bool would be better. I will place a static inline in asm/numa.h. Because arm will have another static inline implementation.

Quoted 52 lines

xen/include/asm-x86/numa.h extern int valid_numa_range(u64 start, u64 end,

Quoted 42 lines
xen/arch/x86/numa.c +4 -0
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/common/page_alloc.c +1 -1
index 5801358b4b..80916205e5 100644
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();
 
     printk("Domain heap initialised");
xen/include/asm-arm/numa.h +5 -0
index 31a6de4e23..9d5739542d 100644
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+static inline unsigned int arch_have_default_dmazone(void)
+{
+    return 0;
+}
+
 #endif /* __ARCH_ARM_NUMA_H */
 /*
  * Local variables:
xen/include/asm-x86/numa.h +0 -0
index 3cf26c2def..8060cbf3f4 100644
@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end,

nodeid_t node);

Quoted 9 lines
xen/include/asm-x86/numa.h +1 -1
 void srat_parse_regions(u64 addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
+unsigned int arch_have_default_dmazone(void);
 
 #endif
--

2.25.1

#14 lore raw

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

In reply to Wei Chen

I realize this series has been pending for a long time, but I don't recall any indication that it would have been dropped. Hence as a first try, a few comments on this relatively simple change. I'm sorry it to have taken so long to get to it.

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377

Quoted 10 lines
xen/arch/x86/numa.c +4 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}

According to the expression and ...

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893

Quoted 9 lines
xen/common/page_alloc.c +1 -1
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();

... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 31

Quoted 10 lines
xen/include/asm-arm/numa.h +4 -0
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+static inline unsigned int arch_have_default_dmazone(void)
+{
+    return 0;
+}

... like this one, x86'es couldn't be inline as well. If indeed it can't be, making it a macro may still be better (and avoid a further comment regarding the lack of __init).

Jan

author
#15 lore raw

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Jan,

Quoted 9 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 5 lines

I realize this series has been pending for a long time, but I don't recall any indication that it would have been dropped. Hence as a first try, a few comments on this relatively simple change. I'm sorry it to have taken so long to get to it.

Thanks for reviewing this series and lift it up. We are still working on this series and will send a new version soon.

Quoted 19 lines · Wei Chen
Quoted 13 lines

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

Quoted 6 lines

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its dma_bitsize to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

Quoted 20 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377

Quoted 11 lines
xen/arch/x86/numa.c +4 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}

According to the expression and ...

Quoted 6 lines
xen/common/page_alloc.c +0 -0
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
Quoted 9 lines

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893

Quoted 3 lines
xen/common/page_alloc.c +1 -1
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();

... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...

I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return ( num_online_nodes() > 1 ) ? 1 : 0;"?

Quoted 6 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +0 -0
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
Quoted 8 lines

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 31

Quoted 4 lines
xen/include/asm-arm/numa.h +4 -0
+static inline unsigned int arch_have_default_dmazone(void)
+{
+    return 0;
+}

... like this one, x86'es couldn't be inline as well. If indeed it can't be, making it a macro may still be better (and avoid a further comment regarding the lack of __init).

Ok, that would be better, I will do it in next version.

Quoted 2 lines

Jan

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00772.html

#16 lore raw

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 29 lines · Wei Chen
Quoted 22 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

Quoted 6 lines

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 6 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its dma_bitsize to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.

Quoted 29 lines
Quoted 19 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377

Quoted 10 lines
xen/arch/x86/numa.c +4 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}

According to the expression and ...

Quoted 6 lines
xen/common/page_alloc.c +0 -0
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
Quoted 9 lines

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893

Quoted 3 lines
xen/common/page_alloc.c +1 -1
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();

... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...

Quoted 4 lines

I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return ( num_online_nodes() > 1 ) ? 1 : 0;"?

Yes. Even without the use of bool such an expression is a more complicated form of

    return num_online_nodes() > 1;

where we'd prefer to use the simpler variant for being easier to read / follow.

Jan

Quoted 3 lines

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00772.html

author
#17 lore raw

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Jan,

Quoted 46 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 29 lines · Wei Chen
Quoted 22 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

Quoted 6 lines

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 6 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its dma_bitsize to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

Quoted 6 lines

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.

I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of dma_bitsize indicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?

Quoted 35 lines
Quoted 30 lines
Quoted 20 lines

xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377

Quoted 11 lines
xen/arch/x86/numa.c +4 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void)
                  + PAGE_SHIFT, 32);
 }
 
+unsigned int arch_have_default_dmazone(void)
+{
+    return ( num_online_nodes() > 1 ) ? 1 : 0;
+}

According to the expression and ...

Quoted 6 lines
xen/common/page_alloc.c +0 -0
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void)
     }
     nr_bootmem_regions = 0;
 
Quoted 9 lines

xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893

Quoted 3 lines
xen/common/page_alloc.c +1 -1
-    if ( !dma_bitsize && (num_online_nodes() > 1) )
+    if ( !dma_bitsize && arch_have_default_dmazone() )
         dma_bitsize = arch_get_dma_bitsize();

... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...

Quoted 4 lines

I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return ( num_online_nodes() > 1 ) ? 1 : 0;"?

Quoted 8 lines

Yes. Even without the use of bool such an expression is a more complicated form of

    return num_online_nodes() > 1;

where we'd prefer to use the simpler variant for being easier to read / follow.

Thanks for clarification, I will fix it.

Quoted 6 lines

Jan

Quoted 1 line

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00772.html

Quoted 2 lines
#18 lore raw

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 46 lines · Wei Chen
Quoted 39 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16

Quoted 28 lines · Wei Chen
Quoted 21 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

Quoted 6 lines

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 6 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its dma_bitsize to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

Quoted 6 lines

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.

Quoted 7 lines

I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of dma_bitsize indicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?

I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...

Jan

author
#19 lore raw

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Jan,

Quoted 65 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 32 lines · Wei Chen
Quoted 32 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16

Quoted 28 lines · Wei Chen
Quoted 27 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its

dma_bitsize

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.

Quoted 7 lines

I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of dma_bitsize indicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?

Quoted 10 lines

I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...

I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?

Thanks,
Wei Chen

Quoted 1 line

Jan

#20 lore raw

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 78 lines · Wei Chen

Hi Jan,

Quoted 65 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 32 lines · Wei Chen
Quoted 32 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16

Quoted 28 lines · Wei Chen
Quoted 27 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 4 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its

dma_bitsize

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.

Quoted 7 lines

I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of dma_bitsize indicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?

Quoted 10 lines

I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...

Quoted 4 lines

I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?

Extending the description is my primary request, or else we may end up having the same discussion every time you submit a new version. As to improving the situation such that preferably per-arch customization wouldn't be necessary - that's perhaps better to be thought about by Arm folks. Otoh, as said, an x86 build with CONFIG_PV=n could probably leverage the new hook to actually not trigger reservation.

Jan

author
#21 lore raw

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Jan,

Quoted 102 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 15:50
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Quoted 9 lines · Wei Chen

Hi Jan,

Quoted 7 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default

dma

Quoted 23 lines
Quoted 23 lines

zone status

Quoted 20 lines · Wei Chen
Quoted 20 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16

Quoted 16 lines · Wei Chen
Quoted 16 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11

Quoted 13 lines · Wei Chen

In current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.

There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to

determine

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

that it need to set dma_bitsize for reserve DMA allocations or not.

How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its

dma_bitsize

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And

some

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

previous discussions are placed here [1].

I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of

such

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

a range to perform I/O to _some_ devices.

I am sorry that my last reply hasn't given you more clues. On Arm,

only

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of

dma_bitsize

Quoted 1 line
Quoted 1 line
Quoted 1 line

indicated. I think these operations meet your above Dom0 special

address

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is

that

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

answers your earlier question?

I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-

like,

Quoted 9 lines
Quoted 4 lines

due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...

I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?

Quoted 7 lines

Extending the description is my primary request, or else we may end up having the same discussion every time you submit a new version. As to improving the situation such that preferably per-arch customization wouldn't be necessary - that's perhaps better to be thought about by Arm folks. Otoh, as said, an x86 build with CONFIG_PV=n could probably leverage the new hook to actually not trigger reservation.

Ok, I will try to extend the description to avoid the same discussion in next version.

Quoted 1 line

Jan

author
#22 lore raw

PATCH 5/37 xen: decouple NUMA from ACPI in Kconfig

In reply to Wei Chen

In current Xen code only implements x86 ACPI-based NUMA support. So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.

In a follow-up patch, we will introduce support for NUMA using the device tree. That means we will have two NUMA implementations, so in this patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.

---
 xen/arch/x86/Kconfig         | 2 +-
 xen/common/Kconfig           | 3 +++
 xen/drivers/acpi/Kconfig     | 3 ++-
 xen/drivers/acpi/Makefile    | 2 +-
 xen/include/asm-x86/config.h | 1 -
 5 files changed, 7 insertions(+), 4 deletions(-)
xen/arch/x86/Kconfig +1 -1
index 1f83518ee0..28d13b9705 100644
@@ -6,6 +6,7 @@ config X86
 	def_bool y
 	select ACPI
 	select ACPI_LEGACY_TABLES_LOOKUP
+	select ACPI_NUMA
 	select ALTERNATIVE_CALL
 	select ARCH_SUPPORTS_INT128
 	select CORE_PARKING
@@ -25,7 +26,6 @@ config X86
 	select HAS_UBSAN
 	select HAS_VPCI if HVM
 	select NEEDS_LIBELF
-	select NUMA
 
 config ARCH_DEFCONFIG
 	string
xen/common/Kconfig +3 -0
index db687b1785..9ebb1c239b 100644
@@ -70,6 +70,9 @@ config MEM_ACCESS
 config NEEDS_LIBELF
 	bool
 
+config NUMA
+	bool
+
 config STATIC_MEMORY
 	bool "Static Allocation Support (UNSUPPORTED)" if UNSUPPORTED
 	depends on ARM
xen/drivers/acpi/Kconfig +2 -1
index b64d3731fb..e3f3d8f4b1 100644
@@ -5,5 +5,6 @@ config ACPI
 config ACPI_LEGACY_TABLES_LOOKUP
 	bool
 
-config NUMA
+config ACPI_NUMA
 	bool
+	select NUMA
xen/drivers/acpi/Makefile +1 -1
index 4f8e97228e..2fc5230253 100644
@@ -3,7 +3,7 @@ obj-y += utilities/
 obj-$(CONFIG_X86) += apei/
 
 obj-bin-y += tables.init.o
-obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_ACPI_NUMA) += numa.o
 obj-y += osl.o
 obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
 
xen/include/asm-x86/config.h +0 -1
index 883c2ef0df..9a6f0a6edf 100644
@@ -31,7 +31,6 @@
 /* Intel P4 currently has largest cache line (L2 line size is 128 bytes). */
 #define CONFIG_X86_L1_CACHE_SHIFT 7
 
-#define CONFIG_ACPI_NUMA 1
 #define CONFIG_ACPI_SRAT 1
 #define CONFIG_ACPI_CSTATE 1
 
Signature · 1 line
2.25.1
author
#23 lore raw

PATCH 6/37 xen/arm: use !CONFIG_NUMA to keep fake NUMA API

In reply to Wei Chen

We have introduced CONFIG_NUMA in previous patch. And this option is enabled only on x86 in current stage. In a follow up patch, we will enable this option for Arm. But we still want users can disable the CONFIG_NUMA through Kconfig. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
 xen/include/asm-arm/numa.h | 4 ++++
 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
index 9d5739542d..8f1c67e3eb 100644
@@ -5,6 +5,8 @@
 
 typedef u8 nodeid_t;
 
+#ifndef CONFIG_NUMA
+
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
 #define node_to_cpumask(node)   (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#endif
+
 static inline unsigned int arch_have_default_dmazone(void)
 {
     return 0;
Signature · 1 line
2.25.1
Reviewed-by on v1 6/37
#24 lore raw

Re: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

We have introduced CONFIG_NUMA in previous patch. And this

                                   ^ a
Quoted 1 line

option is enabled only on x86 in current stage. In a follow

^ at the

Quoted 2 lines

up patch, we will enable this option for Arm. But we still want users can disable the CONFIG_NUMA through Kconfig. In

^ to be able to disable CONFIG_NUMA via Kconfig.

Quoted 1 line

this case, keep current fake NUMA API, will make Arm code

^ the

Quoted 1 line

still can work with NUMA aware memory allocation and scheduler.

^ able to work

Quoted 2 lines

With the small grammar fixes:

Quoted 29 lines
---
 xen/include/asm-arm/numa.h | 4 ++++
 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
index 9d5739542d..8f1c67e3eb 100644
@@ -5,6 +5,8 @@
 
 typedef u8 nodeid_t;
 
+#ifndef CONFIG_NUMA
+
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
 #define node_to_cpumask(node)   (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#endif
+
 static inline unsigned int arch_have_default_dmazone(void)
 {
     return 0;
Signature · 1 line
2.25.1
author
#25 lore raw

RE: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 25 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:05
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

We have introduced CONFIG_NUMA in previous patch. And this

                                   ^ a
Quoted 1 line

option is enabled only on x86 in current stage. In a follow

^ at the

Quoted 2 lines

up patch, we will enable this option for Arm. But we still want users can disable the CONFIG_NUMA through Kconfig. In

^ to be able to disable CONFIG_NUMA via Kconfig.

Quoted 1 line

this case, keep current fake NUMA API, will make Arm code

^ the

Quoted 1 line

still can work with NUMA aware memory allocation and scheduler.

Quoted 10 lines

^ able to work

Quoted 2 lines

With the small grammar fixes:

Thanks, I will fix them in next version.

Quoted 29 lines
Quoted 29 lines
---
 xen/include/asm-arm/numa.h | 4 ++++
 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -1
index 9d5739542d..8f1c67e3eb 100644
@@ -5,6 +5,8 @@
 
 typedef u8 nodeid_t;
 
+#ifndef CONFIG_NUMA
+
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
 #define node_to_cpumask(node)   (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#endif
+
 static inline unsigned int arch_have_default_dmazone(void)
 {
     return 0;
--

2.25.1

author
#26 lore raw

PATCH 7/37 xen/x86: use paddr_t for addresses in NUMA node structure

In reply to Wei Chen

NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to paddr_t. And use pfn_to_paddr and paddr_to_pfn to replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.

---
 xen/arch/x86/numa.c        | 32 +++++++++++++++++---------------
 xen/arch/x86/srat.c        | 26 +++++++++++++-------------
 xen/include/asm-x86/numa.h |  8 ++++----
 3 files changed, 34 insertions(+), 32 deletions(-)
xen/arch/x86/numa.c +17 -15
index 1fabbe8281..6337bbdf31 100644
@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
     return shift;
 }
 /* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{ 
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
+{
     unsigned long start_pfn, end_pfn;
 
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
+    start_pfn = paddr_to_pfn(start);
+    end_pfn = paddr_to_pfn(end);
 
     NODE_DATA(nodeid)->node_start_pfn = start_pfn;
     NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;
 
     /* Kludge needed for the hash function */
     if ( hweight64(sz) > 1 )
@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
     memset(&nodes,0,sizeof(nodes));
     for ( i = 0; i < numa_fake; i++ )
     {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        nodes[i].start = pfn_to_paddr(start_pfn) + i*sz;
         if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+            sz = pfn_to_paddr(end_pfn) - nodes[i].start;
         nodes[i].end = nodes[i].start + sz;
         printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
                i,
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
 { 
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);
+
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration found");
 
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
+           start, end);
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     for ( i = 0; i < nr_cpu_ids; i++ )
         numa_set_node(i, 0);
     cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
+    setup_node_bootmem(0, start, end);
 }
 
 void numa_add_cpu(int cpu)
xen/arch/x86/srat.c +13 -13
index 6b77b98201..7d20d7f222 100644
@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
 
@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)
 	return 0;
 }
 
-static __init int conflicting_memblks(u64 start, u64 end)
+static __init int conflicting_memblks(paddr_t start, paddr_t end)
 {
 	int i;
 
@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)
 	return -1;
 }
 
-static __init void cutoff_node(int i, u64 start, u64 end)
+static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 {
 	struct node *nd = &nodes[i];
 	if (nd->start < start) {
@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 {
-	u64 start, end;
+	paddr_t start, end;
 	unsigned pxm;
 	nodeid_t node;
 	int i;
@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
 		                !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
 		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
+		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
 		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		bad_srat();
@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->end = end;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
+	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
 	       node, pxm, start, end,
 	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
 
@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)
 
 	for (i = 0; i < e820.nr_map; i++) {
 		int j, found;
-		unsigned long long start, end;
+		paddr_t start, end;
 
 		if (e820.map[i].type != E820_RAM) {
 			continue;
@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)
 
 		if (start < end) {
 			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
+				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
 	}
@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,
 	return 0;
 }
 
-void __init srat_parse_regions(u64 addr)
+void __init srat_parse_regions(paddr_t addr)
 {
 	u64 mask;
 	unsigned int i;
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);
 	acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
 			      srat_parse_region, 0);
 
@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
 			       "BIOS Bug or mis-configured hardware?\n", i);
xen/include/asm-x86/numa.h +4 -4
index 8060cbf3f4..50cfd8e7ef 100644
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node { 
-	u64 start,end; 
+	paddr_t start,end;
 };
 
 extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
 
-void srat_parse_regions(u64 addr);
+void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
Signature · 1 line
2.25.1
#27 lore raw

Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 36 lines · Wei Chen

NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to paddr_t. And use pfn_to_paddr and paddr_to_pfn to replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.

---
 xen/arch/x86/numa.c        | 32 +++++++++++++++++---------------
 xen/arch/x86/srat.c        | 26 +++++++++++++-------------
 xen/include/asm-x86/numa.h |  8 ++++----
 3 files changed, 34 insertions(+), 32 deletions(-)
xen/arch/x86/numa.c +4 -4
index 1fabbe8281..6337bbdf31 100644
@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
     return shift;
 }
 /* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{ 
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
+{
     unsigned long start_pfn, end_pfn;
 
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
+    start_pfn = paddr_to_pfn(start);
+    end_pfn = paddr_to_pfn(end);
 
     NODE_DATA(nodeid)->node_start_pfn = start_pfn;
     NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/arch/x86/numa.c void __init numa_init_array(void) · line 205

Quoted 6 lines
xen/arch/x86/numa.c +2 -1
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)

Why not changing numa_emulation to take paddr_t too?

Quoted 21 lines
xen/arch/x86/numa.c +3 -3
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;
 
     /* Kludge needed for the hash function */
     if ( hweight64(sz) > 1 )
@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
     memset(&nodes,0,sizeof(nodes));
     for ( i = 0; i < numa_fake; i++ )
     {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        nodes[i].start = pfn_to_paddr(start_pfn) + i*sz;
         if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+            sz = pfn_to_paddr(end_pfn) - nodes[i].start;
         nodes[i].end = nodes[i].start + sz;
         printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
                i,

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 250

Quoted 1 line
xen/arch/x86/numa.c +0 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)

same here

Quoted 198 lines
xen/arch/x86/numa.c +8 -7
 { 
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);
+
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration found");
 
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
+           start, end);
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     for ( i = 0; i < nr_cpu_ids; i++ )
         numa_set_node(i, 0);
     cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
+    setup_node_bootmem(0, start, end);
 }
 
 void numa_add_cpu(int cpu)
xen/arch/x86/srat.c +13 -13
index 6b77b98201..7d20d7f222 100644
@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
 
@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)
 	return 0;
 }
 
-static __init int conflicting_memblks(u64 start, u64 end)
+static __init int conflicting_memblks(paddr_t start, paddr_t end)
 {
 	int i;
 
@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)
 	return -1;
 }
 
-static __init void cutoff_node(int i, u64 start, u64 end)
+static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 {
 	struct node *nd = &nodes[i];
 	if (nd->start < start) {
@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 {
-	u64 start, end;
+	paddr_t start, end;
 	unsigned pxm;
 	nodeid_t node;
 	int i;
@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
 		                !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
 		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
+		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
 		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		bad_srat();
@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->end = end;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
+	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
 	       node, pxm, start, end,
 	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
 
@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)
 
 	for (i = 0; i < e820.nr_map; i++) {
 		int j, found;
-		unsigned long long start, end;
+		paddr_t start, end;
 
 		if (e820.map[i].type != E820_RAM) {
 			continue;
@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)
 
 		if (start < end) {
 			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
+				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
 	}
@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,
 	return 0;
 }
 
-void __init srat_parse_regions(u64 addr)
+void __init srat_parse_regions(paddr_t addr)
 {
 	u64 mask;
 	unsigned int i;
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);
 	acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
 			      srat_parse_region, 0);
 
@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
 			       "BIOS Bug or mis-configured hardware?\n", i);
xen/include/asm-x86/numa.h +4 -4
index 8060cbf3f4..50cfd8e7ef 100644
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node { 
-	u64 start,end; 
+	paddr_t start,end;
 };
 
 extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
 
-void srat_parse_regions(u64 addr);
+void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
Signature · 1 line
2.25.1
#28 lore raw

Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

You forgot to add the x86 maintainers in CC to all the patches touching x86 code in this series. Adding them now but you should probably resend.

Quoted 272 lines · Stefano Stabellini
Quoted 36 lines · Wei Chen

NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to paddr_t. And use pfn_to_paddr and paddr_to_pfn to replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.

---
 xen/arch/x86/numa.c        | 32 +++++++++++++++++---------------
 xen/arch/x86/srat.c        | 26 +++++++++++++-------------
 xen/include/asm-x86/numa.h |  8 ++++----
 3 files changed, 34 insertions(+), 32 deletions(-)
xen/arch/x86/numa.c +4 -4
index 1fabbe8281..6337bbdf31 100644
@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
     return shift;
 }
 /* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{ 
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
+{
     unsigned long start_pfn, end_pfn;
 
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
+    start_pfn = paddr_to_pfn(start);
+    end_pfn = paddr_to_pfn(end);
 
     NODE_DATA(nodeid)->node_start_pfn = start_pfn;
     NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/arch/x86/numa.c void __init numa_init_array(void) · line 205

Quoted 6 lines
xen/arch/x86/numa.c +2 -1
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)

Why not changing numa_emulation to take paddr_t too?

Quoted 21 lines
xen/arch/x86/numa.c +3 -3
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;
 
     /* Kludge needed for the hash function */
     if ( hweight64(sz) > 1 )
@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
     memset(&nodes,0,sizeof(nodes));
     for ( i = 0; i < numa_fake; i++ )
     {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        nodes[i].start = pfn_to_paddr(start_pfn) + i*sz;
         if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+            sz = pfn_to_paddr(end_pfn) - nodes[i].start;
         nodes[i].end = nodes[i].start + sz;
         printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
                i,

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 250

Quoted 1 line
xen/arch/x86/numa.c +0 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)

same here

Quoted 198 lines
xen/arch/x86/numa.c +8 -7
 { 
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);
+
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration found");
 
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
+           start, end);
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     for ( i = 0; i < nr_cpu_ids; i++ )
         numa_set_node(i, 0);
     cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
+    setup_node_bootmem(0, start, end);
 }
 
 void numa_add_cpu(int cpu)
xen/arch/x86/srat.c +13 -13
index 6b77b98201..7d20d7f222 100644
@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
 
@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)
 	return 0;
 }
 
-static __init int conflicting_memblks(u64 start, u64 end)
+static __init int conflicting_memblks(paddr_t start, paddr_t end)
 {
 	int i;
 
@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)
 	return -1;
 }
 
-static __init void cutoff_node(int i, u64 start, u64 end)
+static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 {
 	struct node *nd = &nodes[i];
 	if (nd->start < start) {
@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 {
-	u64 start, end;
+	paddr_t start, end;
 	unsigned pxm;
 	nodeid_t node;
 	int i;
@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
 		                !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
 		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
+		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
 		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		bad_srat();
@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->end = end;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
+	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
 	       node, pxm, start, end,
 	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
 
@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)
 
 	for (i = 0; i < e820.nr_map; i++) {
 		int j, found;
-		unsigned long long start, end;
+		paddr_t start, end;
 
 		if (e820.map[i].type != E820_RAM) {
 			continue;
@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)
 
 		if (start < end) {
 			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
+				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
 	}
@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,
 	return 0;
 }
 
-void __init srat_parse_regions(u64 addr)
+void __init srat_parse_regions(paddr_t addr)
 {
 	u64 mask;
 	unsigned int i;
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);
 	acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
 			      srat_parse_region, 0);
 
@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
 			       "BIOS Bug or mis-configured hardware?\n", i);
xen/include/asm-x86/numa.h +4 -4
index 8060cbf3f4..50cfd8e7ef 100644
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node { 
-	u64 start,end; 
+	paddr_t start,end;
 };
 
 extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
 
-void srat_parse_regions(u64 addr);
+void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
Signature · 1 line
2.25.1
author
#29 lore raw

RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 13 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:14
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 3 lines

You forgot to add the x86 maintainers in CC to all the patches touching x86 code in this series. Adding them now but you should probably resend.

I am very sorry about it. I realized the problem when I pressed Enter. I had wanted to repost it at that time, but I didn't know whether these patches will be turned into spamming...

I will resend this series ASAP with some changes to address your comments.

Quoted 23 lines · Stefano Stabellini
Quoted 15 lines · Wei Chen
Quoted 14 lines

NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to paddr_t. And use pfn_to_paddr and paddr_to_pfn to replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.

---
 xen/arch/x86/numa.c        | 32 +++++++++++++++++---------------
 xen/arch/x86/srat.c        | 26 +++++++++++++-------------
 xen/include/asm-x86/numa.h |  8 ++++----
 3 files changed, 34 insertions(+), 32 deletions(-)
Quoted 5 lines

xen/arch/x86/numa.c int __init compute_hash_shift(struct node *nodes,

Quoted 5 lines
xen/arch/x86/numa.c +0 -0
index 1fabbe8281..6337bbdf31 100644
@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes,

int numnodes,

Quoted 27 lines
Quoted 6 lines

xen/arch/x86/numa.c int __init compute_hash_shift(struct node *nodes, int numnodes,

Quoted 6 lines
xen/arch/x86/numa.c +1 -2
     return shift;
 }
 /* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start,

paddr_t end)

Quoted 20 lines
Quoted 11 lines
xen/arch/x86/numa.c +3 -2
+{
     unsigned long start_pfn, end_pfn;
 
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
+    start_pfn = paddr_to_pfn(start);
+    end_pfn = paddr_to_pfn(end);
 
     NODE_DATA(nodeid)->node_start_pfn = start_pfn;
     NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/arch/x86/numa.c void __init numa_init_array(void) · line 205

Quoted 6 lines
xen/arch/x86/numa.c +2 -1
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)

Why not changing numa_emulation to take paddr_t too?

numa_emulation parameter is pfn, it's not address. I have discussed with Julien in RFC about pfn. He suggested to use mfn_t or unsigned long for pfn. Comparing to mfn_t, use unsigned long brings less changes.

Quoted 249 lines
Quoted 10 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn,

Quoted 10 lines
xen/arch/x86/numa.c +1 -1
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;
 
     /* Kludge needed for the hash function */
     if ( hweight64(sz) > 1 )
@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn,

u64 end_pfn)

Quoted 10 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Quoted 10 lines
xen/arch/x86/numa.c +2 -2
     memset(&nodes,0,sizeof(nodes));
     for ( i = 0; i < numa_fake; i++ )
     {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        nodes[i].start = pfn_to_paddr(start_pfn) + i*sz;
         if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+            sz = pfn_to_paddr(end_pfn) - nodes[i].start;
         nodes[i].end = nodes[i].start + sz;
         printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"

(%"PRIu64"MB)\n",

Quoted 2 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn,

Quoted 2 lines
xen/arch/x86/numa.c +0 -0
                i,
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,

u64 end_pfn)

Quoted 1 line

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, · line 250

Quoted 1 line
xen/arch/x86/numa.c +0 -0
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 25 lines

same here

Quoted 21 lines
xen/arch/x86/numa.c +5 -2
 {
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);
+
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration

found");

Quoted 5 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Quoted 5 lines
xen/arch/x86/numa.c +1 -3
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-

%016"PRIpaddr"\n",

Quoted 5 lines

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long

Quoted 5 lines
xen/arch/x86/numa.c +1 -0
+           start, end);
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 17 lines
Quoted 17 lines
xen/arch/x86/numa.c +1 -2
     for ( i = 0; i < nr_cpu_ids; i++ )
         numa_set_node(i, 0);
     cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
+    setup_node_bootmem(0, start, end);
 }
 
 void numa_add_cpu(int cpu)
xen/arch/x86/srat.c +0 -0
index 6b77b98201..7d20d7f222 100644
@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
Quoted 6 lines

xen/arch/x86/srat.c int valid_numa_range(u64 start, u64 end, nodeid_t

Quoted 6 lines
xen/arch/x86/srat.c +1 -1
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
 
@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t

node)

Quoted 9 lines

xen/arch/x86/srat.c static __init int conflicting_memblks(u64 start,

Quoted 9 lines
xen/arch/x86/srat.c +1 -1
 	return 0;
 }
 
-static __init int conflicting_memblks(u64 start, u64 end)
+static __init int conflicting_memblks(paddr_t start, paddr_t end)
 {
 	int i;
 
@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start,

u64 end)

Quoted 9 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct

Quoted 9 lines
xen/arch/x86/srat.c +1 -1
 	return -1;
 }
 
-static __init void cutoff_node(int i, u64 start, u64 end)
+static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 {
 	struct node *nd = &nodes[i];
 	if (nd->start < start) {
@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct

acpi_srat_cpu_affinity *pa)

Quoted 2 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct · line 276

Quoted 2 lines
xen/arch/x86/srat.c +0 -0
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity

*ma)

Quoted 7 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 7 lines
xen/arch/x86/srat.c +1 -1
 {
-	u64 start, end;
+	paddr_t start, end;
 	unsigned pxm;
 	nodeid_t node;
 	int i;
@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 4 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 4 lines
xen/arch/x86/srat.c +0 -1
 		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
 		                !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with

itself (%"PRIx64"-%"PRIx64")\n",

Quoted 1 line

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 321

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with

itself (%"PRIpaddr"-%"PRIpaddr")\n",

Quoted 4 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 4 lines
xen/arch/x86/srat.c +0 -0
 		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 4 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 4 lines
xen/arch/x86/srat.c +0 -1
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with

PXM %u (%"PRIx64"-%"PRIx64")\n",

Quoted 1 line

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 330

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with

PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",

Quoted 4 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 4 lines
xen/arch/x86/srat.c +0 -0
 		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		bad_srat();
@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 19 lines
Quoted 19 lines
xen/arch/x86/srat.c +2 -2
 				nd->end = end;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
+	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
 	       node, pxm, start, end,
 	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
 
@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)
 
 	for (i = 0; i < e820.nr_map; i++) {
 		int j, found;
-		unsigned long long start, end;
+		paddr_t start, end;
 
 		if (e820.map[i].type != E820_RAM) {
 			continue;
@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)
 
Quoted 8 lines

xen/arch/x86/srat.c static int __init srat_parse_region(struct

Quoted 8 lines
xen/arch/x86/srat.c +1 -1
 		if (start < end) {
 			printk(KERN_ERR "SRAT: No PXM for e820 range: "
-				"%016Lx - %016Lx\n", start, end);
+				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
 	}
@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct

acpi_subtable_header *header,

Quoted 17 lines
Quoted 17 lines
xen/arch/x86/srat.c +2 -2
 	return 0;
 }
 
-void __init srat_parse_regions(u64 addr)
+void __init srat_parse_regions(paddr_t addr)
 {
 	u64 mask;
 	unsigned int i;
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);
 	acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
 			      srat_parse_region, 0);
 
Quoted 4 lines

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 459

Quoted 4 lines
xen/arch/x86/srat.c +0 -0
@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr)
 }
 
 /* Use the information discovered above to actually set up the nodes.

*/

Quoted 26 lines
Quoted 26 lines
xen/arch/x86/srat.c +2 -2
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
 			       "BIOS Bug or mis-configured hardware?\n", i);
xen/include/asm-x86/numa.h +1 -1
index 8060cbf3f4..50cfd8e7ef 100644
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node {
-	u64 start,end;
+	paddr_t start,end;
 };
 
Quoted 7 lines

xen/include/asm-x86/numa.h extern void numa_set_node(int cpu, nodeid_t node); · line 39

Quoted 7 lines
xen/include/asm-x86/numa.h +1 -1
 extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start,

paddr_t end);

Quoted 4 lines

xen/include/asm-x86/numa.h static inline __attribute__((pure)) nodeid_t

Quoted 4 lines
xen/include/asm-x86/numa.h +0 -0
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t

phys_to_nid(paddr_t addr)

Quoted 5 lines

xen/include/asm-x86/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)

Quoted 5 lines
xen/include/asm-x86/numa.h +1 -1
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t

node);

Quoted 10 lines
Quoted 9 lines
xen/include/asm-x86/numa.h +1 -2
-void srat_parse_regions(u64 addr);
+void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
--

2.25.1

#30 lore raw

Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

In reply to Wei Chen

xen/arch/x86/numa.c void __init numa_init_array(void) · line 209

Quoted 12 lines · Wei Chen
xen/arch/x86/numa.c +3 -2
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;

Nit: Please convert to uint64_t (and alike) whenever you touch a line anyway that uses being-phased-out types.

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261

Quoted 13 lines
xen/arch/x86/numa.c +3 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
 { 
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);

Nit: Would be slightly neater if these were the initializers of the variables.

Quoted 5 lines
xen/arch/x86/numa.c +1 -2
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 272

Quoted 10 lines
xen/arch/x86/numa.c +2 -3
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration found");
 
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
+           start, end);

When switching to PRIpaddr I suppose you did look up what that one expands to? IOW - please drop the 016 from here.

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444

Quoted 6 lines
xen/arch/x86/srat.c +1 -1
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);

I don't see the need for a cast here.

xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 493

Quoted 7 lines
xen/arch/x86/srat.c +1 -1
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )

Please take the opportunity and add the missing blank line between declarations and statements.

xen/include/asm-x86/numa.h extern cpumask_t node_to_cpumask[]; · line 19

Quoted 8 lines
xen/include/asm-x86/numa.h +1 -1
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node { 
-	u64 start,end; 
+	paddr_t start,end;

Please take the opportunity and add the missing blank after the comma.

Jan

author
#31 lore raw

RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Hi Jan,

Quoted 13 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0 -0
@@ -201,11 +201,12 @@ void __init numa_init_array(void)
 static int numa_fake __initdata = 0;
 
Quoted 13 lines

xen/arch/x86/numa.c void __init numa_init_array(void) · line 209

Quoted 9 lines
xen/arch/x86/numa.c +3 -2
 /* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(unsigned long start_pfn,
+                                 unsigned long end_pfn)
 {
     int i;
     struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;

Nit: Please convert to uint64_t (and alike) whenever you touch a line anyway that uses being-phased-out types.

Ok, I will do it.

Quoted 13 lines

Review static int __init numa_emulation(u64 start_pfn,

Quoted 1 line
(quoted diff) +0 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,

u64 end_pfn)

Review static int __init numa_emulation(u64 start_pfn, · line 250

Quoted 1 line
(quoted diff) +0 -0
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 9 lines
xen/arch/x86/numa.c +1 -0
 {
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
Quoted 5 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);

Nit: Would be slightly neater if these were the initializers of the variables.

But if this function returns in numa_fake && !numa_emulation, will the two pfn_to_paddr operations be waste?

Quoted 12 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Quoted 10 lines
xen/arch/x86/numa.c +1 -2
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
+    if ( !numa_off && !acpi_scan_nodes(start, end) )
         return;
 #endif
 
     printk(KERN_INFO "%s\n",
            numa_off ? "NUMA turned off" : "No NUMA configuration

found");

Quoted 1 line
Quoted 8 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 272

Quoted 5 lines
xen/arch/x86/numa.c +2 -3
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
+    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
+           start, end);

When switching to PRIpaddr I suppose you did look up what that one expands to? IOW - please drop the 016 from here.

Oh, yes, I forgot to drop the duplicated 016. I would do it.

Quoted 10 lines

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444

Quoted 7 lines
xen/arch/x86/srat.c +1 -1
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);

I don't see the need for a cast here.

current addr type has been changed to paddr_t, but pdx_init_mask accept parameter type is u64. I know paddr_t is a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?

Quoted 11 lines

xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 493

Quoted 7 lines
xen/arch/x86/srat.c +1 -1
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
 	/* Finally register nodes */
 	for_each_node_mask(i, all_nodes_parsed)
 	{
-		u64 size = nodes[i].end - nodes[i].start;
+		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )

Please take the opportunity and add the missing blank line between declarations and statements.

Ok

Quoted 11 lines

xen/include/asm-x86/numa.h extern cpumask_t node_to_cpumask[]; · line 19

Quoted 8 lines
xen/include/asm-x86/numa.h +1 -1
@@ -16,7 +16,7 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
 struct node {
-	u64 start,end;
+	paddr_t start,end;

Please take the opportunity and add the missing blank after the comma.

Ok

Quoted 1 line

Jan

#32 lore raw

Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 17 lines · Wei Chen
Quoted 17 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23

Review static int __init numa_emulation(u64 start_pfn,

Quoted 1 line · Wei Chen
(quoted diff) +0 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,

u64 end_pfn)

Review static int __init numa_emulation(u64 start_pfn, · line 250

Quoted 1 line
(quoted diff) +0 -0
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 9 lines
xen/arch/x86/numa.c +1 -0
 {
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
Quoted 8 lines
Quoted 5 lines

xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);

Nit: Would be slightly neater if these were the initializers of the variables.

But if this function returns in numa_fake && !numa_emulation, will the two pfn_to_paddr operations be waste?

And what harm would that do?

Quoted 10 lines
Quoted 9 lines

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444

Quoted 6 lines
xen/arch/x86/srat.c +1 -1
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);

I don't see the need for a cast here.

Quoted 5 lines

current addr type has been changed to paddr_t, but pdx_init_mask accept parameter type is u64. I know paddr_t is a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?

It may only ever be an up-cast, yet the compiler would do a widening conversion (according to the usual type conversion rules) for us anyway no matter whether there's a cast. Down-casts (in the general compiler case, i.e. considering a wider set than just gcc and clang) sometimes need making explicit to silence compiler warnings about truncation, but I've not observed any compiler warning when widening values.

Jan

author
#33 lore raw

RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Hi Jan,

Quoted 30 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 15:55
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure

Quoted 20 lines · Wei Chen
Quoted 20 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23

Review static int __init numa_emulation(u64 start_pfn,

Quoted 1 line · Wei Chen
(quoted diff) +0 -0
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,

u64 end_pfn)

Review static int __init numa_emulation(u64 start_pfn, · line 250

Quoted 1 line
(quoted diff) +0 -0
 void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 11 lines
xen/arch/x86/numa.c +3 -0
 {
     int i;
+    paddr_t start, end;
 
 #ifdef CONFIG_NUMA_EMU
     if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
         return;
 #endif
 
+    start = pfn_to_paddr(start_pfn);
+    end = pfn_to_paddr(end_pfn);
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Nit: Would be slightly neater if these were the initializers of the variables.

But if this function returns in numa_fake && !numa_emulation, will the two pfn_to_paddr operations be waste?

And what harm would that do?

Ok, two or three instructions waste in init time will not take too much harm. I will move them above with initializers.

Quoted 17 lines
Quoted 11 lines
Quoted 10 lines

xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444

Quoted 7 lines
xen/arch/x86/srat.c +1 -1
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr)
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
-	srat_region_mask = pdx_init_mask(addr);
+	srat_region_mask = pdx_init_mask((u64)addr);

I don't see the need for a cast here.

Quoted 5 lines

current addr type has been changed to paddr_t, but pdx_init_mask accept parameter type is u64. I know paddr_t is a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?

Quoted 7 lines

It may only ever be an up-cast, yet the compiler would do a widening conversion (according to the usual type conversion rules) for us anyway no matter whether there's a cast. Down-casts (in the general compiler case, i.e. considering a wider set than just gcc and clang) sometimes need making explicit to silence compiler warnings about truncation, but I've not observed any compiler warning when widening values.

Ok, I will drop that cast.

Quoted 2 lines

Jan

author
#34 lore raw

PATCH 8/37 xen/x86: add detection of discontinous node memory range

In reply to Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

---
 xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
xen/arch/x86/srat.c +36 -0
index 7d20d7f222..2f08fa4660 100644
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 		       pxm, pa->apic_id, node);
 }
 
+/*
+ * Check to see if there are other nodes within this node's range.
+ * We just need to check full contains situation. Because overlaps
+ * have been checked before by conflicting_memblks.
+ */
+static bool __init is_node_memory_continuous(nodeid_t nid,
+    paddr_t start, paddr_t end)
+{
+	nodeid_t i;
+
+	struct node *nd = &nodes[nid];
+	for_each_node_mask(i, memory_nodes_parsed)
+	{
+		/* Skip itself */
+		if (i == nid)
+			continue;
+
+		nd = &nodes[i];
+		if (start < nd->start && nd->end < end)
+		{
+			printk(KERN_ERR
+			       "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
+			       nid, start, end, i, nd->start, nd->end);
+			return false;
+		}
+	}
+
+	return true;
+}
+
 /* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->start = start;
 			if (nd->end < end)
 				nd->end = end;
+
+			/* Check whether this range contains memory for other nodes */
+			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
+				bad_srat();
+				return;
+			}
 		}
 	}
 	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
Signature · 1 line
2.25.1
#35 lore raw

Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

Quoted 7 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

Quoted 8 lines

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Quoted 62 lines
---
 xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
xen/arch/x86/srat.c +36 -0
index 7d20d7f222..2f08fa4660 100644
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 		       pxm, pa->apic_id, node);
 }
 
+/*
+ * Check to see if there are other nodes within this node's range.
+ * We just need to check full contains situation. Because overlaps
+ * have been checked before by conflicting_memblks.
+ */
+static bool __init is_node_memory_continuous(nodeid_t nid,
+    paddr_t start, paddr_t end)
+{
+	nodeid_t i;
+
+	struct node *nd = &nodes[nid];
+	for_each_node_mask(i, memory_nodes_parsed)
+	{
+		/* Skip itself */
+		if (i == nid)
+			continue;
+
+		nd = &nodes[i];
+		if (start < nd->start && nd->end < end)
+		{
+			printk(KERN_ERR
+			       "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
+			       nid, start, end, i, nd->start, nd->end);
+			return false;
+		}
+	}
+
+	return true;
+}
+
 /* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->start = start;
 			if (nd->end < end)
 				nd->end = end;
+
+			/* Check whether this range contains memory for other nodes */
+			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
+				bad_srat();
+				return;
+			}
 		}
 	}
 	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
Signature · 1 line
2.25.1
author
#36 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 34 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

Quoted 7 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

Quoted 8 lines

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:

Quoted 10 lines

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Should I need to include/refine above message to commit log?

Quoted 69 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct

Quoted 11 lines
---
 xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
xen/arch/x86/srat.c +0 -0
index 7d20d7f222..2f08fa4660 100644
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct

acpi_srat_cpu_affinity *pa)

Quoted 15 lines
xen/arch/x86/srat.c +12 -0
 		       pxm, pa->apic_id, node);
 }
 
+/*
+ * Check to see if there are other nodes within this node's range.
+ * We just need to check full contains situation. Because overlaps
+ * have been checked before by conflicting_memblks.
+ */
+static bool __init is_node_memory_continuous(nodeid_t nid,
+    paddr_t start, paddr_t end)
+{
+	nodeid_t i;
+
+	struct node *nd = &nodes[nid];
+	for_each_node_mask(i, memory_nodes_parsed)

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)

Quoted 10 lines
xen/arch/x86/srat.c +10 -0
+	{
+		/* Skip itself */
+		if (i == nid)
+			continue;
+
+		nd = &nodes[i];
+		if (start < nd->start && nd->end < end)
+		{
+			printk(KERN_ERR
+			       "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine

with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)

Quoted 9 lines
xen/arch/x86/srat.c +8 -0
+			       nid, start, end, i, nd->start, nd->end);
+			return false;
+		}
+	}
+
+	return true;
+}
+
 /* Callback for parsing of the Proximity Domain <-> Memory Area

mappings */

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 3 lines
xen/arch/x86/srat.c +0 -0
 void __init
 acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 5 lines
xen/arch/x86/srat.c +2 -0
 				nd->start = start;
 			if (nd->end < end)
 				nd->end = end;
+
+			/* Check whether this range contains memory for other

nodes */

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+			if (!is_node_memory_continuous(node, nd->start, nd->end))

{

Quoted 9 lines
xen/arch/x86/srat.c +3 -1
+				bad_srat();
+				return;
+			}
 		}
 	}
 	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
--

2.25.1

#37 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 37 lines · Wei Chen
Quoted 37 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

Quoted 7 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

Quoted 8 lines

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

Quoted 7 lines
Quoted 5 lines

So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved? An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?

Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.

author
#38 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Hi Stefano,

Quoted 60 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 8 lines
Quoted 8 lines

Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

On Thu, 23 Sep 2021, Wei Chen wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is

obviously

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,

the

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM the

check

Quoted 5 lines
Quoted 4 lines

would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Quoted 6 lines
Quoted 1 line

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 3 lines

I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not impossible.

Quoted 4 lines

Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?

Quoted 8 lines

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Quoted 5 lines

I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.

#39 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 80 lines · Wei Chen
Quoted 60 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 8 lines
Quoted 8 lines

Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

On Thu, 23 Sep 2021, Wei Chen wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is

obviously

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,

the

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM the

check

Quoted 5 lines
Quoted 4 lines

would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Quoted 6 lines
Quoted 1 line

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 5 lines
Quoted 3 lines

I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not impossible.

Definitely, I have seen it before.

Quoted 12 lines
Quoted 11 lines

Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

Quoted 9 lines

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.

My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch them.

Quoted 7 lines
Quoted 4 lines

I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.

#40 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 124 lines · Stefano Stabellini
Quoted 80 lines · Wei Chen
Quoted 60 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 8 lines
Quoted 8 lines

Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

CC'ing x86 maintainers

On Thu, 23 Sep 2021, Wei Chen wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is

obviously

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,

the

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved, but

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM the

check

Quoted 5 lines
Quoted 4 lines

would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.

Quoted 6 lines
Quoted 1 line

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 5 lines
Quoted 3 lines

I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not impossible.

Definitely, I have seen it before.

Quoted 12 lines
Quoted 11 lines

Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

Quoted 9 lines

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.

Quoted 7 lines

My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch them.

Looking at numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:

* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.

So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.

I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.

Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.

author
#41 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Hi Stefano,

Quoted 172 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous

node

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[email protected];

Quoted 3 lines
Quoted 3 lines

Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of

discontinous node

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

memory range

CC'ing x86 maintainers

Quoted 1 line · Wei Chen

One NUMA node may contain several memory blocks. In current

Xen

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

code, Xen will maintain a node memory range for each node to

cover

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

all its memory blocks. But here comes the problem, in the gap

of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

one node's two memory blocks, if there are some memory blocks

don't

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

belong to this node (remote memory blocks). This node's memory

range

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is

obviously

Quoted 1 line
Quoted 1 line
Quoted 1 line

not very reasonable. This means current NUMA code only can

support

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

node has continous memory blocks. However, on a physical

machine,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory

blocks

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

for one node. NUMA initializtion will be failed and error

messages

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved,

but

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

check

Quoted 1 line
Quoted 1 line

would look different: we would just look for multiple memory

ranges

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

under a device_type = "memory" node of a NUMA node in device

tree.

Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 1 line
Quoted 1 line
Quoted 1 line

I assume we can deal with this case simply by setting node0 memory

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

0x0-0x30000000 even if there is actually something else, a device,

that

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not

impossible.

Quoted 5 lines

Definitely, I have seen it before.

Quoted 1 line
Quoted 1 line

Is it only other nodes' memory interleaved that cause issues? In

other

Quoted 33 lines
Quoted 11 lines
Quoted 10 lines

words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

Quoted 9 lines

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.

Quoted 7 lines

My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch them.

Looking at numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:

* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.

Quoted 6 lines

So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.

I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.

Yes, I will do it.

Quoted 6 lines

Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.

Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.

(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1 (0000000880000000-00000008c0000000)

#42 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 172 lines · Wei Chen
Quoted 172 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous

node

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[email protected];

Quoted 3 lines
Quoted 3 lines

Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of

discontinous node

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

memory range

CC'ing x86 maintainers

Quoted 1 line · Wei Chen

One NUMA node may contain several memory blocks. In current

Xen

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

code, Xen will maintain a node memory range for each node to

cover

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

all its memory blocks. But here comes the problem, in the gap

of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

one node's two memory blocks, if there are some memory blocks

don't

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

belong to this node (remote memory blocks). This node's memory

range

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is

obviously

Quoted 1 line
Quoted 1 line
Quoted 1 line

not very reasonable. This means current NUMA code only can

support

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

node has continous memory blocks. However, on a physical

machine,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory

blocks

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

for one node. NUMA initializtion will be failed and error

messages

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

will be printed when Xen detect such hardware configuration.

At least on ARM, it is not just memory that can be interleaved,

but

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on ARM

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

check

Quoted 1 line
Quoted 1 line

would look different: we would just look for multiple memory

ranges

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

under a device_type = "memory" node of a NUMA node in device

tree.

Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 1 line
Quoted 1 line
Quoted 1 line

I assume we can deal with this case simply by setting node0 memory

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

0x0-0x30000000 even if there is actually something else, a device,

that

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not

impossible.

Quoted 5 lines

Definitely, I have seen it before.

Quoted 1 line
Quoted 1 line

Is it only other nodes' memory interleaved that cause issues? In

other

Quoted 33 lines
Quoted 11 lines
Quoted 10 lines

words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

Quoted 9 lines

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.

Quoted 7 lines

My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch them.

Looking at numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:

* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.

Quoted 8 lines
Quoted 6 lines

So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.

I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.

Yes, I will do it.

Thank you

Quoted 6 lines
Quoted 5 lines

Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.

Quoted 6 lines

Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.

(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1 (0000000880000000-00000008c0000000)

Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?

author
#43 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Hi Stefano,

Quoted 232 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 1:19
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 20 lines · Wei Chen
Quoted 20 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];

Quoted 3 lines
Quoted 3 lines

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of

discontinous

Quoted 11 lines
Quoted 11 lines

node

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 1 line
Quoted 1 line

Bertrand Marquis <[email protected]>;

[email protected];

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of

discontinous node

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

memory range

CC'ing x86 maintainers

Quoted 1 line · Wei Chen

One NUMA node may contain several memory blocks. In

current

Quoted 2 lines
Quoted 2 lines

Xen

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

code, Xen will maintain a node memory range for each node

to

Quoted 2 lines
Quoted 2 lines

cover

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

all its memory blocks. But here comes the problem, in the

gap

Quoted 2 lines
Quoted 2 lines

of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

one node's two memory blocks, if there are some memory

blocks

Quoted 2 lines
Quoted 2 lines

don't

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

belong to this node (remote memory blocks). This node's

memory

Quoted 4 lines
Quoted 4 lines

range

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this

is

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

obviously

Quoted 1 line
Quoted 1 line
Quoted 1 line

not very reasonable. This means current NUMA code only can

support

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

node has continous memory blocks. However, on a physical

machine,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous

memory

Quoted 4 lines
Quoted 4 lines

blocks

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

for one node. NUMA initializtion will be failed and error

messages

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

will be printed when Xen detect such hardware

configuration.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

At least on ARM, it is not just memory that can be

interleaved,

Quoted 10 lines
Quoted 10 lines

but

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on

ARM

Quoted 9 lines
Quoted 9 lines

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

check

Quoted 1 line
Quoted 1 line

would look different: we would just look for multiple memory

ranges

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

under a device_type = "memory" node of a NUMA node in device

tree.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal

with

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 1 line

cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000 MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000 MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I assume we can deal with this case simply by setting node0

memory

Quoted 2 lines
Quoted 2 lines

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

0x0-0x30000000 even if there is actually something else, a

device,

Quoted 32 lines
Quoted 32 lines

that

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not

impossible.

Quoted 5 lines

Definitely, I have seen it before.

Quoted 1 line
Quoted 1 line

Is it only other nodes' memory interleaved that cause issues? In

other

Quoted 21 lines
Quoted 20 lines
Quoted 10 lines

words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000 MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Yes, I am not too worried about devices with a NUMA id property

because

Quoted 1 line
Quoted 1 line
Quoted 1 line

they are less common and this series doesn't handle them at all,

right?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is

made

Quoted 1 line
Quoted 1 line
Quoted 1 line

of multiple banks. I understand that this patch adds an explicit

check

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they

are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

still made of multiple discontinuous banks, like in the two example above.

My question is whether this patch series in its current form can

handle

Quoted 1 line
Quoted 1 line
Quoted 1 line

the two cases above correctly. If so, I am wondering how it works

given

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above,

my

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch

them.

Quoted 3 lines
Quoted 3 lines

Looking at numa_update_node_memblks, it is clear that the code is

meant

Quoted 6 lines
Quoted 6 lines

to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:

* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.

Quoted 8 lines
Quoted 6 lines

So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.

I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.

Yes, I will do it.

Thank you

Quoted 1 line
Quoted 1 line

Have you had a chance to test this? If not it would be fantastic if

you

Quoted 3 lines
Quoted 3 lines

could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between

in

Quoted 5 lines
Quoted 1 line

device tree, just for the sake of testing.

Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.

Quoted 8 lines
Quoted 3 lines

(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1

(0000000880000000-00000008c0000000)

Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?

Yes, I am using a device tree setting like this:

    memory@80000000 {
        device_type = "memory";
        reg = <0x0 0x80000000 0x0 0x80000000>;
        numa-node-id = <0>;
    };

    memory@880000000 {
        device_type = "memory";
        reg = <0x8 0x80000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@890000000 {
        device_type = "memory";
        reg = <0x8 0x90000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8A0000000 {
        device_type = "memory";
        reg = <0x8 0xA0000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8B0000000 {
        device_type = "memory";
        reg = <0x8 0xB0000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8C0000000 {
        device_type = "memory";
        reg = <0x8 0xC0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8D0000000 {
        device_type = "memory";
        reg = <0x8 0xD0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8E0000000 {
        device_type = "memory";
        reg = <0x8 0xE0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8F0000000 {
        device_type = "memory";
        reg = <0x8 0xF0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

And in Xen we got the output:

(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000 (XEN) SRAT: Node 0 0000000880000000-0000000888000000 (XEN) SRAT: Node 0 0000000890000000-0000000898000000 (XEN) SRAT: Node 0 00000008a0000000-00000008a8000000 (XEN) SRAT: Node 0 00000008b0000000-00000008b8000000 (XEN) SRAT: Node 1 00000008c0000000-00000008c8000000 (XEN) SRAT: Node 1 00000008d0000000-00000008d8000000 (XEN) SRAT: Node 1 00000008e0000000-00000008e8000000 (XEN) SRAT: Node 1 00000008f0000000-00000008f8000000 (XEN) NUMA: parsing numa-distance-map
(XEN) NUMA: distance: NODE#0->NODE#0:10 (XEN) NUMA: distance: NODE#0->NODE#1:20 (XEN) NUMA: distance: NODE#1->NODE#1:10 (XEN) NUMA: Using 16 for the hash shift. (XEN) Domain heap initialised
(XEN) Booting using Device Tree

Dom0 can be boot successfully, xl info got:
xl info
host : X-Dom0
release : 5.12.0
version : #20 SMP PREEMPT Wed Jul 28 13:41:28 CST 2021
machine : aarch64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 2
cores_per_socket : 1
threads_per_core : 1

Xen debug console to dump numa info, we got:

(XEN) 'u' pressed -> dumping numa info (now = 13229372281010) (XEN) NODE0 start->524288 size->8617984 free->388741 (XEN) NODE1 start->9175040 size->229376 free->106460 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0

#44 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 237 lines · Wei Chen

Hi Stefano,

Quoted 235 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 1:19
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 20 lines · Wei Chen
Quoted 20 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];

Quoted 3 lines
Quoted 3 lines

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of

discontinous

Quoted 11 lines
Quoted 11 lines

node

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

memory range

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 1 line
Quoted 1 line

Bertrand Marquis <[email protected]>;

[email protected];

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of

discontinous node

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

memory range

CC'ing x86 maintainers

Quoted 1 line · Wei Chen

One NUMA node may contain several memory blocks. In

current

Quoted 2 lines
Quoted 2 lines

Xen

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

code, Xen will maintain a node memory range for each node

to

Quoted 2 lines
Quoted 2 lines

cover

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

all its memory blocks. But here comes the problem, in the

gap

Quoted 2 lines
Quoted 2 lines

of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

one node's two memory blocks, if there are some memory

blocks

Quoted 2 lines
Quoted 2 lines

don't

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

belong to this node (remote memory blocks). This node's

memory

Quoted 4 lines
Quoted 4 lines

range

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this

is

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

obviously

Quoted 1 line
Quoted 1 line
Quoted 1 line

not very reasonable. This means current NUMA code only can

support

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

node has continous memory blocks. However, on a physical

machine,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous

memory

Quoted 4 lines
Quoted 4 lines

blocks

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

for one node. NUMA initializtion will be failed and error

messages

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

will be printed when Xen detect such hardware

configuration.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

At least on ARM, it is not just memory that can be

interleaved,

Quoted 10 lines
Quoted 10 lines

but

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

also

Quoted 8 lines
Quoted 8 lines

MMIO regions. For instance:

node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000

So I am not familiar with the SRAT format, but I think on

ARM

Quoted 9 lines
Quoted 9 lines

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

check

Quoted 1 line
Quoted 1 line

would look different: we would just look for multiple memory

ranges

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

under a device_type = "memory" node of a NUMA node in device

tree.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Should I need to include/refine above message to commit log?

Let me ask you a question first.

With the NUMA implementation of this patch series, can we deal

with

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 1 line

cases where each node has multiple memory banks, not interleaved?

Yes.

Quoted 11 lines

An an example:

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000 MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000 MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I assume we can deal with this case simply by setting node0

memory

Quoted 2 lines
Quoted 2 lines

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

0x0-0x30000000 even if there is actually something else, a

device,

Quoted 32 lines
Quoted 32 lines

that

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

doesn't belong to node0 in between the two node0 banks?

While this configuration is rare in SoC design, but it is not

impossible.

Quoted 5 lines

Definitely, I have seen it before.

Quoted 1 line
Quoted 1 line

Is it only other nodes' memory interleaved that cause issues? In

other

Quoted 21 lines
Quoted 20 lines
Quoted 10 lines

words, only the following is a problematic scenario?

node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000 MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000

Because node1 is in between the two ranges of node0?

But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Yes, I am not too worried about devices with a NUMA id property

because

Quoted 1 line
Quoted 1 line
Quoted 1 line

they are less common and this series doesn't handle them at all,

right?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I imagine they would be treated like any other device without NUMA awareness.

I am thinking about the case where the memory of each NUMA node is

made

Quoted 1 line
Quoted 1 line
Quoted 1 line

of multiple banks. I understand that this patch adds an explicit

check

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they

are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

still made of multiple discontinuous banks, like in the two example above.

My question is whether this patch series in its current form can

handle

Quoted 1 line
Quoted 1 line
Quoted 1 line

the two cases above correctly. If so, I am wondering how it works

given

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

that we only have a single "start" and "size" parameter per node.

On the other hand if this series cannot handle the two cases above,

my

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch

them.

Quoted 3 lines
Quoted 3 lines

Looking at numa_update_node_memblks, it is clear that the code is

meant

Quoted 6 lines
Quoted 6 lines

to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:

* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.

Quoted 8 lines
Quoted 6 lines

So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.

I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.

Yes, I will do it.

Thank you

Quoted 1 line
Quoted 1 line

Have you had a chance to test this? If not it would be fantastic if

you

Quoted 3 lines
Quoted 3 lines

could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between

in

Quoted 8 lines
Quoted 1 line

device tree, just for the sake of testing.

Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.

(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1

Quoted 7 lines
Quoted 5 lines

(0000000880000000-00000008c0000000)

Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?

Yes, I am using a device tree setting like this:

Perfect! Thank you!

Quoted 102 lines
    memory@80000000 {
        device_type = "memory";
        reg = <0x0 0x80000000 0x0 0x80000000>;
        numa-node-id = <0>;
    };

    memory@880000000 {
        device_type = "memory";
        reg = <0x8 0x80000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@890000000 {
        device_type = "memory";
        reg = <0x8 0x90000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8A0000000 {
        device_type = "memory";
        reg = <0x8 0xA0000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8B0000000 {
        device_type = "memory";
        reg = <0x8 0xB0000000 0x0 0x8000000>;
        numa-node-id = <0>;
    };

    memory@8C0000000 {
        device_type = "memory";
        reg = <0x8 0xC0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8D0000000 {
        device_type = "memory";
        reg = <0x8 0xD0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8E0000000 {
        device_type = "memory";
        reg = <0x8 0xE0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

    memory@8F0000000 {
        device_type = "memory";
        reg = <0x8 0xF0000000 0x0 0x8000000>;
        numa-node-id = <1>;
    };

And in Xen we got the output:

(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000 (XEN) SRAT: Node 0 0000000880000000-0000000888000000 (XEN) SRAT: Node 0 0000000890000000-0000000898000000 (XEN) SRAT: Node 0 00000008a0000000-00000008a8000000 (XEN) SRAT: Node 0 00000008b0000000-00000008b8000000 (XEN) SRAT: Node 1 00000008c0000000-00000008c8000000 (XEN) SRAT: Node 1 00000008d0000000-00000008d8000000 (XEN) SRAT: Node 1 00000008e0000000-00000008e8000000 (XEN) SRAT: Node 1 00000008f0000000-00000008f8000000 (XEN) NUMA: parsing numa-distance-map
(XEN) NUMA: distance: NODE#0->NODE#0:10 (XEN) NUMA: distance: NODE#0->NODE#1:20 (XEN) NUMA: distance: NODE#1->NODE#1:10 (XEN) NUMA: Using 16 for the hash shift. (XEN) Domain heap initialised
(XEN) Booting using Device Tree

Dom0 can be boot successfully, xl info got:
xl info
host : X-Dom0
release : 5.12.0
version : #20 SMP PREEMPT Wed Jul 28 13:41:28 CST 2021
machine : aarch64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 2
cores_per_socket : 1
threads_per_core : 1

Xen debug console to dump numa info, we got:

(XEN) 'u' pressed -> dumping numa info (now = 13229372281010) (XEN) NODE0 start->524288 size->8617984 free->388741 (XEN) NODE1 start->9175040 size->229376 free->106460 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0

#45 lore raw

Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

In reply to Wei Chen
Quoted 7 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

Quoted 8 lines

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

Luckily what you actually check for isn't as strict as "discontinuous":
What you're after is no interleaving of memory. A single nod can still have multiple discontiguous ranges (and that'll often be the case on x86). Please adjust description and function name accordingly.

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 280

Quoted 13 lines
xen/arch/x86/srat.c +7 -0
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 		       pxm, pa->apic_id, node);
 }
 
+/*
+ * Check to see if there are other nodes within this node's range.
+ * We just need to check full contains situation. Because overlaps
+ * have been checked before by conflicting_memblks.
+ */
+static bool __init is_node_memory_continuous(nodeid_t nid,
+    paddr_t start, paddr_t end)

This indentation style demands indenting like ...

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 282

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+{
+	nodeid_t i;

... variable declarations at function scope, i.e. in a Linux-style file by a tab.

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 285

Quoted 3 lines
xen/arch/x86/srat.c +3 -0
+
+	struct node *nd = &nodes[nid];
+	for_each_node_mask(i, memory_nodes_parsed)

Please move the blank line to be between declarations and statements.

Also please make nd pointer-to const.

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 286

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+	{

In a Linux-style file opening braces do not belong on their own lines.

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 295

Quoted 9 lines
xen/arch/x86/srat.c +9 -0
+		/* Skip itself */
+		if (i == nid)
+			continue;
+
+		nd = &nodes[i];
+		if (start < nd->start && nd->end < end)
+		{
+			printk(KERN_ERR
+			       "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",

s/intertwine/interleaves/ ?

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 382

Quoted 10 lines
xen/arch/x86/srat.c +6 -0
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 				nd->start = start;
 			if (nd->end < end)
 				nd->end = end;
+
+			/* Check whether this range contains memory for other nodes */
+			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
+				bad_srat();
+				return;
+			}

I think this check would better come before nodes[] gets updated? Otoh bad_srat() will zap everything anyway ...

Jan

author
#46 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Hi Jan,

Quoted 9 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 13 lines
Quoted 12 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...

Quoted 3 lines
Quoted 3 lines

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...

Quoted 6 lines

Luckily what you actually check for isn't as strict as "discontinuous":
What you're after is no interleaving of memory. A single nod can still have multiple discontiguous ranges (and that'll often be the case on x86). Please adjust description and function name accordingly.

Yes, we're checking for no interlaced memory among nodes. In one node's memory range, the memory block still can be discontinuous.

I will rename the subject to:
"add detection of interlaced memory for different nodes" And I would rename is_node_memory_continuous to:
node_without_interleave_memory.

Quoted 7 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct

Quoted 3 lines
xen/arch/x86/srat.c +0 -0
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct

acpi_srat_cpu_affinity *pa)

Quoted 3 lines
xen/arch/x86/srat.c +0 -0
 		       pxm, pa->apic_id, node);
 }
 
Quoted 10 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 280

Quoted 7 lines
xen/arch/x86/srat.c +7 -0
+/*
+ * Check to see if there are other nodes within this node's range.
+ * We just need to check full contains situation. Because overlaps
+ * have been checked before by conflicting_memblks.
+ */
+static bool __init is_node_memory_continuous(nodeid_t nid,
+    paddr_t start, paddr_t end)

This indentation style demands indenting like ...

Ok.

Quoted 6 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 282

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+{
+	nodeid_t i;

... variable declarations at function scope, i.e. in a Linux-style file by a tab.

Quoted 7 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 285

Quoted 3 lines
xen/arch/x86/srat.c +3 -0
+
+	struct node *nd = &nodes[nid];
+	for_each_node_mask(i, memory_nodes_parsed)

Please move the blank line to be between declarations and statements.

Also please make nd pointer-to const.

Ok.

Quoted 5 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct · line 286

Quoted 2 lines
xen/arch/x86/srat.c +1 -0
+	{

In a Linux-style file opening braces do not belong on their own lines.

Ok.

Quoted 11 lines

xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)

Quoted 9 lines
xen/arch/x86/srat.c +9 -0
+		/* Skip itself */
+		if (i == nid)
+			continue;
+
+		nd = &nodes[i];
+		if (start < nd->start && nd->end < end)
+		{
+			printk(KERN_ERR
+			       "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine

with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",

Quoted 1 line

s/intertwine/interleaves/ ?

Yes, interleaves. I will fix it.

Quoted 9 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 2 lines
xen/arch/x86/srat.c +0 -0
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 5 lines
xen/arch/x86/srat.c +2 -0
 				nd->start = start;
 			if (nd->end < end)
 				nd->end = end;
+
+			/* Check whether this range contains memory for other

nodes */

Quoted 8 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+			if (!is_node_memory_continuous(node, nd->start, nd->end))

{

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 382

Quoted 3 lines
xen/arch/x86/srat.c +3 -0
+				bad_srat();
+				return;
+			}

I think this check would better come before nodes[] gets updated? Otoh bad_srat() will zap everything anyway ...

Yes, when I wrote this check, I considered when the check was failed, bad_srat would make numa initialization be failed. The values in nodes[] will not take any effect. So I didn't adjust the order. But if the bad_srat will be changed in future, and nodes[] will be used in fallback progress, this will take more effort to debug. In this case, I agree with you, I will update the order in next version.

Thanks,
Wei Chen

Quoted 2 lines

Jan

#47 lore raw

Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 21 lines · Wei Chen
Quoted 16 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13

Quoted 12 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...

Taking just this part of your reply (the issue continues later), may I ask that you use a consistent term throughout this single patch? Mixing "interlace" and "interleave" like you do may make people wonder whether the two are intended to express slightly different aspects. Personally, as per my suggestion, I'd prefer "interleave", but I'm not a native speaker.

Jan

author
#48 lore raw

RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Hi Jan,

Quoted 39 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 16:01
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range

Quoted 21 lines · Wei Chen
Quoted 16 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13

Quoted 12 lines · Wei Chen

One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.

One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.

I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.

I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...

Quoted 7 lines

Taking just this part of your reply (the issue continues later), may I ask that you use a consistent term throughout this single patch? Mixing "interlace" and "interleave" like you do may make people wonder whether the two are intended to express slightly different aspects. Personally, as per my suggestion, I'd prefer "interleave", but I'm not a native speaker.

Sorry, I am not a native speaker too, I had checked dict for interlaced before I used it. https://www.merriam-webster.com/thesaurus/interlaced

Obviously, I'm probably using it incorrectly and making it harder to understand, I will use "interleave" in my patches.

Thanks,
Wei Chen

Quoted 1 line

Jan

author
#49 lore raw

PATCH 9/37 xen/x86: introduce two helpers to access memory hotplug end

In reply to Wei Chen

x86 provides a mem_hotplug to maintain the end of memory hotplug end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.

---
 xen/include/asm-x86/mm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +10 -0
index cb90527499..af2fc4b0cd 100644
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,
 
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    mem_hotplug = end;
+}
+
+static inline paddr_t mem_hotplug_boundary(void)
+{
+    return mem_hotplug;
+}
+
 /******************************************************************************
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.
Signature · 1 line
2.25.1
#50 lore raw

Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end

+x86 maintainers

Quoted 1 line · Wei Chen

x86 provides a mem_hotplug to maintain the end of memory hotplug

^ variable

Quoted 2 lines

end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without

^ so that it can be reused

Quoted 3 lines

memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.

^ APIs

Quoted 29 lines
---
 xen/include/asm-x86/mm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +10 -0
index cb90527499..af2fc4b0cd 100644
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,
 
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    mem_hotplug = end;
+}
+
+static inline paddr_t mem_hotplug_boundary(void)
+{
+    return mem_hotplug;
+}
+
 /******************************************************************************
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.
Signature · 1 line
2.25.1
author
#51 lore raw

Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end

Quoted 6 lines · Stefano Stabellini

+x86 maintainers

Quoted 1 line · Wei Chen

x86 provides a mem_hotplug to maintain the end of memory hotplug

^ variable

Quoted 10 lines
Quoted 2 lines

end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without

^ so that it can be reused

Quoted 3 lines

memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.

^ APIs

OK

Quoted 29 lines
Quoted 29 lines
---
  xen/include/asm-x86/mm.h | 10 ++++++++++
  1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +10 -0
index cb90527499..af2fc4b0cd 100644
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,
 
  extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    mem_hotplug = end;
+}
+
+static inline paddr_t mem_hotplug_boundary(void)
+{
+    return mem_hotplug;
+}
+
  /******************************************************************************
   * With shadow pagetables, the different kinds of address start
   * to get get confusing.
Signature · 1 line
2.25.1
#52 lore raw

Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end

In reply to Wei Chen
Quoted 23 lines · Wei Chen

x86 provides a mem_hotplug to maintain the end of memory hotplug end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.

---
 xen/include/asm-x86/mm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +3 -0
index cb90527499..af2fc4b0cd 100644
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,
 
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)
+{
+    mem_hotplug = end;

xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page, · line 490

Quoted 10 lines
xen/include/asm-x86/mm.h +7 -0
+}
+
+static inline paddr_t mem_hotplug_boundary(void)
+{
+    return mem_hotplug;
+}
+
 /******************************************************************************
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.

Imo for this to make sense you want to also use the new functions right away in the place(s) where the direct access(es) get(s) in your way.

Jan

author
#53 lore raw

RE: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end

Hi Jan,

Quoted 44 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:25
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end

xen/include/asm-x86/mm.h static inline int get_page_and_type(struct

Quoted 17 lines · Wei Chen

x86 provides a mem_hotplug to maintain the end of memory hotplug end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.

---
 xen/include/asm-x86/mm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +0 -0
index cb90527499..af2fc4b0cd 100644
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct

page_info *page,

Quoted 4 lines
xen/include/asm-x86/mm.h +1 -0
 extern paddr_t mem_hotplug;
 
+static inline void mem_hotplug_update_boundary(paddr_t end)

xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page,

Quoted 10 lines
xen/include/asm-x86/mm.h +9 -0
+{
+    mem_hotplug = end;
+}
+
+static inline paddr_t mem_hotplug_boundary(void)
+{
+    return mem_hotplug;
+}
+
 

/*************************************************************************
*****

Quoted 7 lines

xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page, · line 490

Quoted 2 lines
xen/include/asm-x86/mm.h +0 -0
  * With shadow pagetables, the different kinds of address start
  * to get get confusing.

Imo for this to make sense you want to also use the new functions right away in the place(s) where the direct access(es) get(s) in your way.

OK, I will merge this patch with the patch that will use these functions firstly.

Quoted 1 line

Jan

author
#54 lore raw

PATCH 10/37 xen/x86: use helpers to access/update mem_hotplug

In reply to Wei Chen

We want to abstract code from acpi_numa_memory_affinity_init. But mem_hotplug is coupled with x86. In this patch, we use helpers to repace mem_hotplug direct accessing. This will allow most code can be common.

---
 xen/arch/x86/srat.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +2 -2
index 2f08fa4660..3334ede7a5 100644
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		if (end > mem_hotplug_boundary())
+			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
Signature · 1 line
2.25.1
#55 lore raw

Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

+x86 maintainers

Quoted 3 lines · Wei Chen

We want to abstract code from acpi_numa_memory_affinity_init. But mem_hotplug is coupled with x86. In this patch, we use helpers to repace mem_hotplug direct accessing. This will

^ replace

Quoted 1 line

allow most code can be common.

^ to be

I think this patch could be merged with the previous patch

Quoted 23 lines
---
 xen/arch/x86/srat.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +2 -2
index 2f08fa4660..3334ede7a5 100644
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		if (end > mem_hotplug_boundary())
+			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
Signature · 1 line
2.25.1
author
#56 lore raw

RE: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

Quoted 20 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月24日 8:32
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

+x86 maintainers

Quoted 3 lines · Wei Chen

We want to abstract code from acpi_numa_memory_affinity_init. But mem_hotplug is coupled with x86. In this patch, we use helpers to repace mem_hotplug direct accessing. This will

Quoted 7 lines

^ replace

Quoted 1 line

allow most code can be common.

^ to be

I think this patch could be merged with the previous patch

Ok, I will do it, and fix above typos

Quoted 25 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 11 lines
---
 xen/arch/x86/srat.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +0 -0
index 2f08fa4660..3334ede7a5 100644
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 13 lines
xen/arch/x86/srat.c +2 -3
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		if (end > mem_hotplug_boundary())
+			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
 }
--

2.25.1

#57 lore raw

Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

In reply to Wei Chen

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395

Quoted 10 lines · Wei Chen
xen/arch/x86/srat.c +2 -2
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		if (end > mem_hotplug_boundary())
+			mem_hotplug_update_boundary(end);

Can the if() please be folded into mem_hotplug_update_boundary(), eliminating (at least for the purpose here) the need for the separate mem_hotplug_boundary()? As said on the previous patch, I think the two want folding.

Jan

author
#58 lore raw

RE: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

Hi Jan,

Quoted 14 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:30
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 3 lines · Wei Chen
xen/arch/x86/srat.c +0 -0
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

Quoted 13 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395

Quoted 7 lines
xen/arch/x86/srat.c +2 -2
 	memblk_nodeid[num_node_memblks] = node;
 	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
 		__set_bit(num_node_memblks, memblk_hotplug);
-		if (end > mem_hotplug)
-			mem_hotplug = end;
+		if (end > mem_hotplug_boundary())
+			mem_hotplug_update_boundary(end);

Can the if() please be folded into mem_hotplug_update_boundary(), eliminating (at least for the purpose here) the need for the separate mem_hotplug_boundary()? As said on the previous patch, I think the two want folding.

Yes, these changes would be more make sense. I will do it in next version.

Quoted 1 line

Jan

author
#59 lore raw

PATCH 11/37 xen/x86: abstract neutral code from acpi_numa_memory_affinity_init

In reply to Wei Chen

There is some code in acpi_numa_memory_affinity_init to update node memory range and update node_memblk_range array. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.

So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.

---
 xen/arch/x86/srat.c        | 131 +++++++++++++++++++++----------------
 xen/include/asm-x86/numa.h |   3 +
 2 files changed, 77 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +74 -57
index 3334ede7a5..18bc6b19bb 100644
@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+bool __init numa_memblks_available(void)
+{
+	if (num_node_memblks < NR_NODE_MEMBLKS)
+		return true;
+
+	return false;
+}
+
 int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,
 	return true;
 }
 
-/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
-void __init
-acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+/* Neutral NUMA memory affinity init function for ACPI and DT */
+int __init numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug)
 {
-	paddr_t start, end;
-	unsigned pxm;
-	nodeid_t node;
+	paddr_t end = start + size;
 	int i;
 
-	if (srat_disabled())
-		return;
-	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
-		bad_srat();
-		return;
-	}
-	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
-		return;
-
-	start = ma->base_address;
-	end = start + ma->length;
-	/* Supplement the heuristics in l1tf_calculations(). */
-	l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(end, PAGE_SIZE));
-
-	if (num_node_memblks >= NR_NODE_MEMBLKS)
-	{
-		dprintk(XENLOG_WARNING,
-                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
-		bad_srat();
-		return;
-	}
-
-	pxm = ma->proximity_domain;
-	if (srat_rev < 2)
-		pxm &= 0xff;
-	node = setup_node(pxm);
-	if (node == NUMA_NO_NODE) {
-		bad_srat();
-		return;
-	}
-	/* It is fine to add this area to the nodes data it will be used later*/
+	/* It is fine to add this area to the nodes data it will be used later */
 	i = conflicting_memblks(start, end);
 	if (i < 0)
 		/* everything fine */;
 	else if (memblk_nodeid[i] == node) {
-		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
-		                !test_bit(i, memblk_hotplug);
+		bool mismatch = !hotplug != !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
-		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
+		printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
+		       mismatch ? KERN_ERR : KERN_WARNING, node, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
-			bad_srat();
-			return;
+			return -1;
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
-		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
+		       "SRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
+		       node, start, end, memblk_nodeid[i],
 		       node_memblk_range[i].start, node_memblk_range[i].end);
-		bad_srat();
-		return;
+		return -1;
 	}
-	if (!(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE)) {
+
+	if (!hotplug) {
 		struct node *nd = &nodes[node];
 
 		if (!node_test_and_set(node, memory_nodes_parsed)) {
@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 			if (nd->end < end)
 				nd->end = end;
 
-			/* Check whether this range contains memory for other nodes */
-			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
-				bad_srat();
-				return;
-			}
+			if (!is_node_memory_continuous(node, nd->start, nd->end))
+				return -1;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
-	       node, pxm, start, end,
-	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
+
+	printk(KERN_INFO "SRAT: Node %u %"PRIpaddr"-%"PRIpaddr"%s\n",
+	       node, start, end, hotplug ? " (hotplug)" : "");
 
 	node_memblk_range[num_node_memblks].start = start;
 	node_memblk_range[num_node_memblks].end = end;
 	memblk_nodeid[num_node_memblks] = node;
-	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
+	if (hotplug) {
 		__set_bit(num_node_memblks, memblk_hotplug);
 		if (end > mem_hotplug_boundary())
 			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
+
+	return 0;
+}
+
+/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
+void __init
+acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+{
+	unsigned pxm;
+	nodeid_t node;
+	int ret;
+
+	if (srat_disabled())
+		return;
+	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
+		bad_srat();
+		return;
+	}
+	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
+		return;
+
+	/* Supplement the heuristics in l1tf_calculations(). */
+	l1tf_safe_maddr = max(l1tf_safe_maddr,
+			ROUNDUP((ma->base_address + ma->length), PAGE_SIZE));
+
+	if (!numa_memblks_available())
+	{
+		dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+		bad_srat();
+		return;
+	}
+
+	pxm = ma->proximity_domain;
+	if (srat_rev < 2)
+		pxm &= 0xff;
+	node = setup_node(pxm);
+	if (node == NUMA_NO_NODE) {
+		bad_srat();
+		return;
+	}
+
+	ret = numa_update_node_memblks(node, ma->base_address, ma->length,
+					ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE);
+	if (ret != 0)
+		bad_srat();
 }
 
 /* Sanity check to catch more bad SRATs (they are amazingly common).
xen/include/asm-x86/numa.h +3 -0
index 50cfd8e7ef..5772a70665 100644
@@ -74,6 +74,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 				 NODE_DATA(nid)->node_spanned_pages)
 
 extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
+extern bool numa_memblks_available(void);
+extern int numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug);
 
 void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
Signature · 1 line
2.25.1
#60 lore raw

Re: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init

+x86 maintainers

Quoted 215 lines · Wei Chen

There is some code in acpi_numa_memory_affinity_init to update node memory range and update node_memblk_range array. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.

So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.

---
 xen/arch/x86/srat.c        | 131 +++++++++++++++++++++----------------
 xen/include/asm-x86/numa.h |   3 +
 2 files changed, 77 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +74 -57
index 3334ede7a5..18bc6b19bb 100644
@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+bool __init numa_memblks_available(void)
+{
+	if (num_node_memblks < NR_NODE_MEMBLKS)
+		return true;
+
+	return false;
+}
+
 int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
 {
 	int i;
@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,
 	return true;
 }
 
-/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
-void __init
-acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+/* Neutral NUMA memory affinity init function for ACPI and DT */
+int __init numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug)
 {
-	paddr_t start, end;
-	unsigned pxm;
-	nodeid_t node;
+	paddr_t end = start + size;
 	int i;
 
-	if (srat_disabled())
-		return;
-	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
-		bad_srat();
-		return;
-	}
-	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
-		return;
-
-	start = ma->base_address;
-	end = start + ma->length;
-	/* Supplement the heuristics in l1tf_calculations(). */
-	l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(end, PAGE_SIZE));
-
-	if (num_node_memblks >= NR_NODE_MEMBLKS)
-	{
-		dprintk(XENLOG_WARNING,
-                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
-		bad_srat();
-		return;
-	}
-
-	pxm = ma->proximity_domain;
-	if (srat_rev < 2)
-		pxm &= 0xff;
-	node = setup_node(pxm);
-	if (node == NUMA_NO_NODE) {
-		bad_srat();
-		return;
-	}
-	/* It is fine to add this area to the nodes data it will be used later*/
+	/* It is fine to add this area to the nodes data it will be used later */
 	i = conflicting_memblks(start, end);
 	if (i < 0)
 		/* everything fine */;
 	else if (memblk_nodeid[i] == node) {
-		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
-		                !test_bit(i, memblk_hotplug);
+		bool mismatch = !hotplug != !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
-		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
+		printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
+		       mismatch ? KERN_ERR : KERN_WARNING, node, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
-			bad_srat();
-			return;
+			return -1;
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
-		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
+		       "SRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
+		       node, start, end, memblk_nodeid[i],
 		       node_memblk_range[i].start, node_memblk_range[i].end);
-		bad_srat();
-		return;
+		return -1;
 	}
-	if (!(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE)) {
+
+	if (!hotplug) {
 		struct node *nd = &nodes[node];
 
 		if (!node_test_and_set(node, memory_nodes_parsed)) {
@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 			if (nd->end < end)
 				nd->end = end;
 
-			/* Check whether this range contains memory for other nodes */
-			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
-				bad_srat();
-				return;
-			}
+			if (!is_node_memory_continuous(node, nd->start, nd->end))
+				return -1;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
-	       node, pxm, start, end,
-	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
+
+	printk(KERN_INFO "SRAT: Node %u %"PRIpaddr"-%"PRIpaddr"%s\n",
+	       node, start, end, hotplug ? " (hotplug)" : "");
 
 	node_memblk_range[num_node_memblks].start = start;
 	node_memblk_range[num_node_memblks].end = end;
 	memblk_nodeid[num_node_memblks] = node;
-	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
+	if (hotplug) {
 		__set_bit(num_node_memblks, memblk_hotplug);
 		if (end > mem_hotplug_boundary())
 			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
+
+	return 0;
+}
+
+/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
+void __init
+acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+{
+	unsigned pxm;
+	nodeid_t node;
+	int ret;
+
+	if (srat_disabled())
+		return;
+	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
+		bad_srat();
+		return;
+	}
+	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
+		return;
+
+	/* Supplement the heuristics in l1tf_calculations(). */
+	l1tf_safe_maddr = max(l1tf_safe_maddr,
+			ROUNDUP((ma->base_address + ma->length), PAGE_SIZE));
+
+	if (!numa_memblks_available())
+	{
+		dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+		bad_srat();
+		return;
+	}
+
+	pxm = ma->proximity_domain;
+	if (srat_rev < 2)
+		pxm &= 0xff;
+	node = setup_node(pxm);
+	if (node == NUMA_NO_NODE) {
+		bad_srat();
+		return;
+	}
+
+	ret = numa_update_node_memblks(node, ma->base_address, ma->length,
+					ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE);
+	if (ret != 0)
+		bad_srat();
 }
 
 /* Sanity check to catch more bad SRATs (they are amazingly common).
xen/include/asm-x86/numa.h +3 -0
index 50cfd8e7ef..5772a70665 100644
@@ -74,6 +74,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 				 NODE_DATA(nid)->node_spanned_pages)
 
 extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
+extern bool numa_memblks_available(void);
+extern int numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug);
 
 void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
Signature · 1 line
2.25.1
#61 lore raw

Re: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init

In reply to Wei Chen
Quoted 11 lines · Wei Chen

There is some code in acpi_numa_memory_affinity_init to update node memory range and update node_memblk_range array. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.

So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.

SRAT is an ACPI concept, which I assume has no meaning with DT. Hence any generically usable logic here wants, I think, separating out into a file which is not SRAT-specific (peeking ahead, specifically not a file named "numa_srat.c"). This may in turn require some more though regarding the proper split between the stuff remaining in srat.c and the stuff becoming kind of library code. In particular this may mean moving some of the static variables as well, and with them perhaps some further functions (while I did peek ahead, I didn't look closely at the later patch doing the actual movement). And it is then hard to see why the separation needs to happen in two steps - you could move the generically usable code to a new file right away.

xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 113

Quoted 13 lines
xen/arch/x86/srat.c +7 -0
@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+bool __init numa_memblks_available(void)
+{
+	if (num_node_memblks < NR_NODE_MEMBLKS)
+		return true;
+
+	return false;
+}

Please can you avoid expressing things in more complex than necessary ways? Here I don't see why it can't just be

bool __init numa_memblks_available(void)

{
	return num_node_memblks < NR_NODE_MEMBLKS;
}

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 314

Quoted 10 lines
xen/arch/x86/srat.c +3 -3
@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,
 	return true;
 }
 
-/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
-void __init
-acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+/* Neutral NUMA memory affinity init function for ACPI and DT */
+int __init numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug)

Indentation.

Quoted 40 lines
xen/arch/x86/srat.c +2 -34
 {
-	paddr_t start, end;
-	unsigned pxm;
-	nodeid_t node;
+	paddr_t end = start + size;
 	int i;
 
-	if (srat_disabled())
-		return;
-	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
-		bad_srat();
-		return;
-	}
-	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
-		return;
-
-	start = ma->base_address;
-	end = start + ma->length;
-	/* Supplement the heuristics in l1tf_calculations(). */
-	l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(end, PAGE_SIZE));
-
-	if (num_node_memblks >= NR_NODE_MEMBLKS)
-	{
-		dprintk(XENLOG_WARNING,
-                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
-		bad_srat();
-		return;
-	}
-
-	pxm = ma->proximity_domain;
-	if (srat_rev < 2)
-		pxm &= 0xff;
-	node = setup_node(pxm);
-	if (node == NUMA_NO_NODE) {
-		bad_srat();
-		return;
-	}
-	/* It is fine to add this area to the nodes data it will be used later*/
+	/* It is fine to add this area to the nodes data it will be used later */
 	i = conflicting_memblks(start, end);

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 326

Quoted 10 lines
xen/arch/x86/srat.c +2 -4
 	if (i < 0)
 		/* everything fine */;
 	else if (memblk_nodeid[i] == node) {
-		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
-		                !test_bit(i, memblk_hotplug);
+		bool mismatch = !hotplug != !test_bit(i, memblk_hotplug);
 
-		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
-		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
+		printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",

Nit: Unlike PXM, which is an acronym, "node" doesn't want to be all upper case.

Also did you check that the node <-> PXM association is known to a reader of a log at this point in time?

Quoted 7 lines
xen/arch/x86/srat.c +2 -2
+		       mismatch ? KERN_ERR : KERN_WARNING, node, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
-			bad_srat();
-			return;
+			return -1;
 		}

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 337

Quoted 10 lines
xen/arch/x86/srat.c +3 -4
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
-		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
+		       "SRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
+		       node, start, end, memblk_nodeid[i],
 		       node_memblk_range[i].start, node_memblk_range[i].end);
-		bad_srat();
-		return;
+		return -1;

Please no -1 return values. Either a function means to return boolean, in which case it should use bool / true / false, or it means to return a proper errno-style error code.

Quoted 9 lines
xen/arch/x86/srat.c +0 -5
@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 			if (nd->end < end)
 				nd->end = end;
 
-			/* Check whether this range contains memory for other nodes */
-			if (!is_node_memory_continuous(node, nd->start, nd->end)) {
-				bad_srat();
-				return;
-			}

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 358

Quoted 10 lines
xen/arch/x86/srat.c +5 -3
+			if (!is_node_memory_continuous(node, nd->start, nd->end))
+				return -1;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
-	       node, pxm, start, end,
-	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
+
+	printk(KERN_INFO "SRAT: Node %u %"PRIpaddr"-%"PRIpaddr"%s\n",
+	       node, start, end, hotplug ? " (hotplug)" : "");

Continuing from a comment further up: Here you remove an instance of logging the node <-> PXM association.

Quoted 24 lines
xen/arch/x86/srat.c +15 -1
 	node_memblk_range[num_node_memblks].start = start;
 	node_memblk_range[num_node_memblks].end = end;
 	memblk_nodeid[num_node_memblks] = node;
-	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
+	if (hotplug) {
 		__set_bit(num_node_memblks, memblk_hotplug);
 		if (end > mem_hotplug_boundary())
 			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
+
+	return 0;
+}
+
+/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
+void __init
+acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+{
+	unsigned pxm;
+	nodeid_t node;
+	int ret;
+
+	if (srat_disabled())
+		return;

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 392

Quoted 10 lines
xen/arch/x86/srat.c +10 -0
+	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
+		bad_srat();
+		return;
+	}
+	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
+		return;
+
+	/* Supplement the heuristics in l1tf_calculations(). */
+	l1tf_safe_maddr = max(l1tf_safe_maddr,
+			ROUNDUP((ma->base_address + ma->length), PAGE_SIZE));

Indentation and unnecessary pair of parentheses.

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+	if (!numa_memblks_available())
+	{

For code you touch, please try to bring it into consistent style. Here the brace wants to move to the previous line, seeing that the file is using Linux style.

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 397

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+		dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");

Here you want to fix indentation and ideally also format and grammar of the actual log message.

Quoted 5 lines
xen/arch/x86/srat.c +5 -0
+		bad_srat();
+		return;
+	}
+
+	pxm = ma->proximity_domain;

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 412

Quoted 10 lines
xen/arch/x86/srat.c +10 -0
+	if (srat_rev < 2)
+		pxm &= 0xff;
+	node = setup_node(pxm);
+	if (node == NUMA_NO_NODE) {
+		bad_srat();
+		return;
+	}
+
+	ret = numa_update_node_memblks(node, ma->base_address, ma->length,
+					ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE);

Indentation again.

Jan

author
#62 lore raw

RE: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init

Hi Jan,

Quoted 20 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:51
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init

Quoted 10 lines · Wei Chen

There is some code in acpi_numa_memory_affinity_init to update node memory range and update node_memblk_range array. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.

So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.

Quoted 6 lines
Quoted 1 line

SRAT is an ACPI concept, which I assume has no meaning with DT. Hence any generically usable logic here wants, I think, separating out into a file which is not SRAT-specific (peeking ahead, specifically not a file named "numa_srat.c"). This may in turn require some more though

When I created the file, I wanted to place non-ACPI/DT specific code in a new file. But I was confused about how to name it. I chose numa_srat.c as the file name because I thought the device tree is also a static resource table. But it seems this name is still misleading, because ACPI SRAT is well known.

Quoted 8 lines

regarding the proper split between the stuff remaining in srat.c and the stuff becoming kind of library code. In particular this may mean moving some of the static variables as well, and with them perhaps some further functions (while I did peek ahead, I didn't look closely at the later patch doing the actual movement). And it is then hard to see why the separation needs to happen in two steps - you could move the generically usable code to a new file right away.

OK, I will reduce the steps. And I think the "new file" can be common/numa.c. Because the generically usable code are some logical functions to check numa memory blocks/ranges and update nodes, we don't need a "numa_srat.c".

Quoted 6 lines
Quoted 6 lines
xen/arch/x86/srat.c +0 -0
@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
Quoted 10 lines

xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 113

Quoted 7 lines
xen/arch/x86/srat.c +7 -0
+bool __init numa_memblks_available(void)
+{
+	if (num_node_memblks < NR_NODE_MEMBLKS)
+		return true;
+
+	return false;
+}

Please can you avoid expressing things in more complex than necessary ways? Here I don't see why it can't just be

OK, I will simplify it.

Quoted 12 lines

bool __init numa_memblks_available(void)

{
	return num_node_memblks < NR_NODE_MEMBLKS;
}

xen/arch/x86/srat.c static bool __init

Quoted 1 line
xen/arch/x86/srat.c +0 -0
@@ -301,69 +309,35 @@ static bool __init

is_node_memory_continuous(nodeid_t nid,

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,

Quoted 4 lines
xen/arch/x86/srat.c +0 -1
 	return true;
 }
 
-/* Callback for parsing of the Proximity Domain <-> Memory Area
Quoted 8 lines

mappings */

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 314

Quoted 5 lines
xen/arch/x86/srat.c +3 -2
-void __init
-acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+/* Neutral NUMA memory affinity init function for ACPI and DT */
+int __init numa_update_node_memblks(nodeid_t node,
+		paddr_t start, paddr_t size, bool hotplug)

Indentation.

OK.

Quoted 50 lines
Quoted 29 lines
xen/arch/x86/srat.c +1 -24
 {
-	paddr_t start, end;
-	unsigned pxm;
-	nodeid_t node;
+	paddr_t end = start + size;
 	int i;
 
-	if (srat_disabled())
-		return;
-	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
-		bad_srat();
-		return;
-	}
-	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
-		return;
-
-	start = ma->base_address;
-	end = start + ma->length;
-	/* Supplement the heuristics in l1tf_calculations(). */
-	l1tf_safe_maddr = max(l1tf_safe_maddr, ROUNDUP(end, PAGE_SIZE));
-
-	if (num_node_memblks >= NR_NODE_MEMBLKS)
-	{
-		dprintk(XENLOG_WARNING,
-                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
-		bad_srat();
-		return;
-	}

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,

Quoted 10 lines
xen/arch/x86/srat.c +0 -10
-
-	pxm = ma->proximity_domain;
-	if (srat_rev < 2)
-		pxm &= 0xff;
-	node = setup_node(pxm);
-	if (node == NUMA_NO_NODE) {
-		bad_srat();
-		return;
-	}
-	/* It is fine to add this area to the nodes data it will be used

later*/

xen/arch/x86/srat.c static bool __init · line 319

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+	/* It is fine to add this area to the nodes data it will be used

later */

Quoted 8 lines
xen/arch/x86/srat.c +1 -2
 	i = conflicting_memblks(start, end);
 	if (i < 0)
 		/* everything fine */;
 	else if (memblk_nodeid[i] == node) {
-		bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
-		                !test_bit(i, memblk_hotplug);
+		bool mismatch = !hotplug != !test_bit(i, memblk_hotplug);
 
Quoted 9 lines

xen/arch/x86/srat.c static bool __init · line 351

Quoted 1 line
xen/arch/x86/srat.c +0 -1
-		printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with

itself (%"PRIpaddr"-%"PRIpaddr")\n",

xen/arch/x86/srat.c static bool __init · line 326

Quoted 2 lines
xen/arch/x86/srat.c +1 -1
-		       mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
+		printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps

with itself (%"PRIpaddr"-%"PRIpaddr")\n",

Nit: Unlike PXM, which is an acronym, "node" doesn't want to be all upper case.

OK.

Quoted 3 lines

Also did you check that the node <-> PXM association is known to a reader of a log at this point in time?

Yes, I read your comment below. The original log contains node <-> PXM mapping. Because PXM is ACPI specific, I think in neutral code, we just need node in log. But this change removed the log of node <-> PXM association, it was my mistake. I will add them back in next version. But I don't think they will stay in neutal code, it would be in ACPI specific code.

I also had wanted to keep PXM <-> node mapping in the log, so I set up PXM to node 1-1 mapping for the device tree. But then I thought it was an unnecessary burden on the device tree, so I selected to remove PXM in log.

Quoted 14 lines

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,

Quoted 10 lines
xen/arch/x86/srat.c +2 -3
+		       mismatch ? KERN_ERR : KERN_WARNING, node, start, end,
 		       node_memblk_range[i].start, node_memblk_range[i].end);
 		if (mismatch) {
-			bad_srat();
-			return;
+			return -1;
 		}
 	} else {
 		printk(KERN_ERR
-		       "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with

PXM %u (%"PRIpaddr"-%"PRIpaddr")\n",

xen/arch/x86/srat.c static bool __init · line 334

Quoted 2 lines
xen/arch/x86/srat.c +1 -1
-		       pxm, start, end, node_to_pxm(memblk_nodeid[i]),
+		       "SRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with

NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",

Quoted 10 lines

xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 337

Quoted 5 lines
xen/arch/x86/srat.c +2 -2
+		       node, start, end, memblk_nodeid[i],
 		       node_memblk_range[i].start, node_memblk_range[i].end);
-		bad_srat();
-		return;
+		return -1;

Please no -1 return values. Either a function means to return boolean, in which case it should use bool / true / false, or it means to return a proper errno-style error code.

Ok, I will do it.

Quoted 9 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct

Quoted 1 line
xen/arch/x86/srat.c +0 -0
@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct

acpi_srat_mem_affinity *ma)

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 4 lines
xen/arch/x86/srat.c +0 -1
 			if (nd->end < end)
 				nd->end = end;
 
-			/* Check whether this range contains memory for other

nodes */

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379

Quoted 1 line
xen/arch/x86/srat.c +0 -1
-			if (!is_node_memory_continuous(node, nd->start, nd->end))

{

Quoted 17 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 358

Quoted 13 lines
xen/arch/x86/srat.c +5 -6
-				bad_srat();
-				return;
-			}
+			if (!is_node_memory_continuous(node, nd->start, nd->end))
+				return -1;
 		}
 	}
-	printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
-	       node, pxm, start, end,
-	       ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
+
+	printk(KERN_INFO "SRAT: Node %u %"PRIpaddr"-%"PRIpaddr"%s\n",
+	       node, start, end, hotplug ? " (hotplug)" : "");

Continuing from a comment further up: Here you remove an instance of logging the node <-> PXM association.

see my above comments.

Quoted 29 lines
Quoted 5 lines
xen/arch/x86/srat.c +1 -1
 	node_memblk_range[num_node_memblks].start = start;
 	node_memblk_range[num_node_memblks].end = end;
 	memblk_nodeid[num_node_memblks] = node;
-	if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
+	if (hotplug) {

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)

Quoted 10 lines
xen/arch/x86/srat.c +5 -0
 		__set_bit(num_node_memblks, memblk_hotplug);
 		if (end > mem_hotplug_boundary())
 			mem_hotplug_update_boundary(end);
 	}
 	num_node_memblks++;
+
+	return 0;
+}
+
+/* Callback for parsing of the Proximity Domain <-> Memory Area

mappings */

Quoted 13 lines
xen/arch/x86/srat.c +13 -0
+void __init
+acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+{
+	unsigned pxm;
+	nodeid_t node;
+	int ret;
+
+	if (srat_disabled())
+		return;
+	if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
+		bad_srat();
+		return;
+	}
Quoted 9 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 392

Quoted 6 lines
xen/arch/x86/srat.c +6 -0
+	if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
+		return;
+
+	/* Supplement the heuristics in l1tf_calculations(). */
+	l1tf_safe_maddr = max(l1tf_safe_maddr,
+			ROUNDUP((ma->base_address + ma->length), PAGE_SIZE));

Indentation and unnecessary pair of parentheses.

OK.

Quoted 7 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+	if (!numa_memblks_available())
+	{

For code you touch, please try to bring it into consistent style. Here the brace wants to move to the previous line, seeing that the file is using Linux style.

OK.

Quoted 6 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 397

Quoted 2 lines
xen/arch/x86/srat.c +2 -0
+		dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");

Here you want to fix indentation and ideally also format and grammar of the actual log message.

I will fix it, thanks.

Quoted 9 lines
Quoted 9 lines
xen/arch/x86/srat.c +9 -0
+		bad_srat();
+		return;
+	}
+
+	pxm = ma->proximity_domain;
+	if (srat_rev < 2)
+		pxm &= 0xff;
+	node = setup_node(pxm);
+	if (node == NUMA_NO_NODE) {
Quoted 8 lines

xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 412

Quoted 6 lines
xen/arch/x86/srat.c +6 -0
+		bad_srat();
+		return;
+	}
+
+	ret = numa_update_node_memblks(node, ma->base_address, ma->length,
+					ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE);

Indentation again.

Ok.

Quoted 2 lines

Jan

author
#63 lore raw

PATCH 12/37 xen/x86: decouple nodes_cover_memory from E820 map

In reply to Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Depends above two helpers, we make nodes_cover_memory become architecture independent. And the only change from an x86 perspective is the additional checks:
!start || !end

---
 xen/arch/x86/numa.c        | 18 ++++++++++++++++++
 xen/arch/x86/srat.c        | 11 ++++-------
 xen/include/asm-x86/numa.h |  3 +++
 3 files changed, 25 insertions(+), 7 deletions(-)
xen/arch/x86/numa.c +18 -0
index 6337bbdf31..6bc4ade411 100644
@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)
     return ( num_online_nodes() > 1 ) ? 1 : 0;
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(uint32_t bank,
+	paddr_t *start, paddr_t *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {
+		return -1;
+	}
+
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/arch/x86/srat.c +4 -7
index 18bc6b19bb..aa07a7e975 100644
@@ -419,17 +419,14 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 static int __init nodes_cover_memory(void)
 {
 	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < nr_banks; i++) {
 		int j, found;
 		paddr_t start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end))
 			continue;
-		}
-
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
 
 		do {
 			found = 0;
@@ -448,7 +445,7 @@ static int __init nodes_cover_memory(void)
 		} while (found && start < end);
 
 		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+			printk(KERN_ERR "SRAT: No NODE for memory map range: "
 				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
xen/include/asm-x86/numa.h +3 -0
index 5772a70665..78e044a390 100644
@@ -82,5 +82,8 @@ void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
+extern uint32_t arch_meminfo_get_nr_bank(void);
+extern int arch_meminfo_get_ram_bank_range(uint32_t bank,
+    paddr_t *start, paddr_t *end);
 
 #endif
Signature · 1 line
2.25.1
#64 lore raw

Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

+x86 maintainers

Quoted 97 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Depends above two helpers, we make nodes_cover_memory become architecture independent. And the only change from an x86 perspective is the additional checks:
!start || !end

---
 xen/arch/x86/numa.c        | 18 ++++++++++++++++++
 xen/arch/x86/srat.c        | 11 ++++-------
 xen/include/asm-x86/numa.h |  3 +++
 3 files changed, 25 insertions(+), 7 deletions(-)
xen/arch/x86/numa.c +18 -0
index 6337bbdf31..6bc4ade411 100644
@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)
     return ( num_online_nodes() > 1 ) ? 1 : 0;
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(uint32_t bank,
+	paddr_t *start, paddr_t *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {
+		return -1;
+	}
+
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();
xen/arch/x86/srat.c +4 -7
index 18bc6b19bb..aa07a7e975 100644
@@ -419,17 +419,14 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
 static int __init nodes_cover_memory(void)
 {
 	int i;
+	uint32_t nr_banks = arch_meminfo_get_nr_bank();
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < nr_banks; i++) {
 		int j, found;
 		paddr_t start, end;
 
-		if (e820.map[i].type != E820_RAM) {
+		if (arch_meminfo_get_ram_bank_range(i, &start, &end))
 			continue;
-		}
-
-		start = e820.map[i].addr;
-		end = e820.map[i].addr + e820.map[i].size;
 
 		do {
 			found = 0;
@@ -448,7 +445,7 @@ static int __init nodes_cover_memory(void)
 		} while (found && start < end);
 
 		if (start < end) {
-			printk(KERN_ERR "SRAT: No PXM for e820 range: "
+			printk(KERN_ERR "SRAT: No NODE for memory map range: "
 				"%"PRIpaddr" - %"PRIpaddr"\n", start, end);
 			return 0;
 		}
xen/include/asm-x86/numa.h +3 -0
index 5772a70665..78e044a390 100644
@@ -82,5 +82,8 @@ void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 unsigned int arch_have_default_dmazone(void);
+extern uint32_t arch_meminfo_get_nr_bank(void);
+extern int arch_meminfo_get_ram_bank_range(uint32_t bank,
+    paddr_t *start, paddr_t *end);
 
 #endif
Signature · 1 line
2.25.1
#65 lore raw

Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

In reply to Wei Chen
Quoted 5 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 381

Quoted 7 lines
xen/arch/x86/numa.c +1 -0
@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)
     return ( num_online_nodes() > 1 ) ? 1 : 0;
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)

unsigned int (also elsewhere)

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 389

Quoted 8 lines
xen/arch/x86/numa.c +8 -0
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(uint32_t bank,
+	paddr_t *start, paddr_t *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {

I see no reason for the checking of start and end.

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 391

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+		return -1;
+	}

No need for braces here.

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 401

Quoted 9 lines
xen/arch/x86/numa.c +6 -0
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();

Judging by just the two pieces of patch context you're inserting a Linux-style code fragment in the middle of two Xen-style ones.

Various other comments given for earlier patches apply here as well.

Jan

author
#66 lore raw

RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Hi Jan,

Quoted 10 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

On 23.09.2021 14:02, Wei Chen wrote:

Quoted 9 lines
Quoted 5 lines

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

In my latest changes, I have updated these two helpers to:
uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.

Quoted 10 lines

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 381

Quoted 7 lines
xen/arch/x86/numa.c +1 -0
@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)
     return ( num_online_nodes() > 1 ) ? 1 : 0;
 }
 
+uint32_t __init arch_meminfo_get_nr_bank(void)

unsigned int (also elsewhere)

OK.

Quoted 11 lines

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 389

Quoted 8 lines
xen/arch/x86/numa.c +8 -0
+{
+	return e820.nr_map;
+}
+
+int __init arch_meminfo_get_ram_bank_range(uint32_t bank,
+	paddr_t *start, paddr_t *end)
+{
+	if (e820.map[bank].type != E820_RAM || !start || !end) {

I see no reason for the checking of start and end.

Yes, I have removed this checking in the latest version.

Quoted 5 lines

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 391

Quoted 2 lines
xen/arch/x86/numa.c +2 -0
+		return -1;
+	}

No need for braces here.

Ok.

Quoted 13 lines

xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 401

Quoted 9 lines
xen/arch/x86/numa.c +6 -0
+	*start = e820.map[bank].addr;
+	*end = e820.map[bank].addr + e820.map[bank].size;
+
+	return 0;
+}
+
 static void dump_numa(unsigned char key)
 {
     s_time_t now = NOW();

Judging by just the two pieces of patch context you're inserting a Linux-style code fragment in the middle of two Xen-style ones.

Quoted 2 lines

Various other comments given for earlier patches apply here as well.

Yes, the original file is xen-style. I will change the inserted code to xen-style too. Thanks!

Quoted 1 line

Jan

#67 lore raw

Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Quoted 10 lines · Wei Chen
Quoted 10 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59

Quoted 5 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

Quoted 8 lines
Quoted 2 lines

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

In my latest changes, I have updated these two helpers to:
uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.

I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the function return "uint32_t" - see ./CODING_STYLE.

Jan

author
#68 lore raw

RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Hi Jan,

Quoted 23 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Quoted 13 lines · Wei Chen
Quoted 12 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59

Quoted 5 lines · Wei Chen

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

Quoted 8 lines
Quoted 5 lines

In my latest changes, I have updated these two helpers to:
uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.

I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the

How about arch_meminfo_get_nr_map/ arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?

Quoted 1 line

function return "uint32_t" - see ./CODING_STYLE.

OK.

Quoted 2 lines

Jan

#69 lore raw

Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Quoted 23 lines · Wei Chen
Quoted 23 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09

Quoted 18 lines · Wei Chen
Quoted 4 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59

On 23.09.2021 14:02, Wei Chen wrote:

Quoted 8 lines
Quoted 5 lines

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

In my latest changes, I have updated these two helpers to:
uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.

Quoted 6 lines
Quoted 2 lines

I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the

How about arch_meminfo_get_nr_map/ arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?

First of all I don't think you need a "get_nr" function at all, which eliminates the need to find a good name for it. The "get_range" function can easily provide back a unique indicator when the passed in index is beyond the number of regions. For this function's name, how about arch_get_memory_map() or arch_get_memory_map_block() or arch_get_memory_map_range() (in order of my personal preference)?

Jan

author
#70 lore raw

RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Hi Jan,

Quoted 40 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map

Quoted 23 lines · Wei Chen
Quoted 23 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09

Quoted 18 lines · Wei Chen
Quoted 4 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59

On 23.09.2021 14:02, Wei Chen wrote:

Quoted 8 lines
Quoted 5 lines

We will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range

I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.

In my latest changes, I have updated these two helpers to:
uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.

Quoted 6 lines
Quoted 2 lines

I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the

How about arch_meminfo_get_nr_map/ arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?

Quoted 7 lines

First of all I don't think you need a "get_nr" function at all, which eliminates the need to find a good name for it. The "get_range" function can easily provide back a unique indicator when the passed in index is beyond the number of regions. For this function's name, how about arch_get_memory_map() or arch_get_memory_map_block() or arch_get_memory_map_range() (in order of my personal preference)?

Thanks, I will try to use arch_get_memory_map for x86 and arm.

Quoted 1 line

Jan

author
#71 lore raw

PATCH 13/37 xen/x86: decouple processor_nodes_parsed from acpi numa functions

In reply to Wei Chen

Xen is using processor_nodes_parsed to record parsed processor nodes from ACPI table or other firmware provided resource table. This variable is used in ACPI numa functions directly. In follow-up patchs, neutral NUMA code will be abstracted and move to other files. So in this patch, we introduce numa_set_processor_nodes_parsed helper to decouple processor_nodes_parsed from acpi numa functions.

---
 xen/arch/x86/srat.c        | 9 +++++++--
 xen/include/asm-x86/numa.h | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +7 -2
index aa07a7e975..9276a52138 100644
@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+void  __init numa_set_processor_nodes_parsed(nodeid_t node)
+{
+	node_set(node, processor_nodes_parsed);
+}
+
 bool __init numa_memblks_available(void)
 {
 	if (num_node_memblks < NR_NODE_MEMBLKS)
@@ -236,7 +241,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 	}
 
 	apicid_to_node[pa->apic_id] = node;
-	node_set(node, processor_nodes_parsed);
+	numa_set_processor_nodes_parsed(node);
 	acpi_numa = 1;
 
 	if (opt_acpi_verbose)
@@ -271,7 +276,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 		return;
 	}
 	apicid_to_node[pa->apic_id] = node;
-	node_set(node, processor_nodes_parsed);
+	numa_set_processor_nodes_parsed(node);
 	acpi_numa = 1;
 
 	if (opt_acpi_verbose)
xen/include/asm-x86/numa.h +1 -0
index 78e044a390..295f875a51 100644
@@ -77,6 +77,7 @@ extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
 extern bool numa_memblks_available(void);
 extern int numa_update_node_memblks(nodeid_t node,
 		paddr_t start, paddr_t size, bool hotplug);
+extern void numa_set_processor_nodes_parsed(nodeid_t node);
 
 void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
Signature · 1 line
2.25.1
#72 lore raw

Re: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions

+x86 maintainers

Quoted 62 lines · Wei Chen

Xen is using processor_nodes_parsed to record parsed processor nodes from ACPI table or other firmware provided resource table. This variable is used in ACPI numa functions directly. In follow-up patchs, neutral NUMA code will be abstracted and move to other files. So in this patch, we introduce numa_set_processor_nodes_parsed helper to decouple processor_nodes_parsed from acpi numa functions.

---
 xen/arch/x86/srat.c        | 9 +++++++--
 xen/include/asm-x86/numa.h | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +7 -2
index aa07a7e975..9276a52138 100644
@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+void  __init numa_set_processor_nodes_parsed(nodeid_t node)
+{
+	node_set(node, processor_nodes_parsed);
+}
+
 bool __init numa_memblks_available(void)
 {
 	if (num_node_memblks < NR_NODE_MEMBLKS)
@@ -236,7 +241,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 	}
 
 	apicid_to_node[pa->apic_id] = node;
-	node_set(node, processor_nodes_parsed);
+	numa_set_processor_nodes_parsed(node);
 	acpi_numa = 1;
 
 	if (opt_acpi_verbose)
@@ -271,7 +276,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 		return;
 	}
 	apicid_to_node[pa->apic_id] = node;
-	node_set(node, processor_nodes_parsed);
+	numa_set_processor_nodes_parsed(node);
 	acpi_numa = 1;
 
 	if (opt_acpi_verbose)
xen/include/asm-x86/numa.h +1 -0
index 78e044a390..295f875a51 100644
@@ -77,6 +77,7 @@ extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
 extern bool numa_memblks_available(void);
 extern int numa_update_node_memblks(nodeid_t node,
 		paddr_t start, paddr_t size, bool hotplug);
+extern void numa_set_processor_nodes_parsed(nodeid_t node);
 
 void srat_parse_regions(paddr_t addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
Signature · 1 line
2.25.1
#73 lore raw

Re: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions

In reply to Wei Chen

xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 107

Quoted 7 lines · Wei Chen
xen/arch/x86/srat.c +1 -0
@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
+void  __init numa_set_processor_nodes_parsed(nodeid_t node)

Besides (nit) the stray blank here, earlier comments apply. The way you do the rearrangement it is close to impossible to see the actual "why" behind the changes. Even if it would make for a bigger patch, I think you want to collapse patches and move things out of srat.c code while you split out generically usable functionality. Or, if the result was deemed to large a patch to have all in one go, make the movement of individual static variables the topic of each patch, introducing accessor functions like the one here.

Jan

author
#74 lore raw

RE: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions

Hi Jan,

Quoted 16 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 17:49
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions

Quoted 6 lines · Wei Chen
xen/arch/x86/srat.c +0 -0
@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
Quoted 11 lines

xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 107

Quoted 1 line
xen/arch/x86/srat.c +1 -0
+void  __init numa_set_processor_nodes_parsed(nodeid_t node)

Besides (nit) the stray blank here, earlier comments apply. The way you do the rearrangement it is close to impossible to see the actual "why" behind the changes. Even if it would make for a bigger patch, I think you want to collapse patches and move things out of srat.c code while you split out generically usable functionality. Or, if the result was deemed to large a patch to have all in one go, make the movement of individual static variables the topic of each patch, introducing accessor functions like the one here.

Thanks, I will fix them in next version.

Quoted 1 line

Jan

author
#75 lore raw

PATCH 14/37 xen/x86: use name fw_numa to replace acpi_numa

In reply to Wei Chen

Xen is using acpi_numa as a switch for ACPI based NUMA. We want to use this switch logic for other firmware based NUMA implementation, like device tree based NUMA in follow-up patches. As Xen will never use both ACPI and device tree based NUMA at runtime. So I rename acpi_numa to a more generic name fw_name. This will also allow to have the code mostly common.

---
 xen/arch/x86/numa.c        |  6 +++---
 xen/arch/x86/setup.c       |  2 +-
 xen/arch/x86/srat.c        | 10 +++++-----
 xen/include/asm-x86/acpi.h |  2 +-
 4 files changed, 10 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +3 -3
index 6bc4ade411..2ef385ae3f 100644
@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
 bool numa_off;
-s8 acpi_numa = 0;
+s8 fw_numa = 0;
 
 int srat_disabled(void)
 {
-    return numa_off || acpi_numa < 0;
+    return numa_off || fw_numa < 0;
 }
 
 /*
@@ -315,7 +315,7 @@ static __init int numa_setup(const char *opt)
     else if ( !strncmp(opt,"noacpi",6) )
     {
         numa_off = false;
-        acpi_numa = -1;
+        fw_numa = -1;
     }
 #endif
     else
xen/arch/x86/setup.c +1 -1
index b101565f14..1a2093b554 100644
@@ -313,7 +313,7 @@ void srat_detect_node(int cpu)
     node_set_online(node);
     numa_set_node(cpu, node);
 
-    if ( opt_cpu_info && acpi_numa > 0 )
+    if ( opt_cpu_info && fw_numa > 0 )
         printk("CPU %d APIC %d -> Node %d\n", cpu, apicid, node);
 }
 
xen/arch/x86/srat.c +5 -5
index 9276a52138..4921830f94 100644
@@ -167,7 +167,7 @@ static __init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
-	acpi_numa = -1;
+	fw_numa = -1;
 	for (i = 0; i < MAX_LOCAL_APIC; i++)
 		apicid_to_node[i] = NUMA_NO_NODE;
 	for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
@@ -242,7 +242,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 
 	apicid_to_node[pa->apic_id] = node;
 	numa_set_processor_nodes_parsed(node);
-	acpi_numa = 1;
+	fw_numa = 1;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %08x -> Node %u\n",
@@ -277,7 +277,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 	}
 	apicid_to_node[pa->apic_id] = node;
 	numa_set_processor_nodes_parsed(node);
-	acpi_numa = 1;
+	fw_numa = 1;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %02x -> Node %u\n",
@@ -492,7 +492,7 @@ void __init srat_parse_regions(paddr_t addr)
 	u64 mask;
 	unsigned int i;
 
-	if (acpi_disabled || acpi_numa < 0 ||
+	if (acpi_disabled || fw_numa < 0 ||
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
@@ -521,7 +521,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
-	if (acpi_numa <= 0)
+	if (fw_numa <= 0)
 		return -1;
 
 	if (!nodes_cover_memory()) {
xen/include/asm-x86/acpi.h +1 -1
index 7032f3a001..83be71fec3 100644
@@ -101,7 +101,7 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
+extern s8 fw_numa;
 extern int acpi_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
Signature · 1 line
2.25.1
#76 lore raw

Re: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa

+x86 maintainers

Quoted 120 lines · Wei Chen

Xen is using acpi_numa as a switch for ACPI based NUMA. We want to use this switch logic for other firmware based NUMA implementation, like device tree based NUMA in follow-up patches. As Xen will never use both ACPI and device tree based NUMA at runtime. So I rename acpi_numa to a more generic name fw_name. This will also allow to have the code mostly common.

---
 xen/arch/x86/numa.c        |  6 +++---
 xen/arch/x86/setup.c       |  2 +-
 xen/arch/x86/srat.c        | 10 +++++-----
 xen/include/asm-x86/acpi.h |  2 +-
 4 files changed, 10 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +3 -3
index 6bc4ade411..2ef385ae3f 100644
@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
 bool numa_off;
-s8 acpi_numa = 0;
+s8 fw_numa = 0;
 
 int srat_disabled(void)
 {
-    return numa_off || acpi_numa < 0;
+    return numa_off || fw_numa < 0;
 }
 
 /*
@@ -315,7 +315,7 @@ static __init int numa_setup(const char *opt)
     else if ( !strncmp(opt,"noacpi",6) )
     {
         numa_off = false;
-        acpi_numa = -1;
+        fw_numa = -1;
     }
 #endif
     else
xen/arch/x86/setup.c +1 -1
index b101565f14..1a2093b554 100644
@@ -313,7 +313,7 @@ void srat_detect_node(int cpu)
     node_set_online(node);
     numa_set_node(cpu, node);
 
-    if ( opt_cpu_info && acpi_numa > 0 )
+    if ( opt_cpu_info && fw_numa > 0 )
         printk("CPU %d APIC %d -> Node %d\n", cpu, apicid, node);
 }
 
xen/arch/x86/srat.c +5 -5
index 9276a52138..4921830f94 100644
@@ -167,7 +167,7 @@ static __init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
-	acpi_numa = -1;
+	fw_numa = -1;
 	for (i = 0; i < MAX_LOCAL_APIC; i++)
 		apicid_to_node[i] = NUMA_NO_NODE;
 	for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
@@ -242,7 +242,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
 
 	apicid_to_node[pa->apic_id] = node;
 	numa_set_processor_nodes_parsed(node);
-	acpi_numa = 1;
+	fw_numa = 1;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %08x -> Node %u\n",
@@ -277,7 +277,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
 	}
 	apicid_to_node[pa->apic_id] = node;
 	numa_set_processor_nodes_parsed(node);
-	acpi_numa = 1;
+	fw_numa = 1;
 
 	if (opt_acpi_verbose)
 		printk(KERN_INFO "SRAT: PXM %u -> APIC %02x -> Node %u\n",
@@ -492,7 +492,7 @@ void __init srat_parse_regions(paddr_t addr)
 	u64 mask;
 	unsigned int i;
 
-	if (acpi_disabled || acpi_numa < 0 ||
+	if (acpi_disabled || fw_numa < 0 ||
 	    acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat))
 		return;
 
@@ -521,7 +521,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 	for (i = 0; i < MAX_NUMNODES; i++)
 		cutoff_node(i, start, end);
 
-	if (acpi_numa <= 0)
+	if (fw_numa <= 0)
 		return -1;
 
 	if (!nodes_cover_memory()) {
xen/include/asm-x86/acpi.h +1 -1
index 7032f3a001..83be71fec3 100644
@@ -101,7 +101,7 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern s8 acpi_numa;
+extern s8 fw_numa;
 extern int acpi_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
Signature · 1 line
2.25.1
#77 lore raw

Re: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa

In reply to Wei Chen

xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 54

Quoted 8 lines · Wei Chen
xen/arch/x86/numa.c +1 -1
@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
 bool numa_off;
-s8 acpi_numa = 0;
+s8 fw_numa = 0;

In x86 code I'd prefer this to remain "acpi_numa". If you need to access the variable from to-become-generic code, introduce an inline wrapper (possibly named numa_mode()), allowing you to do whatever you need in the DT case. It may be helpful to fold this with numa_off then, seeing e.g.
...

xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 58

Quoted 4 lines
xen/arch/x86/numa.c +1 -1
 int srat_disabled(void)
 {
-    return numa_off || acpi_numa < 0;
+    return numa_off || fw_numa < 0;

... this. Actually I think the underlying enumeration could even be made generic:

enum numa_mode {
numa_off,
numa_on,
numa_acpi,
};

is, I believe, sufficient to express the present (numa_off,acpi_numa) tuple. In this context I'd like to point out that the two uses of acpi_numa in srat_parse_regions() and srat_detect_node() should likely be invocations of srat_disabled() instead, to also take numa_off into account. This would then be addressed effectively as a side effect by replacing open-coded uses as well as srat_disabled() by numa_mode() (or whichever name the new helper would gain).

Jan

author
#78 lore raw

RE: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa

Hi Jan,

Quoted 39 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:13
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa

xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES]

Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0 -0
@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES]

__read_mostly;

xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 54

Quoted 5 lines
xen/arch/x86/numa.c +1 -1
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
 bool numa_off;
-s8 acpi_numa = 0;
+s8 fw_numa = 0;

In x86 code I'd prefer this to remain "acpi_numa". If you need to access the variable from to-become-generic code, introduce an inline wrapper (possibly named numa_mode()), allowing you to do whatever you need in the DT case. It may be helpful to fold this with numa_off then, seeing e.g.
...

xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 58

Quoted 4 lines
xen/arch/x86/numa.c +1 -1
 int srat_disabled(void)
 {
-    return numa_off || acpi_numa < 0;
+    return numa_off || fw_numa < 0;

... this. Actually I think the underlying enumeration could even be made generic:

enum numa_mode {
numa_off,
numa_on,
numa_acpi,
};

Quoted 8 lines

is, I believe, sufficient to express the present (numa_off,acpi_numa) tuple. In this context I'd like to point out that the two uses of acpi_numa in srat_parse_regions() and srat_detect_node() should likely be invocations of srat_disabled() instead, to also take numa_off into account. This would then be addressed effectively as a side effect by replacing open-coded uses as well as srat_disabled() by numa_mode() (or whichever name the new helper would gain).

Yes, that seems more reasonable, I will try this in next version.

Quoted 1 line

Jan

author
#79 lore raw

PATCH 15/37 xen/x86: rename acpi_scan_nodes to numa_scan_nodes

In reply to Wei Chen

The most code in acpi_scan_nodes can be reused by other NUMA implementation. Rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace BIOS to Firmware in print message, as BIOS is x86 specific name.

---
 xen/arch/x86/numa.c        | 2 +-
 xen/arch/x86/srat.c        | 4 ++--
 xen/include/asm-x86/acpi.h | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
xen/arch/x86/numa.c +1 -1
index 2ef385ae3f..8a4710df39 100644
@@ -261,7 +261,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     end = pfn_to_paddr(end_pfn);
 
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes(start, end) )
+    if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif
 
xen/arch/x86/srat.c +2 -2
index 4921830f94..0b8b0b0c95 100644
@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+int __init numa_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
+			       "Firmware Bug or mis-configured hardware?\n", i);
 
 		setup_node_bootmem(i, nodes[i].start, nodes[i].end);
 	}
xen/include/asm-x86/acpi.h +1 -1
index 83be71fec3..2add971072 100644
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;
 #define ARCH_HAS_POWER_INIT	1
 
 extern s8 fw_numa;
-extern int acpi_scan_nodes(u64 start, u64 end);
+extern int numa_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
 extern struct acpi_sleep_info acpi_sinfo;
Signature · 1 line
2.25.1
#80 lore raw

Re: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes

+x86 maintainers

Quoted 63 lines · Wei Chen

The most code in acpi_scan_nodes can be reused by other NUMA implementation. Rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace BIOS to Firmware in print message, as BIOS is x86 specific name.

---
 xen/arch/x86/numa.c        | 2 +-
 xen/arch/x86/srat.c        | 4 ++--
 xen/include/asm-x86/acpi.h | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
xen/arch/x86/numa.c +1 -1
index 2ef385ae3f..8a4710df39 100644
@@ -261,7 +261,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     end = pfn_to_paddr(end_pfn);
 
 #ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes(start, end) )
+    if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif
 
xen/arch/x86/srat.c +2 -2
index 4921830f94..0b8b0b0c95 100644
@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+int __init numa_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
+			       "Firmware Bug or mis-configured hardware?\n", i);
 
 		setup_node_bootmem(i, nodes[i].start, nodes[i].end);
 	}
xen/include/asm-x86/acpi.h +1 -1
index 83be71fec3..2add971072 100644
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;
 #define ARCH_HAS_POWER_INIT	1
 
 extern s8 fw_numa;
-extern int acpi_scan_nodes(u64 start, u64 end);
+extern int numa_scan_nodes(u64 start, u64 end);
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
 extern struct acpi_sleep_info acpi_sinfo;
Signature · 1 line
2.25.1
#81 lore raw

Re: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes

In reply to Wei Chen
Quoted 12 lines · Wei Chen
xen/arch/x86/srat.c +1 -1
@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr)
 }
 
 /* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+int __init numa_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;

xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t end) · line 550

Quoted 5 lines
xen/arch/x86/srat.c +1 -1
@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
 		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
+			       "Firmware Bug or mis-configured hardware?\n", i);

Besides the prior theme of in-place renames not making clear why the rename is being done (by deferring to future patches then moving the code) I'm puzzled by you replacing "BIOS" but leaving "SRAT" in place.

Jan

author
#82 lore raw

RE: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes

Hi Jan,

Quoted 24 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:18
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes

xen/arch/x86/srat.c void __init srat_parse_regions(paddr_t addr) · line 514

Quoted 6 lines · Wei Chen
xen/arch/x86/srat.c +0 -0
@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr)
 }
 
 /* Use the information discovered above to actually set up the nodes.

*/

xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t

Quoted 6 lines
xen/arch/x86/srat.c +1 -1
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+int __init numa_scan_nodes(paddr_t start, paddr_t end)
 {
 	int i;
 	nodemask_t all_nodes_parsed;
@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t

end)

Quoted 10 lines

xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t end) · line 550

Quoted 5 lines
xen/arch/x86/srat.c +1 -1
 		paddr_t size = nodes[i].end - nodes[i].start;
 		if ( size == 0 )
 			printk(KERN_WARNING "SRAT: Node %u has no memory. "
-			       "BIOS Bug or mis-configured hardware?\n", i);
+			       "Firmware Bug or mis-configured hardware?\n", i);

Besides the prior theme of in-place renames not making clear why the rename is being done (by deferring to future patches then moving the code) I'm puzzled by you replacing "BIOS" but leaving "SRAT" in place.

The reason is the same as I said in previous patch. At that time, I want to treat device tree as another static resource table. I would change "SRAT" to "NUMA". And I will try to merge this rename patch with the actual affected patch.

Quoted 1 line

Jan

author
#83 lore raw

PATCH 16/37 xen/x86: export srat_bad to external

In reply to Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

---
 xen/arch/x86/srat.c        | 2 +-
 xen/include/asm-x86/numa.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
xen/arch/x86/srat.c +1 -1
index 0b8b0b0c95..94bd5b34da 100644
@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 	}
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
xen/include/asm-x86/numa.h +1 -0
index 295f875a51..a5690a7098 100644
@@ -32,6 +32,7 @@ extern bool numa_off;
 
 
 extern int srat_disabled(void);
+extern void bad_srat(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
Signature · 1 line
2.25.1
#84 lore raw

Re: [PATCH 16/37] xen/x86: export srat_bad to external

+x86 maintainers

Quoted 42 lines · Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

---
 xen/arch/x86/srat.c        | 2 +-
 xen/include/asm-x86/numa.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
xen/arch/x86/srat.c +1 -1
index 0b8b0b0c95..94bd5b34da 100644
@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 	}
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)
 {
 	int i;
 	printk(KERN_ERR "SRAT: SRAT not used.\n");
xen/include/asm-x86/numa.h +1 -0
index 295f875a51..a5690a7098 100644
@@ -32,6 +32,7 @@ extern bool numa_off;
 
 
 extern int srat_disabled(void);
+extern void bad_srat(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
Signature · 1 line
2.25.1
#85 lore raw

Re: [PATCH 16/37] xen/x86: export srat_bad to external

In reply to Wei Chen
Quoted 7 lines · Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.

Perhaps alongside numa_mode() (see the reply to the earlier patch) you want to have a set_numa_off() helper (name subject to improvement)?

xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start, paddr_t end) · line 166

Quoted 8 lines
xen/arch/x86/srat.c +1 -1
@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end)
 	}
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)

Nit: Once again, when touching code, please take the opportunity and adjust style issues (here: __init and void want to change places).

Jan

author
#86 lore raw

RE: [PATCH 16/37] xen/x86: export srat_bad to external

Hi Jan,

Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: export srat_bad to external

Quoted 7 lines · Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

Quoted 8 lines

Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.

Perhaps alongside numa_mode() (see the reply to the earlier patch) you want to have a set_numa_off() helper (name subject to improvement)?

Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use numa_fw_fault to replace srat_bad.

Quoted 4 lines

xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start,

Quoted 3 lines
xen/arch/x86/srat.c +0 -0
@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start,

paddr_t end)

Quoted 9 lines

xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start, paddr_t end) · line 166

Quoted 5 lines
xen/arch/x86/srat.c +1 -1
 	}
 }
 
-static __init void bad_srat(void)
+__init void bad_srat(void)

Nit: Once again, when touching code, please take the opportunity and adjust style issues (here: __init and void want to change places).

Thanks again!

Quoted 1 line

Jan

#87 lore raw

Re: [PATCH 16/37] xen/x86: export srat_bad to external

Quoted 28 lines · Wei Chen

Hi Jan,

Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: export srat_bad to external

Quoted 7 lines · Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

Quoted 8 lines

Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.

Perhaps alongside numa_mode() (see the reply to the earlier patch) you want to have a set_numa_off() helper (name subject to improvement)?

Quoted 4 lines

Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use numa_fw_fault to replace srat_bad.

Why not simply numa_bad() or, say, numa_disable()?

Jan

author
#88 lore raw

RE: [PATCH 16/37] xen/x86: export srat_bad to external

Hi Jan,

Quoted 37 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:37
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: export srat_bad to external

Quoted 28 lines · Wei Chen

Hi Jan,

Quoted 25 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: export srat_bad to external

Quoted 7 lines · Wei Chen

srat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.

In this patch, we export srat_bad to external. This will allow to have the code mostly common.

Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.

Perhaps alongside numa_mode() (see the reply to the earlier patch) you want to have a set_numa_off() helper (name subject to improvement)?

Quoted 7 lines
Quoted 4 lines

Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use numa_fw_fault to replace srat_bad.

Why not simply numa_bad() or, say, numa_disable()?

numa_bad would be OK.

Quoted 1 line

Jan

author
#89 lore raw

PATCH 17/37 xen/x86: use CONFIG_NUMA to gate numa_scan_nodes

In reply to Wei Chen

As we have turned numa_scan_nodes to neutral function. If we still use CONFIG_ACPI_NUMA in numa_initmem_init to gate numa_scan_nodes that doesn't make sense. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So in this patch, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.

---
 xen/arch/x86/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/numa.c +1 -1
index 8a4710df39..509d2738c0 100644
@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     start = pfn_to_paddr(start_pfn);
     end = pfn_to_paddr(end_pfn);
 
-#ifdef CONFIG_ACPI_NUMA
+#ifdef CONFIG_NUMA
     if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif
Signature · 1 line
2.25.1
#90 lore raw

Re: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes

+x86 maintainers

Quoted 27 lines · Wei Chen

As we have turned numa_scan_nodes to neutral function. If we still use CONFIG_ACPI_NUMA in numa_initmem_init to gate numa_scan_nodes that doesn't make sense. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So in this patch, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.

---
 xen/arch/x86/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/numa.c +1 -1
index 8a4710df39..509d2738c0 100644
@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     start = pfn_to_paddr(start_pfn);
     end = pfn_to_paddr(end_pfn);
 
-#ifdef CONFIG_ACPI_NUMA
+#ifdef CONFIG_NUMA
     if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif
Signature · 1 line
2.25.1
#91 lore raw

Re: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes

In reply to Wei Chen
Quoted 16 lines · Wei Chen

As we have turned numa_scan_nodes to neutral function. If we still use CONFIG_ACPI_NUMA in numa_initmem_init to gate numa_scan_nodes that doesn't make sense. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So in this patch, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.

---
 xen/arch/x86/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/numa.c +0 -0
index 8a4710df39..509d2738c0 100644

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 266

Quoted 8 lines
xen/arch/x86/numa.c +1 -1
@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     start = pfn_to_paddr(start_pfn);
     end = pfn_to_paddr(end_pfn);
 
-#ifdef CONFIG_ACPI_NUMA
+#ifdef CONFIG_NUMA
     if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif

Why is this a separate patch, rather than part of the one renaming the function?

Jan

author
#92 lore raw

RE: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes

Hi Jan,

Quoted 30 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:26
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long

Quoted 16 lines · Wei Chen

As we have turned numa_scan_nodes to neutral function. If we still use CONFIG_ACPI_NUMA in numa_initmem_init to gate numa_scan_nodes that doesn't make sense. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So in this patch, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.

---
 xen/arch/x86/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/numa.c +0 -0
index 8a4710df39..509d2738c0 100644
@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 3 lines
xen/arch/x86/numa.c +0 -0
     start = pfn_to_paddr(start_pfn);
     end = pfn_to_paddr(end_pfn);
 
Quoted 8 lines

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 266

Quoted 5 lines
xen/arch/x86/numa.c +1 -1
-#ifdef CONFIG_ACPI_NUMA
+#ifdef CONFIG_NUMA
     if ( !numa_off && !numa_scan_nodes(start, end) )
         return;
 #endif

Why is this a separate patch, rather than part of the one renaming the function?

I will merge these two patches.

Quoted 2 lines

Jan

author
#93 lore raw

PATCH 18/37 xen: move NUMA common code from x86 to common

In reply to Wei Chen

Some common code has been decoupled and abstracted from x86 ACPI based NUMA implementation. In order to make this code can be reused by other NUMA implementation, we move this code from x86 to common folder. And this code is gated by CONFIG_NUMA, it only can be used by those architectures that NUMA is enabled. For those architectures do not support NUMA, they still can implementation NUMA stub API in asm/numa.h to make NUMA-aware components happy.

In this patch, we also remove some unused include headers.

---
 xen/arch/x86/numa.c         | 446 +----------------------------------
 xen/arch/x86/srat.c         | 253 +-------------------
 xen/common/Makefile         |   2 +
 xen/common/numa.c           | 450 ++++++++++++++++++++++++++++++++++++
 xen/common/numa_srat.c      | 264 +++++++++++++++++++++
 xen/include/asm-x86/acpi.h  |   4 -
 xen/include/asm-x86/numa.h  |  68 ------
 xen/include/asm-x86/setup.h |   1 -
 xen/include/xen/numa.h      |  82 +++++++
 9 files changed, 802 insertions(+), 768 deletions(-)
 create mode 100644 xen/common/numa.c
 create mode 100644 xen/common/numa_srat.c
xen/arch/x86/numa.c +2 -444
index 509d2738c0..92b6bdf7b9 100644
@@ -3,24 +3,13 @@
  * Copyright 2002,2003 Andi Kleen, SuSE Labs.
  * Adapted for Xen: Ryan Harper <[email protected]>
  */ 
-
-#include <xen/mm.h>
-#include <xen/string.h>
 #include <xen/init.h>
-#include <xen/ctype.h>
+#include <xen/mm.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
-#include <xen/keyhandler.h>
-#include <xen/param.h>
-#include <xen/time.h>
-#include <xen/smp.h>
-#include <xen/pfn.h>
-#include <asm/acpi.h>
 #include <xen/sched.h>
-#include <xen/softirq.h>
 
-static int numa_setup(const char *s);
-custom_param("numa", numa_setup);
+#include <asm/acpi.h>
 
 #ifndef Dprintk
 #define Dprintk(x...)
@@ -29,300 +18,12 @@ custom_param("numa", numa_setup);
 /* from proto.h */
 #define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
 
-struct node_data node_data[MAX_NUMNODES];
-
-/* Mapping from pdx to node id */
-int memnode_shift;
-static typeof(*memnodemap) _memnodemap[64];
-unsigned long memnodemapsize;
-u8 *memnodemap;
-
-nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
-    [0 ... NR_CPUS-1] = NUMA_NO_NODE
-};
 /*
  * Keep BIOS's CPU2node information, should not be used for memory allocaion
  */
 nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
     [0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
 };
-cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
-
-nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-
-bool numa_off;
-s8 fw_numa = 0;
-
-int srat_disabled(void)
-{
-    return numa_off || fw_numa < 0;
-}
-
-/*
- * Given a shift value, try to populate memnodemap[]
- * Returns :
- * 1 if OK
- * 0 if memnodmap[] too small (of shift too small)
- * -1 if node overlap or lost ram (shift too big)
- */
-static int __init populate_memnodemap(const struct node *nodes,
-                                      int numnodes, int shift, nodeid_t *nodeids)
-{
-    unsigned long spdx, epdx;
-    int i, res = -1;
-
-    memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        if ( (epdx >> shift) >= memnodemapsize )
-            return 0;
-        do {
-            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
-                return -1;
-
-            if ( !nodeids )
-                memnodemap[spdx >> shift] = i;
-            else
-                memnodemap[spdx >> shift] = nodeids[i];
-
-            spdx += (1UL << shift);
-        } while ( spdx < epdx );
-        res = 1;
-    }
-
-    return res;
-}
-
-static int __init allocate_cachealigned_memnodemap(void)
-{
-    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
-    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
-
-    memnodemap = mfn_to_virt(mfn);
-    mfn <<= PAGE_SHIFT;
-    size <<= PAGE_SHIFT;
-    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
-           mfn, mfn + size);
-    memnodemapsize = size / sizeof(*memnodemap);
-
-    return 0;
-}
-
-/*
- * The LSB of all start and end addresses in the node map is the value of the
- * maximum possible shift.
- */
-static int __init extract_lsb_from_nodes(const struct node *nodes,
-                                         int numnodes)
-{
-    int i, nodes_used = 0;
-    unsigned long spdx, epdx;
-    unsigned long bitfield = 0, memtop = 0;
-
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        bitfield |= spdx;
-        nodes_used++;
-        if ( epdx > memtop )
-            memtop = epdx;
-    }
-    if ( nodes_used <= 1 )
-        i = BITS_PER_LONG - 1;
-    else
-        i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
-    memnodemapsize = (memtop >> i) + 1;
-    return i;
-}
-
-int __init compute_hash_shift(struct node *nodes, int numnodes,
-                              nodeid_t *nodeids)
-{
-    int shift;
-
-    shift = extract_lsb_from_nodes(nodes, numnodes);
-    if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
-        memnodemap = _memnodemap;
-    else if ( allocate_cachealigned_memnodemap() )
-        return -1;
-    printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
-
-    if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
-    {
-        printk(KERN_INFO "Your memory is not aligned you need to "
-               "rebuild your hypervisor with a bigger NODEMAPSIZE "
-               "shift=%d\n", shift);
-        return -1;
-    }
-
-    return shift;
-}
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
-{
-    unsigned long start_pfn, end_pfn;
-
-    start_pfn = paddr_to_pfn(start);
-    end_pfn = paddr_to_pfn(end);
-
-    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
-    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
-
-    node_set_online(nodeid);
-} 
-
-void __init numa_init_array(void)
-{
-    int rr, i;
-
-    /* There are unfortunately some poorly designed mainboards around
-       that only connect memory to a single CPU. This breaks the 1:1 cpu->node
-       mapping. To avoid this fill in the mapping for all possible
-       CPUs, as the number of CPUs is not known yet.
-       We round robin the existing nodes. */
-    rr = first_node(node_online_map);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-    {
-        if ( cpu_to_node[i] != NUMA_NO_NODE )
-            continue;
-        numa_set_node(i, rr);
-        rr = cycle_node(rr, node_online_map);
-    }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
-
-/* Numa emulation */
-static int __init numa_emulation(unsigned long start_pfn,
-                                 unsigned long end_pfn)
-{
-    int i;
-    struct node nodes[MAX_NUMNODES];
-    u64 sz = pfn_to_paddr(end_pfn - start_pfn) / numa_fake;
-
-    /* Kludge needed for the hash function */
-    if ( hweight64(sz) > 1 )
-    {
-        u64 x = 1;
-        while ( (x << 1) < sz )
-            x <<= 1;
-        if ( x < sz/2 )
-            printk(KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n");
-        sz = x;
-    }
-
-    memset(&nodes,0,sizeof(nodes));
-    for ( i = 0; i < numa_fake; i++ )
-    {
-        nodes[i].start = pfn_to_paddr(start_pfn) + i*sz;
-        if ( i == numa_fake - 1 )
-            sz = pfn_to_paddr(end_pfn) - nodes[i].start;
-        nodes[i].end = nodes[i].start + sz;
-        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
-               i,
-               nodes[i].start, nodes[i].end,
-               (nodes[i].end - nodes[i].start) >> 20);
-        node_set_online(i);
-    }
-    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
-    if ( memnode_shift < 0 )
-    {
-        memnode_shift = 0;
-        printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
-        return -1;
-    }
-    for_each_online_node ( i )
-        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-    numa_init_array();
-
-    return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
-{ 
-    int i;
-    paddr_t start, end;
-
-#ifdef CONFIG_NUMA_EMU
-    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
-        return;
-#endif
-
-    start = pfn_to_paddr(start_pfn);
-    end = pfn_to_paddr(end_pfn);
-
-#ifdef CONFIG_NUMA
-    if ( !numa_off && !numa_scan_nodes(start, end) )
-        return;
-#endif
-
-    printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration found");
-
-    printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
-           start, end);
-    /* setup dummy node covering all memory */
-    memnode_shift = BITS_PER_LONG - 1;
-    memnodemap = _memnodemap;
-    /* Dummy node only uses 1 slot in reality */
-    memnodemap[0] = 0;
-    memnodemapsize = 1;
-
-    nodes_clear(node_online_map);
-    node_set_online(0);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-        numa_set_node(i, 0);
-    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, start, end);
-}
-
-void numa_add_cpu(int cpu)
-{
-    cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-} 
-
-void numa_set_node(int cpu, nodeid_t node)
-{
-    cpu_to_node[cpu] = node;
-}
-
-/* [numa=off] */
-static __init int numa_setup(const char *opt)
-{
-    if ( !strncmp(opt,"off",3) )
-        numa_off = true;
-    else if ( !strncmp(opt,"on",2) )
-        numa_off = false;
-#ifdef CONFIG_NUMA_EMU
-    else if ( !strncmp(opt, "fake=", 5) )
-    {
-        numa_off = false;
-        numa_fake = simple_strtoul(opt+5,NULL,0);
-        if ( numa_fake >= MAX_NUMNODES )
-            numa_fake = MAX_NUMNODES;
-    }
-#endif
-#ifdef CONFIG_ACPI_NUMA
-    else if ( !strncmp(opt,"noacpi",6) )
-    {
-        numa_off = false;
-        fw_numa = -1;
-    }
-#endif
-    else
-        return -EINVAL;
-
-    return 0;
-} 
 
 /*
  * Setup early cpu_to_node.
@@ -395,146 +96,3 @@ int __init arch_meminfo_get_ram_bank_range(uint32_t bank,
 
 	return 0;
 }
-
-static void dump_numa(unsigned char key)
-{
-    s_time_t now = NOW();
-    unsigned int i, j, n;
-    struct domain *d;
-    struct page_info *page;
-    unsigned int page_num_node[MAX_NUMNODES];
-    const struct vnuma_info *vnuma;
-
-    printk("'%c' pressed -> dumping numa info (now = %"PRI_stime")\n", key,
-           now);
-
-    for_each_online_node ( i )
-    {
-        paddr_t pa = pfn_to_paddr(node_start_pfn(i) + 1);
-
-        printk("NODE%u start->%lu size->%lu free->%lu\n",
-               i, node_start_pfn(i), node_spanned_pages(i),
-               avail_node_heap_pages(i));
-        /* sanity check phys_to_nid() */
-        if ( phys_to_nid(pa) != i )
-            printk("phys_to_nid(%"PRIpaddr") -> %d should be %u\n",
-                   pa, phys_to_nid(pa), i);
-    }
-
-    j = cpumask_first(&cpu_online_map);
-    n = 0;
-    for_each_online_cpu ( i )
-    {
-        if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] )
-        {
-            if ( n > 1 )
-                printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
-            else
-                printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
-            j = i;
-            n = 1;
-        }
-        else
-            ++n;
-    }
-    if ( n > 1 )
-        printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
-    else
-        printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
-
-    rcu_read_lock(&domlist_read_lock);
-
-    printk("Memory location of each domain:\n");
-    for_each_domain ( d )
-    {
-        process_pending_softirqs();
-
-        printk("Domain %u (total: %u):\n", d->domain_id, domain_tot_pages(d));
-
-        for_each_online_node ( i )
-            page_num_node[i] = 0;
-
-        spin_lock(&d->page_alloc_lock);
-        page_list_for_each(page, &d->page_list)
-        {
-            i = phys_to_nid(page_to_maddr(page));
-            page_num_node[i]++;
-        }
-        spin_unlock(&d->page_alloc_lock);
-
-        for_each_online_node ( i )
-            printk("    Node %u: %u\n", i, page_num_node[i]);
-
-        if ( !read_trylock(&d->vnuma_rwlock) )
-            continue;
-
-        if ( !d->vnuma )
-        {
-            read_unlock(&d->vnuma_rwlock);
-            continue;
-        }
-
-        vnuma = d->vnuma;
-        printk("     %u vnodes, %u vcpus, guest physical layout:\n",
-               vnuma->nr_vnodes, d->max_vcpus);
-        for ( i = 0; i < vnuma->nr_vnodes; i++ )
-        {
-            unsigned int start_cpu = ~0U;
-
-            if ( vnuma->vnode_to_pnode[i] == NUMA_NO_NODE )
-                printk("       %3u: pnode ???,", i);
-            else
-                printk("       %3u: pnode %3u,", i, vnuma->vnode_to_pnode[i]);
-
-            printk(" vcpus ");
-
-            for ( j = 0; j < d->max_vcpus; j++ )
-            {
-                if ( !(j & 0x3f) )
-                    process_pending_softirqs();
-
-                if ( vnuma->vcpu_to_vnode[j] == i )
-                {
-                    if ( start_cpu == ~0U )
-                    {
-                        printk("%d", j);
-                        start_cpu = j;
-                    }
-                }
-                else if ( start_cpu != ~0U )
-                {
-                    if ( j - 1 != start_cpu )
-                        printk("-%d ", j - 1);
-                    else
-                        printk(" ");
-                    start_cpu = ~0U;
-                }
-            }
-
-            if ( start_cpu != ~0U  && start_cpu != j - 1 )
-                printk("-%d", j - 1);
-
-            printk("\n");
-
-            for ( j = 0; j < vnuma->nr_vmemranges; j++ )
-            {
-                if ( vnuma->vmemrange[j].nid == i )
-                    printk("           %016"PRIx64" - %016"PRIx64"\n",
-                           vnuma->vmemrange[j].start,
-                           vnuma->vmemrange[j].end);
-            }
-        }
-
-        read_unlock(&d->vnuma_rwlock);
-    }
-
-    rcu_read_unlock(&domlist_read_lock);
-}
-
-static __init int register_numa_trigger(void)
-{
-    register_keyhandler('u', dump_numa, "dump NUMA info", 1);
-    return 0;
-}
-__initcall(register_numa_trigger);
-
xen/arch/x86/srat.c +2 -251

300 lines not on this page · raw message

xen/common/Makefile +2 -0
index 54de70d422..90e5bf3efb 100644
@@ -26,6 +26,8 @@ obj-$(CONFIG_MEM_ACCESS) += mem_access.o
 obj-y += memory.o
 obj-y += multicall.o
 obj-y += notifier.o
+obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_NUMA) += numa_srat.o
 obj-y += page_alloc.o
 obj-$(CONFIG_HAS_PDX) += pdx.o
 obj-$(CONFIG_PERF_COUNTERS) += perfc.o
xen/common/numa.c new file +450 -0

450 lines not on this page · raw message

xen/common/numa_srat.c new file +264 -0

264 lines not on this page · raw message

xen/include/asm-x86/acpi.h +0 -4
index 2add971072..2140461ff3 100644
@@ -101,10 +101,6 @@ extern unsigned long acpi_wakeup_address;
 
 #define ARCH_HAS_POWER_INIT	1
 
-extern s8 fw_numa;
-extern int numa_scan_nodes(u64 start, u64 end);
-#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
-
 extern struct acpi_sleep_info acpi_sinfo;
 #define acpi_video_flags bootsym(video_flags)
 struct xenpf_enter_acpi_sleep;
xen/include/asm-x86/numa.h +0 -68

85 lines not on this page · raw message

xen/include/asm-x86/setup.h +0 -1
index 24be46115d..63838ba2d1 100644
@@ -17,7 +17,6 @@ void early_time_init(void);
 
 void set_nr_cpu_ids(unsigned int max_cpus);
 
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 void arch_init_memory(void);
 void subarch_init_memory(void);
 
xen/include/xen/numa.h +82 -0

92 lines not on this page · raw message

Signature · 1 line
2.25.1
author
#94 lore raw

PATCH 19/37 xen/x86: promote VIRTUAL_BUG_ON to ASSERT in

In reply to Wei Chen

VIRTUAL_BUG_ON that is using in phys_to_nid is an empty macro. This results in two lines of error-checking code in phys_to_nid are not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is because MAX_NUMNODES is defined in xen/numa.h. But asm/numa.h is a dependent file of xen/numa.h, we can't include xen/numa.h in asm/numa.h. This error has been fixed after we move phys_to_nid to xen/numa.h.
2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT in VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages.

Because when node_spanned_pages is 0, this node has no memory. numa_scan_node will print warning message for such kind of nodes:
"Firmware Bug or mis-configured hardware?". Even Xen allows to online such kind of nodes. I still think it's impossible for phys_to_nid to return a no memory node for a physical address.

---
 xen/include/xen/numa.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
xen/include/xen/numa.h +2 -2
index 51391a2440..1978e2be1b 100644
@@ -38,7 +38,7 @@ struct node {
 extern int compute_hash_shift(struct node *nodes, int numnodes,
 			      nodeid_t *nodeids);
 
-#define VIRTUAL_BUG_ON(x)
+#define VIRTUAL_BUG_ON(x) ASSERT(!(x))
 
 extern void numa_add_cpu(int cpu);
 extern void numa_init_array(void);
@@ -75,7 +75,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 	nodeid_t nid;
 	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
 	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
-	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid].node_spanned_pages);
 	return nid;
 }
 
Signature · 1 line
2.25.1
#95 lore raw

Re: [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in

Quoted 8 lines · Wei Chen

VIRTUAL_BUG_ON that is using in phys_to_nid is an empty macro. This results in two lines of error-checking code in phys_to_nid are not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is because MAX_NUMNODES is defined in xen/numa.h. But asm/numa.h is a dependent file of xen/numa.h, we can't include xen/numa.h in asm/numa.h. This error has been fixed after we move phys_to_nid to xen/numa.h.

This could easily be taken care of by move MAX_NUMNODES up ahead of the asm/numa.h inclusion point. And then the change here would become independent of the rest of the series (and could hence go in early).

Quoted 6 lines

2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

So, in this patch, we use ASSERT in VIRTUAL_BUG_ON to enable the two lines of error-checking code.

May I suggest to drop VIRTUAL_BUG_ON() and instead use ASSERT() directly?

Jan

author
#96 lore raw

RE: [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in

Hi Jan,

Quoted 26 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in

Quoted 8 lines · Wei Chen

VIRTUAL_BUG_ON that is using in phys_to_nid is an empty macro. This results in two lines of error-checking code in phys_to_nid are not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is because MAX_NUMNODES is defined in xen/numa.h. But asm/numa.h is a dependent file of xen/numa.h, we can't include xen/numa.h in asm/numa.h. This error has been fixed after we move phys_to_nid to xen/numa.h.

This could easily be taken care of by move MAX_NUMNODES up ahead of the asm/numa.h inclusion point. And then the change here would become independent of the rest of the series (and could hence go in early).

Quoted 4 lines

2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.

Quoted 6 lines
Quoted 2 lines

So, in this patch, we use ASSERT in VIRTUAL_BUG_ON to enable the two lines of error-checking code.

May I suggest to drop VIRTUAL_BUG_ON() and instead use ASSERT() directly?

Sure!

Quoted 1 line

Jan

author
#97 lore raw

PATCH 20/37 xen: introduce CONFIG_EFI to stub API for non-EFI architecture

In reply to Wei Chen

Some architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this Kconfig option to give Xen the ability of stubing EFI API for non-EFI supported architectures.

---
 xen/arch/arm/Kconfig  |  1 +
 xen/arch/arm/Makefile |  2 +-
 xen/arch/x86/Kconfig  |  1 +
 xen/common/Kconfig    | 11 +++++++++++
 xen/include/xen/efi.h |  4 ++++
 5 files changed, 18 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -0
index ecfa6822e4..865ad83a89 100644
@@ -6,6 +6,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
+	select EFI
 	select HAS_FAST_MULTIPLY
 
 config ARM
xen/arch/arm/Makefile +1 -1
index 3d3b97b5b4..ae4efbf76e 100644
@@ -1,6 +1,6 @@
 obj-$(CONFIG_ARM_32) += arm32/
 obj-$(CONFIG_ARM_64) += arm64/
-obj-$(CONFIG_ARM_64) += efi/
+obj-$(CONFIG_EFI) += efi/
 obj-$(CONFIG_ACPI) += acpi/
 ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
xen/arch/x86/Kconfig +1 -0
index 28d13b9705..b9ed187f6b 100644
@@ -10,6 +10,7 @@ config X86
 	select ALTERNATIVE_CALL
 	select ARCH_SUPPORTS_INT128
 	select CORE_PARKING
+	select EFI
 	select HAS_ALTERNATIVE
 	select HAS_COMPAT
 	select HAS_CPUFREQ
xen/common/Kconfig +11 -0
index 9ebb1c239b..f998746a1a 100644
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.
+
 config GRANT_TABLE
 	bool "Grant table support" if EXPERT
 	default y
@@ -196,6 +206,7 @@ config KEXEC
 
 config EFI_SET_VIRTUAL_ADDRESS_MAP
     bool "EFI: call SetVirtualAddressMap()" if EXPERT
+    depends on EFI
     ---help---
       Call EFI SetVirtualAddressMap() runtime service to setup memory map for
       further runtime services. According to UEFI spec, it isn't strictly
xen/include/xen/efi.h +4 -0
index 94a7e547f9..661a48286a 100644
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __XEN_EFI_H__ */
Signature · 1 line
2.25.1
#98 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 55 lines · Wei Chen

Some architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this Kconfig option to give Xen the ability of stubing EFI API for non-EFI supported architectures.

---
 xen/arch/arm/Kconfig  |  1 +
 xen/arch/arm/Makefile |  2 +-
 xen/arch/x86/Kconfig  |  1 +
 xen/common/Kconfig    | 11 +++++++++++
 xen/include/xen/efi.h |  4 ++++
 5 files changed, 18 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -0
index ecfa6822e4..865ad83a89 100644
@@ -6,6 +6,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
+	select EFI
 	select HAS_FAST_MULTIPLY
 
 config ARM
xen/arch/arm/Makefile +1 -1
index 3d3b97b5b4..ae4efbf76e 100644
@@ -1,6 +1,6 @@
 obj-$(CONFIG_ARM_32) += arm32/
 obj-$(CONFIG_ARM_64) += arm64/
-obj-$(CONFIG_ARM_64) += efi/
+obj-$(CONFIG_EFI) += efi/
 obj-$(CONFIG_ACPI) += acpi/
 ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
xen/arch/x86/Kconfig +1 -0
index 28d13b9705..b9ed187f6b 100644
@@ -10,6 +10,7 @@ config X86
 	select ALTERNATIVE_CALL
 	select ARCH_SUPPORTS_INT128
 	select CORE_PARKING
+	select EFI
 	select HAS_ALTERNATIVE
 	select HAS_COMPAT
 	select HAS_CPUFREQ
xen/common/Kconfig +0 -0
index 9ebb1c239b..f998746a1a 100644

xen/common/Kconfig config COMPAT · line 15

Quoted 5 lines
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 44 lines
xen/common/Kconfig +9 -0
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.
+
 config GRANT_TABLE
 	bool "Grant table support" if EXPERT
 	default y
@@ -196,6 +206,7 @@ config KEXEC
 
 config EFI_SET_VIRTUAL_ADDRESS_MAP
     bool "EFI: call SetVirtualAddressMap()" if EXPERT
+    depends on EFI
     ---help---
       Call EFI SetVirtualAddressMap() runtime service to setup memory map for
       further runtime services. According to UEFI spec, it isn't strictly
xen/include/xen/efi.h +4 -0
index 94a7e547f9..661a48286a 100644
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __XEN_EFI_H__ */
Signature · 1 line
2.25.1
author
#99 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Stefano,

Quoted 74 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 55 lines · Wei Chen

Some architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this Kconfig option to give Xen the ability of stubing EFI API for non-EFI supported architectures.

---
 xen/arch/arm/Kconfig  |  1 +
 xen/arch/arm/Makefile |  2 +-
 xen/arch/x86/Kconfig  |  1 +
 xen/common/Kconfig    | 11 +++++++++++
 xen/include/xen/efi.h |  4 ++++
 5 files changed, 18 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -0
index ecfa6822e4..865ad83a89 100644
@@ -6,6 +6,7 @@ config ARM_64
 	def_bool y
 	depends on !ARM_32
 	select 64BIT
+	select EFI
 	select HAS_FAST_MULTIPLY
 
 config ARM
xen/arch/arm/Makefile +1 -1
index 3d3b97b5b4..ae4efbf76e 100644
@@ -1,6 +1,6 @@
 obj-$(CONFIG_ARM_32) += arm32/
 obj-$(CONFIG_ARM_64) += arm64/
-obj-$(CONFIG_ARM_64) += efi/
+obj-$(CONFIG_EFI) += efi/
 obj-$(CONFIG_ACPI) += acpi/
 ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
xen/arch/x86/Kconfig +1 -0
index 28d13b9705..b9ed187f6b 100644
@@ -10,6 +10,7 @@ config X86
 	select ALTERNATIVE_CALL
 	select ARCH_SUPPORTS_INT128
 	select CORE_PARKING
+	select EFI
 	select HAS_ALTERNATIVE
 	select HAS_COMPAT
 	select HAS_CPUFREQ
xen/common/Kconfig +0 -0
index 9ebb1c239b..f998746a1a 100644

xen/common/Kconfig config COMPAT · line 15

Quoted 5 lines
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.

Quoted 7 lines

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

Quoted 47 lines
Quoted 13 lines
xen/common/Kconfig +8 -0
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.
+
 config GRANT_TABLE
 	bool "Grant table support" if EXPERT
 	default y

xen/common/Kconfig config KEXEC · line 211

Quoted 6 lines
xen/common/Kconfig +1 -0
@@ -196,6 +206,7 @@ config KEXEC
 
 config EFI_SET_VIRTUAL_ADDRESS_MAP
     bool "EFI: call SetVirtualAddressMap()" if EXPERT
+    depends on EFI
     ---help---
       Call EFI SetVirtualAddressMap() runtime service to setup memory

map for

xen/common/Kconfig config KEXEC · line 212

Quoted 1 line
xen/common/Kconfig +0 -0
       further runtime services. According to UEFI spec, it isn't

strictly

Quoted 25 lines
xen/include/xen/efi.h +4 -1
index 94a7e547f9..661a48286a 100644
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __XEN_EFI_H__ */
--

2.25.1

#100 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 16 lines · Wei Chen
Quoted 16 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.

Quoted 8 lines
Quoted 5 lines

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

Jan

author
#101 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 32 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 22 lines · Wei Chen
Quoted 21 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 8 lines
Quoted 2 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.

Quoted 1 line

Jan

#102 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 35 lines · Wei Chen
Quoted 26 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 22 lines · Wei Chen
Quoted 21 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 8 lines
Quoted 2 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

Quoted 6 lines

I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Jan

author
#103 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 51 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Jan Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 10 lines · Wei Chen
Quoted 10 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

Quoted 6 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

xen/common/Kconfig config COMPAT · line 15

Quoted 2 lines
xen/common/Kconfig +2 -0
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

selectable).

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 8 lines
Quoted 2 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.

Quoted 6 lines
Quoted 2 lines

So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00808.html

Quoted 2 lines

Jan

author
#104 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Julien, Stefano,

Quoted 72 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Hi Jan,

Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of

Jan

Quoted 6 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 43 lines

EFI architecture

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 10 lines · Wei Chen
Quoted 10 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

Quoted 6 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

xen/common/Kconfig config COMPAT · line 15

Quoted 2 lines
xen/common/Kconfig +2 -0
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

selectable).

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 8 lines
Quoted 2 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.

Quoted 6 lines
Quoted 2 lines

So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Quoted 8 lines

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

Quoted 2 lines
Quoted 2 lines

Jan

#105 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 87 lines · Wei Chen
Quoted 72 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Hi Jan,

Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of

Jan

Quoted 6 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 43 lines

EFI architecture

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 10 lines · Wei Chen
Quoted 10 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

Quoted 6 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

xen/common/Kconfig config COMPAT · line 15

Quoted 2 lines
xen/common/Kconfig +2 -0
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

selectable).

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help text.

Quoted 8 lines
Quoted 2 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.

Quoted 6 lines
Quoted 2 lines

So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Quoted 8 lines

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

Quoted 5 lines

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.

On x86 EFI is detected based on compiler support, setting XEN_BUILD_EFI in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.

On ARM32, XEN_BUILD_EFI should be always unset.

On ARM64 XEN_BUILD_EFI should be always set.

That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better names than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.

author
#106 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Stefano,

Quoted 122 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 2 lines · Wei Chen
Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 6 lines
Quoted 6 lines

Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 6 lines
Quoted 6 lines

EFI architecture

Hi Jan,

Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf

Of

Quoted 6 lines
Quoted 6 lines

Jan

Quoted 5 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini

<[email protected]>

Quoted 23 lines
Quoted 23 lines
Quoted 1 line

Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 21 lines

EFI architecture

Quoted 18 lines · Wei Chen
Quoted 18 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

selectable).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help

text.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending

on

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this

option

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not

a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

selectable option in Kconfig. I agree with you, we can't use

Kconfig

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on

Arm,

Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line

we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Quoted 8 lines

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

Quoted 5 lines

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.

On x86 EFI is detected based on compiler support, setting XEN_BUILD_EFI in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.

Quoted 8 lines

On ARM32, XEN_BUILD_EFI should be always unset.

On ARM64 XEN_BUILD_EFI should be always set.

That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better names than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.

Thanks, this is a good approach. But if we place XEN_BUILD_EFI in Kconfig it will be transfer to CONFIG_XEN_BUILD_EFI. How about using another name in Kconfig like ARM_EFI, but use CONFIG_ARM_EFI in config.h to define XEN_BUILD_EFI?

#107 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 131 lines · Wei Chen
Quoted 122 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 2 lines · Wei Chen
Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 6 lines
Quoted 6 lines

Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 6 lines
Quoted 6 lines

EFI architecture

Hi Jan,

Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf

Of

Quoted 6 lines
Quoted 6 lines

Jan

Quoted 5 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini

<[email protected]>

Quoted 23 lines
Quoted 23 lines
Quoted 1 line

Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 21 lines

EFI architecture

Quoted 18 lines · Wei Chen
Quoted 18 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

selectable).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help

text.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending

on

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this

option

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not

a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

selectable option in Kconfig. I agree with you, we can't use

Kconfig

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on

Arm,

Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line

we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Quoted 8 lines

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

Quoted 5 lines

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.

On x86 EFI is detected based on compiler support, setting XEN_BUILD_EFI in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.

Quoted 8 lines

On ARM32, XEN_BUILD_EFI should be always unset.

On ARM64 XEN_BUILD_EFI should be always set.

That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better names than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.

Quoted 4 lines

Thanks, this is a good approach. But if we place XEN_BUILD_EFI in Kconfig it will be transfer to CONFIG_XEN_BUILD_EFI. How about using another name in Kconfig like ARM_EFI, but use CONFIG_ARM_EFI in config.h to define XEN_BUILD_EFI?

I am OK with that. Another option is to rename XEN_BUILD_EFI to CONFIG_XEN_BUILD_EFI on x86. Either way is fine by me. Jan, do you havea preference?

#108 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 132 lines · Stefano Stabellini
Quoted 131 lines · Wei Chen
Quoted 130 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 2 lines · Wei Chen
Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 6 lines
Quoted 6 lines

Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 6 lines
Quoted 6 lines

EFI architecture

Hi Jan,

Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf

Of

Quoted 6 lines
Quoted 6 lines

Jan

Quoted 5 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini

<[email protected]>

Quoted 23 lines
Quoted 23 lines
Quoted 1 line

Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 21 lines

EFI architecture

Quoted 18 lines · Wei Chen
Quoted 18 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

selectable).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help

text.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending

on

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this

option

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not

a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

selectable option in Kconfig. I agree with you, we can't use

Kconfig

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on

Arm,

Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line

we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you want to use the option for subsequently.

Quoted 8 lines

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

Quoted 5 lines

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.

On x86 EFI is detected based on compiler support, setting XEN_BUILD_EFI in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.

On ARM32, XEN_BUILD_EFI should be always unset.

On ARM64 XEN_BUILD_EFI should be always set.

That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better names than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.

Quoted 8 lines
Quoted 4 lines

Thanks, this is a good approach. But if we place XEN_BUILD_EFI in Kconfig it will be transfer to CONFIG_XEN_BUILD_EFI. How about using another name in Kconfig like ARM_EFI, but use CONFIG_ARM_EFI in config.h to define XEN_BUILD_EFI?

I am OK with that. Another option is to rename XEN_BUILD_EFI to CONFIG_XEN_BUILD_EFI on x86. Either way is fine by me. Jan, do you havea preference?

Yes, I do: No new CONFIG_* settings please that don't originate from Kconfig. Hence I'm afraid this is a "no" to your suggestion.

Mid-term we should try to get rid of the remaining CONFIG_* which get #define-d in e.g. asm/config.h.

Jan

author
#109 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 165 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月28日 16:03
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 9 lines · Stefano Stabellini
Quoted 8 lines · Wei Chen
Quoted 8 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

EFI architecture

Quoted 2 lines · Wei Chen
Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf

Of

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Wei

Quoted 5 lines
Quoted 5 lines

Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini

<[email protected]>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line

Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 4 lines
Quoted 4 lines

EFI architecture

Hi Jan,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf

Of

Quoted 5 lines
Quoted 5 lines

Jan

Quoted 4 lines

Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand

Marquis

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

<[email protected]>; Stefano Stabellini

<[email protected]>

Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API

for

Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 22 lines
Quoted 22 lines

non-

Quoted 21 lines

EFI architecture

Quoted 18 lines · Wei Chen
Quoted 18 lines

From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59

Quoted 14 lines · Wei Chen
Quoted 14 lines

From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15

xen/common/Kconfig config COMPAT · line 15

Quoted 8 lines · Wei Chen
xen/common/Kconfig +2 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
+config EFI
+	bool

Without the title the option is not user-selectable (or de-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

selectable).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

So the help message below can never be seen.

Either add a title, e.g.:

bool "EFI support"

Or fully make the option a silent option by removing the help

text.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.

But that's not true: At least on x86 EFI gets compiled depending

on

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this

option

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

at all may be misleading on x86.

I check the build scripts, yes, you're right. For x86, EFI is not

a

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

selectable option in Kconfig. I agree with you, we can't use

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Kconfig

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

system to decide to enable EFI build for x86 or not.

So how about we just use this EFI option for Arm only? Because on

Arm,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

we do not have such toolchain dependency.

To be honest - don't know. That's because I don't know what you

want

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

to use the option for subsequently.

In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.

[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html

As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.

If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?

EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.

On x86 EFI is detected based on compiler support, setting

XEN_BUILD_EFI

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.

On ARM32, XEN_BUILD_EFI should be always unset.

On ARM64 XEN_BUILD_EFI should be always set.

That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better

names

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select

it

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

only on ARM64.

Thanks, this is a good approach. But if we place XEN_BUILD_EFI in

Kconfig

Quoted 1 line
Quoted 1 line

it will be transfer to CONFIG_XEN_BUILD_EFI. How about using another

name

Quoted 6 lines
Quoted 2 lines

in Kconfig like ARM_EFI, but use CONFIG_ARM_EFI in config.h to define XEN_BUILD_EFI?

I am OK with that. Another option is to rename XEN_BUILD_EFI to CONFIG_XEN_BUILD_EFI on x86. Either way is fine by me. Jan, do you havea preference?

Quoted 6 lines

Yes, I do: No new CONFIG_* settings please that don't originate from Kconfig. Hence I'm afraid this is a "no" to your suggestion.

Mid-term we should try to get rid of the remaining CONFIG_* which get #define-d in e.g. asm/config.h.

I will do something like this:
- introduce an ARM_EFI invisible symbol in kconfig, selected by ARM64 only
- use CONFIG_ARM_EFI to define XEN_BUILD_EFI in config.h

Quoted 1 line

Jan

#110 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

In reply to Wei Chen
Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

Quoted 12 lines
xen/include/xen/efi.h +2 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Jan

author
#111 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 33 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Quoted 3 lines

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 24 lines
Quoted 12 lines
xen/include/xen/efi.h +2 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Jan

author
#112 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 60 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2022年1月27日 16:45
To: Jan Beulich <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Hi Jan,

Quoted 7 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for

non-

Quoted 26 lines

EFI architecture

Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Quoted 3 lines

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
Quoted 18 lines
Quoted 6 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call

Quoted 6 lines
xen/include/xen/efi.h +2 -0
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call

*);

Quoted 11 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h

Quoted 1 line
Quoted 1 line

Jan

#113 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 63 lines · Wei Chen
Quoted 45 lines

From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2022年1月27日 16:45

Quoted 27 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35

Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Quoted 3 lines

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
Quoted 17 lines
Quoted 6 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call

Quoted 6 lines
xen/include/xen/efi.h +2 -0
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call

*);

Quoted 10 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Quoted 9 lines

I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h

But you realize we already have such a stub file on x86?

Jan

author
#114 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 84 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 1 line · Wei Chen
Quoted 1 line

From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 62 lines
Quoted 44 lines

Chen
Sent: 2022年1月27日 16:45

Quoted 27 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35

Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Quoted 3 lines

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
Quoted 17 lines
Quoted 6 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call

Quoted 6 lines
xen/include/xen/efi.h +2 -0
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call

*);

Quoted 10 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Quoted 9 lines

I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h

Quoted 2 lines

But you realize we already have such a stub file on x86?

Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

Quoted 1 line

Jan

#115 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 81 lines · Wei Chen
Quoted 78 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00

Quoted 1 line · Wei Chen
Quoted 1 line

From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 62 lines
Quoted 44 lines

Chen
Sent: 2022年1月27日 16:45

Quoted 27 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35

Quoted 5 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool

xen/common/Kconfig config COMPAT · line 22

Quoted 10 lines
xen/common/Kconfig +9 -0
 
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Quoted 3 lines

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
 #ifndef __ASSEMBLY__
 
Quoted 17 lines
Quoted 6 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call

Quoted 6 lines
xen/include/xen/efi.h +2 -0
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call

*);

Quoted 10 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Quoted 9 lines

I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h

Quoted 2 lines

But you realize we already have such a stub file on x86?

Quoted 9 lines

Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.

Jan

author
#116 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Jan,

Quoted 105 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Quoted 19 lines · Wei Chen
Quoted 19 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00

Quoted 1 line · Wei Chen
Quoted 1 line

From: Xen-devel <[email protected]> On Behalf Of

Wei

Quoted 13 lines
Quoted 13 lines

Chen
Sent: 2022年1月27日 16:45

Quoted 10 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35

Quoted 6 lines · Wei Chen
xen/common/Kconfig +0 -0
@@ -11,6 +11,16 @@ config COMPAT
 config CORE_PARKING
 	bool
 
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/common/Kconfig +4 -0
+config EFI
+	bool
+	---help---
+      This option provides support for runtime services provided
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/common/Kconfig config COMPAT · line 22

Quoted 5 lines
xen/common/Kconfig +5 -0
+      by UEFI firmware (such as non-volatile variables, realtime
+      clock, and platform reset). A UEFI stub is also provided to
+      allow the kernel to be booted as an EFI application. This
+      is only useful for kernels that may run on systems that have
+      UEFI firmware.

The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very

least

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the help text would need to call out the extra dependencies. Plus

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

help text of course then needs to be generic (i.e. applicable to

both

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.

Also (nit): Indentation. And please don't use ---help--- anymore in new code.

I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.

Quoted 4 lines
Quoted 4 lines
xen/include/xen/efi.h +0 -0
@@ -25,6 +25,8 @@ extern struct efi efi;
 
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/include/xen/efi.h int efi_runtime_call(struct

Quoted 8 lines
xen/include/xen/efi.h +2 -0
 #ifndef __ASSEMBLY__
 
+#ifdef CONFIG_EFI
+
 union xenpf_efi_info;
 union compat_pf_efi_info;
 
@@ -45,6 +47,8 @@ int efi_runtime_call(struct

xenpf_efi_runtime_call

Quoted 25 lines
Quoted 22 lines
Quoted 12 lines
Quoted 11 lines

*);

Quoted 10 lines

xen/include/xen/efi.h int efi_runtime_call(struct · line 50

Quoted 4 lines
xen/include/xen/efi.h +1 -0
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#endif /* CONFIG_EFI*/

I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).

Also (nit): Missing blank in the comment.

Quoted 9 lines

I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h

Quoted 2 lines

But you realize we already have such a stub file on x86?

Quoted 9 lines

Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

Quoted 3 lines

While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.

Some dummy functions in stub.c can be shared by arm or other architectures. But some functions like efi_multiboot2 are architecture dependent.

Quoted 2 lines

Jan

#117 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Quoted 20 lines · Wei Chen
Quoted 20 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17

Quoted 15 lines · Wei Chen
Quoted 5 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00

But you realize we already have such a stub file on x86?

Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

Quoted 6 lines
Quoted 3 lines

While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.

Some dummy functions in stub.c can be shared by arm or other architectures. But some functions like efi_multiboot2 are architecture dependent.

Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.

Jan

#118 lore raw

Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi,

Quoted 25 lines · Jan Beulich
Quoted 24 lines · Wei Chen
Quoted 23 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17

Quoted 6 lines · Wei Chen
Quoted 5 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00

But you realize we already have such a stub file on x86?

Quoted 9 lines

Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.

Quoted 7 lines
Quoted 2 lines

Some dummy functions in stub.c can be shared by arm or other architectures. But some functions like efi_multiboot2 are architecture dependent.

Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.

If the stubs are mostly the same then they should be common rather than duplicated.

Quoted 2 lines

x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me.

But that would be a build risk right? If so, it is quite easy to verify/catch it compare to runtime issue.

Cheers,

Quoted 3 lines

Jan

Signature · 1 line
Julien Grall
author
#119 lore raw

RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture

Hi Julien,

Quoted 47 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年1月27日 18:01
To: Jan Beulich <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non- EFI architecture

Hi,

Quoted 4 lines · Jan Beulich
Quoted 3 lines · Wei Chen
Quoted 3 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 7 lines
Quoted 5 lines

From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00

But you realize we already have such a stub file on x86?

Yes, we found the redefinition errors that are caused by x86 stub

file

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFI
XEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.

Some dummy functions in stub.c can be shared by arm or other

architectures.

Quoted 6 lines
Quoted 1 line

But some functions like efi_multiboot2 are architecture dependent.

Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.

Quoted 9 lines

If the stubs are mostly the same then they should be common rather than duplicated.

Quoted 2 lines

x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me.

But that would be a build risk right? If so, it is quite easy to verify/catch it compare to runtime issue.

Risk may be not very accurate word. I mean, I was afraid that my changes to x86 EFI build would introduce some unforeseen problems to x86. Because I am not very familiar with that. As you and Jan think it's better, I will try to do it.

Quoted 8 lines

Cheers,

Quoted 3 lines

Jan

--
Julien Grall

author
#120 lore raw

PATCH 21/37 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

In reply to Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prevent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent

---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/xen/efi.h       |  7 +++++++
 3 files changed, 14 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
index afaa0e249b..6bc5a465ec 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If Xen has been booted via UEFI, the memory banks will already
+     * be populated. So we should skip the parsing.
+     */
+    if ( !efi_enabled(EFI_BOOT) &&
+         device_tree_node_matches(fdt, node, "memory"))
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells, &bootinfo.mem);
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -25
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
     int status;
     u32 fdt_val32;
     u64 fdt_val64;
-    int prev;
     int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
    /*
     * Delete all memory reserve map entries. When booting via UEFI,
     * kernel will use the UEFI memory map to find reserved regions.
xen/include/xen/efi.h +7 -0
index 661a48286a..b52a4678e9 100644
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#else
+
+static inline bool efi_enabled(unsigned int feature)
+{
+    return false;
+}
+
 #endif /* CONFIG_EFI*/
 
 #endif /* !__ASSEMBLY__ */
Signature · 1 line
2.25.1
#121 lore raw

Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 12 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

Quoted 7 lines

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prevent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent

This last sentence is incomplete.

But aside from that, this patch looks good to me.

Quoted 92 lines
---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/xen/efi.h       |  7 +++++++
 3 files changed, 14 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
index afaa0e249b..6bc5a465ec 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If Xen has been booted via UEFI, the memory banks will already
+     * be populated. So we should skip the parsing.
+     */
+    if ( !efi_enabled(EFI_BOOT) &&
+         device_tree_node_matches(fdt, node, "memory"))
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells, &bootinfo.mem);
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -25
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
     int status;
     u32 fdt_val32;
     u64 fdt_val64;
-    int prev;
     int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
    /*
     * Delete all memory reserve map entries. When booting via UEFI,
     * kernel will use the UEFI memory map to find reserved regions.
xen/include/xen/efi.h +7 -0
index 661a48286a..b52a4678e9 100644
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#else
+
+static inline bool efi_enabled(unsigned int feature)
+{
+    return false;
+}
+
 #endif /* CONFIG_EFI*/
 
 #endif /* !__ASSEMBLY__ */
Signature · 1 line
2.25.1
author
#122 lore raw

RE: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 28 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:23
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 17 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prevent early_scan_node to parse memory nodes in EFI boot.

Quoted 7 lines
Quoted 2 lines

As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent

This last sentence is incomplete.

But aside from that, this patch looks good to me.

Ah, it truncated by accident. I will fix it.

Quoted 95 lines
Quoted 23 lines
---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/xen/efi.h       |  7 +++++++
 3 files changed, 14 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index afaa0e249b..6bc5a465ec 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 381

Quoted 10 lines
xen/arch/arm/bootfdt.c +6 -1
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If Xen has been booted via UEFI, the memory banks will already
+     * be populated. So we should skip the parsing.
+     */
+    if ( !efi_enabled(EFI_BOOT) &&
+         device_tree_node_matches(fdt, node, "memory"))
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells,

&bootinfo.mem);

xen/arch/arm/efi/efi-boot.h EFI_STATUS __init

Quoted 6 lines
xen/arch/arm/bootfdt.c +0 -0
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -0
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init

fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,

Quoted 54 lines
xen/arch/arm/efi/efi-boot.h +0 -25
     int status;
     u32 fdt_val32;
     u64 fdt_val64;
-    int prev;
     int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
    /*
     * Delete all memory reserve map entries. When booting via UEFI,
     * kernel will use the UEFI memory map to find reserved regions.
xen/include/xen/efi.h +7 -1
index 661a48286a..b52a4678e9 100644
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#else
+
+static inline bool efi_enabled(unsigned int feature)
+{
+    return false;
+}
+
 #endif /* CONFIG_EFI*/
 
 #endif /* !__ASSEMBLY__ */
--

2.25.1

#123 lore raw

Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

In reply to Wei Chen
Quoted 4 lines · Wei Chen
xen/include/xen/efi.h +0 -0
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 57

Quoted 10 lines
xen/include/xen/efi.h +7 -0
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
+#else
+
+static inline bool efi_enabled(unsigned int feature)
+{
+    return false;
+}
+
 #endif /* CONFIG_EFI*/

As already hinted at for the earlier patch, I think this hunk wants moving ahead in the series.

Jan

author
#124 lore raw

RE: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Hi Jan,

Quoted 16 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:39
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 6 lines · Wei Chen
xen/include/xen/efi.h +0 -0
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);
 int efi_compat_get_info(uint32_t idx, union compat_pf_efi_info *);
 int efi_compat_runtime_call(struct compat_pf_efi_runtime_call *);
 
Quoted 11 lines

xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 57

Quoted 8 lines
xen/include/xen/efi.h +7 -0
+#else
+
+static inline bool efi_enabled(unsigned int feature)
+{
+    return false;
+}
+
 #endif /* CONFIG_EFI*/

As already hinted at for the earlier patch, I think this hunk wants moving ahead in the series.

Yes, I will do it.
Thanks!

Quoted 2 lines

Jan

author
#125 lore raw

PATCH 22/37 xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

In reply to Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header. And keep default NR_NODE_MEMBLKS in common header for those architectures NUMA is disabled.

---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +2 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
#126 lore raw

Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

Quoted 2 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is no x86 definition of it)

Quoted 44 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +2 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
author
#127 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Hi Stefano,

Quoted 9 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 8 lines
Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't thought so deeply. We just thought a memory range described in DT cannot be split across multiple nodes. So NR_MEM_BANKS should be equal to NR_MEM_BANKS.

https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00974.html

Quoted 10 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

But I agree with Julien's suggestion, NR_MEM_BANKS and NR_NODE_MEMBLKS must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

Quoted 8 lines
Quoted 3 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is no x86 definition of it)

Yes.

Quoted 45 lines
Quoted 45 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +2 -1
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
--

2.25.1

#128 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 40 lines · Wei Chen
Quoted 9 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 8 lines
Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't thought so deeply. We just thought a memory range described in DT cannot be split across multiple nodes. So NR_MEM_BANKS should be equal to NR_MEM_BANKS.

https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00974.html

Quoted 10 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

Quoted 4 lines

But I agree with Julien's suggestion, NR_MEM_BANKS and NR_NODE_MEMBLKS must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 47 lines
Quoted 8 lines
Quoted 3 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is no x86 definition of it)

Yes.

Quoted 36 lines
Quoted 36 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

author
#129 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Hi Stefano,

Quoted 114 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 7 lines
Quoted 7 lines

NR_NODE_MEMBLKS

Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

Quoted 5 lines
Quoted 3 lines

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't

thought

Quoted 1 line

so deeply. We just thought a memory range described in DT cannot be

split

Quoted 3 lines

across multiple nodes. So NR_MEM_BANKS should be equal to NR_MEM_BANKS.

https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00974.html

Quoted 17 lines
Quoted 11 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

Quoted 4 lines

But I agree with Julien's suggestion, NR_MEM_BANKS and NR_NODE_MEMBLKS must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is

no

Quoted 41 lines
Quoted 2 lines

x86 definition of it)

Yes.

Quoted 36 lines
Quoted 36 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all NR_NODE_MEMBLKS in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of CONFIG_NR_NUMA_NODES.

#130 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

+x86 maintainers

Quoted 120 lines · Wei Chen
Quoted 114 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 7 lines
Quoted 7 lines

NR_NODE_MEMBLKS

Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

Quoted 5 lines
Quoted 3 lines

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't

thought

Quoted 1 line

so deeply. We just thought a memory range described in DT cannot be

split

Quoted 3 lines

across multiple nodes. So NR_MEM_BANKS should be equal to NR_MEM_BANKS.

https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00974.html

Quoted 17 lines
Quoted 11 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

Quoted 4 lines

But I agree with Julien's suggestion, NR_MEM_BANKS and NR_NODE_MEMBLKS must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is

no

Quoted 41 lines
Quoted 2 lines

x86 definition of it)

Yes.

Quoted 36 lines
Quoted 36 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
index 8f1c67e3eb..21569e634b 100644
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

Quoted 10 lines

How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all NR_NODE_MEMBLKS in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of CONFIG_NR_NUMA_NODES.

x86 doesn't have NR_MEM_BANKS as far as I can tell. I guess you also meant to rename NR_NODE_MEMBLKS to NR_MEM_BANKS?

But NR_MEM_BANKS is not directly related to CONFIG_NR_NUMA_NODES because there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:

#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)

Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?

On the other hand, NR_MEM_BANKS and NR_NODE_MEMBLKS seem to be related. In fact, what's the difference?

NR_MEM_BANKS is the max number of memory banks (with or without numa-node-id).

NR_NODE_MEMBLKS is the max number of memory banks with NUMA support (with numa-node-id)?

They are basically the same thing. On ARM I would just do:

#define NR_NODE_MEMBLKS MAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))

And maybe the definition could be common with x86 if we define NR_MEM_BANKS to 128 on x86 too.

#131 lore raw

Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Quoted 162 lines · On Mon, 27 Sep 2021, 06:59 Stefano Stabellini, <[email protected]> wrote:

+x86 maintainers

Quoted 8 lines · Wei Chen
Quoted 8 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 23 lines
Quoted 23 lines

NR_NODE_MEMBLKS

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 11 lines
Quoted 10 lines

NR_NODE_MEMBLKS

Quoted 3 lines · Wei Chen

As a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

No, when Julien and I discussed this in last version[1], we hadn't

thought

Quoted 1 line

so deeply. We just thought a memory range described in DT cannot be

split

Quoted 1 line

across multiple nodes. So NR_MEM_BANKS should be equal to

NR_MEM_BANKS.

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00974.html

Quoted 6 lines
Quoted 6 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at

least

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

doesn't make any difference unless CONFIG_NR_NUMA_NODES is

increased.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

But I agree with Julien's suggestion, NR_MEM_BANKS and

NR_NODE_MEMBLKS

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there

is

Quoted 7 lines
Quoted 7 lines

no

Quoted 6 lines
Quoted 2 lines

x86 definition of it)

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Review

Quoted 7 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/xen/include/asm-arm/numa.h

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

index 8f1c67e3eb..21569e634b 100644

xen/include/asm-arm/numa.h +7 -1
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

Quoted 8 lines

How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all

NR_NODE_MEMBLKS

Quoted 2 lines

in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of

CONFIG_NR_NUMA_NODES.

x86 doesn't have NR_MEM_BANKS as far as I can tell. I guess you also meant to rename NR_NODE_MEMBLKS to NR_MEM_BANKS?

But NR_MEM_BANKS is not directly related to CONFIG_NR_NUMA_NODES because there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:

#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)

Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?

On the other hand, NR_MEM_BANKS and NR_NODE_MEMBLKS seem to be related. In fact, what's the difference?

NR_MEM_BANKS is the max number of memory banks (with or without numa-node-id).

Quoted 7 lines

NR_NODE_MEMBLKS is the max number of memory banks with NUMA support (with numa-node-id)?

They are basically the same thing. On ARM I would just do:

#define NR_NODE_MEMBLKS MAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))

As you wrote above, the second part of the MAX is totally arbitrary. In fact, it is very likely than if you have more than 64 nodes, you may need a lot more than 2 regions per node.

So, for Arm, I would just define NR_NODE_MEMBLKS as an alias to NR_MEM_BANKS so it can be used by common code.

author
#132 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

In reply to Stefano Stabellini

Hi Stefano, Julien,

Quoted 152 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:00
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

+x86 maintainers

Quoted 8 lines · Wei Chen
Quoted 8 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 8 lines
Quoted 8 lines

NR_NODE_MEMBLKS

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

Quoted 8 lines
Quoted 8 lines

[email protected];

Quoted 2 lines
Quoted 2 lines

Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 4 lines
Quoted 4 lines

NR_NODE_MEMBLKS

Quoted 1 line · Wei Chen

As a memory range described in device tree cannot be split

across

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header.

This statement is true but what is the goal of this patch? Is it

to

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't

thought

Quoted 1 line

so deeply. We just thought a memory range described in DT cannot be

split

Quoted 1 line

across multiple nodes. So NR_MEM_BANKS should be equal to

NR_MEM_BANKS.

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00974.html

Quoted 6 lines
Quoted 6 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at

least

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

doesn't make any difference unless CONFIG_NR_NUMA_NODES is

increased.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES

is

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

higher than 64?

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

But I agree with Julien's suggestion, NR_MEM_BANKS and

NR_NODE_MEMBLKS

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there

is

Quoted 7 lines
Quoted 7 lines

no

Quoted 6 lines
Quoted 2 lines

x86 definition of it)

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xen/include/asm-

Quoted 7 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm- renamed from xen/include/asm-arm/numa.h +0 -0

arm/numa.h

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

index 8f1c67e3eb..21569e634b 100644

xen/include/asm-arm/numa.h +7 -1
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

Quoted 8 lines

How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all

Quoted 8 lines

NR_NODE_MEMBLKS

Quoted 2 lines

in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of

CONFIG_NR_NUMA_NODES.

x86 doesn't have NR_MEM_BANKS as far as I can tell. I guess you also meant to rename NR_NODE_MEMBLKS to NR_MEM_BANKS?

Yes.

Quoted 9 lines

But NR_MEM_BANKS is not directly related to CONFIG_NR_NUMA_NODES because there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:

#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)

Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?

At that time, this was probably the most cost-effective approach. Enough and easy. But, if more nodes need to be supported in the future, it may bring more memory blocks. And this maximum value might not apply. The maximum may need to support dynamic extension.

Quoted 4 lines

On the other hand, NR_MEM_BANKS and NR_NODE_MEMBLKS seem to be related. In fact, what's the difference?

Quoted 8 lines

NR_MEM_BANKS is the max number of memory banks (with or without numa-node-id).

NR_NODE_MEMBLKS is the max number of memory banks with NUMA support (with numa-node-id)?

They are basically the same thing. On ARM I would just do:

Probably not, NR_MEM_BANKS will count those memory ranges without numa-node-id in boot memory parsing stage (process_memory_node or EFI parser). But NR_NODE_MEMBLKS will only count those memory ranges with numa-node-id.

Quoted 5 lines

#define NR_NODE_MEMBLKS MAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))

And maybe the definition could be common with x86 if we define NR_MEM_BANKS to 128 on x86 too.

Julien had comment here, I will continue in that email.

author
#133 lore raw

RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Hi Julien,

Quoted 206 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月27日 14:46
To: Stefano Stabellini <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

Hi Stefano, Julien,

Quoted 32 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:00
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS

+x86 maintainers

Quoted 8 lines · Wei Chen
Quoted 8 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 9 lines
Quoted 9 lines

NR_NODE_MEMBLKS

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override

default

Quoted 4 lines
Quoted 4 lines

NR_NODE_MEMBLKS

Quoted 1 line · Wei Chen

As a memory range described in device tree cannot be split

across

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS

in

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

arch header.

This statement is true but what is the goal of this patch? Is it

to

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

reduce code size and memory consumption?

No, when Julien and I discussed this in last version[1], we hadn't

thought

Quoted 1 line

so deeply. We just thought a memory range described in DT cannot

be

Quoted 21 lines
Quoted 2 lines
Quoted 2 lines

split

Quoted 1 line

across multiple nodes. So NR_MEM_BANKS should be equal to

NR_MEM_BANKS.

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

https://lists.xenproject.org/archives/html/xen-devel/2021-

08/msg00974.html

Quoted 6 lines
Quoted 6 lines

I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.

In other words, this patch alone doesn't make any difference; at

least

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

doesn't make any difference unless CONFIG_NR_NUMA_NODES is

increased.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES

is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

higher than 64?

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.

But I agree with Julien's suggestion, NR_MEM_BANKS and

NR_NODE_MEMBLKS

Quoted 1 line
Quoted 1 line
Quoted 1 line

must be aware of each other. I had thought to add some ASSERT

check,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

but I don't know how to do it better. So I post this patch for

more

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

suggestion.

OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

And keep default NR_NODE_MEMBLKS in common header
for those architectures NUMA is disabled.

This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h

(there

Quoted 58 lines

is

Quoted 7 lines
Quoted 7 lines

no

Quoted 6 lines
Quoted 2 lines

x86 definition of it)

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xen/include/asm-

Quoted 7 lines
---
 xen/include/asm-arm/numa.h | 8 +++++++-
 xen/include/xen/numa.h     | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm- renamed from xen/include/asm-arm/numa.h +0 -0

arm/numa.h

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

index 8f1c67e3eb..21569e634b 100644

xen/include/asm-arm/numa.h +7 -1
@@ -3,9 +3,15 @@
 
 #include <xen/mm.h>
 
+#include <asm/setup.h>
+
 typedef u8 nodeid_t;
 
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS NR_MEM_BANKS
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
xen/include/xen/numa.h +0 -0
index 1978e2be1b..1731e1cc6b 100644
@@ -12,7 +12,9 @@
 #define MAX_NUMNODES    1
 #endif
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

xen/include/xen/numa.h line 17

Quoted 3 lines
xen/include/xen/numa.h +2 -0
+#ifndef NR_NODE_MEMBLKS
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
+#endif

This one we can remove it completely right?

Quoted 8 lines

How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all

Quoted 8 lines

NR_NODE_MEMBLKS

Quoted 2 lines

in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of

CONFIG_NR_NUMA_NODES.

x86 doesn't have NR_MEM_BANKS as far as I can tell. I guess you also meant to rename NR_NODE_MEMBLKS to NR_MEM_BANKS?

Yes.

Quoted 9 lines

But NR_MEM_BANKS is not directly related to CONFIG_NR_NUMA_NODES because there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:

#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)

Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?

At that time, this was probably the most cost-effective approach. Enough and easy. But, if more nodes need to be supported in the future, it may bring more memory blocks. And this maximum value might not apply. The maximum may need to support dynamic extension.

Quoted 10 lines

On the other hand, NR_MEM_BANKS and NR_NODE_MEMBLKS seem to be related. In fact, what's the difference?

NR_MEM_BANKS is the max number of memory banks (with or without numa-node-id).

NR_NODE_MEMBLKS is the max number of memory banks with NUMA support (with numa-node-id)?

Quoted 11 lines
Quoted 2 lines

They are basically the same thing. On ARM I would just do:

Probably not, NR_MEM_BANKS will count those memory ranges without numa-node-id in boot memory parsing stage (process_memory_node or EFI parser). But NR_NODE_MEMBLKS will only count those memory ranges with numa-node-id.

Quoted 3 lines

#define NR_NODE_MEMBLKS MAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))

Quote Julien's comment from HTML email to here:
" As you wrote above, the second part of the MAX is totally arbitrary. In fact, it is very likely than if you have more than 64 nodes, you may need a lot more than 2 regions per node.

So, for Arm, I would just define NR_NODE_MEMBLKS as an alias to NR_MEM_BANKS so it can be used by common code.
"

But here comes the problem:
How can we set the NR_MEM_BANKS maximum value, 128 seems an arbitrary too? If #define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * N)? And what N should be?

Quoted 4 lines
Quoted 2 lines

And maybe the definition could be common with x86 if we define NR_MEM_BANKS to 128 on x86 too.

Julien had comment here, I will continue in that email.