Skip to contentSkip to conversation

Add device tree based NUMA support to Arm64

RFC PATCH In review Wei Chen 5 participants 196 messages Aug 11, 2021 – Sep 9, 2021 Page 1 of 2

lore
Share & tools
lore

Wei Chen

author
#0 lore raw

cover Add device tree based NUMA support to Arm64

Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.

Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.

So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.

Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.

So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.

Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common

 tools/libs/util/libxlu_pci.c    |   3 +-
 xen/arch/arm/Kconfig            |  10 +
 xen/arch/arm/Makefile           |   2 +
 xen/arch/arm/arm64/head.S       |   9 +-
 xen/arch/arm/bootfdt.c          |   8 +-
 xen/arch/arm/domain_build.c     |  17 +-
 xen/arch/arm/efi/efi-boot.h     |  25 --
 xen/arch/arm/numa.c             | 162 +++++++++
 xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
 xen/arch/arm/platform.c         |   4 +-
 xen/arch/arm/setup.c            |  14 +
 xen/arch/arm/smpboot.c          |  37 +-
 xen/arch/x86/Kconfig            |   2 +-
 xen/arch/x86/numa.c             | 421 +----------------------
 xen/arch/x86/srat.c             | 147 +-------
 xen/common/Kconfig              |   3 +
 xen/common/Makefile             |   1 +
 xen/common/libfdt/fdt_ro.c      |  15 +
 xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
 xen/common/page_alloc.c         |   2 +-
 xen/drivers/acpi/Kconfig        |   3 +-
 xen/drivers/acpi/Makefile       |   2 +-
 xen/include/asm-arm/numa.h      |  33 ++
 xen/include/asm-arm/setup.h     |   6 +
 xen/include/asm-x86/acpi.h      |   4 -
 xen/include/asm-x86/config.h    |   1 -
 xen/include/asm-x86/numa.h      |  65 +---
 xen/include/asm-x86/setup.h     |   1 -
 xen/include/xen/libfdt/libfdt.h |  25 ++
 xen/include/xen/nodemask.h      |   2 +
 xen/include/xen/numa.h          |  80 +++++
 31 files changed, 1325 insertions(+), 659 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
author
#1 lore raw

RFC PATCH 1/40 tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf

| libxlu_pci.c: In function 'xlu_pci_parse_bdf':
| libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    32 |     pcidev->func = func;
|       |     ~~~~~~~~~~~~~^~~~~~
| libxlu_pci.c:51:29: note: 'func' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                             ^~~~
| libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    31 |     pcidev->dev = dev;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:24: note: 'dev' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                        ^~~
| libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    30 |     pcidev->bus = bus;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:19: note: 'bus' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                   ^~~
| libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    78 |                 if ( dom & ~0xff )
|       |                      ~~~~^~~~~~~
---
 tools/libs/util/libxlu_pci.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
tools/libs/util/libxlu_pci.c +1 -2
index 551d8e3aed..b38e9aab40 100644
@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
 {
     const char *ptr = str;
     unsigned int colons = 0;
-    unsigned int domain, bus, dev, func;
+    unsigned int domain = 0, bus = 0, dev = 0, func = 0;
     int n;
 
     /* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
     ptr = str;
     switch (colons) {
     case 1:
-        domain = 0;
         if (sscanf(ptr, "%x:%x.%n", &bus, &dev, &n) != 2)
             return ERROR_INVAL;
         break;
Signature · 1 line
2.25.1
#2 lore raw

Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf

Quoted 4 lines · Wei Chen
| libxlu_pci.c: In function 'xlu_pci_parse_bdf':
| libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    32 |     pcidev->func = func;
|       |     ~~~~~~~~~~~~~^~~~~~

I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is

    pci->domain = domain;
    pci->bus = bus;
    pci->dev = dev;
    pci->func = func;
Quoted 10 lines
| libxlu_pci.c:51:29: note: 'func' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                             ^~~~
| libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    31 |     pcidev->dev = dev;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:24: note: 'dev' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                        ^~~
| libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized]
Quoted 8 lines
|    30 |     pcidev->bus = bus;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:19: note: 'bus' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                   ^~~
| libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized]
|    78 |                 if ( dom & ~0xff )
|       |                      ~~~~^~~~~~~

I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar if(). May I ask what code base these were observed with? Is the change needed at all anymore?

Quoted 13 lines
tools/libs/util/libxlu_pci.c +1 -1
@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
 {
     const char *ptr = str;
     unsigned int colons = 0;
-    unsigned int domain, bus, dev, func;
+    unsigned int domain = 0, bus = 0, dev = 0, func = 0;
     int n;
 
     /* Count occurrences of ':' to detrmine presence/absence of the 'domain' */

tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33

Quoted 7 lines
tools/libs/util/libxlu_pci.c +0 -1
@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
     ptr = str;
     switch (colons) {
     case 1:
-        domain = 0;
         if (sscanf(ptr, "%x:%x.%n", &bus, &dev, &n) != 2)
             return ERROR_INVAL;
         break;

Also - which compiler did you encounter this with?

Finally please don't forget to Cc maintainers.

Jan

author
#3 lore raw

RE: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf

Hi Jan,

Quoted 16 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:50
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf

Quoted 2 lines · Wei Chen

| libxlu_pci.c: In function 'xlu_pci_parse_bdf':
| libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this

function [-Werror=maybe-uninitialized]

Quoted 2 lines
|    32 |     pcidev->func = func;
|       |     ~~~~~~~~~~~~~^~~~~~
Quoted 8 lines

I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is

    pci->domain = domain;
    pci->bus = bus;
    pci->dev = dev;
    pci->func = func;

Sorry, I forgot to update my commit log with the latest code base. I revert this change in my current code, I can't reproduce it. I'm not sure if it's because I upgraded my build environment. Give me sometime, if I can reproduce it I will update the commit log in next version. If it's no longer needed, I will remove this patch from this series.

Quoted 18 lines
Quoted 4 lines
| libxlu_pci.c:51:29: note: 'func' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                             ^~~~
| libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this

function [-Werror=maybe-uninitialized]

Quoted 6 lines
|    31 |     pcidev->dev = dev;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:24: note: 'dev' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                        ^~~
| libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this

function [-Werror=maybe-uninitialized]

Quoted 6 lines
|    30 |     pcidev->bus = bus;
|       |     ~~~~~~~~~~~~^~~~~
| libxlu_pci.c:51:19: note: 'bus' was declared here
|    51 |     unsigned dom, bus, dev, func, vslot = 0;
|       |                   ^~~
| libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this
Quoted 8 lines

function [-Werror=maybe-uninitialized]

Quoted 2 lines
|    78 |                 if ( dom & ~0xff )
|       |                      ~~~~^~~~~~~

I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar if(). May I ask what code base these were observed with? Is the change needed at all anymore?

same as above.

Quoted 24 lines

tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char

Quoted 3 lines
tools/libs/util/libxlu_pci.c +0 -0
@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char

*str, const char **endp)

tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)

Quoted 8 lines
tools/libs/util/libxlu_pci.c +1 -1
 {
     const char *ptr = str;
     unsigned int colons = 0;
-    unsigned int domain, bus, dev, func;
+    unsigned int domain = 0, bus = 0, dev = 0, func = 0;
     int n;
 
     /* Count occurrences of ':' to detrmine presence/absence of the

'domain' */

tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char

Quoted 1 line
tools/libs/util/libxlu_pci.c +0 -0
@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char

*str, const char **endp)

tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33

Quoted 8 lines
tools/libs/util/libxlu_pci.c +0 -1
     ptr = str;
     switch (colons) {
     case 1:
-        domain = 0;
         if (sscanf(ptr, "%x:%x.%n", &bus, &dev, &n) != 2)
             return ERROR_INVAL;
         break;
Quoted 4 lines

Also - which compiler did you encounter this with?

Finally please don't forget to Cc maintainers.

If this patch still needed, I will do in next version.

Quoted 1 line

Jan

author
#4 lore raw

RFC PATCH 2/40 xen/arm: Print a 64-bit number in hex from early uart

In reply to Wei Chen

Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -

In this patch, we increased the print loops and shift bits to make putn print 64-bit number.

---
 xen/arch/arm/arm64/head.S | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +5 -4
index aa1f88c764..b32639d7d6 100644
@@ -862,17 +862,18 @@ puts:
         ret
 ENDPROC(puts)
 
-/* Print a 32-bit number in hex.  Specific to the PL011 UART.
+/* Print a 64-bit number in hex.  Specific to the PL011 UART.
  * x0: Number to print.
  * x23: Early UART base address
  * Clobbers x0-x3 */
+#define PRINT_MASK 0xf000000000000000
 putn:
         adr   x1, hex
-        mov   x3, #8
+        mov   x3, #16
 1:
         early_uart_ready x23, 2
-        and   x2, x0, #0xf0000000    /* Mask off the top nybble */
-        lsr   x2, x2, #28
+        and   x2, x0, #PRINT_MASK    /* Mask off the top nybble */
+        lsr   x2, x2, #60
         ldrb  w2, [x1, x2]           /* Convert to a char */
         early_uart_transmit x23, w2
         lsl   x0, x0, #4             /* Roll it through one nybble at a time */
Signature · 1 line
2.25.1
Acked-by on v1 2/40
#5 lore raw

Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart

Hi Wei,

Quoted 10 lines · Wei Chen

Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -

Quoted 4 lines

In this patch, we increased the print loops and shift bits to make putn print 64-bit number.

Quoted 9 lines
---
  xen/arch/arm/arm64/head.S | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +0 -0
index aa1f88c764..b32639d7d6 100644

xen/arch/arm/arm64/head.S puts: · line 865

Quoted 5 lines
xen/arch/arm/arm64/head.S +1 -1
@@ -862,17 +862,18 @@ puts:
          ret
  ENDPROC(puts)
 
-/* Print a 32-bit number in hex.  Specific to the PL011 UART.
+/* Print a 64-bit number in hex.  Specific to the PL011 UART.

As you modify the line, can you take the opportunity to write:

/*
* Print a 64-bit...

And also drop the second sentence as it the code has not been PL011 specific for quite a while now.

Quoted 8 lines
xen/arch/arm/arm64/head.S +1 -1
   * x0: Number to print.
   * x23: Early UART base address
   * Clobbers x0-x3 */
+#define PRINT_MASK 0xf000000000000000
  putn:
          adr   x1, hex
-        mov   x3, #8

xen/arch/arm/arm64/head.S puts: · line 879

Quoted 10 lines
xen/arch/arm/arm64/head.S +3 -2
+        mov   x3, #16
  1:
          early_uart_ready x23, 2
-        and   x2, x0, #0xf0000000    /* Mask off the top nybble */
-        lsr   x2, x2, #28
+        and   x2, x0, #PRINT_MASK    /* Mask off the top nybble */
+        lsr   x2, x2, #60
          ldrb  w2, [x1, x2]           /* Convert to a char */
          early_uart_transmit x23, w2
          lsl   x0, x0, #4             /* Roll it through one nybble at a time */

Cheers,

Signature · 1 line
Julien Grall
author
#6 lore raw

RE: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart

Hi Julien,

Quoted 46 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:05
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart

Hi Wei,

Quoted 10 lines · Wei Chen

Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -

Quoted 4 lines

In this patch, we increased the print loops and shift bits to make putn print 64-bit number.

Quoted 9 lines
---
  xen/arch/arm/arm64/head.S | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +0 -0
index aa1f88c764..b32639d7d6 100644

xen/arch/arm/arm64/head.S puts: · line 865

Quoted 5 lines
xen/arch/arm/arm64/head.S +1 -1
@@ -862,17 +862,18 @@ puts:
          ret
  ENDPROC(puts)
 
-/* Print a 32-bit number in hex.  Specific to the PL011 UART.
+/* Print a 64-bit number in hex.  Specific to the PL011 UART.

As you modify the line, can you take the opportunity to write:

Quoted 6 lines

/*
* Print a 64-bit...

And also drop the second sentence as it the code has not been PL011 specific for quite a while now.

Ok, I will do it in next version

Quoted 24 lines
Quoted 7 lines
xen/arch/arm/arm64/head.S +1 -1
   * x0: Number to print.
   * x23: Early UART base address
   * Clobbers x0-x3 */
+#define PRINT_MASK 0xf000000000000000
  putn:
          adr   x1, hex
-        mov   x3, #8

xen/arch/arm/arm64/head.S puts:

Quoted 10 lines
xen/arch/arm/arm64/head.S +3 -2
+        mov   x3, #16
  1:
          early_uart_ready x23, 2
-        and   x2, x0, #0xf0000000    /* Mask off the top nybble */
-        lsr   x2, x2, #28
+        and   x2, x0, #PRINT_MASK    /* Mask off the top nybble */
+        lsr   x2, x2, #60
          ldrb  w2, [x1, x2]           /* Convert to a char */
          early_uart_transmit x23, w2
          lsl   x0, x0, #4             /* Roll it through one nybble at

a time */

Quoted 1 line

Cheers,

--
Julien Grall

author
#7 lore raw

RFC PATCH 3/40 xen/x86: Initialize memnodemapsize while faking NUMA node

In reply to Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of _memnodemap. Xen hadn't done it, and Xen should assert in phys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.

In this patch, we set memnodemapsize to ARRAY_SIZE(_memnodemap) to fix it.

---
 xen/arch/x86/numa.c | 2 ++
 1 file changed, 2 insertions(+)
xen/arch/x86/numa.c +2 -0
index f1066c59c7..d23f4f7919 100644
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);
+
     nodes_clear(node_online_map);
     node_set_online(0);
     for ( i = 0; i < nr_cpu_ids; i++ )
Signature · 1 line
2.25.1
#8 lore raw

Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 10 lines · Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of _memnodemap. Xen hadn't done it, and Xen should assert in phys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.

How about we promote VIRTUAL_BUG_ON() to expand to at least ASSERT()?

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273

Quoted 7 lines
xen/arch/x86/numa.c +1 -0
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);

But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.

Jan

author
#9 lore raw

RE: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node

Hi Jan,

Quoted 14 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 4 lines · Wei Chen

When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.

Quoted 9 lines
Quoted 6 lines

In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of _memnodemap. Xen hadn't done it, and Xen should assert in phys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.

How about we promote VIRTUAL_BUG_ON() to expand to at least ASSERT()?

That would be good. Frankly, we discovered this because we used ASSERT in Arm and then noticed that x86 was using VIRTUAL_BUG_ON.

Quoted 4 lines

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long

Quoted 3 lines
xen/arch/x86/numa.c +0 -0
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 8 lines

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273

Quoted 4 lines
xen/arch/x86/numa.c +1 -0
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);

But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.

Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in acpi_scan_nodes, but acpi_scan_nodes doesn't reset memnodemap when it failed. I think maybe we can add:

    memnodemap[0] = 0;
    memnodemapsize = 1;

How do you think about it?

Quoted 1 line

Jan

#10 lore raw

Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node

Quoted 17 lines · Wei Chen
Quoted 8 lines

From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long

Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0 -0
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long

start_pfn, unsigned long end_pfn)

Quoted 8 lines

xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273

Quoted 4 lines
xen/arch/x86/numa.c +1 -0
     /* setup dummy node covering all memory */
     memnode_shift = BITS_PER_LONG - 1;
     memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);

But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.

Quoted 6 lines

Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in acpi_scan_nodes, but acpi_scan_nodes doesn't reset memnodemap when it failed. I think maybe we can add:

    memnodemap[0] = 0;
    memnodemapsize = 1;

How do you think about it?

Well, yes, if data may have been put there, then resetting of course makes sense.

Jan

author
#11 lore raw

RFC PATCH 4/40 xen/arm: return default DMA bit width when platform is not set

In reply to Wei Chen

From: Hongda Deng <[email protected]>

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm, we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function. But in NUMA system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware. Only platform setting can override dma_bitsize. So in this patch, we return default dma_bitsize, when platform and platorm->dma_bitsize are not set.

---
 xen/arch/arm/platform.c | 4 +++-
 xen/common/page_alloc.c | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
xen/arch/arm/platform.c +3 -1
index 4db5bbb4c5..0a27fef9a4 100644
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];
 /* Pointer to the current platform description */
 static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;
 
 static bool __init platform_is_compatible(const struct platform_desc *plat)
 {
@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)
 
 unsigned int arch_get_dma_bitsize(void)
 {
-    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize
+                                                 : dma_bitsize;
 }
 
 /*
xen/common/page_alloc.c +1 -1
index 958ba0cd92..0f0cae5a4e 100644
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
  * Bit width of the DMA heap -- used to override NUMA-node-first.
  * allocation strategy, which can otherwise exhaust low memory.
  */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;
 integer_param("dma_bits", dma_bitsize);
 
 /* Offlined page list, protected by heap_lock. */
Signature · 1 line
2.25.1
#12 lore raw

Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30

Quoted 7 lines · Wei Chen
xen/arch/arm/platform.c +1 -0
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];
 /* Pointer to the current platform description */
 static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;

This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...

xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230

Quoted 8 lines
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
  * Bit width of the DMA heap -- used to override NUMA-node-first.
  * allocation strategy, which can otherwise exhaust low memory.
  */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;

... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.

Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.

Jan

author
#13 lore raw

RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi Jan,

Quoted 14 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

xen/arch/arm/platform.c extern const struct platform_desc _splatform[],

Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0 -0
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],

_eplatform[];

Quoted 7 lines

xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30

Quoted 4 lines
xen/arch/arm/platform.c +1 -0
 /* Pointer to the current platform description */
 static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;

This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...

Ok, I will place it to a header.

Quoted 13 lines

xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230

Quoted 9 lines
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
  * Bit width of the DMA heap -- used to override NUMA-node-first.
  * allocation strategy, which can otherwise exhaust low memory.
  */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;

... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.

Sorry, I am not very clear about this comment.

Quoted 4 lines

Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.

Different Arm platforms may have different DMA bitsize. So in my previous thought, If Arm platform doesn't provide any DMA bitsize info, I will return the system DMA bitsize (dma_bitsize) in arch_get_dma_bitsize. But your comment made me think again. My current change may have fallen into a logical loophole. dma_bitsize as a high level variable, it's value depends on xen boot command line or arch_get_dma_bitsize. I can't use it for arch_get_dma_bitsize's input.

So I think, in next version, I will discard the changes of dma_bitsize, just keep the changes in arch_get_dma_bitsize to return 0, when platform hasn't specify the DMA bitsize. Just like:

unsigned int arch_get_dma_bitsize(void)
{
- return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+ return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize
+ : 0;
}

Thanks,
Wei Chen

Quoted 1 line

Jan

#14 lore raw

Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 32 lines · Wei Chen
Quoted 8 lines

From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54

xen/arch/arm/platform.c extern const struct platform_desc _splatform[],

Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0 -0
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],

_eplatform[];

Quoted 7 lines

xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30

Quoted 4 lines
xen/arch/arm/platform.c +1 -0
 /* Pointer to the current platform description */
 static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;

This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...

Ok, I will place it to a header.

Quoted 13 lines

xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230

Quoted 9 lines
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
  * Bit width of the DMA heap -- used to override NUMA-node-first.
  * allocation strategy, which can otherwise exhaust low memory.
  */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;

... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.

Quoted 1 line

Sorry, I am not very clear about this comment.

I've merely been trying to explain _why_ the declaration needs to be in a header.

Jan

#15 lore raw

Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

In reply to Wei Chen

Hi,

Quoted 4 lines · Wei Chen

From: Hongda Deng <[email protected]>

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.

Quoted 5 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.

What do you mean by "hardware design"? Are you referring to the server you boot Xen on?

Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).

So I would be cautious to change the default limit.

At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).

So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.

Quoted 40 lines

Only
platform setting can override dma_bitsize. So in this patch, we return default dma_bitsize, when platform and platorm->dma_bitsize are not set.

---
  xen/arch/arm/platform.c | 4 +++-
  xen/common/page_alloc.c | 2 +-
  2 files changed, 4 insertions(+), 2 deletions(-)
xen/arch/arm/platform.c +3 -1
index 4db5bbb4c5..0a27fef9a4 100644
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];
  /* Pointer to the current platform description */
  static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;
 
  static bool __init platform_is_compatible(const struct platform_desc *plat)
  {
@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)
 
  unsigned int arch_get_dma_bitsize(void)
  {
-    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize
+                                                 : dma_bitsize;
  }
 
  /*
xen/common/page_alloc.c +0 -0
index 958ba0cd92..0f0cae5a4e 100644

xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233

Quoted 8 lines
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
   * Bit width of the DMA heap -- used to override NUMA-node-first.
   * allocation strategy, which can otherwise exhaust low memory.
   */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;
  integer_param("dma_bits", dma_bitsize);
 
  /* Offlined page list, protected by heap_lock. */

Cheers,

Signature · 1 line
Julien Grall
author
#16 lore raw

RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi Julien,

Quoted 11 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi,

Quoted 8 lines
Quoted 4 lines · Wei Chen

From: Hongda Deng <[email protected]>

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Ok

Quoted 7 lines
Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.

Quoted 9 lines
Quoted 5 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.

What do you mean by "hardware design"? Are you referring to the server you boot Xen on?

Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?

Quoted 5 lines

Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).

So I would be cautious to change the default limit.

How about return 0 when platform doesn't specify the limit?
In my opinion, arbitrary to give 32 on AArch64 doesn't make sense. But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from platform->dma_bitsize. In this case, return 0 will also cause some issue.

Quoted 3 lines

At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).

For Dom0, we squash the patch#5 into this patch?

Quoted 3 lines

So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.

Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...

      if ( !dma_bitsize && (num_online_nodes() > 1) )
          dma_bitsize = arch_get_dma_bitsize();
Quoted 58 lines

xen/arch/arm/platform.c extern const struct platform_desc _splatform[],

Quoted 17 lines

Only
platform setting can override dma_bitsize. So in this patch, we return default dma_bitsize, when platform and platorm->dma_bitsize are not set.

---
  xen/arch/arm/platform.c | 4 +++-
  xen/common/page_alloc.c | 2 +-
  2 files changed, 4 insertions(+), 2 deletions(-)
xen/arch/arm/platform.c +0 -0
index 4db5bbb4c5..0a27fef9a4 100644
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],

_eplatform[];

xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[];

Quoted 6 lines
xen/arch/arm/platform.c +1 -0
  /* Pointer to the current platform description */
  static const struct platform_desc *platform;
 
+extern unsigned int dma_bitsize;
 
  static bool __init platform_is_compatible(const struct platform_desc

*plat)

xen/arch/arm/platform.c bool platform_device_is_blacklisted(const struct

Quoted 2 lines
xen/arch/arm/platform.c +0 -0
  {
@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct

dt_device_node *node)

Quoted 17 lines
xen/arch/arm/platform.c +1 -1
  unsigned int arch_get_dma_bitsize(void)
  {
-    return ( platform && platform->dma_bitsize ) ? platform-
 dma_bitsize : 32;
+    return ( platform && platform->dma_bitsize ) ? platform-

dma_bitsize

xen/arch/arm/platform.c +1 -0
+                                                 : dma_bitsize;
  }
 
  /*
xen/common/page_alloc.c +0 -0
index 958ba0cd92..0f0cae5a4e 100644

xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233

Quoted 8 lines
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;
   * Bit width of the DMA heap -- used to override NUMA-node-first.
   * allocation strategy, which can otherwise exhaust low memory.
   */
-static unsigned int dma_bitsize;
+unsigned int dma_bitsize;
  integer_param("dma_bits", dma_bitsize);
 
  /* Offlined page list, protected by heap_lock. */

Cheers,

--
Julien Grall

#17 lore raw

Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 35 lines
Quoted 11 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi,

Quoted 8 lines
Quoted 4 lines · Wei Chen

From: Hongda Deng <[email protected]>

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Ok

Quoted 13 lines
Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.

Quoted 5 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.

Quoted 7 lines
Quoted 3 lines

What do you mean by "hardware design"? Are you referring to the server you boot Xen on?

Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?

The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?

Quoted 10 lines
Quoted 7 lines

Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).

So I would be cautious to change the default limit.

How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.

We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.

Quoted 10 lines

But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from platform->dma_bitsize. In this case, return 0 will also cause some issue.

Quoted 3 lines

At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).

For Dom0, we squash the patch#5 into this patch?

Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...

Quoted 10 lines
Quoted 4 lines

So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.

Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...

       if ( !dma_bitsize && (num_online_nodes() > 1) )
           dma_bitsize = arch_get_dma_bitsize();

... Why do you think we need this check on Arm when NUMA is enabled?

We can discuss how to remove it once this is answered.

Cheers,

Signature · 1 line
Julien Grall
author
#18 lore raw

RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi Julien,

Quoted 59 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit

width

Quoted 7 lines
Quoted 7 lines

when platform is not set

Hi,

Quoted 2 lines · Wei Chen

From: Hongda Deng <[email protected]>

Quoted 14 lines
Quoted 5 lines
Quoted 2 lines

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Ok

Quoted 6 lines
Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every

devices.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.

Quoted 6 lines
Quoted 3 lines

What do you mean by "hardware design"? Are you referring to the server you boot Xen on?

Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may

Quoted 8 lines

not

Quoted 1 line

have kind of restriction on hardware." ?

The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?

OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )
Anyway, keep the arch_get_dma_bitsize as original maybe the best approach for now.

Quoted 22 lines
Quoted 10 lines
Quoted 7 lines

Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).

So I would be cautious to change the default limit.

How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.

We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.

Quoted 1 line

But as you mentioned, if Xen is running on a platform with DMA

limitation,

Quoted 6 lines

but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from platform->dma_bitsize. In this case, return 0 will also cause some issue.

Quoted 2 lines

At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for

Quoted 7 lines

now).

Quoted 3 lines
Quoted 1 line

For Dom0, we squash the patch#5 into this patch?

Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...

As I answered above, let's keep arch_get_dma_bitsize as original then we don't need to modify Arm code.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.

Quoted 8 lines
Quoted 5 lines

Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...

       if ( !dma_bitsize && (num_online_nodes() > 1) )
           dma_bitsize = arch_get_dma_bitsize();

... Why do you think we need this check on Arm when NUMA is enabled?

I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.

Quoted 2 lines

We can discuss how to remove it once this is answered.

I think we can start to discuss it.

Quoted 4 lines

Cheers,

--
Julien Grall

#19 lore raw

Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 68 lines
Quoted 59 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit

width

Quoted 7 lines
Quoted 7 lines

when platform is not set

Hi,

Quoted 2 lines · Wei Chen

From: Hongda Deng <[email protected]>

Quoted 14 lines
Quoted 5 lines
Quoted 2 lines

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Ok

Quoted 6 lines
Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every

devices.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.

Quoted 6 lines
Quoted 3 lines

What do you mean by "hardware design"? Are you referring to the server you boot Xen on?

Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may

Quoted 8 lines

not

Quoted 1 line

have kind of restriction on hardware." ?

The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?

Quoted 3 lines

OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )

I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.

My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).

If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.

Quoted 8 lines
Quoted 5 lines

Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...

        if ( !dma_bitsize && (num_online_nodes() > 1) )
            dma_bitsize = arch_get_dma_bitsize();

... Why do you think we need this check on Arm when NUMA is enabled?

Quoted 7 lines

I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.

Quoted 2 lines

We can discuss how to remove it once this is answered.

I think we can start to discuss it.

How about replacing the second part of the check with a new helper arch_have_default_dma_zone() (or a different name)?

Cheers,

Signature · 1 line
Julien Grall
author
#20 lore raw

RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Hi Julien,

Quoted 120 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:18
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit

width

Quoted 30 lines
Quoted 30 lines

when platform is not set

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit

width

Quoted 13 lines
Quoted 12 lines

when platform is not set

Hi,

Quoted 4 lines · Wei Chen

From: Hongda Deng <[email protected]>

In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm,

s/resonable/reasonable/

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Ok

Quoted 5 lines
Quoted 3 lines

we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function.

arch_get_dma_bitsize() is also used to allocate dom0 memory. We need

to

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line

be able to allocate some DMA-able memory that can be used by every

devices.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our

hardware

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

design. We don't have that kind of restriction on hardware.

What do you mean by "hardware design"? Are you referring to the

server

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

you boot Xen on?

Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may

not

Quoted 1 line

have kind of restriction on hardware." ?

The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged).

What

Quoted 5 lines
Quoted 3 lines

you seem to suggest is no-one will ever plug such card on your platform. Is that correct?

OK, I understand now. Let's keep 32-bit as default value, but even in

this

Quoted 1 line

case, how about DMA-16 devices? Although these devices are very rare,

they

Quoted 1 line

still exist : )

I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.

My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).

If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

So I would explore to remove the NUMA check for drop the DMA zone.

FAOD,

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

both suggestion are for Arm only. For x86, they need to be kept.

Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...

        if ( !dma_bitsize && (num_online_nodes() > 1) )
            dma_bitsize = arch_get_dma_bitsize();
Quoted 5 lines
Quoted 2 lines

... Why do you think we need this check on Arm when NUMA is enabled?

I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this

check.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

We can discuss how to remove it once this is answered.

I think we can start to discuss it.

How about replacing the second part of the check with a new helper arch_have_default_dma_zone() (or a different name)?

This seems like a method worth trying, I will try to follow this way to compose next version.

Quoted 5 lines

Cheers,

--
Julien Grall

author
#21 lore raw

RFC PATCH 5/40 xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

In reply to Wei Chen

From: Hongda Deng <[email protected]>

In previous patch, we make arch_get_dma_bitsize return 0 when dma_bitsize and platform->dma_bitsize are not set. But this will affect lowmem_bitsize in allocate_memory_11 for domain0. Because this function depends lowmem_bitsize to allocate memory below 4GB.

In current code, when arch_get_dma_bitsize return 0, lowmem_bitsize will be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.

This behavior is inconsistent with what allocate_memory_11 done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before set lowmem_bitsize. Avoid setting lowmem_bitsize to 0 by mistake.

---
 xen/arch/arm/domain_build.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +10 -1
index 6c86d52781..cf341f349f 100644
@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,
     int i;
 
     bool lowmem = true;
-    unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());
+    unsigned int lowmem_bitsize = arch_get_dma_bitsize();
     unsigned int bits;
 
+    /*
+       When dma_bitsize and platform->dma_bitsize are not set,
+       arch_get_dma_bitsize will return 0. That means this system
+       doesn't need to reserve memory for DMA. But in order to
+       meet above requirements, we still need to try to allocate
+       memory below 4GB for Dom0.
+    */
+    lowmem_bitsize = lowmem_bitsize ? min(32U, lowmem_bitsize) : 32U;
+
     /*
      * TODO: Implement memory bank allocation when DOM0 is not direct
      * mapped
Signature · 1 line
2.25.1
#22 lore raw

Re: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Hi,

I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.

Quoted 13 lines · Wei Chen

From: Hongda Deng <[email protected]>

In previous patch, we make arch_get_dma_bitsize return 0 when dma_bitsize and platform->dma_bitsize are not set. But this will affect lowmem_bitsize in allocate_memory_11 for domain0. Because this function depends lowmem_bitsize to allocate memory below 4GB.

In current code, when arch_get_dma_bitsize return 0, lowmem_bitsize will be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.

Quoted 4 lines

This behavior is inconsistent with what allocate_memory_11 done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before set lowmem_bitsize. Avoid setting lowmem_bitsize to 0 by mistake.

In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.

Quoted 16 lines
---
  xen/arch/arm/domain_build.c | 11 ++++++++++-
  1 file changed, 10 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +0 -1
index 6c86d52781..cf341f349f 100644
@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,
      int i;
 
      bool lowmem = true;
-    unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());

xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277

Quoted 10 lines
xen/arch/arm/domain_build.c +8 -0
+    unsigned int lowmem_bitsize = arch_get_dma_bitsize();
      unsigned int bits;
 
+    /*
+       When dma_bitsize and platform->dma_bitsize are not set,
+       arch_get_dma_bitsize will return 0. That means this system
+       doesn't need to reserve memory for DMA. But in order to
+       meet above requirements, we still need to try to allocate
+       memory below 4GB for Dom0.
+    */

The coding style for comments is:

/*
* A
* B
*/

xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282

Quoted 6 lines
xen/arch/arm/domain_build.c +2 -0
+    lowmem_bitsize = lowmem_bitsize ? min(32U, lowmem_bitsize) : 32U;
+
      /*
       * TODO: Implement memory bank allocation when DOM0 is not direct
       * mapped

Cheers,

Signature · 1 line
Julien Grall
author
#23 lore raw

RE: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Hi Julien,

Quoted 9 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:32
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Quoted 5 lines

Hi,

I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.

Ok

Quoted 50 lines · Wei Chen
Quoted 13 lines

From: Hongda Deng <[email protected]>

In previous patch, we make arch_get_dma_bitsize return 0 when dma_bitsize and platform->dma_bitsize are not set. But this will affect lowmem_bitsize in allocate_memory_11 for domain0. Because this function depends lowmem_bitsize to allocate memory below 4GB.

In current code, when arch_get_dma_bitsize return 0, lowmem_bitsize will be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.

Quoted 4 lines

This behavior is inconsistent with what allocate_memory_11 done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before set lowmem_bitsize. Avoid setting lowmem_bitsize to 0 by mistake.

In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.

xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain

Quoted 12 lines
---
  xen/arch/arm/domain_build.c | 11 ++++++++++-
  1 file changed, 10 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +0 -0
index 6c86d52781..cf341f349f 100644
@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain

*d,

Quoted 4 lines
xen/arch/arm/domain_build.c +0 -1
      int i;
 
      bool lowmem = true;
-    unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());

xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277

Quoted 10 lines
xen/arch/arm/domain_build.c +8 -0
+    unsigned int lowmem_bitsize = arch_get_dma_bitsize();
      unsigned int bits;
 
+    /*
+       When dma_bitsize and platform->dma_bitsize are not set,
+       arch_get_dma_bitsize will return 0. That means this system
+       doesn't need to reserve memory for DMA. But in order to
+       meet above requirements, we still need to try to allocate
+       memory below 4GB for Dom0.
+    */
Quoted 7 lines

The coding style for comments is:

/*
* A
* B
*/

I will fix it.

Quoted 11 lines

xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282

Quoted 6 lines
xen/arch/arm/domain_build.c +2 -0
+    lowmem_bitsize = lowmem_bitsize ? min(32U, lowmem_bitsize) : 32U;
+
      /*
       * TODO: Implement memory bank allocation when DOM0 is not direct
       * mapped

Cheers,

--
Julien Grall

author
#24 lore raw

RFC PATCH 6/40 xen: decouple NUMA from ACPI in Kconfig

In reply to Wei Chen

In current Xen code only implments x86 ACPI-based NUMA support. So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.

In this patch series, we introduced device tree based NUMA for Arm. That means we will have two NUMA implemetations, so in this patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.

---
 xen/arch/x86/Kconfig         | 2 +-
 xen/common/Kconfig           | 3 +++
 xen/drivers/acpi/Kconfig     | 3 ++-
 xen/drivers/acpi/Makefile    | 2 +-
 xen/include/asm-x86/config.h | 1 -
 5 files changed, 7 insertions(+), 4 deletions(-)
xen/arch/x86/Kconfig +1 -1
index 9b164db641..7414aef113 100644
@@ -24,7 +24,7 @@ config X86
 	select HAS_UBSAN
 	select HAS_VPCI if HVM
 	select NEEDS_LIBELF
-	select NUMA
+	select ACPI_NUMA
 
 config ARCH_DEFCONFIG
 	string
xen/common/Kconfig +3 -0
index 0ddd18e11a..b1f1145613 100644
@@ -67,6 +67,9 @@ config MEM_ACCESS
 config NEEDS_LIBELF
 	bool
 
+config NUMA
+	bool
+
 menu "Speculative hardening"
 
 config SPECULATIVE_HARDEN_ARRAY
xen/drivers/acpi/Kconfig +2 -1
index b64d3731fb..e3f3d8f4b1 100644
@@ -5,5 +5,6 @@ config ACPI
 config ACPI_LEGACY_TABLES_LOOKUP
 	bool
 
-config NUMA
+config ACPI_NUMA
 	bool
+	select NUMA
xen/drivers/acpi/Makefile +1 -1
index 4f8e97228e..2fc5230253 100644
@@ -3,7 +3,7 @@ obj-y += utilities/
 obj-$(CONFIG_X86) += apei/
 
 obj-bin-y += tables.init.o
-obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_ACPI_NUMA) += numa.o
 obj-y += osl.o
 obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
 
xen/include/asm-x86/config.h +0 -1
index 883c2ef0df..9a6f0a6edf 100644
@@ -31,7 +31,6 @@
 /* Intel P4 currently has largest cache line (L2 line size is 128 bytes). */
 #define CONFIG_X86_L1_CACHE_SHIFT 7
 
-#define CONFIG_ACPI_NUMA 1
 #define CONFIG_ACPI_SRAT 1
 #define CONFIG_ACPI_CSTATE 1
 
Signature · 1 line
2.25.1
Reviewed-by on v1 6/40
#25 lore raw

Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

xen/arch/x86/Kconfig config X86 · line 27

Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1 -1
@@ -24,7 +24,7 @@ config X86
 	select HAS_UBSAN
 	select HAS_VPCI if HVM
 	select NEEDS_LIBELF
-	select NUMA
+	select ACPI_NUMA

We try to keep this alphabetically sorted, so please move up the replacement line. Then

Jan

author
#26 lore raw

RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

Hi Jan,

Quoted 20 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:37
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

xen/arch/x86/Kconfig config X86 · line 27

Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1 -1
@@ -24,7 +24,7 @@ config X86
 	select HAS_UBSAN
 	select HAS_VPCI if HVM
 	select NEEDS_LIBELF
-	select NUMA
+	select ACPI_NUMA

We try to keep this alphabetically sorted, so please move up the replacement line. Then

Quoted 2 lines

Thanks, I will do it in next version.

Quoted 1 line

Jan

#27 lore raw

Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

In reply to Wei Chen

Hi Wei,

Quoted 1 line · Wei Chen

In current Xen code only implments x86 ACPI-based NUMA support.

s/implments/implements/

Quoted 5 lines

So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.

In this patch series, we introduced device tree based NUMA for Arm.

The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:

"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".

Quoted 1 line

That means we will have two NUMA implemetations, so in this

s/implemetations/implementations/

Quoted 2 lines

patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.

Cheers,

Signature · 1 line
Julien Grall
author
#28 lore raw

RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

Hi Julien,

Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月13日 0:55
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig

Quoted 6 lines

Hi Wei,

Quoted 1 line · Wei Chen

In current Xen code only implments x86 ACPI-based NUMA support.

s/implments/implements/

Got it.

Quoted 10 lines
Quoted 6 lines

So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.

In this patch series, we introduced device tree based NUMA for Arm.

The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:

Quoted 7 lines

"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".

Quoted 1 line

That means we will have two NUMA implemetations, so in this

s/implemetations/implementations/

Thanks, I will update the commit log in next version.

Quoted 7 lines
Quoted 2 lines

patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.

Cheers,

--
Julien Grall

author
#29 lore raw

RFC PATCH 7/40 xen/arm: use !CONFIG_NUMA to keep fake NUMA API

In reply to Wei Chen

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32. Even in Arm64, users still can disable the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
 xen/include/asm-arm/numa.h | 4 ++++
 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
index 31a6de4e23..ab9c4a2448 100644
@@ -5,6 +5,8 @@
 
 typedef u8 nodeid_t;
 
+#if !defined(CONFIG_NUMA)
+
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
 #define node_to_cpumask(node)   (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#endif
+
 #endif /* __ARCH_ARM_NUMA_H */
 /*
  * Local variables:
Signature · 1 line
2.25.1
#30 lore raw

Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Hi Wei,

Quoted 2 lines · Wei Chen

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

What do you mean by "could not be enabled"?

Quoted 15 lines

Even in Arm64, users still can disable
the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
  xen/include/asm-arm/numa.h | 4 ++++
  1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +0 -0
index 31a6de4e23..ab9c4a2448 100644

xen/include/asm-arm/numa.h line 8

Quoted 4 lines
xen/include/asm-arm/numa.h +1 -0
@@ -5,6 +5,8 @@
 
  typedef u8 nodeid_t;
 
+#if !defined(CONFIG_NUMA)

NIT: We tend to use #ifndef rather than #if !defined(...)

Quoted 6 lines
xen/include/asm-arm/numa.h +1 -0
+
  /* Fake one node for now. See also node_online_map. */
  #define cpu_to_node(cpu) 0
  #define node_to_cpumask(node)   (cpu_online_map)

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34

Quoted 8 lines
xen/include/asm-arm/numa.h +2 -0
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
  #define __node_distance(a, b) (20)
 
+#endif
+
  #endif /* __ARCH_ARM_NUMA_H */
  /*
   * Local variables:

Cheers,

Signature · 1 line
Julien Grall
author
#31 lore raw

RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Hi Julien,

Quoted 9 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 7 lines

Hi Wei,

Quoted 2 lines · Wei Chen

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA. In this case, this Kconfig option could not be enabled on Arm32.

Quoted 17 lines
Quoted 17 lines

Even in Arm64, users still can disable
the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
  xen/include/asm-arm/numa.h | 4 ++++
  1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +0 -0
index 31a6de4e23..ab9c4a2448 100644
@@ -5,6 +5,8 @@
 
Quoted 6 lines

xen/include/asm-arm/numa.h line 8

Quoted 3 lines
xen/include/asm-arm/numa.h +1 -0
  typedef u8 nodeid_t;
 
+#if !defined(CONFIG_NUMA)

NIT: We tend to use #ifndef rather than #if !defined(...)

OK, I will change related changes in this series.

Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1 -0
+
  /* Fake one node for now. See also node_online_map. */
  #define cpu_to_node(cpu) 0
  #define node_to_cpumask(node)   (cpu_online_map)

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34

Quoted 8 lines
xen/include/asm-arm/numa.h +2 -0
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
  #define __node_distance(a, b) (20)
 
+#endif
+
  #endif /* __ARCH_ARM_NUMA_H */
  /*
   * Local variables:

Cheers,

--
Julien Grall

#32 lore raw

Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 12 lines
Quoted 12 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Hi Wei,

Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.

I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...

Quoted 2 lines

In this case, this Kconfig
option could not be enabled on Arm32.

... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.

To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Even in Arm64, users still can disable
the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
   xen/include/asm-arm/numa.h | 4 ++++
   1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +0 -0
index 31a6de4e23..ab9c4a2448 100644
@@ -5,6 +5,8 @@
 
Quoted 28 lines
Quoted 6 lines

xen/include/asm-arm/numa.h line 8

Quoted 3 lines
xen/include/asm-arm/numa.h +1 -0
   typedef u8 nodeid_t;
 
+#if !defined(CONFIG_NUMA)

NIT: We tend to use #ifndef rather than #if !defined(...)

OK, I will change related changes in this series.

Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1 -0
+
   /* Fake one node for now. See also node_online_map. */
   #define cpu_to_node(cpu) 0
   #define node_to_cpumask(node)   (cpu_online_map)

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34

Quoted 8 lines
xen/include/asm-arm/numa.h +2 -0
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
   #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
   #define __node_distance(a, b) (20)
 
+#endif
+
   #endif /* __ARCH_ARM_NUMA_H */
   /*
    * Local variables:

Cheers,

--
Julien Grall

Cheers,

Signature · 1 line
Julien Grall
author
#33 lore raw

RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Hi Julien,

Quoted 41 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 4 lines
Quoted 4 lines

NUMA API

Hi Wei,

Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.

I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...

Quoted 6 lines
Quoted 2 lines

In this case, this Kconfig
option could not be enabled on Arm32.

... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.

Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed. I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.

Quoted 4 lines

To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.

Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use? And how can we test this code? Before this patch, I had checked Linux, and found that OF_NUMA only selected by Arm64 not Arm32. But if you feel the need to add to arm32, I have no problem with that.

Quoted 51 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Even in Arm64, users still can disable
the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.

---
   xen/include/asm-arm/numa.h | 4 ++++
   1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +0 -0
index 31a6de4e23..ab9c4a2448 100644
@@ -5,6 +5,8 @@
 
Quoted 28 lines
Quoted 6 lines

xen/include/asm-arm/numa.h line 8

Quoted 3 lines
xen/include/asm-arm/numa.h +1 -0
   typedef u8 nodeid_t;
 
+#if !defined(CONFIG_NUMA)

NIT: We tend to use #ifndef rather than #if !defined(...)

OK, I will change related changes in this series.

Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1 -0
+
   /* Fake one node for now. See also node_online_map. */
   #define cpu_to_node(cpu) 0
   #define node_to_cpumask(node)   (cpu_online_map)

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34

Quoted 8 lines
xen/include/asm-arm/numa.h +2 -0
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;
   #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
   #define __node_distance(a, b) (20)
 
+#endif
+
   #endif /* __ARCH_ARM_NUMA_H */
   /*
    * Local variables:

Cheers,

--
Julien Grall

Cheers,

--
Julien Grall

#34 lore raw

Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 45 lines
Quoted 45 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 4 lines
Quoted 4 lines

NUMA API

Hi Wei,

Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.

I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...

Quoted 2 lines

In this case, this Kconfig
option could not be enabled on Arm32.

Quoted 7 lines
Quoted 3 lines

... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.

Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.

When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?

Quoted 11 lines

I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.

Quoted 4 lines

To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.

Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?

Technically you already added the code because arch/arm/ is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...

And how can we test this code?

I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.

Cheers,

Signature · 1 line
Julien Grall
author
#35 lore raw

RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Hi Julien,

Quoted 68 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 26 lines
Quoted 26 lines

NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 8 lines
Quoted 8 lines

NUMA API

Hi Wei,

Quoted 2 lines · Wei Chen

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.

I understand that there may not be 32-bit platform with NUMA. And

that's

Quoted 6 lines
Quoted 6 lines

fine stating that in the commit message. However...

Quoted 2 lines

In this case, this Kconfig
option could not be enabled on Arm32.

... you continue to say "couldn't be enabled" without clarifying

whether

Quoted 3 lines
Quoted 2 lines

this mean that the build will break or this was just not tested because you don't have any platform.

Quoted 9 lines
Quoted 1 line

Ok, I understand your concern. Yes, my words would lead to mis-

understanding.

Quoted 2 lines

If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.

When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?

It could not build because I have not tried to enable device_tree_numa option for Arm32 but enabled NUMA for arm32.

I have tested it again, yes, simple enable device_tree_numa and NUMA for arm32 can build a image successfully.

So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work? And I will remove "could not enable for Arm32" from commit log.

Quoted 12 lines
Quoted 11 lines

I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.

Quoted 4 lines

To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.

Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?

Quoted 8 lines

Technically you already added the code because arch/arm/ is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...

And how can we test this code?

I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.

I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.

Quoted 5 lines

Cheers,

--
Julien Grall

#36 lore raw

Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 81 lines
Quoted 68 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 26 lines
Quoted 26 lines

NUMA API

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep

fake

Quoted 8 lines
Quoted 8 lines

NUMA API

Hi Wei,

Quoted 2 lines · Wei Chen

Only Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

What do you mean by "could not be enabled"?

I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.

I understand that there may not be 32-bit platform with NUMA. And

that's

Quoted 6 lines
Quoted 6 lines

fine stating that in the commit message. However...

Quoted 2 lines

In this case, this Kconfig
option could not be enabled on Arm32.

... you continue to say "couldn't be enabled" without clarifying

whether

Quoted 3 lines
Quoted 2 lines

this mean that the build will break or this was just not tested because you don't have any platform.

Quoted 9 lines
Quoted 1 line

Ok, I understand your concern. Yes, my words would lead to mis-

understanding.

Quoted 2 lines

If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.

When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?

It could not build because I have not tried to enable device_tree_numa option for Arm32 but enabled NUMA for arm32.

Quoted 6 lines

I have tested it again, yes, simple enable device_tree_numa and NUMA for arm32 can build a image successfully.

So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work?

Yes, we still need to keep the FAKE APIs. I was only commenting about the wording in the commit message.

Quoted 21 lines

And I will remove "could not
enable for Arm32" from commit log.

Quoted 18 lines
Quoted 11 lines

I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.

Quoted 4 lines

To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.

Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?

Technically you already added the code because arch/arm/ is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...

And how can we test this code?

Quoted 7 lines
Quoted 2 lines

I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.

I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.

I understood you don't have any Arm32 NUMA machine. But I don't see the lack of testing as an issue because the code doesn't look to contain bits that may rely on arm64. So there are very limited reasons for the code to break on arm32.

If we really want to test it, then it should be feasible to fake the NUMA node in the DT. However, I don't expect you to do it.

Cheers,

Signature · 1 line
Julien Grall
author
#37 lore raw

RFC PATCH 8/40 xen/x86: Move NUMA memory node map functions to common

In reply to Wei Chen

In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from arch/x86 to common.

---
 xen/arch/x86/numa.c        | 114 --------------------------------
 xen/common/Makefile        |   1 +
 xen/common/numa.c          | 131 +++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/numa.h |  29 --------
 xen/include/xen/numa.h     |  35 ++++++++++
 5 files changed, 167 insertions(+), 143 deletions(-)
 create mode 100644 xen/common/numa.c
xen/arch/x86/numa.c +0 -114
index d23f4f7919..a6211be121 100644
@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);
 /* from proto.h */
 #define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
 
-struct node_data node_data[MAX_NUMNODES];
-
-/* Mapping from pdx to node id */
-int memnode_shift;
-static typeof(*memnodemap) _memnodemap[64];
-unsigned long memnodemapsize;
-u8 *memnodemap;
-
 nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
     [0 ... NR_CPUS-1] = NUMA_NO_NODE
 };
@@ -58,112 +50,6 @@ int srat_disabled(void)
     return numa_off || acpi_numa < 0;
 }
 
-/*
- * Given a shift value, try to populate memnodemap[]
- * Returns :
- * 1 if OK
- * 0 if memnodmap[] too small (of shift too small)
- * -1 if node overlap or lost ram (shift too big)
- */
-static int __init populate_memnodemap(const struct node *nodes,
-                                      int numnodes, int shift, nodeid_t *nodeids)
-{
-    unsigned long spdx, epdx;
-    int i, res = -1;
-
-    memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        if ( (epdx >> shift) >= memnodemapsize )
-            return 0;
-        do {
-            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
-                return -1;
-
-            if ( !nodeids )
-                memnodemap[spdx >> shift] = i;
-            else
-                memnodemap[spdx >> shift] = nodeids[i];
-
-            spdx += (1UL << shift);
-        } while ( spdx < epdx );
-        res = 1;
-    }
-
-    return res;
-}
-
-static int __init allocate_cachealigned_memnodemap(void)
-{
-    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
-    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
-
-    memnodemap = mfn_to_virt(mfn);
-    mfn <<= PAGE_SHIFT;
-    size <<= PAGE_SHIFT;
-    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
-           mfn, mfn + size);
-    memnodemapsize = size / sizeof(*memnodemap);
-
-    return 0;
-}
-
-/*
- * The LSB of all start and end addresses in the node map is the value of the
- * maximum possible shift.
- */
-static int __init extract_lsb_from_nodes(const struct node *nodes,
-                                         int numnodes)
-{
-    int i, nodes_used = 0;
-    unsigned long spdx, epdx;
-    unsigned long bitfield = 0, memtop = 0;
-
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        bitfield |= spdx;
-        nodes_used++;
-        if ( epdx > memtop )
-            memtop = epdx;
-    }
-    if ( nodes_used <= 1 )
-        i = BITS_PER_LONG - 1;
-    else
-        i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
-    memnodemapsize = (memtop >> i) + 1;
-    return i;
-}
-
-int __init compute_hash_shift(struct node *nodes, int numnodes,
-                              nodeid_t *nodeids)
-{
-    int shift;
-
-    shift = extract_lsb_from_nodes(nodes, numnodes);
-    if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
-        memnodemap = _memnodemap;
-    else if ( allocate_cachealigned_memnodemap() )
-        return -1;
-    printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
-
-    if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
-    {
-        printk(KERN_INFO "Your memory is not aligned you need to "
-               "rebuild your hypervisor with a bigger NODEMAPSIZE "
-               "shift=%d\n", shift);
-        return -1;
-    }
-
-    return shift;
-}
 /* initialize NODE_DATA given nodeid and start/end */
 void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
 { 
xen/common/Makefile +1 -0
index 54de70d422..f8f667e90a 100644
@@ -54,6 +54,7 @@ obj-y += wait.o
 obj-bin-y += warning.init.o
 obj-$(CONFIG_XENOPROF) += xenoprof.o
 obj-y += xmalloc_tlsf.o
+obj-$(CONFIG_NUMA) += numa.o
 
 obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
 
xen/common/numa.c new file +131 -0

131 lines not on this page · raw message

xen/include/asm-x86/numa.h +0 -29
index bada2c0bb9..abe5617d01 100644
@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,
 extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-#define VIRTUAL_BUG_ON(x) 
 
 extern void numa_add_cpu(int cpu);
 extern void numa_init_array(void);
@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)
 	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 }
 
-/* Simple perfect hash to map pdx to node numbers */
-extern int memnode_shift; 
-extern unsigned long memnodemapsize;
-extern u8 *memnodemap;
-
-struct node_data {
-    unsigned long node_start_pfn;
-    unsigned long node_spanned_pages;
-};
-
-extern struct node_data node_data[];
-
-static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
-{ 
-	nodeid_t nid;
-	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
-	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift]; 
-	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]); 
-	return nid; 
-} 
-
-#define NODE_DATA(nid)		(&(node_data[nid]))
-
-#define node_start_pfn(nid)	(NODE_DATA(nid)->node_start_pfn)
-#define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
-#define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
-				 NODE_DATA(nid)->node_spanned_pages)
-
 extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
 void srat_parse_regions(u64 addr);
xen/include/xen/numa.h +35 -0
index 7aef1a88dc..39e8a4e00a 100644
@@ -18,4 +18,39 @@
   (((d)->vcpu != NULL && (d)->vcpu[0] != NULL) \
    ? vcpu_to_node((d)->vcpu[0]) : NUMA_NO_NODE)
 
+/* The following content can be used when NUMA feature is enabled */
+#if defined(CONFIG_NUMA)
+
+/* Simple perfect hash to map pdx to node numbers */
+extern int memnode_shift;
+extern unsigned long memnodemapsize;
+extern u8 *memnodemap;
+extern typeof(*memnodemap) _memnodemap[64];
+
+struct node_data {
+    unsigned long node_start_pfn;
+    unsigned long node_spanned_pages;
+};
+
+extern struct node_data node_data[];
+#define VIRTUAL_BUG_ON(x)
+
+static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
+{
+	nodeid_t nid;
+	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
+	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
+	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+	return nid;
+}
+
+#define NODE_DATA(nid)		(&(node_data[nid]))
+
+#define node_start_pfn(nid)	(NODE_DATA(nid)->node_start_pfn)
+#define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
+#define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
+				 NODE_DATA(nid)->node_spanned_pages)
+
+#endif /* CONFIG_NUMA */
+
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#38 lore raw

Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common

Hi Wei,

Quoted 4 lines · Wei Chen

In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from arch/x86 to common.

I would add "No functional changes intended" to make clear this patch is only moving code.

Quoted 148 lines
---
  xen/arch/x86/numa.c        | 114 --------------------------------
  xen/common/Makefile        |   1 +
  xen/common/numa.c          | 131 +++++++++++++++++++++++++++++++++++++
  xen/include/asm-x86/numa.h |  29 --------
  xen/include/xen/numa.h     |  35 ++++++++++
  5 files changed, 167 insertions(+), 143 deletions(-)

create mode 100644 xen/common/numa.c

xen/arch/x86/numa.c +0 -114
index d23f4f7919..a6211be121 100644
@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);
  /* from proto.h */
  #define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
 
-struct node_data node_data[MAX_NUMNODES];
-
-/* Mapping from pdx to node id */
-int memnode_shift;
-static typeof(*memnodemap) _memnodemap[64];
-unsigned long memnodemapsize;
-u8 *memnodemap;
-
  nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
      [0 ... NR_CPUS-1] = NUMA_NO_NODE
  };
@@ -58,112 +50,6 @@ int srat_disabled(void)
      return numa_off || acpi_numa < 0;
  }
 
-/*
- * Given a shift value, try to populate memnodemap[]
- * Returns :
- * 1 if OK
- * 0 if memnodmap[] too small (of shift too small)
- * -1 if node overlap or lost ram (shift too big)
- */
-static int __init populate_memnodemap(const struct node *nodes,
-                                      int numnodes, int shift, nodeid_t *nodeids)
-{
-    unsigned long spdx, epdx;
-    int i, res = -1;
-
-    memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        if ( (epdx >> shift) >= memnodemapsize )
-            return 0;
-        do {
-            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
-                return -1;
-
-            if ( !nodeids )
-                memnodemap[spdx >> shift] = i;
-            else
-                memnodemap[spdx >> shift] = nodeids[i];
-
-            spdx += (1UL << shift);
-        } while ( spdx < epdx );
-        res = 1;
-    }
-
-    return res;
-}
-
-static int __init allocate_cachealigned_memnodemap(void)
-{
-    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
-    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
-
-    memnodemap = mfn_to_virt(mfn);
-    mfn <<= PAGE_SHIFT;
-    size <<= PAGE_SHIFT;
-    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
-           mfn, mfn + size);
-    memnodemapsize = size / sizeof(*memnodemap);
-
-    return 0;
-}
-
-/*
- * The LSB of all start and end addresses in the node map is the value of the
- * maximum possible shift.
- */
-static int __init extract_lsb_from_nodes(const struct node *nodes,
-                                         int numnodes)
-{
-    int i, nodes_used = 0;
-    unsigned long spdx, epdx;
-    unsigned long bitfield = 0, memtop = 0;
-
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        bitfield |= spdx;
-        nodes_used++;
-        if ( epdx > memtop )
-            memtop = epdx;
-    }
-    if ( nodes_used <= 1 )
-        i = BITS_PER_LONG - 1;
-    else
-        i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
-    memnodemapsize = (memtop >> i) + 1;
-    return i;
-}
-
-int __init compute_hash_shift(struct node *nodes, int numnodes,
-                              nodeid_t *nodeids)
-{
-    int shift;
-
-    shift = extract_lsb_from_nodes(nodes, numnodes);
-    if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
-        memnodemap = _memnodemap;
-    else if ( allocate_cachealigned_memnodemap() )
-        return -1;
-    printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
-
-    if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
-    {
-        printk(KERN_INFO "Your memory is not aligned you need to "
-               "rebuild your hypervisor with a bigger NODEMAPSIZE "
-               "shift=%d\n", shift);
-        return -1;
-    }
-
-    return shift;
-}
  /* initialize NODE_DATA given nodeid and start/end */
  void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
  {
xen/common/Makefile +0 -0
index 54de70d422..f8f667e90a 100644

xen/common/Makefile obj-y += wait.o · line 57

Quoted 4 lines
xen/common/Makefile +1 -0
@@ -54,6 +54,7 @@ obj-y += wait.o
  obj-bin-y += warning.init.o
  obj-$(CONFIG_XENOPROF) += xenoprof.o
  obj-y += xmalloc_tlsf.o
+obj-$(CONFIG_NUMA) += numa.o

AFAICT, the Makefile is listing the file in alphabetical order. So please add numa.o in the correct position.

Quoted 9 lines
xen/common/Makefile +0 -0
  obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
 
xen/common/numa.c new file +0 -0
new file mode 100644
index 0000000000..e65b6a6676

xen/common/numa.c line 10

Quoted 10 lines
xen/common/numa.c new file +10 -0
@@ -0,0 +1,131 @@
+/*
+ * Generic VM initialization for x86-64 NUMA setups.
+ * Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Adapted for Xen: Ryan Harper <[email protected]>
+ */
+
+#include <xen/mm.h>
+#include <xen/string.h>
+#include <xen/init.h>
+#include <xen/ctype.h>

You don't seem to use any helpers./types directly defined by at least this header...

xen/common/numa.c line 13

Quoted 3 lines
xen/common/numa.c new file +3 -0
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/time.h>

... this one and ...

xen/common/numa.c line 14

Quoted 1 line
xen/common/numa.c new file +1 -0
+#include <xen/smp.h>

... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.

xen/common/numa.c line 16

Quoted 2 lines
xen/common/numa.c new file +2 -0
+#include <xen/pfn.h>
+#include <xen/sched.h>

Please sort the includes in alphabetical order.

Quoted 167 lines
xen/common/numa.c new file +115 -0
+
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+int memnode_shift;
+typeof(*memnodemap) _memnodemap[64];
+unsigned long memnodemapsize;
+u8 *memnodemap;
+
+/*
+ * Given a shift value, try to populate memnodemap[]
+ * Returns :
+ * 1 if OK
+ * 0 if memnodmap[] too small (of shift too small)
+ * -1 if node overlap or lost ram (shift too big)
+ */
+static int __init populate_memnodemap(const struct node *nodes,
+                                      int numnodes, int shift, nodeid_t *nodeids)
+{
+    unsigned long spdx, epdx;
+    int i, res = -1;
+
+    memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
+    for ( i = 0; i < numnodes; i++ )
+    {
+        spdx = paddr_to_pdx(nodes[i].start);
+        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
+        if ( spdx >= epdx )
+            continue;
+        if ( (epdx >> shift) >= memnodemapsize )
+            return 0;
+        do {
+            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
+                return -1;
+
+            if ( !nodeids )
+                memnodemap[spdx >> shift] = i;
+            else
+                memnodemap[spdx >> shift] = nodeids[i];
+
+            spdx += (1UL << shift);
+        } while ( spdx < epdx );
+        res = 1;
+    }
+
+    return res;
+}
+
+static int __init allocate_cachealigned_memnodemap(void)
+{
+    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
+    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
+
+    memnodemap = mfn_to_virt(mfn);
+    mfn <<= PAGE_SHIFT;
+    size <<= PAGE_SHIFT;
+    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
+           mfn, mfn + size);
+    memnodemapsize = size / sizeof(*memnodemap);
+
+    return 0;
+}
+
+/*
+ * The LSB of all start and end addresses in the node map is the value of the
+ * maximum possible shift.
+ */
+static int __init extract_lsb_from_nodes(const struct node *nodes,
+                                         int numnodes)
+{
+    int i, nodes_used = 0;
+    unsigned long spdx, epdx;
+    unsigned long bitfield = 0, memtop = 0;
+
+    for ( i = 0; i < numnodes; i++ )
+    {
+        spdx = paddr_to_pdx(nodes[i].start);
+        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
+        if ( spdx >= epdx )
+            continue;
+        bitfield |= spdx;
+        nodes_used++;
+        if ( epdx > memtop )
+            memtop = epdx;
+    }
+    if ( nodes_used <= 1 )
+        i = BITS_PER_LONG - 1;
+    else
+        i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
+    memnodemapsize = (memtop >> i) + 1;
+    return i;
+}
+
+int __init compute_hash_shift(struct node *nodes, int numnodes,
+                              nodeid_t *nodeids)
+{
+    int shift;
+
+    shift = extract_lsb_from_nodes(nodes, numnodes);
+    if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
+        memnodemap = _memnodemap;
+    else if ( allocate_cachealigned_memnodemap() )
+        return -1;
+    printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
+
+    if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
+    {
+        printk(KERN_INFO "Your memory is not aligned you need to "
+               "rebuild your hypervisor with a bigger NODEMAPSIZE "
+               "shift=%d\n", shift);
+        return -1;
+    }
+
+    return shift;
+}
xen/include/asm-x86/numa.h +0 -29
index bada2c0bb9..abe5617d01 100644
@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,
  extern nodeid_t pxm_to_node(unsigned int pxm);
 
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-#define VIRTUAL_BUG_ON(x)
 
  extern void numa_add_cpu(int cpu);
  extern void numa_init_array(void);
@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)
  	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
  }
 
-/* Simple perfect hash to map pdx to node numbers */
-extern int memnode_shift;
-extern unsigned long memnodemapsize;
-extern u8 *memnodemap;
-
-struct node_data {
-    unsigned long node_start_pfn;
-    unsigned long node_spanned_pages;
-};
-
-extern struct node_data node_data[];
-
-static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
-{
-	nodeid_t nid;
-	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
-	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
-	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
-	return nid;
-}
-
-#define NODE_DATA(nid)		(&(node_data[nid]))
-
-#define node_start_pfn(nid)	(NODE_DATA(nid)->node_start_pfn)
-#define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
-#define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
-				 NODE_DATA(nid)->node_spanned_pages)
-
  extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
  void srat_parse_regions(u64 addr);
xen/include/xen/numa.h +0 -0
index 7aef1a88dc..39e8a4e00a 100644

xen/include/xen/numa.h line 22

Quoted 5 lines
xen/include/xen/numa.h +2 -0
@@ -18,4 +18,39 @@
    (((d)->vcpu != NULL && (d)->vcpu[0] != NULL) \
     ? vcpu_to_node((d)->vcpu[0]) : NUMA_NO_NODE)
 
+/* The following content can be used when NUMA feature is enabled */
+#if defined(CONFIG_NUMA)

Please use #ifdef CONFIG_NUMA

xen/include/xen/numa.h line 28

Quoted 6 lines
xen/include/xen/numa.h +6 -0
+
+/* Simple perfect hash to map pdx to node numbers */
+extern int memnode_shift;
+extern unsigned long memnodemapsize;
+extern u8 *memnodemap;
+extern typeof(*memnodemap) _memnodemap[64];

AFAICT, this will be turned static against in a follow-up patch. Can this be avoided?

Quoted 19 lines
xen/include/xen/numa.h +18 -0
+
+struct node_data {
+    unsigned long node_start_pfn;
+    unsigned long node_spanned_pages;
+};
+
+extern struct node_data node_data[];
+#define VIRTUAL_BUG_ON(x)
+
+static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
+{
+	nodeid_t nid;
+	VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
+	nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
+	VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
+	return nid;
+}
+

xen/include/xen/numa.h line 56

Quoted 10 lines
xen/include/xen/numa.h +9 -0
+#define NODE_DATA(nid)		(&(node_data[nid]))
+
+#define node_start_pfn(nid)	(NODE_DATA(nid)->node_start_pfn)
+#define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
+#define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
+				 NODE_DATA(nid)->node_spanned_pages)
+
+#endif /* CONFIG_NUMA */
+
  #endif /* _XEN_NUMA_H */

Cheers,

Signature · 1 line
Julien Grall
author
#39 lore raw

RE: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common

Hi Julien,

Quoted 11 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:47
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common

Hi Wei,

Quoted 8 lines
Quoted 4 lines · Wei Chen

In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from arch/x86 to common.

I would add "No functional changes intended" to make clear this patch is only moving code.

Ok, I will do it.

Quoted 151 lines
Quoted 34 lines
---
  xen/arch/x86/numa.c        | 114 --------------------------------
  xen/common/Makefile        |   1 +
  xen/common/numa.c          | 131 +++++++++++++++++++++++++++++++++++++
  xen/include/asm-x86/numa.h |  29 --------
  xen/include/xen/numa.h     |  35 ++++++++++
  5 files changed, 167 insertions(+), 143 deletions(-)

create mode 100644 xen/common/numa.c

xen/arch/x86/numa.c +0 -8
index d23f4f7919..a6211be121 100644
@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);
  /* from proto.h */
  #define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
 
-struct node_data node_data[MAX_NUMNODES];
-
-/* Mapping from pdx to node id */
-int memnode_shift;
-static typeof(*memnodemap) _memnodemap[64];
-unsigned long memnodemapsize;
-u8 *memnodemap;
-
  nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
      [0 ... NR_CPUS-1] = NUMA_NO_NODE
  };
@@ -58,112 +50,6 @@ int srat_disabled(void)
      return numa_off || acpi_numa < 0;
  }

xen/arch/x86/numa.c int srat_disabled(void) · line 69

Quoted 10 lines
xen/arch/x86/numa.c +0 -9
 
-/*
- * Given a shift value, try to populate memnodemap[]
- * Returns :
- * 1 if OK
- * 0 if memnodmap[] too small (of shift too small)
- * -1 if node overlap or lost ram (shift too big)
- */
-static int __init populate_memnodemap(const struct node *nodes,
-                                      int numnodes, int shift, nodeid_t

*nodeids)

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 5 lines
xen/arch/x86/numa.c +0 -5
-{
-    unsigned long spdx, epdx;
-    int i, res = -1;
-
-    memset(memnodemap, NUMA_NO_NODE, memnodemapsize *

sizeof(*memnodemap));

Quoted 32 lines
xen/arch/x86/numa.c +0 -32
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        if ( (epdx >> shift) >= memnodemapsize )
-            return 0;
-        do {
-            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
-                return -1;
-
-            if ( !nodeids )
-                memnodemap[spdx >> shift] = i;
-            else
-                memnodemap[spdx >> shift] = nodeids[i];
-
-            spdx += (1UL << shift);
-        } while ( spdx < epdx );
-        res = 1;
-    }
-
-    return res;
-}
-
-static int __init allocate_cachealigned_memnodemap(void)
-{
-    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
-    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
-
-    memnodemap = mfn_to_virt(mfn);
-    mfn <<= PAGE_SHIFT;

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-    size <<= PAGE_SHIFT;
-    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
-           mfn, mfn + size);
-    memnodemapsize = size / sizeof(*memnodemap);
-
-    return 0;
-}
-
-/*
- * The LSB of all start and end addresses in the node map is the value

of the

Quoted 57 lines
xen/arch/x86/numa.c +0 -50
- * maximum possible shift.
- */
-static int __init extract_lsb_from_nodes(const struct node *nodes,
-                                         int numnodes)
-{
-    int i, nodes_used = 0;
-    unsigned long spdx, epdx;
-    unsigned long bitfield = 0, memtop = 0;
-
-    for ( i = 0; i < numnodes; i++ )
-    {
-        spdx = paddr_to_pdx(nodes[i].start);
-        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
-        if ( spdx >= epdx )
-            continue;
-        bitfield |= spdx;
-        nodes_used++;
-        if ( epdx > memtop )
-            memtop = epdx;
-    }
-    if ( nodes_used <= 1 )
-        i = BITS_PER_LONG - 1;
-    else
-        i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
-    memnodemapsize = (memtop >> i) + 1;
-    return i;
-}
-
-int __init compute_hash_shift(struct node *nodes, int numnodes,
-                              nodeid_t *nodeids)
-{
-    int shift;
-
-    shift = extract_lsb_from_nodes(nodes, numnodes);
-    if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
-        memnodemap = _memnodemap;
-    else if ( allocate_cachealigned_memnodemap() )
-        return -1;
-    printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
-
-    if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
-    {
-        printk(KERN_INFO "Your memory is not aligned you need to "
-               "rebuild your hypervisor with a bigger NODEMAPSIZE "
-               "shift=%d\n", shift);
-        return -1;
-    }
-
-    return shift;
-}
  /* initialize NODE_DATA given nodeid and start/end */
  void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
  {
xen/common/Makefile +0 -0
index 54de70d422..f8f667e90a 100644
Quoted 9 lines

xen/common/Makefile obj-y += wait.o · line 57

Quoted 5 lines
xen/common/Makefile +1 -0
@@ -54,6 +54,7 @@ obj-y += wait.o
  obj-bin-y += warning.init.o
  obj-$(CONFIG_XENOPROF) += xenoprof.o
  obj-y += xmalloc_tlsf.o
+obj-$(CONFIG_NUMA) += numa.o

AFAICT, the Makefile is listing the file in alphabetical order. So please add numa.o in the correct position.

Thanks for the reminder, I will fix it.

Quoted 30 lines

xen/common/Makefile obj-y += wait.o · line 58

Quoted 2 lines
xen/common/Makefile +0 -0
  obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma

lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)

Quoted 7 lines
xen/common/numa.c new file +0 -0
new file mode 100644
index 0000000000..e65b6a6676

xen/common/numa.c line 10

Quoted 10 lines
xen/common/numa.c new file +10 -0
@@ -0,0 +1,131 @@
+/*
+ * Generic VM initialization for x86-64 NUMA setups.
+ * Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Adapted for Xen: Ryan Harper <[email protected]>
+ */
+
+#include <xen/mm.h>
+#include <xen/string.h>
+#include <xen/init.h>
+#include <xen/ctype.h>

You don't seem to use any helpers./types directly defined by at least this header...

xen/common/numa.c line 13

Quoted 3 lines
xen/common/numa.c new file +3 -0
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/time.h>

... this one and ...

Quoted 6 lines

xen/common/numa.c line 14

Quoted 1 line
xen/common/numa.c new file +1 -0
+#include <xen/smp.h>

... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.

I will check it in next version. If it isn't needed, I will remove it.

Quoted 5 lines

xen/common/numa.c line 16

Quoted 2 lines
xen/common/numa.c new file +2 -0
+#include <xen/pfn.h>
+#include <xen/sched.h>

Please sort the includes in alphabetical order.

OK

Quoted 180 lines
Quoted 8 lines
xen/common/numa.c new file +8 -0
+
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+int memnode_shift;
+typeof(*memnodemap) _memnodemap[64];
+unsigned long memnodemapsize;
+u8 *memnodemap;

xen/common/numa.c

Quoted 10 lines
xen/common/numa.c new file +10 -0
+
+/*
+ * Given a shift value, try to populate memnodemap[]
+ * Returns :
+ * 1 if OK
+ * 0 if memnodmap[] too small (of shift too small)
+ * -1 if node overlap or lost ram (shift too big)
+ */
+static int __init populate_memnodemap(const struct node *nodes,
+                                      int numnodes, int shift, nodeid_t

*nodeids)

xen/common/numa.c

Quoted 5 lines
xen/common/numa.c new file +5 -0
+{
+    unsigned long spdx, epdx;
+    int i, res = -1;
+
+    memset(memnodemap, NUMA_NO_NODE, memnodemapsize *

sizeof(*memnodemap));

Quoted 32 lines
xen/common/numa.c new file +32 -0
+    for ( i = 0; i < numnodes; i++ )
+    {
+        spdx = paddr_to_pdx(nodes[i].start);
+        epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
+        if ( spdx >= epdx )
+            continue;
+        if ( (epdx >> shift) >= memnodemapsize )
+            return 0;
+        do {
+            if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
+                return -1;
+
+            if ( !nodeids )
+                memnodemap[spdx >> shift] = i;
+            else
+                memnodemap[spdx >> shift] = nodeids[i];
+
+            spdx += (1UL << shift);
+        } while ( spdx < epdx );
+        res = 1;
+    }
+
+    return res;
+}
+
+static int __init allocate_cachealigned_memnodemap(void)
+{
+    unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
+    unsigned long mfn = mfn_x(alloc_boot_pages(size, 1));
+
+    memnodemap = mfn_to_virt(mfn);
+    mfn <<= PAGE_SHIFT;

xen/common/numa.c

Quoted 10 lines
xen/common/numa.c new file +10 -0
+    size <<= PAGE_SHIFT;
+    printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
+           mfn, mfn + size);
+    memnodemapsize = size / sizeof(*memnodemap);
+
+    return 0;
+}
+
+/*
+ * The LSB of all start and end addresses in the node map is the value

of the

110 lines not shown · raw message

Quoted 10 lines

8 lines not shown · raw message

36 lines not shown · raw message

Part of this message is not shown here. View the raw message.

author
#40 lore raw

RFC PATCH 9/40 xen/x86: Move numa_add_cpu_node to common

In reply to Wei Chen

This function will be reused by Arm later, so we move it from arch/x86 to common. But we keep cpu_to_node and node_to_cpumask to x86 header file. Because cpu_to_node and node_to_cpumask have different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.

---
 xen/arch/x86/numa.c        |  9 ---------
 xen/common/numa.c          | 11 +++++++++++
 xen/include/asm-x86/numa.h |  1 -
 xen/include/xen/numa.h     |  2 ++
 4 files changed, 13 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +0 -9
index a6211be121..f2626b3968 100644
@@ -29,16 +29,12 @@ custom_param("numa", numa_setup);
 /* from proto.h */
 #define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
 
-nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
-    [0 ... NR_CPUS-1] = NUMA_NO_NODE
-};
 /*
  * Keep BIOS's CPU2node information, should not be used for memory allocaion
  */
 nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
     [0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
 };
-cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
@@ -167,11 +163,6 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
                     (u64)end_pfn << PAGE_SHIFT);
 }
 
-void numa_add_cpu(int cpu)
-{
-    cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-} 
-
 void numa_set_node(int cpu, nodeid_t node)
 {
     cpu_to_node[cpu] = node;
xen/common/numa.c +11 -0
index e65b6a6676..9b6f23dfc1 100644
@@ -23,6 +23,12 @@ typeof(*memnodemap) _memnodemap[64];
 unsigned long memnodemapsize;
 u8 *memnodemap;
 
+nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
+    [0 ... NR_CPUS-1] = NUMA_NO_NODE
+};
+
+cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
+
 /*
  * Given a shift value, try to populate memnodemap[]
  * Returns :
@@ -129,3 +135,8 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
 
     return shift;
 }
+
+void numa_add_cpu(int cpu)
+{
+    cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+}
xen/include/asm-x86/numa.h +0 -1
index abe5617d01..07ff78ea1b 100644
@@ -27,7 +27,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
-extern void numa_add_cpu(int cpu);
 extern void numa_init_array(void);
 extern bool numa_off;
 
xen/include/xen/numa.h +2 -0
index 39e8a4e00a..f9769cba4b 100644
@@ -51,6 +51,8 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
 
+extern void numa_add_cpu(int cpu);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#41 lore raw

Re: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common

Hi Wei,

Quoted 6 lines · Wei Chen

This function will be reused by Arm later, so we move it from arch/x86 to common. But we keep cpu_to_node and node_to_cpumask to x86 header file. Because cpu_to_node and node_to_cpumask have different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.

AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?

Cheers,

Signature · 1 line
Julien Grall
author
#42 lore raw

RE: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common

Hi Julien,

Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:54
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common

Hi Wei,

Quoted 6 lines · Wei Chen

This function will be reused by Arm later, so we move it from arch/x86 to common. But we keep cpu_to_node and node_to_cpumask to x86 header file. Because cpu_to_node and node_to_cpumask have different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.

Quoted 4 lines

AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?

Yes, you're right. After we had introduced !CONFIG_NUMA, we can move node_to_cpumask and cpu_to_node in this patch too. I will fix it in next version.

Quoted 4 lines

Cheers,

--
Julien Grall

author
#43 lore raw

RFC PATCH 10/40 xen/x86: Move NR_NODE_MEMBLKS macro to common

In reply to Wei Chen

Not only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number. So we move it from x86 ACPI header file to common NUMA header file.

---
 xen/include/asm-x86/acpi.h | 1 -
 xen/include/xen/numa.h     | 1 +
 2 files changed, 1 insertion(+), 1 deletion(-)
xen/include/asm-x86/acpi.h +0 -1
index 7032f3a001..d347500a3c 100644
@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;
 
 extern s8 acpi_numa;
 extern int acpi_scan_nodes(u64 start, u64 end);
-#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
 extern struct acpi_sleep_info acpi_sinfo;
 #define acpi_video_flags bootsym(video_flags)
xen/include/xen/numa.h +1 -0
index f9769cba4b..5af74b357f 100644
@@ -11,6 +11,7 @@
 #define NUMA_NO_DISTANCE 0xFF
 
 #define MAX_NUMNODES    (1 << NODES_SHIFT)
+#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
 #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 
Signature · 1 line
2.25.1
#44 lore raw

Re: [XEN RFC PATCH 10/40] xen/x86: Move NR_NODE_MEMBLKS macro to common

Hi Wei,

Quoted 2 lines · Wei Chen

Not only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number.

AFAICT, a memory range described in DT cannot be split across multiple nodes. So I think we want to define NR_NODE_MEMBLKS as NR_MEM_BANKS.

Quoted 27 lines

So we move it from x86 ACPI header file to common NUMA header file.

---
  xen/include/asm-x86/acpi.h | 1 -
  xen/include/xen/numa.h     | 1 +
  2 files changed, 1 insertion(+), 1 deletion(-)
xen/include/asm-x86/acpi.h +0 -1
index 7032f3a001..d347500a3c 100644
@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;
 
  extern s8 acpi_numa;
  extern int acpi_scan_nodes(u64 start, u64 end);
-#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
  extern struct acpi_sleep_info acpi_sinfo;
  #define acpi_video_flags bootsym(video_flags)
xen/include/xen/numa.h +0 -0
index f9769cba4b..5af74b357f 100644

xen/include/xen/numa.h line 17

Quoted 7 lines
xen/include/xen/numa.h +1 -0
@@ -11,6 +11,7 @@
  #define NUMA_NO_DISTANCE 0xFF
 
  #define MAX_NUMNODES    (1 << NODES_SHIFT)
+#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
  #define vcpu_to_node(v) (cpu_to_node((v)->processor))
 

Cheers,

Signature · 1 line
Julien Grall
author
#45 lore raw

RFC PATCH 11/40 xen/x86: Move NUMA nodes and memory block ranges to common

In reply to Wei Chen

These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.

---
 xen/arch/x86/srat.c        | 50 -------------------------------------
 xen/common/numa.c          | 51 ++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/numa.h |  8 ------
 xen/include/xen/numa.h     | 15 +++++++++++
 4 files changed, 66 insertions(+), 58 deletions(-)
xen/arch/x86/srat.c +0 -50
index 6b77b98201..6d68b8a614 100644
@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;
 
 static nodemask_t memory_nodes_parsed __initdata;
 static nodemask_t processor_nodes_parsed __initdata;
-static struct node nodes[MAX_NUMNODES] __initdata;
 
 struct pxm2node {
 	unsigned pxm;
@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
 
 static unsigned node_to_pxm(nodeid_t n);
 
-static int num_node_memblks;
-static struct node node_memblk_range[NR_NODE_MEMBLKS];
-static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
 
 static inline bool node_found(unsigned idx, unsigned pxm)
@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)
 	return node;
 }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-
-		if (nd->start <= start && nd->end >= end &&
-			memblk_nodeid[i] == node)
-			return 1;
-	}
-
-	return 0;
-}
-
-static __init int conflicting_memblks(u64 start, u64 end)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-		if (nd->start == nd->end)
-			continue;
-		if (nd->end > start && nd->start < end)
-			return i;
-		if (nd->end == end && nd->start == start)
-			return i;
-	}
-	return -1;
-}
-
-static __init void cutoff_node(int i, u64 start, u64 end)
-{
-	struct node *nd = &nodes[i];
-	if (nd->start < start) {
-		nd->start = start;
-		if (nd->end < nd->start)
-			nd->start = nd->end;
-	}
-	if (nd->end > end) {
-		nd->end = end;
-		if (nd->start > nd->end)
-			nd->start = nd->end;
-	}
-}
-
 static __init void bad_srat(void)
 {
 	int i;
xen/common/numa.c +51 -0

63 lines not on this page · raw message

xen/include/asm-x86/numa.h +0 -8
index 07ff78ea1b..e8a92ad9df 100644
@@ -17,12 +17,6 @@ extern cpumask_t     node_to_cpumask[];
 #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
 #define node_to_cpumask(node)    (node_to_cpumask[node])
 
-struct node { 
-	u64 start,end; 
-};
-
-extern int compute_hash_shift(struct node *nodes, int numnodes,
-			      nodeid_t *nodeids);
 extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)
 	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 }
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
-
 void srat_parse_regions(u64 addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
xen/include/xen/numa.h +15 -0
index 5af74b357f..67b79a73a3 100644
@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 
 extern void numa_add_cpu(int cpu);
 
+struct node {
+	u64 start,end;
+};
+
+extern struct node nodes[MAX_NUMNODES];
+extern int num_node_memblks;
+extern struct node node_memblk_range[NR_NODE_MEMBLKS];
+extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
+
+extern int compute_hash_shift(struct node *nodes, int numnodes,
+			      nodeid_t *nodeids);
+extern int conflicting_memblks(u64 start, u64 end);
+extern void cutoff_node(int i, u64 start, u64 end);
+extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#46 lore raw

Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common

Hi Wei,

Quoted 189 lines · Wei Chen

These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.

---
  xen/arch/x86/srat.c        | 50 -------------------------------------
  xen/common/numa.c          | 51 ++++++++++++++++++++++++++++++++++++++
  xen/include/asm-x86/numa.h |  8 ------
  xen/include/xen/numa.h     | 15 +++++++++++
  4 files changed, 66 insertions(+), 58 deletions(-)
xen/arch/x86/srat.c +0 -50
index 6b77b98201..6d68b8a614 100644
@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;
 
  static nodemask_t memory_nodes_parsed __initdata;
  static nodemask_t processor_nodes_parsed __initdata;
-static struct node nodes[MAX_NUMNODES] __initdata;
 
  struct pxm2node {
  	unsigned pxm;
@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
 
  static unsigned node_to_pxm(nodeid_t n);
 
-static int num_node_memblks;
-static struct node node_memblk_range[NR_NODE_MEMBLKS];
-static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
  static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
 
  static inline bool node_found(unsigned idx, unsigned pxm)
@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)
  	return node;
  }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-
-		if (nd->start <= start && nd->end >= end &&
-			memblk_nodeid[i] == node)
-			return 1;
-	}
-
-	return 0;
-}
-
-static __init int conflicting_memblks(u64 start, u64 end)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-		if (nd->start == nd->end)
-			continue;
-		if (nd->end > start && nd->start < end)
-			return i;
-		if (nd->end == end && nd->start == start)
-			return i;
-	}
-	return -1;
-}
-
-static __init void cutoff_node(int i, u64 start, u64 end)
-{
-	struct node *nd = &nodes[i];
-	if (nd->start < start) {
-		nd->start = start;
-		if (nd->end < nd->start)
-			nd->start = nd->end;
-	}
-	if (nd->end > end) {
-		nd->end = end;
-		if (nd->start > nd->end)
-			nd->start = nd->end;
-	}
-}
-
  static __init void bad_srat(void)
  {
  	int i;
xen/common/numa.c +51 -0
index 9b6f23dfc1..1facc8fe2b 100644
@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
 
  cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 
+struct node nodes[MAX_NUMNODES] __initdata;
+int num_node_memblks;
+struct node node_memblk_range[NR_NODE_MEMBLKS];
+nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
+
  /*
   * Given a shift value, try to populate memnodemap[]
   * Returns :
@@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
      return shift;
  }
 
+int valid_numa_range(u64 start, u64 end, nodeid_t node)
+{
+	int i;
+
+	for (i = 0; i < num_node_memblks; i++) {
+		struct node *nd = &node_memblk_range[i];
+
+		if (nd->start <= start && nd->end >= end &&
+			memblk_nodeid[i] == node)
+			return 1;
+	}
+
+	return 0;
+}
+
+int __init conflicting_memblks(u64 start, u64 end)
+{
+	int i;
+
+	for (i = 0; i < num_node_memblks; i++) {
+		struct node *nd = &node_memblk_range[i];
+		if (nd->start == nd->end)
+			continue;
+		if (nd->end > start && nd->start < end)
+			return i;
+		if (nd->end == end && nd->start == start)
+			return i;
+	}
+	return -1;
+}
+
+void __init cutoff_node(int i, u64 start, u64 end)
+{
+	struct node *nd = &nodes[i];
+	if (nd->start < start) {
+		nd->start = start;
+		if (nd->end < nd->start)
+			nd->start = nd->end;
+	}
+	if (nd->end > end) {
+		nd->end = end;
+		if (nd->start > nd->end)
+			nd->start = nd->end;
+	}
+}
+
  void numa_add_cpu(int cpu)
  {
      cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
xen/include/asm-x86/numa.h +0 -8
index 07ff78ea1b..e8a92ad9df 100644
@@ -17,12 +17,6 @@ extern cpumask_t     node_to_cpumask[];
  #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
  #define node_to_cpumask(node)    (node_to_cpumask[node])
 
-struct node {
-	u64 start,end;
-};
-
-extern int compute_hash_shift(struct node *nodes, int numnodes,
-			      nodeid_t *nodeids);
  extern nodeid_t pxm_to_node(unsigned int pxm);
 
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)
  	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
  }
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
-
  void srat_parse_regions(u64 addr);
  extern u8 __node_distance(nodeid_t a, nodeid_t b);
  unsigned int arch_get_dma_bitsize(void);
xen/include/xen/numa.h +0 -0
index 5af74b357f..67b79a73a3 100644
@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64

Quoted 10 lines
xen/include/xen/numa.h +8 -0
  extern void numa_add_cpu(int cpu);
 
+struct node {
+	u64 start,end;
+};
+
+extern struct node nodes[MAX_NUMNODES];
+extern int num_node_memblks;
+extern struct node node_memblk_range[NR_NODE_MEMBLKS];
+extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];

I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for acpi_numa_memory_affinity_init() and dtb_numa_memory_affinity_init().

There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in printk() and the hotplug bits.

So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 74

Quoted 11 lines
xen/include/xen/numa.h +7 -0
+
+extern int compute_hash_shift(struct node *nodes, int numnodes,
+			      nodeid_t *nodeids);
+extern int conflicting_memblks(u64 start, u64 end);
+extern void cutoff_node(int i, u64 start, u64 end);
+extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+
  #endif /* CONFIG_NUMA */
 
  #endif /* _XEN_NUMA_H */

[1] https://lore.kernel.org/xen-devel/[email protected]/

Signature · 1 line
Julien Grall
author
#47 lore raw

RE: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common

Hi Julien,

Quoted 219 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 1:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common

Hi Wei,

xen/arch/x86/srat.c static struct pxm2node __read_mostly

Quoted 26 lines · Wei Chen

These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.

---
  xen/arch/x86/srat.c        | 50 -------------------------------------
  xen/common/numa.c          | 51 ++++++++++++++++++++++++++++++++++++++
  xen/include/asm-x86/numa.h |  8 ------
  xen/include/xen/numa.h     | 15 +++++++++++
  4 files changed, 66 insertions(+), 58 deletions(-)
xen/arch/x86/srat.c +0 -1
index 6b77b98201..6d68b8a614 100644
@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;
 
  static nodemask_t memory_nodes_parsed __initdata;
  static nodemask_t processor_nodes_parsed __initdata;
-static struct node nodes[MAX_NUMNODES] __initdata;
 
  struct pxm2node {
  	unsigned pxm;
@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly

pxm2node[MAX_NUMNODES] =

xen/common/numa.c int __init compute_hash_shift(struct node *nodes,

Quoted 79 lines
xen/arch/x86/srat.c +0 -49
  static unsigned node_to_pxm(nodeid_t n);
 
-static int num_node_memblks;
-static struct node node_memblk_range[NR_NODE_MEMBLKS];
-static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
  static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
 
  static inline bool node_found(unsigned idx, unsigned pxm)
@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)
  	return node;
  }
 
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-
-		if (nd->start <= start && nd->end >= end &&
-			memblk_nodeid[i] == node)
-			return 1;
-	}
-
-	return 0;
-}
-
-static __init int conflicting_memblks(u64 start, u64 end)
-{
-	int i;
-
-	for (i = 0; i < num_node_memblks; i++) {
-		struct node *nd = &node_memblk_range[i];
-		if (nd->start == nd->end)
-			continue;
-		if (nd->end > start && nd->start < end)
-			return i;
-		if (nd->end == end && nd->start == start)
-			return i;
-	}
-	return -1;
-}
-
-static __init void cutoff_node(int i, u64 start, u64 end)
-{
-	struct node *nd = &nodes[i];
-	if (nd->start < start) {
-		nd->start = start;
-		if (nd->end < nd->start)
-			nd->start = nd->end;
-	}
-	if (nd->end > end) {
-		nd->end = end;
-		if (nd->start > nd->end)
-			nd->start = nd->end;
-	}
-}
-
  static __init void bad_srat(void)
  {
  	int i;
xen/common/numa.c +5 -0
index 9b6f23dfc1..1facc8fe2b 100644
@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
 
  cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
 
+struct node nodes[MAX_NUMNODES] __initdata;
+int num_node_memblks;
+struct node node_memblk_range[NR_NODE_MEMBLKS];
+nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
+
  /*
   * Given a shift value, try to populate memnodemap[]
   * Returns :
@@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes,

int numnodes,

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t

Quoted 83 lines
xen/common/numa.c +46 -0
      return shift;
  }
 
+int valid_numa_range(u64 start, u64 end, nodeid_t node)
+{
+	int i;
+
+	for (i = 0; i < num_node_memblks; i++) {
+		struct node *nd = &node_memblk_range[i];
+
+		if (nd->start <= start && nd->end >= end &&
+			memblk_nodeid[i] == node)
+			return 1;
+	}
+
+	return 0;
+}
+
+int __init conflicting_memblks(u64 start, u64 end)
+{
+	int i;
+
+	for (i = 0; i < num_node_memblks; i++) {
+		struct node *nd = &node_memblk_range[i];
+		if (nd->start == nd->end)
+			continue;
+		if (nd->end > start && nd->start < end)
+			return i;
+		if (nd->end == end && nd->start == start)
+			return i;
+	}
+	return -1;
+}
+
+void __init cutoff_node(int i, u64 start, u64 end)
+{
+	struct node *nd = &nodes[i];
+	if (nd->start < start) {
+		nd->start = start;
+		if (nd->end < nd->start)
+			nd->start = nd->end;
+	}
+	if (nd->end > end) {
+		nd->end = end;
+		if (nd->start > nd->end)
+			nd->start = nd->end;
+	}
+}
+
  void numa_add_cpu(int cpu)
  {
      cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
xen/include/asm-x86/numa.h +0 -8
index 07ff78ea1b..e8a92ad9df 100644
@@ -17,12 +17,6 @@ extern cpumask_t     node_to_cpumask[];
  #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
  #define node_to_cpumask(node)    (node_to_cpumask[node])
 
-struct node {
-	u64 start,end;
-};
-
-extern int compute_hash_shift(struct node *nodes, int numnodes,
-			      nodeid_t *nodeids);
  extern nodeid_t pxm_to_node(unsigned int pxm);
 
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)
  	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
  }
 
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
-
  void srat_parse_regions(u64 addr);
  extern u8 __node_distance(nodeid_t a, nodeid_t b);
  unsigned int arch_get_dma_bitsize(void);
xen/include/xen/numa.h +0 -0
index 5af74b357f..67b79a73a3 100644
@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t

phys_to_nid(paddr_t addr)

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64

Quoted 11 lines
xen/include/xen/numa.h +8 -0
  extern void numa_add_cpu(int cpu);
 
+struct node {
+	u64 start,end;
+};
+
+extern struct node nodes[MAX_NUMNODES];
+extern int num_node_memblks;
+extern struct node node_memblk_range[NR_NODE_MEMBLKS];
+extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];

I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for acpi_numa_memory_affinity_init() and dtb_numa_memory_affinity_init().

Quoted 8 lines

There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in printk() and the hotplug bits.

So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.

Ok, I will look at that thread. If it's useful, I will do it in next version.

Quoted 19 lines

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)

Quoted 12 lines
xen/include/xen/numa.h +7 -0
+
+extern int compute_hash_shift(struct node *nodes, int numnodes,
+			      nodeid_t *nodeids);
+extern int conflicting_memblks(u64 start, u64 end);
+extern void cutoff_node(int i, u64 start, u64 end);
+extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+
  #endif /* CONFIG_NUMA */
 
  #endif /* _XEN_NUMA_H */
 

[1]
https://lore.kernel.org/xen-devel/1500378106-2620-1-git-send-email-
[email protected]/

--
Julien Grall

author
#48 lore raw

RFC PATCH 12/40 xen/x86: Move numa_initmem_init to common

In reply to Wei Chen

This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem, numa_init_array and numa_emulation.

As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.

---
 xen/arch/x86/numa.c         | 118 ----------------------------------
 xen/common/numa.c           | 122 +++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/numa.h  |   5 --
 xen/include/asm-x86/setup.h |   1 -
 xen/include/xen/numa.h      |   8 ++-
 5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -118
index f2626b3968..6908738305 100644
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-bool numa_off;
 s8 acpi_numa = 0;
 
 int srat_disabled(void)
@@ -46,123 +45,6 @@ int srat_disabled(void)
     return numa_off || acpi_numa < 0;
 }
 
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{ 
-    unsigned long start_pfn, end_pfn;
-
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
-
-    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
-    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
-
-    node_set_online(nodeid);
-} 
-
-void __init numa_init_array(void)
-{
-    int rr, i;
-
-    /* There are unfortunately some poorly designed mainboards around
-       that only connect memory to a single CPU. This breaks the 1:1 cpu->node
-       mapping. To avoid this fill in the mapping for all possible
-       CPUs, as the number of CPUs is not known yet.
-       We round robin the existing nodes. */
-    rr = first_node(node_online_map);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-    {
-        if ( cpu_to_node[i] != NUMA_NO_NODE )
-            continue;
-        numa_set_node(i, rr);
-        rr = cycle_node(rr, node_online_map);
-    }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
-
-/* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
-{
-    int i;
-    struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
-
-    /* Kludge needed for the hash function */
-    if ( hweight64(sz) > 1 )
-    {
-        u64 x = 1;
-        while ( (x << 1) < sz )
-            x <<= 1;
-        if ( x < sz/2 )
-            printk(KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n");
-        sz = x;
-    }
-
-    memset(&nodes,0,sizeof(nodes));
-    for ( i = 0; i < numa_fake; i++ )
-    {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
-        if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
-        nodes[i].end = nodes[i].start + sz;
-        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
-               i,
-               nodes[i].start, nodes[i].end,
-               (nodes[i].end - nodes[i].start) >> 20);
-        node_set_online(i);
-    }
-    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
-    if ( memnode_shift < 0 )
-    {
-        memnode_shift = 0;
-        printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
-        return -1;
-    }
-    for_each_online_node ( i )
-        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-    numa_init_array();
-
-    return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
-{ 
-    int i;
-
-#ifdef CONFIG_NUMA_EMU
-    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
-        return;
-#endif
-
-#ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
-        return;
-#endif
-
-    printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration found");
-
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
-    /* setup dummy node covering all memory */
-    memnode_shift = BITS_PER_LONG - 1;
-    memnodemap = _memnodemap;
-    memnodemapsize = ARRAY_SIZE(_memnodemap);
-
-    nodes_clear(node_online_map);
-    node_set_online(0);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-        numa_set_node(i, 0);
-    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
-}
-
 void numa_set_node(int cpu, nodeid_t node)
 {
     cpu_to_node[cpu] = node;
xen/common/numa.c +121 -1

142 lines not on this page · raw message

xen/include/asm-x86/numa.h +0 -5
index e8a92ad9df..f8e4e15586 100644
@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
-extern void numa_init_array(void);
-extern bool numa_off;
-
-
 extern int srat_disabled(void);
 extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
xen/include/asm-x86/setup.h +0 -1
index 24be46115d..63838ba2d1 100644
@@ -17,7 +17,6 @@ void early_time_init(void);
 
 void set_nr_cpu_ids(unsigned int max_cpus);
 
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
 void arch_init_memory(void);
 void subarch_init_memory(void);
 
xen/include/xen/numa.h +7 -1
index 67b79a73a3..258a5cb3db 100644
@@ -26,7 +26,6 @@
 extern int memnode_shift;
 extern unsigned long memnodemapsize;
 extern u8 *memnodemap;
-extern typeof(*memnodemap) _memnodemap[64];
 
 struct node_data {
     unsigned long node_start_pfn;
@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);
 extern void cutoff_node(int i, u64 start, u64 end);
 extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
+extern void numa_init_array(void);
+extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+extern bool numa_off;
+extern int numa_fake;
+
+extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
#49 lore raw

Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common

Hi Wei,

Quoted 8 lines · Wei Chen

This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem, numa_init_array and numa_emulation.

As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.

As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.

Quoted 151 lines
---
  xen/arch/x86/numa.c         | 118 ----------------------------------
  xen/common/numa.c           | 122 +++++++++++++++++++++++++++++++++++-
  xen/include/asm-x86/numa.h  |   5 --
  xen/include/asm-x86/setup.h |   1 -
  xen/include/xen/numa.h      |   8 ++-
  5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -118
index f2626b3968..6908738305 100644
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
 
  nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-bool numa_off;
  s8 acpi_numa = 0;
 
  int srat_disabled(void)
@@ -46,123 +45,6 @@ int srat_disabled(void)
      return numa_off || acpi_numa < 0;
  }
 
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{
-    unsigned long start_pfn, end_pfn;
-
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
-
-    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
-    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
-
-    node_set_online(nodeid);
-}
-
-void __init numa_init_array(void)
-{
-    int rr, i;
-
-    /* There are unfortunately some poorly designed mainboards around
-       that only connect memory to a single CPU. This breaks the 1:1 cpu->node
-       mapping. To avoid this fill in the mapping for all possible
-       CPUs, as the number of CPUs is not known yet.
-       We round robin the existing nodes. */
-    rr = first_node(node_online_map);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-    {
-        if ( cpu_to_node[i] != NUMA_NO_NODE )
-            continue;
-        numa_set_node(i, rr);
-        rr = cycle_node(rr, node_online_map);
-    }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
-
-/* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
-{
-    int i;
-    struct node nodes[MAX_NUMNODES];
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
-
-    /* Kludge needed for the hash function */
-    if ( hweight64(sz) > 1 )
-    {
-        u64 x = 1;
-        while ( (x << 1) < sz )
-            x <<= 1;
-        if ( x < sz/2 )
-            printk(KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n");
-        sz = x;
-    }
-
-    memset(&nodes,0,sizeof(nodes));
-    for ( i = 0; i < numa_fake; i++ )
-    {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
-        if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
-        nodes[i].end = nodes[i].start + sz;
-        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
-               i,
-               nodes[i].start, nodes[i].end,
-               (nodes[i].end - nodes[i].start) >> 20);
-        node_set_online(i);
-    }
-    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
-    if ( memnode_shift < 0 )
-    {
-        memnode_shift = 0;
-        printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
-        return -1;
-    }
-    for_each_online_node ( i )
-        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-    numa_init_array();
-
-    return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
-{
-    int i;
-
-#ifdef CONFIG_NUMA_EMU
-    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
-        return;
-#endif
-
-#ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
-        return;
-#endif
-
-    printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration found");
-
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
-    /* setup dummy node covering all memory */
-    memnode_shift = BITS_PER_LONG - 1;
-    memnodemap = _memnodemap;
-    memnodemapsize = ARRAY_SIZE(_memnodemap);
-
-    nodes_clear(node_online_map);
-    node_set_online(0);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-        numa_set_node(i, 0);
-    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
-}
-
  void numa_set_node(int cpu, nodeid_t node)
  {
      cpu_to_node[cpu] = node;
xen/common/numa.c +0 -0
index 1facc8fe2b..26c0006d04 100644

xen/common/numa.c line 16

Quoted 3 lines
xen/common/numa.c +0 -0
@@ -14,12 +14,13 @@
  #include <xen/smp.h>
  #include <xen/pfn.h>
  #include <xen/sched.h>

NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.

Quoted 21 lines
xen/common/numa.c +4 -1
+#include <asm/acpi.h>
 
  struct node_data node_data[MAX_NUMNODES];
 
  /* Mapping from pdx to node id */
  int memnode_shift;
-typeof(*memnodemap) _memnodemap[64];
+static typeof(*memnodemap) _memnodemap[64];
  unsigned long memnodemapsize;
  u8 *memnodemap;
 
@@ -34,6 +35,8 @@ int num_node_memblks;
  struct node node_memblk_range[NR_NODE_MEMBLKS];
  nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
+bool numa_off;
+
  /*
   * Given a shift value, try to populate memnodemap[]
   * Returns :

xen/common/numa.c void numa_add_cpu(int cpu) · line 199

Quoted 6 lines
xen/common/numa.c +3 -0
@@ -191,3 +194,120 @@ void numa_add_cpu(int cpu)
  {
      cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
  }
+
+/* initialize NODE_DATA given nodeid and start/end */
+void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)

From an abstract PoV, start and end should be paddr_t. This should be done on a separate patch though.

Quoted 26 lines
xen/common/numa.c +26 -0
+{
+    unsigned long start_pfn, end_pfn;
+
+    start_pfn = start >> PAGE_SHIFT;
+    end_pfn = end >> PAGE_SHIFT;
+
+    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
+    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
+
+    node_set_online(nodeid);
+}
+
+void __init numa_init_array(void)
+{
+    int rr, i;
+
+    /* There are unfortunately some poorly designed mainboards around
+       that only connect memory to a single CPU. This breaks the 1:1 cpu->node
+       mapping. To avoid this fill in the mapping for all possible
+       CPUs, as the number of CPUs is not known yet.
+       We round robin the existing nodes. */
+    rr = first_node(node_online_map);
+    for ( i = 0; i < nr_cpu_ids; i++ )
+    {
+        if ( cpu_to_node[i] != NUMA_NO_NODE )
+            continue;

xen/common/numa.c void numa_add_cpu(int cpu) · line 235

Quoted 10 lines
xen/common/numa.c +10 -0
+        numa_set_node(i, rr);
+        rr = cycle_node(rr, node_online_map);
+    }
+}
+
+#ifdef CONFIG_NUMA_EMU
+int numa_fake __initdata = 0;
+
+/* Numa emulation */
+static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Here, this should be either "unsigned long" or ideally "mfn_t". Although, if you use "unsigned long", you will need to...

xen/common/numa.c void numa_add_cpu(int cpu) · line 239

Quoted 4 lines
xen/common/numa.c +4 -0
+{
+    int i;
+    struct node nodes[MAX_NUMNODES];
+    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;

... cast "(end_pfn - start_pfn)" to uin64_t or use pfn_to_paddr().

Quoted 42 lines
xen/common/numa.c +42 -0
+
+    /* Kludge needed for the hash function */
+    if ( hweight64(sz) > 1 )
+    {
+        u64 x = 1;
+        while ( (x << 1) < sz )
+            x <<= 1;
+        if ( x < sz/2 )
+            printk(KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n");
+        sz = x;
+    }
+
+    memset(&nodes,0,sizeof(nodes));
+    for ( i = 0; i < numa_fake; i++ )
+    {
+        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        if ( i == numa_fake - 1 )
+            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+        nodes[i].end = nodes[i].start + sz;
+        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
+               i,
+               nodes[i].start, nodes[i].end,
+               (nodes[i].end - nodes[i].start) >> 20);
+        node_set_online(i);
+    }
+    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
+    if ( memnode_shift < 0 )
+    {
+        memnode_shift = 0;
+        printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
+        return -1;
+    }
+    for_each_online_node ( i )
+        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
+    numa_init_array();
+
+    return 0;
+}
+#endif
+
+void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
+{

xen/common/numa.c void numa_add_cpu(int cpu) · line 291

Quoted 10 lines
xen/common/numa.c +10 -0
+    int i;
+
+#ifdef CONFIG_NUMA_EMU
+    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
+        return;
+#endif
+
+#ifdef CONFIG_ACPI_NUMA
+    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
+         (u64)end_pfn << PAGE_SHIFT) )

(u64)v << PAGE_SHIFT should be switched to use pfn_to_paddr() or mfn_to_paddr() if you decide to make start_pfn and end_pfn typesafe.

xen/common/numa.c void numa_add_cpu(int cpu) · line 300

Quoted 9 lines
xen/common/numa.c +9 -0
+        return;
+#endif
+
+    printk(KERN_INFO "%s\n",
+           numa_off ? "NUMA turned off" : "No NUMA configuration found");
+
+    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
+           (u64)start_pfn << PAGE_SHIFT,
+           (u64)end_pfn << PAGE_SHIFT);

Same remark here. PRIx64 would also have to be switched to PRIpaddr.

Quoted 63 lines
xen/common/numa.c +13 -0
+    /* setup dummy node covering all memory */
+    memnode_shift = BITS_PER_LONG - 1;
+    memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);
+
+    nodes_clear(node_online_map);
+    node_set_online(0);
+    for ( i = 0; i < nr_cpu_ids; i++ )
+        numa_set_node(i, 0);
+    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
+    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
+                    (u64)end_pfn << PAGE_SHIFT);
+}
xen/include/asm-x86/numa.h +0 -5
index e8a92ad9df..f8e4e15586 100644
@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
-extern void numa_init_array(void);
-extern bool numa_off;
-
-
  extern int srat_disabled(void);
  extern void numa_set_node(int cpu, nodeid_t node);
  extern nodeid_t setup_node(unsigned int pxm);
  extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
  extern nodeid_t apicid_to_node[];
  extern void init_cpu_to_node(void);
 
xen/include/asm-x86/setup.h +0 -1
index 24be46115d..63838ba2d1 100644
@@ -17,7 +17,6 @@ void early_time_init(void);
 
  void set_nr_cpu_ids(unsigned int max_cpus);
 
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
  void arch_init_memory(void);
  void subarch_init_memory(void);
 
xen/include/xen/numa.h +0 -1
index 67b79a73a3..258a5cb3db 100644
@@ -26,7 +26,6 @@
  extern int memnode_shift;
  extern unsigned long memnodemapsize;
  extern u8 *memnodemap;
-extern typeof(*memnodemap) _memnodemap[64];
 
  struct node_data {
      unsigned long node_start_pfn;
@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);
  extern void cutoff_node(int i, u64 start, u64 end);
  extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 

xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80

Quoted 10 lines
xen/include/xen/numa.h +7 -0
+extern void numa_init_array(void);
+extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+extern bool numa_off;
+extern int numa_fake;
+
+extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+
  #endif /* CONFIG_NUMA */
 
  #endif /* _XEN_NUMA_H */

Cheers,

Signature · 1 line
Julien Grall
author
#50 lore raw

RE: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common

Hi Julien,

Quoted 17 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common

Hi Wei,

Quoted 5 lines · Wei Chen

This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem, numa_init_array and numa_emulation.

Quoted 7 lines
Quoted 3 lines

As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.

As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.

Yes, thanks!

Quoted 155 lines
Quoted 36 lines
---
  xen/arch/x86/numa.c         | 118 ----------------------------------
  xen/common/numa.c           | 122 +++++++++++++++++++++++++++++++++++-
  xen/include/asm-x86/numa.h  |   5 --
  xen/include/asm-x86/setup.h |   1 -
  xen/include/xen/numa.h      |   8 ++-
  5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -11
index f2626b3968..6908738305 100644
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
 
  nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-bool numa_off;
  s8 acpi_numa = 0;
 
  int srat_disabled(void)
@@ -46,123 +45,6 @@ int srat_disabled(void)
      return numa_off || acpi_numa < 0;
  }
 
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{
-    unsigned long start_pfn, end_pfn;
-
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
-
-    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
-    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/arch/x86/numa.c int srat_disabled(void) · line 68

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-
-    node_set_online(nodeid);
-}
-
-void __init numa_init_array(void)
-{
-    int rr, i;
-
-    /* There are unfortunately some poorly designed mainboards around
-       that only connect memory to a single CPU. This breaks the 1:1

cpu->node

Quoted 21 lines
xen/arch/x86/numa.c +0 -21
-       mapping. To avoid this fill in the mapping for all possible
-       CPUs, as the number of CPUs is not known yet.
-       We round robin the existing nodes. */
-    rr = first_node(node_online_map);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-    {
-        if ( cpu_to_node[i] != NUMA_NO_NODE )
-            continue;
-        numa_set_node(i, rr);
-        rr = cycle_node(rr, node_online_map);
-    }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
-
-/* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
-{
-    int i;
-    struct node nodes[MAX_NUMNODES];

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
-
-    /* Kludge needed for the hash function */
-    if ( hweight64(sz) > 1 )
-    {
-        u64 x = 1;
-        while ( (x << 1) < sz )
-            x <<= 1;
-        if ( x < sz/2 )
-            printk(KERN_ERR "Numa emulation unbalanced. Complain to

maintainer\n");

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 11 lines
xen/arch/x86/numa.c +0 -11
-        sz = x;
-    }
-
-    memset(&nodes,0,sizeof(nodes));
-    for ( i = 0; i < numa_fake; i++ )
-    {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
-        if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
-        nodes[i].end = nodes[i].start + sz;
-        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"

(%"PRIu64"MB)\n",

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-               i,
-               nodes[i].start, nodes[i].end,
-               (nodes[i].end - nodes[i].start) >> 20);
-        node_set_online(i);
-    }
-    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
-    if ( memnode_shift < 0 )
-    {
-        memnode_shift = 0;
-        printk(KERN_ERR "No NUMA hash function found. Emulation

disabled.\n");

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 11 lines
xen/arch/x86/numa.c +0 -11
-        return -1;
-    }
-    for_each_online_node ( i )
-        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-    numa_init_array();
-
-    return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 6 lines
xen/arch/x86/numa.c +0 -6
-{
-    int i;
-
-#ifdef CONFIG_NUMA_EMU
-    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
-        return;

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-#endif
-
-#ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
-        return;
-#endif
-
-    printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration

found");

Quoted 24 lines
xen/arch/x86/numa.c +0 -18
-
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
-    /* setup dummy node covering all memory */
-    memnode_shift = BITS_PER_LONG - 1;
-    memnodemap = _memnodemap;
-    memnodemapsize = ARRAY_SIZE(_memnodemap);
-
-    nodes_clear(node_online_map);
-    node_set_online(0);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-        numa_set_node(i, 0);
-    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
-}
-
  void numa_set_node(int cpu, nodeid_t node)
  {
      cpu_to_node[cpu] = node;
xen/common/numa.c +0 -1
index 1facc8fe2b..26c0006d04 100644
--- a/xen/common/numa.c
Quoted 9 lines

xen/common/numa.c line 16

Quoted 5 lines

+++ b/xen/common/numa.c

xen/common/numa.c +0 -0
@@ -14,12 +14,13 @@
  #include <xen/smp.h>
  #include <xen/pfn.h>
  #include <xen/sched.h>

NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.

got it

Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2 -1
+#include <asm/acpi.h>
 
  struct node_data node_data[MAX_NUMNODES];
 
  /* Mapping from pdx to node id */
  int memnode_shift;
-typeof(*memnodemap) _memnodemap[64];
+static typeof(*memnodemap) _memnodemap[64];
  unsigned long memnodemapsize;
  u8 *memnodemap;
 
@@ -34,6 +35,8 @@ int num_node_memblks;
  struct node node_memblk_range[NR_NODE_MEMBLKS];
  nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
Quoted 16 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 199

Quoted 12 lines
xen/common/numa.c +5 -0
+bool numa_off;
+
  /*
   * Given a shift value, try to populate memnodemap[]
   * Returns :
@@ -191,3 +194,120 @@ void numa_add_cpu(int cpu)
  {
      cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
  }
+
+/* initialize NODE_DATA given nodeid and start/end */
+void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)

From an abstract PoV, start and end should be paddr_t. This should be
done on a separate patch though.

Ok.

Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8 -0
+{
+    unsigned long start_pfn, end_pfn;
+
+    start_pfn = start >> PAGE_SHIFT;
+    end_pfn = end >> PAGE_SHIFT;
+
+    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
+    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 10 lines
xen/common/numa.c +10 -0
+
+    node_set_online(nodeid);
+}
+
+void __init numa_init_array(void)
+{
+    int rr, i;
+
+    /* There are unfortunately some poorly designed mainboards around
+       that only connect memory to a single CPU. This breaks the 1:1

cpu->node

Quoted 13 lines
xen/common/numa.c +13 -0
+       mapping. To avoid this fill in the mapping for all possible
+       CPUs, as the number of CPUs is not known yet.
+       We round robin the existing nodes. */
+    rr = first_node(node_online_map);
+    for ( i = 0; i < nr_cpu_ids; i++ )
+    {
+        if ( cpu_to_node[i] != NUMA_NO_NODE )
+            continue;
+        numa_set_node(i, rr);
+        rr = cycle_node(rr, node_online_map);
+    }
+}
+
Quoted 9 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 235

Quoted 5 lines
xen/common/numa.c +5 -0
+#ifdef CONFIG_NUMA_EMU
+int numa_fake __initdata = 0;
+
+/* Numa emulation */
+static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Here, this should be either "unsigned long" or ideally "mfn_t". Although, if you use "unsigned long", you will need to...

Do we need a separate patch to do it?

Quoted 7 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 239

Quoted 4 lines
xen/common/numa.c +4 -0
+{
+    int i;
+    struct node nodes[MAX_NUMNODES];
+    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;

... cast "(end_pfn - start_pfn)" to uin64_t or use pfn_to_paddr().

Ok

Quoted 45 lines

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 9 lines
xen/common/numa.c +9 -0
+
+    /* Kludge needed for the hash function */
+    if ( hweight64(sz) > 1 )
+    {
+        u64 x = 1;
+        while ( (x << 1) < sz )
+            x <<= 1;
+        if ( x < sz/2 )
+            printk(KERN_ERR "Numa emulation unbalanced. Complain to

maintainer\n");

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 11 lines
xen/common/numa.c +11 -0
+        sz = x;
+    }
+
+    memset(&nodes,0,sizeof(nodes));
+    for ( i = 0; i < numa_fake; i++ )
+    {
+        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        if ( i == numa_fake - 1 )
+            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+        nodes[i].end = nodes[i].start + sz;
+        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"

(%"PRIu64"MB)\n",

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 10 lines
xen/common/numa.c +10 -0
+               i,
+               nodes[i].start, nodes[i].end,
+               (nodes[i].end - nodes[i].start) >> 20);
+        node_set_online(i);
+    }
+    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
+    if ( memnode_shift < 0 )
+    {
+        memnode_shift = 0;
+        printk(KERN_ERR "No NUMA hash function found. Emulation

disabled.\n");

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 11 lines
xen/common/numa.c +11 -0
+        return -1;
+    }
+    for_each_online_node ( i )
+        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
+    numa_init_array();
+
+    return 0;
+}
+#endif
+
+void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 15 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 291

Quoted 11 lines
xen/common/numa.c +11 -0
+{
+    int i;
+
+#ifdef CONFIG_NUMA_EMU
+    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
+        return;
+#endif
+
+#ifdef CONFIG_ACPI_NUMA
+    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
+         (u64)end_pfn << PAGE_SHIFT) )

(u64)v << PAGE_SHIFT should be switched to use pfn_to_paddr() or mfn_to_paddr() if you decide to make start_pfn and end_pfn typesafe.

Still need a separate patch to change it before move?

Quoted 6 lines

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 5 lines
xen/common/numa.c +5 -0
+        return;
+#endif
+
+    printk(KERN_INFO "%s\n",
+           numa_off ? "NUMA turned off" : "No NUMA configuration

found");

Quoted 7 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 300

Quoted 4 lines
xen/common/numa.c +4 -0
+
+    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
+           (u64)start_pfn << PAGE_SHIFT,
+           (u64)end_pfn << PAGE_SHIFT);

Same remark here. PRIx64 would also have to be switched to PRIpaddr.

Hmm, It seems I'd better to use a separate patch to do PRIpaddr clean up before move code.

Quoted 79 lines
Quoted 59 lines
xen/common/numa.c +13 -0
+    /* setup dummy node covering all memory */
+    memnode_shift = BITS_PER_LONG - 1;
+    memnodemap = _memnodemap;
+    memnodemapsize = ARRAY_SIZE(_memnodemap);
+
+    nodes_clear(node_online_map);
+    node_set_online(0);
+    for ( i = 0; i < nr_cpu_ids; i++ )
+        numa_set_node(i, 0);
+    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
+    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
+                    (u64)end_pfn << PAGE_SHIFT);
+}
xen/include/asm-x86/numa.h +0 -5
index e8a92ad9df..f8e4e15586 100644
@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
-extern void numa_init_array(void);
-extern bool numa_off;
-
-
  extern int srat_disabled(void);
  extern void numa_set_node(int cpu, nodeid_t node);
  extern nodeid_t setup_node(unsigned int pxm);
  extern void srat_detect_node(int cpu);
 
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
  extern nodeid_t apicid_to_node[];
  extern void init_cpu_to_node(void);
 
xen/include/asm-x86/setup.h +0 -1
index 24be46115d..63838ba2d1 100644
@@ -17,7 +17,6 @@ void early_time_init(void);
 
  void set_nr_cpu_ids(unsigned int max_cpus);
 
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
  void arch_init_memory(void);
  void subarch_init_memory(void);
 
xen/include/xen/numa.h +0 -1
index 67b79a73a3..258a5cb3db 100644
@@ -26,7 +26,6 @@
  extern int memnode_shift;
  extern unsigned long memnodemapsize;
  extern u8 *memnodemap;
-extern typeof(*memnodemap) _memnodemap[64];
 
  struct node_data {
      unsigned long node_start_pfn;

xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 72

Quoted 5 lines
xen/include/xen/numa.h +2 -0
@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);
  extern void cutoff_node(int i, u64 start, u64 end);
  extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
+extern void numa_init_array(void);
+extern void numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn);

xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80

Quoted 9 lines
xen/include/xen/numa.h +5 -0
+extern bool numa_off;
+extern int numa_fake;
+
+extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+
  #endif /* CONFIG_NUMA */
 
  #endif /* _XEN_NUMA_H */

Cheers,

--
Julien Grall

#51 lore raw

Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 270 lines
Quoted 17 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common

Hi Wei,

Quoted 5 lines · Wei Chen

This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem, numa_init_array and numa_emulation.

Quoted 7 lines
Quoted 3 lines

As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.

As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.

Yes, thanks!

Quoted 155 lines
Quoted 36 lines
---
   xen/arch/x86/numa.c         | 118 ----------------------------------
   xen/common/numa.c           | 122 +++++++++++++++++++++++++++++++++++-
   xen/include/asm-x86/numa.h  |   5 --
   xen/include/asm-x86/setup.h |   1 -
   xen/include/xen/numa.h      |   8 ++-
   5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -11
index f2626b3968..6908738305 100644
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
 
   nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-bool numa_off;
   s8 acpi_numa = 0;
 
   int srat_disabled(void)
@@ -46,123 +45,6 @@ int srat_disabled(void)
       return numa_off || acpi_numa < 0;
   }
 
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{
-    unsigned long start_pfn, end_pfn;
-
-    start_pfn = start >> PAGE_SHIFT;
-    end_pfn = end >> PAGE_SHIFT;
-
-    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
-    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/arch/x86/numa.c int srat_disabled(void) · line 68

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-
-    node_set_online(nodeid);
-}
-
-void __init numa_init_array(void)
-{
-    int rr, i;
-
-    /* There are unfortunately some poorly designed mainboards around
-       that only connect memory to a single CPU. This breaks the 1:1

cpu->node

Quoted 21 lines
xen/arch/x86/numa.c +0 -21
-       mapping. To avoid this fill in the mapping for all possible
-       CPUs, as the number of CPUs is not known yet.
-       We round robin the existing nodes. */
-    rr = first_node(node_online_map);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-    {
-        if ( cpu_to_node[i] != NUMA_NO_NODE )
-            continue;
-        numa_set_node(i, rr);
-        rr = cycle_node(rr, node_online_map);
-    }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
-
-/* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
-{
-    int i;
-    struct node nodes[MAX_NUMNODES];

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
-
-    /* Kludge needed for the hash function */
-    if ( hweight64(sz) > 1 )
-    {
-        u64 x = 1;
-        while ( (x << 1) < sz )
-            x <<= 1;
-        if ( x < sz/2 )
-            printk(KERN_ERR "Numa emulation unbalanced. Complain to

maintainer\n");

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 11 lines
xen/arch/x86/numa.c +0 -11
-        sz = x;
-    }
-
-    memset(&nodes,0,sizeof(nodes));
-    for ( i = 0; i < numa_fake; i++ )
-    {
-        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
-        if ( i == numa_fake - 1 )
-            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
-        nodes[i].end = nodes[i].start + sz;
-        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"

(%"PRIu64"MB)\n",

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-               i,
-               nodes[i].start, nodes[i].end,
-               (nodes[i].end - nodes[i].start) >> 20);
-        node_set_online(i);
-    }
-    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
-    if ( memnode_shift < 0 )
-    {
-        memnode_shift = 0;
-        printk(KERN_ERR "No NUMA hash function found. Emulation

disabled.\n");

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 11 lines
xen/arch/x86/numa.c +0 -11
-        return -1;
-    }
-    for_each_online_node ( i )
-        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
-    numa_init_array();
-
-    return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 6 lines
xen/arch/x86/numa.c +0 -6
-{
-    int i;
-
-#ifdef CONFIG_NUMA_EMU
-    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
-        return;

xen/arch/x86/numa.c int srat_disabled(void)

Quoted 10 lines
xen/arch/x86/numa.c +0 -10
-#endif
-
-#ifdef CONFIG_ACPI_NUMA
-    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
-         (u64)end_pfn << PAGE_SHIFT) )
-        return;
-#endif
-
-    printk(KERN_INFO "%s\n",
-           numa_off ? "NUMA turned off" : "No NUMA configuration

found");

Quoted 24 lines
xen/arch/x86/numa.c +0 -18
-
-    printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
-           (u64)start_pfn << PAGE_SHIFT,
-           (u64)end_pfn << PAGE_SHIFT);
-    /* setup dummy node covering all memory */
-    memnode_shift = BITS_PER_LONG - 1;
-    memnodemap = _memnodemap;
-    memnodemapsize = ARRAY_SIZE(_memnodemap);
-
-    nodes_clear(node_online_map);
-    node_set_online(0);
-    for ( i = 0; i < nr_cpu_ids; i++ )
-        numa_set_node(i, 0);
-    cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
-    setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
-                    (u64)end_pfn << PAGE_SHIFT);
-}
-
   void numa_set_node(int cpu, nodeid_t node)
   {
       cpu_to_node[cpu] = node;
xen/common/numa.c +0 -1
index 1facc8fe2b..26c0006d04 100644
--- a/xen/common/numa.c
Quoted 9 lines

xen/common/numa.c line 16

Quoted 5 lines

+++ b/xen/common/numa.c

xen/common/numa.c +0 -0
@@ -14,12 +14,13 @@
   #include <xen/smp.h>
   #include <xen/pfn.h>
   #include <xen/sched.h>

NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.

got it

Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2 -1
+#include <asm/acpi.h>
 
   struct node_data node_data[MAX_NUMNODES];
 
   /* Mapping from pdx to node id */
   int memnode_shift;
-typeof(*memnodemap) _memnodemap[64];
+static typeof(*memnodemap) _memnodemap[64];
   unsigned long memnodemapsize;
   u8 *memnodemap;
 
@@ -34,6 +35,8 @@ int num_node_memblks;
   struct node node_memblk_range[NR_NODE_MEMBLKS];
   nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
 
Quoted 16 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 199

Quoted 12 lines
xen/common/numa.c +5 -0
+bool numa_off;
+
   /*
    * Given a shift value, try to populate memnodemap[]
    * Returns :
@@ -191,3 +194,120 @@ void numa_add_cpu(int cpu)
   {
       cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
   }
+
+/* initialize NODE_DATA given nodeid and start/end */
+void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)

From an abstract PoV, start and end should be paddr_t. This should be
done on a separate patch though.

Ok.

Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8 -0
+{
+    unsigned long start_pfn, end_pfn;
+
+    start_pfn = start >> PAGE_SHIFT;
+    end_pfn = end >> PAGE_SHIFT;
+
+    NODE_DATA(nodeid)->node_start_pfn = start_pfn;
+    NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 10 lines
xen/common/numa.c +10 -0
+
+    node_set_online(nodeid);
+}
+
+void __init numa_init_array(void)
+{
+    int rr, i;
+
+    /* There are unfortunately some poorly designed mainboards around
+       that only connect memory to a single CPU. This breaks the 1:1

cpu->node

Quoted 13 lines
xen/common/numa.c +13 -0
+       mapping. To avoid this fill in the mapping for all possible
+       CPUs, as the number of CPUs is not known yet.
+       We round robin the existing nodes. */
+    rr = first_node(node_online_map);
+    for ( i = 0; i < nr_cpu_ids; i++ )
+    {
+        if ( cpu_to_node[i] != NUMA_NO_NODE )
+            continue;
+        numa_set_node(i, rr);
+        rr = cycle_node(rr, node_online_map);
+    }
+}
+
Quoted 9 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 235

Quoted 5 lines
xen/common/numa.c +5 -0
+#ifdef CONFIG_NUMA_EMU
+int numa_fake __initdata = 0;
+
+/* Numa emulation */
+static int __init numa_emulation(u64 start_pfn, u64 end_pfn)

Here, this should be either "unsigned long" or ideally "mfn_t". Although, if you use "unsigned long", you will need to...

Quoted 1 line

Do we need a separate patch to do it?

I would prefer if the cleanups are done separately as this makes easier to review code movement.

Quoted 72 lines
Quoted 8 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 239

Quoted 5 lines
xen/common/numa.c +4 -0
+{
+    int i;
+    struct node nodes[MAX_NUMNODES];
+    u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;

... cast "(end_pfn - start_pfn)" to uin64_t or use pfn_to_paddr().

Ok

Quoted 45 lines

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 9 lines
xen/common/numa.c +9 -0
+
+    /* Kludge needed for the hash function */
+    if ( hweight64(sz) > 1 )
+    {
+        u64 x = 1;
+        while ( (x << 1) < sz )
+            x <<= 1;
+        if ( x < sz/2 )
+            printk(KERN_ERR "Numa emulation unbalanced. Complain to

maintainer\n");

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 11 lines
xen/common/numa.c +11 -0
+        sz = x;
+    }
+
+    memset(&nodes,0,sizeof(nodes));
+    for ( i = 0; i < numa_fake; i++ )
+    {
+        nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+        if ( i == numa_fake - 1 )
+            sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+        nodes[i].end = nodes[i].start + sz;
+        printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"

(%"PRIu64"MB)\n",

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 10 lines
xen/common/numa.c +10 -0
+               i,
+               nodes[i].start, nodes[i].end,
+               (nodes[i].end - nodes[i].start) >> 20);
+        node_set_online(i);
+    }
+    memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
+    if ( memnode_shift < 0 )
+    {
+        memnode_shift = 0;
+        printk(KERN_ERR "No NUMA hash function found. Emulation

disabled.\n");

xen/common/numa.c void numa_add_cpu(int cpu)

Quoted 11 lines
xen/common/numa.c +11 -0
+        return -1;
+    }
+    for_each_online_node ( i )
+        setup_node_bootmem(i, nodes[i].start, nodes[i].end);
+    numa_init_array();
+
+    return 0;
+}
+#endif
+
+void __init numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn)

Quoted 15 lines

xen/common/numa.c void numa_add_cpu(int cpu) · line 291

Quoted 11 lines
xen/common/numa.c +11 -0
+{
+    int i;
+
+#ifdef CONFIG_NUMA_EMU
+    if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
+        return;
+#endif
+
+#ifdef CONFIG_ACPI_NUMA
+    if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
+         (u64)end_pfn << PAGE_SHIFT) )

(u64)v << PAGE_SHIFT should be switched to use pfn_to_paddr() or mfn_to_paddr() if you decide to make start_pfn and end_pfn typesafe.

Quoted 1 line

Still need a separate patch to change it before move?

Yes.

Cheers,

Signature · 1 line
Julien Grall
author
#52 lore raw

RFC PATCH 13/40 xen/arm: introduce numa_set_node for Arm

In reply to Wei Chen

This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.

---
 xen/arch/arm/Makefile      |  1 +
 xen/arch/arm/numa.c        | 31 +++++++++++++++++++++++++++++++
 xen/include/asm-arm/numa.h |  2 ++
 xen/include/asm-x86/numa.h |  1 -
 xen/include/xen/numa.h     |  1 +
 5 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/arm/numa.c
xen/arch/arm/Makefile +1 -0
index 3d3b97b5b4..6e3fb8033e 100644
@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o
 obj-y += mem_access.o
 obj-y += mm.o
 obj-y += monitor.o
+obj-$(CONFIG_NUMA) += numa.o
 obj-y += p2m.o
 obj-y += percpu.o
 obj-y += platform.o
xen/arch/arm/numa.c new file +31 -0
new file mode 100644
index 0000000000..1e30c5bb13
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+void numa_set_node(int cpu, nodeid_t nid)
+{
+    if ( nid >= MAX_NUMNODES ||
+        !nodemask_test(nid, &node_online_map) )
+        nid = 0;
+
+    cpu_to_node[cpu] = nid;
+}
xen/include/asm-arm/numa.h +2 -0
index ab9c4a2448..1162c702df 100644
@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#define numa_set_node(x, y) do { } while (0)
+
 #endif
 
 #endif /* __ARCH_ARM_NUMA_H */
xen/include/asm-x86/numa.h +0 -1
index f8e4e15586..69859b0a57 100644
@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
 extern int srat_disabled(void);
-extern void numa_set_node(int cpu, nodeid_t node);
 extern nodeid_t setup_node(unsigned int pxm);
 extern void srat_detect_node(int cpu);
 
xen/include/xen/numa.h +1 -0
index 258a5cb3db..3972aa6b93 100644
@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
 extern void numa_init_array(void);
 extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+extern void numa_set_node(int cpu, nodeid_t node);
 extern bool numa_off;
 extern int numa_fake;
 
Signature · 1 line
2.25.1
#53 lore raw

Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Wei,

Quoted 6 lines · Wei Chen

This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.

IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.

From the commit message, I don't quite understand why the check is necessary on Arm but not on x86. Can you clarify it?

Quoted 50 lines
---
  xen/arch/arm/Makefile      |  1 +
  xen/arch/arm/numa.c        | 31 +++++++++++++++++++++++++++++++
  xen/include/asm-arm/numa.h |  2 ++
  xen/include/asm-x86/numa.h |  1 -
  xen/include/xen/numa.h     |  1 +
  5 files changed, 35 insertions(+), 1 deletion(-)

create mode 100644 xen/arch/arm/numa.c

xen/arch/arm/Makefile +1 -0
index 3d3b97b5b4..6e3fb8033e 100644
@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o
  obj-y += mem_access.o
  obj-y += mm.o
  obj-y += monitor.o
+obj-$(CONFIG_NUMA) += numa.o
  obj-y += p2m.o
  obj-y += percpu.o
  obj-y += platform.o
xen/arch/arm/numa.c new file +21 -0
new file mode 100644
index 0000000000..1e30c5bb13
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>

xen/arch/arm/numa.c line 31

Quoted 10 lines
xen/arch/arm/numa.c new file +10 -0
+#include <xen/numa.h>
+
+void numa_set_node(int cpu, nodeid_t nid)
+{
+    if ( nid >= MAX_NUMNODES ||
+        !nodemask_test(nid, &node_online_map) )
+        nid = 0;
+
+    cpu_to_node[cpu] = nid;
+}

I think numa_set_node() will want to be implemented in common code.

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30

Quoted 9 lines
xen/include/asm-arm/numa.h +1 -0
index ab9c4a2448..1162c702df 100644
@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;
  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
  #define __node_distance(a, b) (20)
 
+#define numa_set_node(x, y) do { } while (0)

I would define it in xen/numa.h so other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.

Quoted 22 lines
xen/include/asm-arm/numa.h +1 -0
+
  #endif
 
  #endif /* __ARCH_ARM_NUMA_H */
xen/include/asm-x86/numa.h +0 -1
index f8e4e15586..69859b0a57 100644
@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
  extern int srat_disabled(void);
-extern void numa_set_node(int cpu, nodeid_t node);
  extern nodeid_t setup_node(unsigned int pxm);
  extern void srat_detect_node(int cpu);
 
xen/include/xen/numa.h +0 -0
index 258a5cb3db..3972aa6b93 100644

xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node); · line 76

Quoted 7 lines
xen/include/xen/numa.h +1 -0
@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
 
  extern void numa_init_array(void);
  extern void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+extern void numa_set_node(int cpu, nodeid_t node);
  extern bool numa_off;
  extern int numa_fake;
 

Cheers,

Signature · 1 line
Julien Grall
author
#54 lore raw

RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Julien,

Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Wei,

Quoted 6 lines · Wei Chen

This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.

Quoted 6 lines

IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.

From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?

Yes, in patch#27, in smpboot.c, dt_smp_init_cpus function. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invoke numa_set_node to create CPU-NODE map. But in our testing, we found when NUMA init failed, numa_set_node still can set CPU to a offline or invalid NODE. So we're using node_online_map to prevent this behavior. Otherwise we have to check node_online_map everywhere before we call numa_set_node.

x86 actually is doing the same way, but it handles node_online_map check out of numa_set_node:

57 void __init init_cpu_to_node(void)
58 {
59 unsigned int i;
60 nodeid_t node;
61
62 for ( i = 0; i < nr_cpu_ids; i++ )
63 {
64 u32 apicid = x86_cpu_to_apicid[i];
65 if ( apicid == BAD_APICID )
66 continue;
67 node = apicid < MAX_LOCAL_APIC ? apicid_to_node[apicid] : NUMA_NO_NODE;
68 if ( node == NUMA_NO_NODE || !node_online(node) )
69 node = 0;
70 numa_set_node(i, node);
71 }
72 }

Quoted 53 lines
Quoted 53 lines
---
  xen/arch/arm/Makefile      |  1 +
  xen/arch/arm/numa.c        | 31 +++++++++++++++++++++++++++++++
  xen/include/asm-arm/numa.h |  2 ++
  xen/include/asm-x86/numa.h |  1 -
  xen/include/xen/numa.h     |  1 +
  5 files changed, 35 insertions(+), 1 deletion(-)

create mode 100644 xen/arch/arm/numa.c

xen/arch/arm/Makefile +1 -0
index 3d3b97b5b4..6e3fb8033e 100644
@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o
  obj-y += mem_access.o
  obj-y += mm.o
  obj-y += monitor.o
+obj-$(CONFIG_NUMA) += numa.o
  obj-y += p2m.o
  obj-y += percpu.o
  obj-y += platform.o
xen/arch/arm/numa.c new file +24 -0
new file mode 100644
index 0000000000..1e30c5bb13
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+void numa_set_node(int cpu, nodeid_t nid)
Quoted 9 lines

xen/arch/arm/numa.c line 31

Quoted 7 lines
xen/arch/arm/numa.c new file +7 -0
+{
+    if ( nid >= MAX_NUMNODES ||
+        !nodemask_test(nid, &node_online_map) )
+        nid = 0;
+
+    cpu_to_node[cpu] = nid;
+}

I think numa_set_node() will want to be implemented in common code.

See my above comment. If x86 is ok, I think yes, we can do it in common code.

Quoted 8 lines
Quoted 8 lines
xen/include/asm-arm/numa.h +0 -0
index ab9c4a2448..1162c702df 100644
@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;
  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
  #define __node_distance(a, b) (20)
 
Quoted 5 lines

xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30

Quoted 1 line
xen/include/asm-arm/numa.h +1 -0
+#define numa_set_node(x, y) do { } while (0)

I would define it in xen/numa.h so other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.

Ok

Quoted 36 lines

xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end,

Quoted 21 lines
xen/include/asm-arm/numa.h +1 -0
+
  #endif
 
  #endif /* __ARCH_ARM_NUMA_H */
xen/include/asm-x86/numa.h +0 -1
index f8e4e15586..69859b0a57 100644
@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);
  #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
 
  extern int srat_disabled(void);
-extern void numa_set_node(int cpu, nodeid_t node);
  extern nodeid_t setup_node(unsigned int pxm);
  extern void srat_detect_node(int cpu);
 
xen/include/xen/numa.h +0 -0
index 258a5cb3db..3972aa6b93 100644
@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end,

nodeid_t node);

xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, · line 71

Quoted 3 lines
xen/include/xen/numa.h +0 -0
  extern void numa_init_array(void);
  extern void numa_initmem_init(unsigned long start_pfn, unsigned long

end_pfn);

xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node);

Quoted 5 lines
xen/include/xen/numa.h +1 -0
+extern void numa_set_node(int cpu, nodeid_t node);
  extern bool numa_off;
  extern int numa_fake;
 
 

Cheers,

--
Julien Grall

#55 lore raw

Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 26 lines
Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Wei,

Quoted 6 lines · Wei Chen

This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.

Quoted 6 lines

IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.

From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?

Quoted 8 lines

Yes, in patch#27, in smpboot.c, dt_smp_init_cpus function. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invoke numa_set_node to create CPU-NODE map. But in our testing, we found when NUMA init failed, numa_set_node still can set CPU to a offline or invalid NODE. So we're using node_online_map to prevent this behavior. Otherwise we have to check node_online_map everywhere before we call numa_set_node.

What do you mean by invalid NODE? Is it 0xFF (NUMA_NO_NODE)?

Quoted 3 lines

x86 actually is doing the same way, but it handles node_online_map check out of numa_set_node:

Right...

Quoted 5 lines
Quoted 2 lines

I think numa_set_node() will want to be implemented in common code.

See my above comment. If x86 is ok, I think yes, we can do it in common code.

... on x86, this check is performed outside of numa_set_node() for one caller whereas on Arm you are adding it in numa_set_node().

For example, numa_set_node() can be called with NUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.

This will change the behavior of users to cpu_to_node() later on (such as XEN_SYSCTL_cputopoinfo).

NUMA is not something architecture specific, so I dont't think the implementation should differ here.

In this case, I think numa_set_node() shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.

Cheers,

Signature · 1 line
Julien Grall
author
#56 lore raw

RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Julien,

Quoted 51 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 26 lines
Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm

Hi Wei,

Quoted 6 lines · Wei Chen

This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.

Quoted 6 lines

IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.

From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?

Quoted 8 lines

Yes, in patch#27, in smpboot.c, dt_smp_init_cpus function. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invoke numa_set_node to create CPU-NODE map. But in our testing, we found when NUMA init failed, numa_set_node still can set CPU to a offline or invalid NODE. So we're using node_online_map to prevent this behavior. Otherwise we have to check node_online_map everywhere before we call numa_set_node.

Quoted 1 line

What do you mean by invalid NODE? Is it 0xFF (NUMA_NO_NODE)?

No, I mean some wrong content in device tree. For example, if the dtb set a wrong numa-node-id in CPU dt-node.

Quoted 20 lines
Quoted 4 lines

x86 actually is doing the same way, but it handles node_online_map check out of numa_set_node:

Right...

Quoted 5 lines
Quoted 2 lines

I think numa_set_node() will want to be implemented in common code.

See my above comment. If x86 is ok, I think yes, we can do it in common code.

... on x86, this check is performed outside of numa_set_node() for one caller whereas on Arm you are adding it in numa_set_node().

For example, numa_set_node() can be called with NUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.

Quoted 9 lines

This will change the behavior of users to cpu_to_node() later on (such as XEN_SYSCTL_cputopoinfo).

NUMA is not something architecture specific, so I dont't think the implementation should differ here.

In this case, I think numa_set_node() shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.

Yes, I agree. I will change it in next version.

Quoted 4 lines

Cheers,

--
Julien Grall

author
#57 lore raw

RFC PATCH 14/40 xen/arm: set NUMA nodes max number to 64 by default

In reply to Wei Chen

Today's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.

---
 xen/include/asm-arm/numa.h | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +9 -1
index 1162c702df..b2982f9053 100644
@@ -5,7 +5,15 @@
 
 typedef u8 nodeid_t;
 
-#if !defined(CONFIG_NUMA)
+#if defined(CONFIG_NUMA)
+
+/*
+ * Same as x86, we set the max number of NUMA nodes to 64 and
+ * set the number of NUMA memory block number to 128.
+ */
+#define NODES_SHIFT      6
+
+#else
 
 /* Fake one node for now. See also node_online_map. */
 #define cpu_to_node(cpu) 0
Signature · 1 line
2.25.1
#58 lore raw

Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default

Hi Wei,

Quoted 13 lines · Wei Chen

Today's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.

---
  xen/include/asm-arm/numa.h | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +0 -0
index 1162c702df..b2982f9053 100644

xen/include/asm-arm/numa.h line 13

Quoted 10 lines
xen/include/asm-arm/numa.h +6 -1
@@ -5,7 +5,15 @@
 
  typedef u8 nodeid_t;
 
-#if !defined(CONFIG_NUMA)
+#if defined(CONFIG_NUMA)
+
+/*
+ * Same as x86, we set the max number of NUMA nodes to 64 and
+ * set the number of NUMA memory block number to 128.
+ */

Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to xen/numa.h.

xen/include/asm-arm/numa.h line 19

Quoted 7 lines
xen/include/asm-arm/numa.h +3 -0
+#define NODES_SHIFT      6
+
+#else
 
  /* Fake one node for now. See also node_online_map. */
  #define cpu_to_node(cpu) 0

Cheers,

Signature · 1 line
Julien Grall
#59 lore raw

Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default

Quoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0 -0
@@ -5,7 +5,15 @@
 
  typedef u8 nodeid_t;
 
Quoted 11 lines

xen/include/asm-arm/numa.h line 13

Quoted 7 lines
xen/include/asm-arm/numa.h +6 -1
-#if !defined(CONFIG_NUMA)
+#if defined(CONFIG_NUMA)
+
+/*
+ * Same as x86, we set the max number of NUMA nodes to 64 and
+ * set the number of NUMA memory block number to 128.
+ */

Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to xen/numa.h.

To be honest - if this gets moved, please at least consider making it a proper Kconfig setting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...

Jan

author
#60 lore raw

RE: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default

Hi Jan, Julien,

Quoted 29 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月25日 21:36
To: Julien Grall <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default

Quoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0 -0
@@ -5,7 +5,15 @@
 
  typedef u8 nodeid_t;
 
Quoted 11 lines

xen/include/asm-arm/numa.h line 13

Quoted 7 lines
xen/include/asm-arm/numa.h +6 -1
-#if !defined(CONFIG_NUMA)
+#if defined(CONFIG_NUMA)
+
+/*
+ * Same as x86, we set the max number of NUMA nodes to 64 and
+ * set the number of NUMA memory block number to 128.
+ */

Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to xen/numa.h.

Quoted 5 lines

To be honest - if this gets moved, please at least consider making it a proper Kconfig setting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...

Ok

Quoted 1 line

Jan

author
#61 lore raw

RFC PATCH 15/40 xen/x86: move NUMA API from x86 header to common header

In reply to Wei Chen

As all functions and macros that depends on these NUMA API, like clear_node_cpumask, have been moved to common header file. In this case, we can move NUMA API from x86 header file to common header file, and will not trigger the symbols not found error when functions in arch NUMA header file depends on the symbols in common NUMA header file.

---
 xen/include/asm-x86/numa.h | 13 -------------
 xen/include/xen/numa.h     | 13 +++++++++++++
 2 files changed, 13 insertions(+), 13 deletions(-)
xen/include/asm-x86/numa.h +0 -13
index 69859b0a57..5a57a51e26 100644
@@ -9,14 +9,6 @@ typedef u8 nodeid_t;
 
 extern int srat_rev;
 
-extern nodeid_t      cpu_to_node[NR_CPUS];
-extern cpumask_t     node_to_cpumask[];
-
-#define cpu_to_node(cpu)		(cpu_to_node[cpu])
-#define parent_node(node)		(node)
-#define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
-#define node_to_cpumask(node)    (node_to_cpumask[node])
-
 extern nodeid_t pxm_to_node(unsigned int pxm);
 
 #define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -28,11 +20,6 @@ extern void srat_detect_node(int cpu);
 extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
-static inline void clear_node_cpumask(int cpu)
-{
-	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-}
-
 void srat_parse_regions(u64 addr);
 extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
xen/include/xen/numa.h +13 -0
index 3972aa6b93..cb08d2eca9 100644
@@ -22,6 +22,14 @@
 /* The following content can be used when NUMA feature is enabled */
 #if defined(CONFIG_NUMA)
 
+extern nodeid_t      cpu_to_node[NR_CPUS];
+extern cpumask_t     node_to_cpumask[];
+
+#define cpu_to_node(cpu)		(cpu_to_node[cpu])
+#define parent_node(node)		(node)
+#define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
+#define node_to_cpumask(node)    (node_to_cpumask[node])
+
 /* Simple perfect hash to map pdx to node numbers */
 extern int memnode_shift;
 extern unsigned long memnodemapsize;
@@ -76,6 +84,11 @@ extern int numa_fake;
 
 extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
 
+static inline void clear_node_cpumask(int cpu)
+{
+	cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+}
+
 #endif /* CONFIG_NUMA */
 
 #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
author
#62 lore raw

RFC PATCH 16/40 xen/arm: Create a fake NUMA node to use common code

In reply to Wei Chen

When CONFIG_NUMA is enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, like cpu_to_node, as single node NUMA system.

So in this patch, we introduce a numa_init function for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.

---
 xen/arch/arm/numa.c        | 53 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/setup.c       |  8 ++++++
 xen/include/asm-arm/numa.h | 11 ++++++++
 3 files changed, 72 insertions(+)
xen/arch/arm/numa.c +53 -0
index 1e30c5bb13..566ad1e52b 100644
@@ -20,6 +20,8 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/pfn.h>
+#include <asm/setup.h>
 
 void numa_set_node(int cpu, nodeid_t nid)
 {
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
 
     cpu_to_node[cpu] = nid;
 }
+
+void __init numa_init(bool acpi_off)
+{
+    uint32_t idx;
+    paddr_t ram_start = ~0;
+    paddr_t ram_size = 0;
+    paddr_t ram_end = 0;
+
+    printk(XENLOG_WARNING
+        "NUMA has not been supported yet, NUMA off!\n");
+    /* Arm NUMA has not been implemented until this patch */
+    numa_off = true;
+
+    /*
+     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
+     * function return 0 as previous fake cpu_to_node API.
+     */
+    for ( idx = 0; idx < NR_CPUS; idx++ )
+        cpu_to_node[idx] = 0;
+
+    /*
+     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
+     * return as previous fake APIs.
+     */
+    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
+        node_to_cpumask[idx] = cpu_online_map;
+        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
+        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    }
+
+    /*
+     * Find the minimal and maximum address of RAM, NUMA will
+     * build a memory to node mapping table for the whole range.
+     */
+    ram_start = bootinfo.mem.bank[0].start;
+    ram_size  = bootinfo.mem.bank[0].size;
+    ram_end   = ram_start + ram_size;
+    for ( idx = 1 ; idx < bootinfo.mem.nr_banks; idx++ )
+    {
+        paddr_t bank_start = bootinfo.mem.bank[idx].start;
+        paddr_t bank_size = bootinfo.mem.bank[idx].size;
+        paddr_t bank_end = bank_start + bank_size;
+
+        ram_size  = ram_size + bank_size;
+        ram_start = min(ram_start, bank_start);
+        ram_end   = max(ram_end, bank_end);
+    }
+
+    numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
+    return;
+}
xen/arch/arm/setup.c +8 -0
index 63a908e325..3c58d2d441 100644
@@ -30,6 +30,7 @@
 #include <xen/init.h>
 #include <xen/irq.h>
 #include <xen/mm.h>
+#include <xen/numa.h>
 #include <xen/param.h>
 #include <xen/softirq.h>
 #include <xen/keyhandler.h>
@@ -874,6 +875,13 @@ void __init start_xen(unsigned long boot_phys_offset,
     /* Parse the ACPI tables for possible boot-time configuration */
     acpi_boot_table_init();
 
+    /*
+     * Try to initialize NUMA system, if failed, the system will
+     * fallback to uniform system which means system has only 1
+     * NUMA node.
+     */
+    numa_init(acpi_disabled);
+
     end_boot_allocator();
 
     /*
xen/include/asm-arm/numa.h +11 -0
index b2982f9053..bb495a24e1 100644
@@ -13,6 +13,16 @@ typedef u8 nodeid_t;
  */
 #define NODES_SHIFT      6
 
+extern void numa_init(bool acpi_off);
+
+/*
+ * Temporary for fake NUMA node, when CPU, memory and distance
+ * matrix will be read from DTB or ACPI SRAT. The following
+ * symbols will be removed.
+ */
+extern mfn_t first_valid_mfn;
+#define __node_distance(a, b) (20)
+
 #else
 
 /* Fake one node for now. See also node_online_map. */
@@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#define numa_init(x) do { } while (0)
 #define numa_set_node(x, y) do { } while (0)
 
 #endif
Signature · 1 line
2.25.1
#63 lore raw

Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

Quoted 35 lines · Wei Chen

When CONFIG_NUMA is enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, like cpu_to_node, as single node NUMA system.

So in this patch, we introduce a numa_init function for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.

---
 xen/arch/arm/numa.c        | 53 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/setup.c       |  8 ++++++
 xen/include/asm-arm/numa.h | 11 ++++++++
 3 files changed, 72 insertions(+)
xen/arch/arm/numa.c +2 -0
index 1e30c5bb13..566ad1e52b 100644
@@ -20,6 +20,8 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/pfn.h>
+#include <asm/setup.h>
 
 void numa_set_node(int cpu, nodeid_t nid)
 {
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
 
     cpu_to_node[cpu] = nid;
 }

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+void __init numa_init(bool acpi_off)
+{
+    uint32_t idx;
+    paddr_t ram_start = ~0;
+    paddr_t ram_size = 0;
+    paddr_t ram_end = 0;
+
+    printk(XENLOG_WARNING
+        "NUMA has not been supported yet, NUMA off!\n");

NIT: please align

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44

Quoted 1 line
xen/arch/arm/numa.c +1 -0
+    /* Arm NUMA has not been implemented until this patch */

"Arm NUMA is not implemented yet"

Quoted 8 lines
xen/arch/arm/numa.c +8 -0
+    numa_off = true;
+
+    /*
+     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
+     * function return 0 as previous fake cpu_to_node API.
+     */
+    for ( idx = 0; idx < NR_CPUS; idx++ )
+        cpu_to_node[idx] = 0;

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+    /*
+     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
+     * return as previous fake APIs.
+     */
+    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
+        node_to_cpumask[idx] = cpu_online_map;
+        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
+        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    }

I just want to note that this works because MAX_NUMNODES is 1. If MAX_NUMNODES was > 1 then it would be wrong to set node_to_cpumask, node_spanned_pages and node_start_pfn for all nodes to the same values.

It might be worth writing something about it in the in-code comment.

Quoted 4 lines
xen/arch/arm/numa.c +4 -0
+    /*
+     * Find the minimal and maximum address of RAM, NUMA will
+     * build a memory to node mapping table for the whole range.
+     */

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 77

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+    ram_start = bootinfo.mem.bank[0].start;
+    ram_size  = bootinfo.mem.bank[0].size;
+    ram_end   = ram_start + ram_size;
+    for ( idx = 1 ; idx < bootinfo.mem.nr_banks; idx++ )
+    {
+        paddr_t bank_start = bootinfo.mem.bank[idx].start;
+        paddr_t bank_size = bootinfo.mem.bank[idx].size;
+        paddr_t bank_end = bank_start + bank_size;
+
+        ram_size  = ram_size + bank_size;

ram_size is updated but not utilized

Quoted 65 lines
xen/arch/arm/numa.c +7 -0
+        ram_start = min(ram_start, bank_start);
+        ram_end   = max(ram_end, bank_end);
+    }
+
+    numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
+    return;
+}
xen/arch/arm/setup.c +8 -0
index 63a908e325..3c58d2d441 100644
@@ -30,6 +30,7 @@
 #include <xen/init.h>
 #include <xen/irq.h>
 #include <xen/mm.h>
+#include <xen/numa.h>
 #include <xen/param.h>
 #include <xen/softirq.h>
 #include <xen/keyhandler.h>
@@ -874,6 +875,13 @@ void __init start_xen(unsigned long boot_phys_offset,
     /* Parse the ACPI tables for possible boot-time configuration */
     acpi_boot_table_init();
 
+    /*
+     * Try to initialize NUMA system, if failed, the system will
+     * fallback to uniform system which means system has only 1
+     * NUMA node.
+     */
+    numa_init(acpi_disabled);
+
     end_boot_allocator();
 
     /*
xen/include/asm-arm/numa.h +11 -0
index b2982f9053..bb495a24e1 100644
@@ -13,6 +13,16 @@ typedef u8 nodeid_t;
  */
 #define NODES_SHIFT      6
 
+extern void numa_init(bool acpi_off);
+
+/*
+ * Temporary for fake NUMA node, when CPU, memory and distance
+ * matrix will be read from DTB or ACPI SRAT. The following
+ * symbols will be removed.
+ */
+extern mfn_t first_valid_mfn;
+#define __node_distance(a, b) (20)
+
 #else
 
 /* Fake one node for now. See also node_online_map. */
@@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#define numa_init(x) do { } while (0)
 #define numa_set_node(x, y) do { } while (0)
 
 #endif
Signature · 1 line
2.25.1
author
#64 lore raw

RE: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

Hi Stefano,

Quoted 43 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:10
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

Quoted 33 lines · Wei Chen

When CONFIG_NUMA is enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, like cpu_to_node, as single node NUMA system.

So in this patch, we introduce a numa_init function for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.

---
 xen/arch/arm/numa.c        | 53 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/setup.c       |  8 ++++++
 xen/include/asm-arm/numa.h | 11 ++++++++
 3 files changed, 72 insertions(+)
xen/arch/arm/numa.c +2 -0
index 1e30c5bb13..566ad1e52b 100644
@@ -20,6 +20,8 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/pfn.h>
+#include <asm/setup.h>
 
 void numa_set_node(int cpu, nodeid_t nid)
 {
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
 
Quoted 15 lines

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43

Quoted 12 lines
xen/arch/arm/numa.c +10 -0
     cpu_to_node[cpu] = nid;
 }
+
+void __init numa_init(bool acpi_off)
+{
+    uint32_t idx;
+    paddr_t ram_start = ~0;
+    paddr_t ram_size = 0;
+    paddr_t ram_end = 0;
+
+    printk(XENLOG_WARNING
+        "NUMA has not been supported yet, NUMA off!\n");

NIT: please align

OK

Quoted 5 lines

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44

Quoted 2 lines
xen/arch/arm/numa.c +1 -0
+    /* Arm NUMA has not been implemented until this patch */

"Arm NUMA is not implemented yet"

OK

Quoted 24 lines
Quoted 9 lines
xen/arch/arm/numa.c +8 -0
+    numa_off = true;
+
+    /*
+     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
+     * function return 0 as previous fake cpu_to_node API.
+     */
+    for ( idx = 0; idx < NR_CPUS; idx++ )
+        cpu_to_node[idx] = 0;

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+    /*
+     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
+     * return as previous fake APIs.
+     */
+    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
+        node_to_cpumask[idx] = cpu_online_map;
+        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
+        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    }

I just want to note that this works because MAX_NUMNODES is 1. If MAX_NUMNODES was > 1 then it would be wrong to set node_to_cpumask, node_spanned_pages and node_start_pfn for all nodes to the same values.

Quoted 2 lines

It might be worth writing something about it in the in-code comment.

OK, I will do it.

Quoted 16 lines
Quoted 5 lines
xen/arch/arm/numa.c +4 -0
+    /*
+     * Find the minimal and maximum address of RAM, NUMA will
+     * build a memory to node mapping table for the whole range.
+     */

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 77

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+    ram_start = bootinfo.mem.bank[0].start;
+    ram_size  = bootinfo.mem.bank[0].size;
+    ram_end   = ram_start + ram_size;
+    for ( idx = 1 ; idx < bootinfo.mem.nr_banks; idx++ )
+    {
+        paddr_t bank_start = bootinfo.mem.bank[idx].start;
+        paddr_t bank_size = bootinfo.mem.bank[idx].size;
+        paddr_t bank_end = bank_start + bank_size;
+
+        ram_size  = ram_size + bank_size;
Quoted 2 lines

ram_size is updated but not utilized

Ok, I will remove it.

Quoted 67 lines

xen/arch/arm/setup.c void __init start_xen(unsigned long

Quoted 21 lines
xen/arch/arm/numa.c +7 -0
+        ram_start = min(ram_start, bank_start);
+        ram_end   = max(ram_end, bank_end);
+    }
+
+    numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
+    return;
+}
xen/arch/arm/setup.c +1 -0
index 63a908e325..3c58d2d441 100644
@@ -30,6 +30,7 @@
 #include <xen/init.h>
 #include <xen/irq.h>
 #include <xen/mm.h>
+#include <xen/numa.h>
 #include <xen/param.h>
 #include <xen/softirq.h>
 #include <xen/keyhandler.h>
@@ -874,6 +875,13 @@ void __init start_xen(unsigned long

boot_phys_offset,

Quoted 45 lines
xen/arch/arm/setup.c +7 -0
     /* Parse the ACPI tables for possible boot-time configuration */
     acpi_boot_table_init();
 
+    /*
+     * Try to initialize NUMA system, if failed, the system will
+     * fallback to uniform system which means system has only 1
+     * NUMA node.
+     */
+    numa_init(acpi_disabled);
+
     end_boot_allocator();
 
     /*
xen/include/asm-arm/numa.h +11 -1
index b2982f9053..bb495a24e1 100644
@@ -13,6 +13,16 @@ typedef u8 nodeid_t;
  */
 #define NODES_SHIFT      6
 
+extern void numa_init(bool acpi_off);
+
+/*
+ * Temporary for fake NUMA node, when CPU, memory and distance
+ * matrix will be read from DTB or ACPI SRAT. The following
+ * symbols will be removed.
+ */
+extern mfn_t first_valid_mfn;
+#define __node_distance(a, b) (20)
+
 #else
 
 /* Fake one node for now. See also node_online_map. */
@@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
+#define numa_init(x) do { } while (0)
 #define numa_set_node(x, y) do { } while (0)
 
 #endif
--

2.25.1

#65 lore raw

Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

In reply to Stefano Stabellini
Quoted 47 lines · Stefano Stabellini
Quoted 4 lines · Wei Chen
xen/arch/arm/numa.c +0 -0
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
 
     cpu_to_node[cpu] = nid;
 }

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+void __init numa_init(bool acpi_off)
+{
+    uint32_t idx;
+    paddr_t ram_start = ~0;
+    paddr_t ram_size = 0;
+    paddr_t ram_end = 0;
+
+    printk(XENLOG_WARNING
+        "NUMA has not been supported yet, NUMA off!\n");

NIT: please align

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44

Quoted 1 line
xen/arch/arm/numa.c +1 -0
+    /* Arm NUMA has not been implemented until this patch */

"Arm NUMA is not implemented yet"

Quoted 8 lines
xen/arch/arm/numa.c +8 -0
+    numa_off = true;
+
+    /*
+     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
+     * function return 0 as previous fake cpu_to_node API.
+     */
+    for ( idx = 0; idx < NR_CPUS; idx++ )
+        cpu_to_node[idx] = 0;

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+    /*
+     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
+     * return as previous fake APIs.
+     */
+    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
+        node_to_cpumask[idx] = cpu_online_map;
+        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
+        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    }

I just want to note that this works because MAX_NUMNODES is 1. If MAX_NUMNODES was > 1 then it would be wrong to set node_to_cpumask, node_spanned_pages and node_start_pfn for all nodes to the same values.

Quoted 1 line

It might be worth writing something about it in the in-code comment.

Plus perhaps BUILD_BUG_ON(MAX_NUMNODES != 1), so the issue is actually noticed at build time once the constant gets changed?

Jan

author
#66 lore raw

RE: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

Hi Jan,

Quoted 58 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月27日 14:18
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code

Quoted 47 lines · Stefano Stabellini
Quoted 4 lines · Wei Chen
xen/arch/arm/numa.c +0 -0
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
 
     cpu_to_node[cpu] = nid;
 }

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+void __init numa_init(bool acpi_off)
+{
+    uint32_t idx;
+    paddr_t ram_start = ~0;
+    paddr_t ram_size = 0;
+    paddr_t ram_end = 0;
+
+    printk(XENLOG_WARNING
+        "NUMA has not been supported yet, NUMA off!\n");

NIT: please align

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44

Quoted 1 line
xen/arch/arm/numa.c +1 -0
+    /* Arm NUMA has not been implemented until this patch */

"Arm NUMA is not implemented yet"

Quoted 8 lines
xen/arch/arm/numa.c +8 -0
+    numa_off = true;
+
+    /*
+     * Set all cpu_to_node mapping to 0, this will make cpu_to_node
+     * function return 0 as previous fake cpu_to_node API.
+     */
+    for ( idx = 0; idx < NR_CPUS; idx++ )
+        cpu_to_node[idx] = 0;

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 10 lines
xen/arch/arm/numa.c +10 -0
+
+    /*
+     * Make node_to_cpumask, node_spanned_pages and node_start_pfn
+     * return as previous fake APIs.
+     */
+    for ( idx = 0; idx < MAX_NUMNODES; idx++ ) {
+        node_to_cpumask[idx] = cpu_online_map;
+        node_spanned_pages(idx) = (max_page - mfn_x(first_valid_mfn));
+        node_start_pfn(idx) = (mfn_x(first_valid_mfn));
+    }

I just want to note that this works because MAX_NUMNODES is 1. If MAX_NUMNODES was > 1 then it would be wrong to set node_to_cpumask, node_spanned_pages and node_start_pfn for all nodes to the same values.

Quoted 5 lines
Quoted 1 line

It might be worth writing something about it in the in-code comment.

Plus perhaps BUILD_BUG_ON(MAX_NUMNODES != 1), so the issue is actually noticed at build time once the constant gets changed?

It would be better. I will use it in next version.

Quoted 1 line

Jan

author
#67 lore raw

RFC PATCH 17/40 xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

In reply to Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
 xen/arch/arm/Kconfig | 10 ++++++++++
 1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +10 -0
index ecfa6822e4..678cc98ea3 100644
@@ -33,6 +33,16 @@ config ACPI
 	  Advanced Configuration and Power Interface (ACPI) support for Xen is
 	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA
+	bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED
+	depends on ARM_64
+	select NUMA
+	---help---
+
+	  Non-Uniform Memory Access (NUMA) is a computer memory design used in
+	  multiprocessing, where the memory access time depends on the memory
+	  location relative to the processor.
+
 config GICV3
 	bool "GICv3 driver"
 	depends on ARM_64 && !NEW_VGIC
Signature · 1 line
2.25.1
#68 lore raw

Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi,

Quoted 14 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
  xen/arch/arm/Kconfig | 10 ++++++++++
  1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644

xen/arch/arm/Kconfig config ACPI · line 36

Quoted 4 lines
xen/arch/arm/Kconfig +1 -0
@@ -33,6 +33,16 @@ config ACPI
  	  Advanced Configuration and Power Interface (ACPI) support for Xen is
  	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.

So we should have a generic config that will then select DT (and ACPI in the future).

xen/arch/arm/Kconfig config ACPI · line 45

Quoted 9 lines
xen/arch/arm/Kconfig +9 -0
+	bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED
+	depends on ARM_64
+	select NUMA
+	---help---
+
+	  Non-Uniform Memory Access (NUMA) is a computer memory design used in
+	  multiprocessing, where the memory access time depends on the memory
+	  location relative to the processor.
+

Cheers,

Signature · 1 line
Julien Grall
author
#69 lore raw

RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi Julien,

Quoted 35 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi,

Quoted 14 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
  xen/arch/arm/Kconfig | 10 ++++++++++
  1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644

xen/arch/arm/Kconfig config ACPI · line 33

Quoted 1 line
xen/arch/arm/Kconfig +0 -0
@@ -33,6 +33,16 @@ config ACPI
  	  Advanced Configuration and Power Interface (ACPI) support for Xen

is

xen/arch/arm/Kconfig config ACPI · line 36

Quoted 3 lines
xen/arch/arm/Kconfig +1 -0
  	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

Quoted 7 lines

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.

So we should have a generic config that will then select DT (and ACPI in the future).

How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?

If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?

Quoted 17 lines

xen/arch/arm/Kconfig config ACPI · line 37

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if

UNSUPPORTED

xen/arch/arm/Kconfig config ACPI

Quoted 5 lines
xen/arch/arm/Kconfig +5 -0
+	depends on ARM_64
+	select NUMA
+	---help---
+
+	  Non-Uniform Memory Access (NUMA) is a computer memory design used

in

xen/arch/arm/Kconfig config ACPI · line 43

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+	  multiprocessing, where the memory access time depends on the

memory

xen/arch/arm/Kconfig config ACPI · line 45

Quoted 2 lines
xen/arch/arm/Kconfig +2 -0
+	  location relative to the processor.
+

Cheers,

--
Julien Grall

#70 lore raw

Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 39 lines
Quoted 39 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi,

Quoted 14 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
   xen/arch/arm/Kconfig | 10 ++++++++++
   1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644

xen/arch/arm/Kconfig config ACPI · line 33

Quoted 1 line
xen/arch/arm/Kconfig +0 -0
@@ -33,6 +33,16 @@ config ACPI
   	  Advanced Configuration and Power Interface (ACPI) support for Xen

is

xen/arch/arm/Kconfig config ACPI · line 36

Quoted 3 lines
xen/arch/arm/Kconfig +1 -0
   	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.

Quoted 6 lines
Quoted 3 lines

So we should have a generic config that will then select DT (and ACPI in the future).

How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?

I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.

Quoted 4 lines

If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?

I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?

Cheers,

Signature · 1 line
Julien Grall
author
#71 lore raw

RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi Julien,

Quoted 56 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 21 lines
Quoted 21 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi,

Quoted 9 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
   xen/arch/arm/Kconfig | 10 ++++++++++
   1 file changed, 10 insertions(+)
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/Kconfig config ACPI · line 33

Quoted 6 lines
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644
@@ -33,6 +33,16 @@ config ACPI
   	  Advanced Configuration and Power Interface (ACPI) support

for Xen

Quoted 5 lines
Quoted 5 lines

is

Quoted 3 lines
xen/arch/arm/Kconfig +1 -0
   	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA
Quoted 8 lines
Quoted 8 lines

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.

So we should have a generic config that will then select DT (and ACPI

in

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

the future).

How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?

I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.

Ok. I agree with this.

Quoted 8 lines
Quoted 4 lines

If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?

I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?

No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.

Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already have CONFIG_NUMA in common layer. So we need to add another one for Arm like CONFIG_ARM_NUMA?
And in this option, we can select CONFIG_DEVICE_TREE_NUMA automatically if device tree is enabled. If CONFIG_ACPI is enabled, we will select CONFIG_ACPI_NUMA too (in the future)
In Xen code, DT_NUMA and ACPI_NUMA code can co-exist, Xen will check the system ACPI support status to decide to use DT_NUMA or ACPI_NUMA?

Quoted 4 lines

Cheers,

--
Julien Grall

#72 lore raw

Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 78 lines
Quoted 57 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 21 lines
Quoted 21 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi,

Quoted 9 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
    xen/arch/arm/Kconfig | 10 ++++++++++
    1 file changed, 10 insertions(+)
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/Kconfig config ACPI · line 33

Quoted 6 lines
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644
@@ -33,6 +33,16 @@ config ACPI
    	  Advanced Configuration and Power Interface (ACPI) support

for Xen

Quoted 5 lines
Quoted 5 lines

is

Quoted 3 lines
xen/arch/arm/Kconfig +1 -0
    	  an alternative to device tree on ARM64.
 
+config DEVICE_TREE_NUMA
Quoted 8 lines
Quoted 8 lines

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.

So we should have a generic config that will then select DT (and ACPI

in

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

the future).

How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?

I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.

Ok. I agree with this.

Quoted 8 lines
Quoted 4 lines

If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?

I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?

Quoted 7 lines

No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.

Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already have CONFIG_NUMA in common layer. So we need to add another one for Arm like CONFIG_ARM_NUMA?

I think so.

Quoted 5 lines

And in this option, we can select CONFIG_DEVICE_TREE_NUMA automatically if device tree is enabled. If CONFIG_ACPI is enabled, we will select CONFIG_ACPI_NUMA too (in the future)
In Xen code, DT_NUMA and ACPI_NUMA code can co-exist, Xen

Distributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.

Quoted 2 lines

will check the system ACPI support status to decide to use DT_NUMA or ACPI_NUMA?

Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.

Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".

Cheers,

Signature · 1 line
Julien Grall
author
#73 lore raw

RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Hi Julien,

Quoted 118 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:29
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 15 lines
Quoted 15 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce

DEVICE_TREE_NUMA

Quoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines

Kconfig for arm64

Hi,

Quoted 9 lines · Wei Chen

We need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:
DEVICE_TREE_NUMA in this patch for Arm64.

---
    xen/arch/arm/Kconfig | 10 ++++++++++
    1 file changed, 10 insertions(+)
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/Kconfig config ACPI · line 33

Quoted 6 lines
xen/arch/arm/Kconfig +0 -0
index ecfa6822e4..678cc98ea3 100644
@@ -33,6 +33,16 @@ config ACPI
    	  Advanced Configuration and Power Interface (ACPI) support

for Xen

Quoted 3 lines
Quoted 3 lines

is

Quoted 2 lines
xen/arch/arm/Kconfig +0 -0
    	  an alternative to device tree on ARM64.
 
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

xen/arch/arm/Kconfig config ACPI · line 36

Quoted 1 line
xen/arch/arm/Kconfig +1 -0
+config DEVICE_TREE_NUMA

The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.

However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the

choice

Quoted 25 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

to enable/disable the parsing for DT/ACPI.

So we should have a generic config that will then select DT (and ACPI

in

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

the future).

How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?

I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.

Ok. I agree with this.

Quoted 8 lines
Quoted 4 lines

If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?

I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?

Quoted 7 lines

No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.

Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already have CONFIG_NUMA in common layer. So we need to add another one for Arm like CONFIG_ARM_NUMA?

I think so.

Quoted 5 lines

And in this option, we can select CONFIG_DEVICE_TREE_NUMA automatically if device tree is enabled. If CONFIG_ACPI is enabled, we will select CONFIG_ACPI_NUMA too (in the future)
In Xen code, DT_NUMA and ACPI_NUMA code can co-exist, Xen

Distributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.

Quoted 2 lines

will check the system ACPI support status to decide to use DT_NUMA or ACPI_NUMA?

Quoted 7 lines

Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.

Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".

I am glad we are now in the same page. Ok, I will change the Kconfig like this in next version.

Quoted 4 lines

Cheers,

--
Julien Grall

author
#74 lore raw

RFC PATCH 18/40 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

In reply to Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/asm-arm/setup.h |  6 ++++++
 3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells, &bootinfo.mem);
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -25
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
     int status;
     u32 fdt_val32;
     u64 fdt_val64;
-    int prev;
     int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
    /*
     * Delete all memory reserve map entries. When booting via UEFI,
     * kernel will use the UEFI memory map to find reserved regions.
xen/include/asm-arm/setup.h +6 -0
index c4b6af6029..e4fb5f0d49 100644
@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,
 u32 device_tree_get_u32(const void *fdt, int node,
                         const char *prop_name, u32 dflt);
 
+#if defined(CONFIG_ARM_64)
+#define arch_efi_enabled(x) efi_enabled(x)
+#else
+#define arch_efi_enabled(x) (0)
+#endif
+
 #endif
 /*
  * Local variables:
Signature · 1 line
2.25.1
#75 lore raw

Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Hi Wei,

Quoted 9 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

Quoted 6 lines

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to

s/prvent/prevent/

Quoted 25 lines

parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
  xen/arch/arm/bootfdt.c      |  8 +++++++-
  xen/arch/arm/efi/efi-boot.h | 25 -------------------------
  xen/include/asm-arm/setup.h |  6 ++++++
  3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
  #include <xen/lib.h>
  #include <xen/kernel.h>
  #include <xen/init.h>
+#include <xen/efi.h>
  #include <xen/device_tree.h>
  #include <xen/libfdt/libfdt.h>
  #include <xen/sort.h>

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340

Quoted 6 lines
xen/arch/arm/bootfdt.c +2 -1
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
  {
      int rc = 0;
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,

"If the system boot". Although, I would suggest to write:

"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344

Quoted 4 lines
xen/arch/arm/bootfdt.c +4 -0
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )

arch_efi_enabled() is going to be less expensive than device_tree_node_matches(). So I would suggest to re-order the operands.

Quoted 46 lines
xen/arch/arm/bootfdt.c +0 -0
          rc = process_memory_node(fdt, node, name, depth,
                                   address_cells, size_cells, &bootinfo.mem);
      else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -25
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
      int status;
      u32 fdt_val32;
      u64 fdt_val64;
-    int prev;
      int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
     /*
      * Delete all memory reserve map entries. When booting via UEFI,
      * kernel will use the UEFI memory map to find reserved regions.
xen/include/asm-arm/setup.h +0 -0
index c4b6af6029..e4fb5f0d49 100644

xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130

Quoted 8 lines
xen/include/asm-arm/setup.h +5 -0
@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,
  u32 device_tree_get_u32(const void *fdt, int node,
                          const char *prop_name, u32 dflt);
 
+#if defined(CONFIG_ARM_64)
+#define arch_efi_enabled(x) efi_enabled(x)
+#else
+#define arch_efi_enabled(x) (0)
+#endif

I would prefer if we introduce CONFIG_EFI that would stub efi_enabled for architecture not supporting EFI.

Cheers,

Signature · 1 line
Julien Grall
author
#76 lore raw

RE: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Hi Julien,

Quoted 21 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:35
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Hi Wei,

Quoted 9 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

Quoted 9 lines
Quoted 6 lines

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to

s/prvent/prevent/

Ok

Quoted 28 lines
Quoted 28 lines

parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
  xen/arch/arm/bootfdt.c      |  8 +++++++-
  xen/arch/arm/efi/efi-boot.h | 25 -------------------------
  xen/include/asm-arm/setup.h |  6 ++++++
  3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
  #include <xen/lib.h>
  #include <xen/kernel.h>
  #include <xen/init.h>
+#include <xen/efi.h>
  #include <xen/device_tree.h>
  #include <xen/libfdt/libfdt.h>
  #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
  {
      int rc = 0;
 
Quoted 9 lines

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340

Quoted 3 lines
xen/arch/arm/bootfdt.c +2 -1
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,

"If the system boot". Although, I would suggest to write:

"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."

Yes, that would be better. I will change it in next version.

Quoted 8 lines

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344

Quoted 4 lines
xen/arch/arm/bootfdt.c +4 -0
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )

arch_efi_enabled() is going to be less expensive than device_tree_node_matches(). So I would suggest to re-order the operands.

yes.

Quoted 52 lines

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 346

Quoted 2 lines
xen/arch/arm/bootfdt.c +0 -0
          rc = process_memory_node(fdt, node, name, depth,
                                   address_cells, size_cells,

&bootinfo.mem);

xen/arch/arm/efi/efi-boot.h EFI_STATUS __init

Quoted 6 lines
xen/arch/arm/bootfdt.c +0 -0
      else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
xen/arch/arm/efi/efi-boot.h +0 -0
index cf9c37153f..d0a9987fa4 100644
@@ -197,33 +197,8 @@ EFI_STATUS __init

fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,

xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32

Quoted 38 lines
xen/arch/arm/efi/efi-boot.h +0 -25
      int status;
      u32 fdt_val32;
      u64 fdt_val64;
-    int prev;
      int num_rsv;
 
-    /*
-     * Delete any memory nodes present.  The EFI memory map is the only
-     * memory description provided to Xen.
-     */
-    prev = 0;
-    for (;;)
-    {
-        const char *type;
-        int len;
-
-        node = fdt_next_node(fdt, prev, NULL);
-        if ( node < 0 )
-            break;
-
-        type = fdt_getprop(fdt, node, "device_type", &len);
-        if ( type && strncmp(type, "memory", len) == 0 )
-        {
-            fdt_del_node(fdt, node);
-            continue;
-        }
-
-        prev = node;
-    }
-
     /*
      * Delete all memory reserve map entries. When booting via UEFI,
      * kernel will use the UEFI memory map to find reserved regions.
xen/include/asm-arm/setup.h +0 -0
index c4b6af6029..e4fb5f0d49 100644
@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32

address_cells,

Quoted 3 lines
xen/include/asm-arm/setup.h +0 -0
  u32 device_tree_get_u32(const void *fdt, int node,
                          const char *prop_name, u32 dflt);
 
Quoted 9 lines

xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130

Quoted 5 lines
xen/include/asm-arm/setup.h +5 -0
+#if defined(CONFIG_ARM_64)
+#define arch_efi_enabled(x) efi_enabled(x)
+#else
+#define arch_efi_enabled(x) (0)
+#endif

I would prefer if we introduce CONFIG_EFI that would stub efi_enabled for architecture not supporting EFI.

Yes, that's a good idea.
I also feel a little awkward with the current arch helpers.

Quoted 4 lines

Cheers,

--
Julien Grall

#77 lore raw

Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

In reply to Wei Chen
Quoted 43 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/asm-arm/setup.h |  6 ++++++
 3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;
 

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347

Quoted 10 lines
xen/arch/arm/bootfdt.c +6 -1
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells, &bootinfo.mem);
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )

If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?

I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?

Reviewed-by Matt Fleming Acked-by Steve Capper
#78 lore raw

Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Hi Stefano,

Quoted 44 lines · Stefano Stabellini
Quoted 43 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
  xen/arch/arm/bootfdt.c      |  8 +++++++-
  xen/arch/arm/efi/efi-boot.h | 25 -------------------------
  xen/include/asm-arm/setup.h |  6 ++++++
  3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
  #include <xen/lib.h>
  #include <xen/kernel.h>
  #include <xen/init.h>
+#include <xen/efi.h>
  #include <xen/device_tree.h>
  #include <xen/libfdt/libfdt.h>
  #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
  {
      int rc = 0;
 
Quoted 15 lines

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347

Quoted 10 lines
xen/arch/arm/bootfdt.c +6 -1
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )
          rc = process_memory_node(fdt, node, name, depth,
                                   address_cells, size_cells, &bootinfo.mem);
      else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )

If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?

Yes. AFAIK, when booting using EFI, the Device-Tree may not contain all the reserved regions. Furthermore, we are still too early to know whether we boot using ACPI and DT.

Quoted 3 lines

I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?

I looked at Linux when I first reviewed this patch because I was wondering what happens if the DT and UEFI map disagrees.

Linux and Xen are the same after this patch:
1) The memory map is coming from UEFI map
2) NUMA ID is coming from the DT

The commit that introduced the change in Linux is:

commit 500899c2cc3e3f06140373b587a69d30650f2d9d

Author: Ard Biesheuvel <[email protected]>
Date:   Fri Apr 8 15:50:23 2016 -0700

efi: ARM/arm64: ignore DT memory nodes instead of removing them

There are two problems with the UEFI stub DT memory node removal routine:
- it deletes nodes as it traverses the tree, which happens to work but is not supported, as deletion invalidates the node iterator;
- deleting memory nodes entirely may discard annotations in the form of additional properties on the nodes.

Since the discovery of DT memory nodes occurs strictly before the UEFI init sequence, we can simply clear the memblock memory table before parsing the UEFI memory map. This way, it is no longer necessary to remove the nodes, so we can remove that logic from the stub as well.

Cheers,

Signature · 1 line
Julien Grall
#79 lore raw

Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 70 lines · Julien Grall

Hi Stefano,

Quoted 51 lines · Stefano Stabellini
Quoted 50 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
  xen/arch/arm/bootfdt.c      |  8 +++++++-
  xen/arch/arm/efi/efi-boot.h | 25 -------------------------
  xen/include/asm-arm/setup.h |  6 ++++++
  3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
  #include <xen/lib.h>
  #include <xen/kernel.h>
  #include <xen/init.h>
+#include <xen/efi.h>
  #include <xen/device_tree.h>
  #include <xen/libfdt/libfdt.h>
  #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
  {
      int rc = 0;
  -    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )
          rc = process_memory_node(fdt, node, name, depth,
Quoted 8 lines

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt,

Quoted 3 lines
xen/arch/arm/bootfdt.c +0 -0
                                   address_cells, size_cells,
 &bootinfo.mem);
      else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )

If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?

Yes. AFAIK, when booting using EFI, the Device-Tree may not contain all the reserved regions. Furthermore, we are still too early to know whether we boot using ACPI and DT.

Quoted 3 lines

I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?

Quoted 8 lines

I looked at Linux when I first reviewed this patch because I was wondering what happens if the DT and UEFI map disagrees.

Linux and Xen are the same after this patch:
1) The memory map is coming from UEFI map
2) NUMA ID is coming from the DT

The commit that introduced the change in Linux is:

[...]

Thanks both you and Wei for the investigation :-)

author
#80 lore raw

RE: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

In reply to Stefano Stabellini

Hi Stefano,

Quoted 66 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:25
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI

Quoted 42 lines · Wei Chen

EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.

But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.

So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.

As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.

As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.

---
 xen/arch/arm/bootfdt.c      |  8 +++++++-
 xen/arch/arm/efi/efi-boot.h | 25 -------------------------
 xen/include/asm-arm/setup.h |  6 ++++++
 3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +1 -0
index 476e32e0f5..7df149dbca 100644
@@ -11,6 +11,7 @@
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
+#include <xen/efi.h>
 #include <xen/device_tree.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt,
 {
     int rc = 0;

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 346

Quoted 10 lines
xen/arch/arm/bootfdt.c +6 -1
 
-    if ( device_tree_node_matches(fdt, node, "memory") )
+    /*
+     * If system boot from EFI, bootinfo.mem has been set by EFI,
+     * so we don't need to parse memory node from DTB.
+     */
+    if ( device_tree_node_matches(fdt, node, "memory") &&
+         !arch_efi_enabled(EFI_BOOT) )
         rc = process_memory_node(fdt, node, name, depth,
                                  address_cells, size_cells,

&bootinfo.mem);

xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347

Quoted 1 line
xen/arch/arm/bootfdt.c +0 -0
     else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )
Quoted 6 lines

If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?

I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?

We don't use device tree for memory map on EFI boot. We just reply on device tree to provide memory NUMA node info. Because EFI system table doesn't contain this kind of data.

I have a quick look into Linux. Linux efi stub has a update_fdt function in:
drivers/firmware/efi/libstub/fdt.c. In this function, efi stub only delete reserve memory nodes. Usable memory nodes will not be touched in this function.

Before Linux efi_init, early_init_dt_scan will be invoked. This means, if the efi stub doesn't remove the normal memory nodes, these nodes will be found and added to memblock by early_init_dt_add_memory_arch();

Later, in efi_init, sytemtable->memory_description will also be parsed, the memory block will be added to memblock by early_init_dt_add_memory_arch. So the duplicated memory nodes will be merged/ignored in memblock_add.

In Linux NUMA init, if ACPI is off, the NUMA info will be parsed from DT.

So I think in EFI boot, we use system table to parse memory, but use DT to parse NUMA info. It doesn't seem particularly strange : )

Cheers,
Wei Chen

author
#81 lore raw

RFC PATCH 19/40 xen: fdt: Introduce a helper to check fdt node type

In reply to Wei Chen

In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce fdt_node_check_type in this patch to avoid redundant code in subsequent patches.

---
 xen/common/libfdt/fdt_ro.c      | 15 +++++++++++++++
 xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
xen/common/libfdt/fdt_ro.c +15 -0
index 36f9b480d1..ae7794d870 100644
@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,
 		return 1;
 }
 
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type)
+{
+	const void *prop;
+	int len;
+
+	prop = fdt_getprop(fdt, nodeoffset, "device_type", &len);
+	if (!prop)
+		return len;
+	if (fdt_stringlist_contains(prop, len, type))
+		return 0;
+	else
+		return 1;
+}
+
 int fdt_node_offset_by_compatible(const void *fdt, int startoffset,
 				  const char *compatible)
 {
xen/include/xen/libfdt/libfdt.h +25 -0
index 7c75688a39..7e4930dbcd 100644
@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
 int fdt_node_check_compatible(const void *fdt, int nodeoffset,
 			      const char *compatible);
 
+/**
+ * fdt_node_check_type: check a node's device_type property
+ * @fdt: pointer to the device tree blob
+ * @nodeoffset: offset of a tree node
+ * @type: string to match against
+ *
+ *
+ * fdt_node_check_type() returns 0 if the given node contains a 'device_type'
+ * property with the given string as one of its elements, it returns non-zero
+ * otherwise, or on error.
+ *
+ * returns:
+ *	0, if the node has a 'device_type' property listing the given string
+ *	1, if the node has a 'device_type' property, but it does not list
+ *		the given string
+ *	-FDT_ERR_NOTFOUND, if the given node has no 'device_type' property
+ * 	-FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag
+ *	-FDT_ERR_BADMAGIC,
+ *	-FDT_ERR_BADVERSION,
+ *	-FDT_ERR_BADSTATE,
+ *	-FDT_ERR_BADSTRUCTURE, standard meanings
+ */
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type);
+
 /**
  * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value
  * @fdt: pointer to the device tree blob
Signature · 1 line
2.25.1
#82 lore raw

Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type

Hi Wei,

Quoted 9 lines · Wei Chen

In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce fdt_node_check_type in this patch to avoid redundant code in subsequent patches.

---
  xen/common/libfdt/fdt_ro.c      | 15 +++++++++++++++
  xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++

This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.

Quoted 10 lines
  2 files changed, 40 insertions(+)
xen/common/libfdt/fdt_ro.c +0 -0
index 36f9b480d1..ae7794d870 100644
@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,
  		return 1;
  }
 

xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557

Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10 -0
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type)
+{
+	const void *prop;
+	int len;
+
+	prop = fdt_getprop(fdt, nodeoffset, "device_type", &len);
+	if (!prop)
+		return len;
+	if (fdt_stringlist_contains(prop, len, type))

The "device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?

Quoted 35 lines
xen/common/libfdt/fdt_ro.c +5 -0
+		return 0;
+	else
+		return 1;
+}
+
  int fdt_node_offset_by_compatible(const void *fdt, int startoffset,
  				  const char *compatible)
  {
xen/include/xen/libfdt/libfdt.h +18 -0
index 7c75688a39..7e4930dbcd 100644
@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
  int fdt_node_check_compatible(const void *fdt, int nodeoffset,
  			      const char *compatible);
 
+/**
+ * fdt_node_check_type: check a node's device_type property
+ * @fdt: pointer to the device tree blob
+ * @nodeoffset: offset of a tree node
+ * @type: string to match against
+ *
+ *
+ * fdt_node_check_type() returns 0 if the given node contains a 'device_type'
+ * property with the given string as one of its elements, it returns non-zero
+ * otherwise, or on error.
+ *
+ * returns:
+ *	0, if the node has a 'device_type' property listing the given string
+ *	1, if the node has a 'device_type' property, but it does not list
+ *		the given string
+ *	-FDT_ERR_NOTFOUND, if the given node has no 'device_type' property
+ * 	-FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag
+ *	-FDT_ERR_BADMAGIC,

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829

Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +7 -0
+ *	-FDT_ERR_BADVERSION,
+ *	-FDT_ERR_BADSTATE,
+ *	-FDT_ERR_BADSTRUCTURE, standard meanings
+ */
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type);
+
  /**
   * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value
   * @fdt: pointer to the device tree blob

Cheers,

Signature · 1 line
Julien Grall
author
#83 lore raw

RE: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type

Hi Julien,

Quoted 18 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:39
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type

Hi Wei,

Quoted 5 lines · Wei Chen

In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce fdt_node_check_type in this patch to avoid redundant code in subsequent patches.

Quoted 9 lines
Quoted 4 lines
---
  xen/common/libfdt/fdt_ro.c      | 15 +++++++++++++++
  xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++

This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.

Oh, if we need to upstream this change in libfdt. I think I'd better to remove this change in libfdt. Because we can implement type checking in other place, and I don't want to introduce a dependency on external repo upstream in this series.

Quoted 11 lines

xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int

Quoted 7 lines
  2 files changed, 40 insertions(+)
xen/common/libfdt/fdt_ro.c +0 -0
index 36f9b480d1..ae7794d870 100644
@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int

nodeoffset,

Quoted 3 lines
xen/common/libfdt/fdt_ro.c +0 -0
  		return 1;
  }
 
Quoted 15 lines

xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557

Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10 -0
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type)
+{
+	const void *prop;
+	int len;
+
+	prop = fdt_getprop(fdt, nodeoffset, "device_type", &len);
+	if (!prop)
+		return len;
+	if (fdt_stringlist_contains(prop, len, type))

The "device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?

Yes, I think you're right. This function was based on the modification of fdt_node_check_compatible, and I forgot to replace fdt_stringlist_contains. And, as I reply above, we can simply the check. And I will implement it out of libfdt.

Quoted 56 lines

Review

Quoted 9 lines
xen/common/libfdt/fdt_ro.c +5 -0
+		return 0;
+	else
+		return 1;
+}
+
  int fdt_node_offset_by_compatible(const void *fdt, int startoffset,
  				  const char *compatible)
  {
(quoted diff) +0 -0

b/xen/include/xen/libfdt/libfdt.h

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt,

Quoted 4 lines

index 7c75688a39..7e4930dbcd 100644

xen/include/xen/libfdt/libfdt.h +0 -0
@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt,

uint32_t phandle);

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);

Quoted 11 lines
xen/include/xen/libfdt/libfdt.h +8 -0
  int fdt_node_check_compatible(const void *fdt, int nodeoffset,
  			      const char *compatible);
 
+/**
+ * fdt_node_check_type: check a node's device_type property
+ * @fdt: pointer to the device tree blob
+ * @nodeoffset: offset of a tree node
+ * @type: string to match against
+ *
+ *
+ * fdt_node_check_type() returns 0 if the given node contains a

'device_type'

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, · line 810

Quoted 1 line
xen/include/xen/libfdt/libfdt.h +1 -0
+ * property with the given string as one of its elements, it returns

non-zero

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);

Quoted 8 lines
xen/include/xen/libfdt/libfdt.h +8 -0
+ * otherwise, or on error.
+ *
+ * returns:
+ *	0, if the node has a 'device_type' property listing the given string
+ *	1, if the node has a 'device_type' property, but it does not list
+ *		the given string
+ *	-FDT_ERR_NOTFOUND, if the given node has no 'device_type' property
+ * 	-FDT_ERR_BADOFFSET, if nodeoffset does not refer to a

BEGIN_NODE tag

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);

Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +8 -0
+ *	-FDT_ERR_BADMAGIC,
+ *	-FDT_ERR_BADVERSION,
+ *	-FDT_ERR_BADSTATE,
+ *	-FDT_ERR_BADSTRUCTURE, standard meanings
+ */
+int fdt_node_check_type(const void *fdt, int nodeoffset,
+			      const char *type);
+
  /**
   * fdt_node_offset_by_compatible - find nodes with a given

'compatible' value

xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829

Quoted 2 lines
xen/include/xen/libfdt/libfdt.h +0 -0
   * @fdt: pointer to the device tree blob

Cheers,

--
Julien Grall

author
#84 lore raw

RFC PATCH 20/40 xen/arm: implement node distance helpers for Arm64

In reply to Wei Chen

In current Xen code, __node_distance is a fake API, it always returns NUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide a set_node_distance API to set the distance for any two nodes in this patch.

---
 xen/arch/arm/numa.c        | 44 ++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/numa.h | 12 ++++++++++-
 xen/include/asm-x86/numa.h |  1 -
 xen/include/xen/numa.h     |  2 +-
 4 files changed, 56 insertions(+), 3 deletions(-)
xen/arch/arm/numa.c +44 -0
index 566ad1e52b..f61a8df645 100644
@@ -23,6 +23,11 @@
 #include <xen/pfn.h>
 #include <asm/setup.h>
 
+static uint8_t __read_mostly
+node_distance_map[MAX_NUMNODES][MAX_NUMNODES] = {
+    { NUMA_REMOTE_DISTANCE }
+};
+
 void numa_set_node(int cpu, nodeid_t nid)
 {
     if ( nid >= MAX_NUMNODES ||
@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid)
     cpu_to_node[cpu] = nid;
 }
 
+void __init numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance)
+{
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+    {
+        printk(KERN_WARNING
+            "NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",
+            from, to, distance);
+        return;
+    }
+
+    /* NUMA defines 0xff as an unreachable node and 0-9 are undefined */
+    if ( distance >= NUMA_NO_DISTANCE ||
+        (distance >= NUMA_DISTANCE_UDF_MIN &&
+         distance <= NUMA_DISTANCE_UDF_MAX) ||
+        (from == to && distance != NUMA_LOCAL_DISTANCE) )
+    {
+        printk(KERN_WARNING
+            "Invalid NUMA node distance, from:%d to:%d distance=%d\n",
+            from, to, distance);
+        return;
+    }
+
+    node_distance_map[from][to] = distance;
+}
+
+uint8_t __node_distance(nodeid_t from, nodeid_t to)
+{
+    /*
+     * Check whether the nodes are in the matrix range.
+     * When any node is out of range, except from and to nodes are the
+     * same, we treat them as unreachable (return 0xFF)
+     */
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+        return from == to ? NUMA_LOCAL_DISTANCE : NUMA_NO_DISTANCE;
+
+    return node_distance_map[from][to];
+}
+EXPORT_SYMBOL(__node_distance);
+
 void __init numa_init(bool acpi_off)
 {
     uint32_t idx;
xen/include/asm-arm/numa.h +11 -1
index bb495a24e1..559b028a01 100644
@@ -12,8 +12,19 @@ typedef u8 nodeid_t;
  * set the number of NUMA memory block number to 128.
  */
 #define NODES_SHIFT      6
+/*
+ * In ACPI spec, 0-9 are the reserved values for node distance,
+ * 10 indicates local node distance, 20 indicates remote node
+ * distance. Set node distance map in device tree will follow
+ * the ACPI's definition.
+ */
+#define NUMA_DISTANCE_UDF_MIN   0
+#define NUMA_DISTANCE_UDF_MAX   9
+#define NUMA_LOCAL_DISTANCE     10
+#define NUMA_REMOTE_DISTANCE    20
 
 extern void numa_init(bool acpi_off);
+extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 
 /*
  * Temporary for fake NUMA node, when CPU, memory and distance
@@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);
  * symbols will be removed.
  */
 extern mfn_t first_valid_mfn;
-#define __node_distance(a, b) (20)
 
 #else
 
xen/include/asm-x86/numa.h +0 -1
index 5a57a51e26..e0253c20b7 100644
@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
 void srat_parse_regions(u64 addr);
-extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 
 #endif
xen/include/xen/numa.h +1 -1
index cb08d2eca9..0475823b13 100644
@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
-
+extern u8 __node_distance(nodeid_t a, nodeid_t b);
 extern void numa_add_cpu(int cpu);
 
 struct node {
Signature · 1 line
2.25.1
#85 lore raw

Re: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64

Quoted 31 lines · Wei Chen

In current Xen code, __node_distance is a fake API, it always returns NUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide a set_node_distance API to set the distance for any two nodes in this patch.

---
 xen/arch/arm/numa.c        | 44 ++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/numa.h | 12 ++++++++++-
 xen/include/asm-x86/numa.h |  1 -
 xen/include/xen/numa.h     |  2 +-
 4 files changed, 56 insertions(+), 3 deletions(-)
xen/arch/arm/numa.c +5 -0
index 566ad1e52b..f61a8df645 100644
@@ -23,6 +23,11 @@
 #include <xen/pfn.h>
 #include <asm/setup.h>
 
+static uint8_t __read_mostly
+node_distance_map[MAX_NUMNODES][MAX_NUMNODES] = {
+    { NUMA_REMOTE_DISTANCE }
+};
+
 void numa_set_node(int cpu, nodeid_t nid)
 {
     if ( nid >= MAX_NUMNODES ||

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 46

Quoted 10 lines
xen/arch/arm/numa.c +7 -0
@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid)
     cpu_to_node[cpu] = nid;
 }
 
+void __init numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance)
+{
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+    {
+        printk(KERN_WARNING
+            "NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",
+            from, to, distance);

NIT: please align. Example:

printk(KERN_WARNING
"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",

Also please use PRIu32 for uint32_t. Probably should use PRIu8 for nodeids.

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 58

Quoted 12 lines
xen/arch/arm/numa.c +12 -0
+        return;
+    }
+
+    /* NUMA defines 0xff as an unreachable node and 0-9 are undefined */
+    if ( distance >= NUMA_NO_DISTANCE ||
+        (distance >= NUMA_DISTANCE_UDF_MIN &&
+         distance <= NUMA_DISTANCE_UDF_MAX) ||
+        (from == to && distance != NUMA_LOCAL_DISTANCE) )
+    {
+        printk(KERN_WARNING
+            "Invalid NUMA node distance, from:%d to:%d distance=%d\n",
+            from, to, distance);

NIT: please align

Also you used %u before for nodeids, which is better because from and to are unsigned. Distance should be uint32_t.

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 4 lines
xen/arch/arm/numa.c +4 -0
+        return;
+    }
+
+    node_distance_map[from][to] = distance;

Shouldn't we also be setting:

    node_distance_map[to][from] = distance;

?

Quoted 79 lines
xen/arch/arm/numa.c +16 -0
+}
+
+uint8_t __node_distance(nodeid_t from, nodeid_t to)
+{
+    /*
+     * Check whether the nodes are in the matrix range.
+     * When any node is out of range, except from and to nodes are the
+     * same, we treat them as unreachable (return 0xFF)
+     */
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+        return from == to ? NUMA_LOCAL_DISTANCE : NUMA_NO_DISTANCE;
+
+    return node_distance_map[from][to];
+}
+EXPORT_SYMBOL(__node_distance);
+
 void __init numa_init(bool acpi_off)
 {
     uint32_t idx;
xen/include/asm-arm/numa.h +11 -1
index bb495a24e1..559b028a01 100644
@@ -12,8 +12,19 @@ typedef u8 nodeid_t;
  * set the number of NUMA memory block number to 128.
  */
 #define NODES_SHIFT      6
+/*
+ * In ACPI spec, 0-9 are the reserved values for node distance,
+ * 10 indicates local node distance, 20 indicates remote node
+ * distance. Set node distance map in device tree will follow
+ * the ACPI's definition.
+ */
+#define NUMA_DISTANCE_UDF_MIN   0
+#define NUMA_DISTANCE_UDF_MAX   9
+#define NUMA_LOCAL_DISTANCE     10
+#define NUMA_REMOTE_DISTANCE    20
 
 extern void numa_init(bool acpi_off);
+extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 
 /*
  * Temporary for fake NUMA node, when CPU, memory and distance
@@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);
  * symbols will be removed.
  */
 extern mfn_t first_valid_mfn;
-#define __node_distance(a, b) (20)
 
 #else
 
xen/include/asm-x86/numa.h +0 -1
index 5a57a51e26..e0253c20b7 100644
@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
 void srat_parse_regions(u64 addr);
-extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 
 #endif
xen/include/xen/numa.h +1 -1
index cb08d2eca9..0475823b13 100644
@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
 #define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
-
+extern u8 __node_distance(nodeid_t a, nodeid_t b);
 extern void numa_add_cpu(int cpu);
 
 struct node {
Signature · 1 line
2.25.1
author
#86 lore raw

RE: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64

Hi Stefano,

Quoted 53 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:52
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64

Quoted 31 lines · Wei Chen

In current Xen code, __node_distance is a fake API, it always returns NUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide a set_node_distance API to set the distance for any two nodes in this patch.

---
 xen/arch/arm/numa.c        | 44 ++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/numa.h | 12 ++++++++++-
 xen/include/asm-x86/numa.h |  1 -
 xen/include/xen/numa.h     |  2 +-
 4 files changed, 56 insertions(+), 3 deletions(-)
xen/arch/arm/numa.c +5 -0
index 566ad1e52b..f61a8df645 100644
@@ -23,6 +23,11 @@
 #include <xen/pfn.h>
 #include <asm/setup.h>
 
+static uint8_t __read_mostly
+node_distance_map[MAX_NUMNODES][MAX_NUMNODES] = {
+    { NUMA_REMOTE_DISTANCE }
+};
+
 void numa_set_node(int cpu, nodeid_t nid)
 {
     if ( nid >= MAX_NUMNODES ||

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 40

Quoted 4 lines
xen/arch/arm/numa.c +1 -0
@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid)
     cpu_to_node[cpu] = nid;
 }
 
+void __init numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance)

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 46

Quoted 6 lines
xen/arch/arm/numa.c +6 -0
+{
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+    {
+        printk(KERN_WARNING
+            "NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",
+            from, to, distance);
Quoted 8 lines

NIT: please align. Example:

printk(KERN_WARNING
"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",

Also please use PRIu32 for uint32_t. Probably should use PRIu8 for nodeids.

OK

Quoted 15 lines

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid)

Quoted 5 lines
xen/arch/arm/numa.c +4 -0
+        return;
+    }
+
+    /* NUMA defines 0xff as an unreachable node and 0-9 are undefined

*/

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 58

Quoted 8 lines
xen/arch/arm/numa.c +8 -0
+    if ( distance >= NUMA_NO_DISTANCE ||
+        (distance >= NUMA_DISTANCE_UDF_MIN &&
+         distance <= NUMA_DISTANCE_UDF_MAX) ||
+        (from == to && distance != NUMA_LOCAL_DISTANCE) )
+    {
+        printk(KERN_WARNING
+            "Invalid NUMA node distance, from:%d to:%d distance=%d\n",
+            from, to, distance);
Quoted 5 lines

NIT: please align

Also you used %u before for nodeids, which is better because from and to are unsigned. Distance should be uint32_t.

OK

Quoted 6 lines

xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62

Quoted 5 lines
xen/arch/arm/numa.c +4 -0
+        return;
+    }
+
+    node_distance_map[from][to] = distance;
Quoted 6 lines

Shouldn't we also be setting:

    node_distance_map[to][from] = distance;

?

No, we want numa_set_distance behavior is single. "node_distance_map[to][from] = distance" is handled in caller.

Quoted 82 lines
Quoted 31 lines
xen/arch/arm/numa.c +16 -0
+}
+
+uint8_t __node_distance(nodeid_t from, nodeid_t to)
+{
+    /*
+     * Check whether the nodes are in the matrix range.
+     * When any node is out of range, except from and to nodes are the
+     * same, we treat them as unreachable (return 0xFF)
+     */
+    if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
+        return from == to ? NUMA_LOCAL_DISTANCE : NUMA_NO_DISTANCE;
+
+    return node_distance_map[from][to];
+}
+EXPORT_SYMBOL(__node_distance);
+
 void __init numa_init(bool acpi_off)
 {
     uint32_t idx;
xen/include/asm-arm/numa.h +3 -0
index bb495a24e1..559b028a01 100644
@@ -12,8 +12,19 @@ typedef u8 nodeid_t;
  * set the number of NUMA memory block number to 128.
  */
 #define NODES_SHIFT      6
+/*
+ * In ACPI spec, 0-9 are the reserved values for node distance,
+ * 10 indicates local node distance, 20 indicates remote node

xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 27

Quoted 10 lines
xen/include/asm-arm/numa.h +8 -0
+ * distance. Set node distance map in device tree will follow
+ * the ACPI's definition.
+ */
+#define NUMA_DISTANCE_UDF_MIN   0
+#define NUMA_DISTANCE_UDF_MAX   9
+#define NUMA_LOCAL_DISTANCE     10
+#define NUMA_REMOTE_DISTANCE    20
 
 extern void numa_init(bool acpi_off);
+extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance);

xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t

Quoted 28 lines
xen/include/asm-arm/numa.h +0 -1
 /*
  * Temporary for fake NUMA node, when CPU, memory and distance
@@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);
  * symbols will be removed.
  */
 extern mfn_t first_valid_mfn;
-#define __node_distance(a, b) (20)
 
 #else
 
xen/include/asm-x86/numa.h +0 -1
index 5a57a51e26..e0253c20b7 100644
@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];
 extern void init_cpu_to_node(void);
 
 void srat_parse_regions(u64 addr);
-extern u8 __node_distance(nodeid_t a, nodeid_t b);
 unsigned int arch_get_dma_bitsize(void);
 
 #endif
xen/include/xen/numa.h +0 -0
index cb08d2eca9..0475823b13 100644
@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t

phys_to_nid(paddr_t addr)

Quoted 11 lines
xen/include/xen/numa.h +1 -2
 #define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
 #define node_end_pfn(nid)       (NODE_DATA(nid)->node_start_pfn + \
 				 NODE_DATA(nid)->node_spanned_pages)
-
+extern u8 __node_distance(nodeid_t a, nodeid_t b);
 extern void numa_add_cpu(int cpu);
 
 struct node {
--

2.25.1

author
#87 lore raw

RFC PATCH 21/40 xen/arm: introduce device_tree_numa as a switch for device tree NUMA

In reply to Wei Chen

Like acpi_numa in x86 as a switch for ACPI based NUMA, we introduce device_tree_numa as a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.

Keep using bad_srat and srat_disabled functions name, because we will reuse node_covers_memory and acpi_scan_nodes code for Arm. These functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.

---
 xen/arch/arm/Makefile           |  1 +
 xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++
 xen/include/asm-arm/numa.h      |  2 ++
 3 files changed, 38 insertions(+)
 create mode 100644 xen/arch/arm/numa_device_tree.c
xen/arch/arm/Makefile +1 -0
index 6e3fb8033e..13e1549be0 100644
@@ -36,6 +36,7 @@ obj-y += mem_access.o
 obj-y += mm.o
 obj-y += monitor.o
 obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o
 obj-y += p2m.o
 obj-y += percpu.o
 obj-y += platform.o
xen/arch/arm/numa_device_tree.c new file +35 -0
new file mode 100644
index 0000000000..1c74ad135d
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+s8 device_tree_numa = 0;
+
+int srat_disabled(void)
+{
+    return numa_off || device_tree_numa < 0;
+}
+
+void __init bad_srat(void)
+{
+    printk(KERN_ERR "DT: NUMA information is not used.\n");
+    device_tree_numa = -1;
+}
xen/include/asm-arm/numa.h +2 -0
index 559b028a01..756ad82d07 100644
@@ -23,6 +23,8 @@ typedef u8 nodeid_t;
 #define NUMA_LOCAL_DISTANCE     10
 #define NUMA_REMOTE_DISTANCE    20
 
+extern s8 device_tree_numa;
+
 extern void numa_init(bool acpi_off);
 extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 
Signature · 1 line
2.25.1
#88 lore raw

Re: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA

Hi Wei,

Quoted 5 lines · Wei Chen

Like acpi_numa in x86 as a switch for ACPI based NUMA, we introduce device_tree_numa as a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.

The hypervisor will never use both ACPI and DT at runtime. In fact...

Quoted 3 lines

Keep using bad_srat and srat_disabled functions name, because we will reuse node_covers_memory and acpi_scan_nodes code for Arm.

... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.

So I think we should either rename acpi_numa to something more generic (maybe fw_numa) or convert numa_off to a tri-state.

This will allow to have the code mostly common.

Quoted 47 lines

These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.

---
  xen/arch/arm/Makefile           |  1 +
  xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++
  xen/include/asm-arm/numa.h      |  2 ++
  3 files changed, 38 insertions(+)

create mode 100644 xen/arch/arm/numa_device_tree.c

xen/arch/arm/Makefile +1 -0
index 6e3fb8033e..13e1549be0 100644
@@ -36,6 +36,7 @@ obj-y += mem_access.o
  obj-y += mm.o
  obj-y += monitor.o
  obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o
  obj-y += p2m.o
  obj-y += percpu.o
  obj-y += platform.o
xen/arch/arm/numa_device_tree.c new file +16 -0
new file mode 100644
index 0000000000..1c74ad135d
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License

xen/arch/arm/numa_device_tree.c line 26

Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10 -0
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+s8 device_tree_numa = 0;
+
+int srat_disabled(void)

You export this one and ...

xen/arch/arm/numa_device_tree.c line 31

Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1 -0
+void __init bad_srat(void)

... this one without providing in a prototype.

Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.

In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).

In this case, the helpers should be exported from now.

Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +4 -0
+{
+    printk(KERN_ERR "DT: NUMA information is not used.\n");
+    device_tree_numa = -1;
+}
xen/include/asm-arm/numa.h +0 -0
index 559b028a01..756ad82d07 100644

xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 30

Quoted 8 lines
xen/include/asm-arm/numa.h +2 -0
@@ -23,6 +23,8 @@ typedef u8 nodeid_t;
  #define NUMA_LOCAL_DISTANCE     10
  #define NUMA_REMOTE_DISTANCE    20
 
+extern s8 device_tree_numa;
+
  extern void numa_init(bool acpi_off);
  extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t distance);
 

Cheers,

Signature · 1 line
Julien Grall
author
#89 lore raw

RE: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA

Hi Julien,

Quoted 11 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:45
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA

Hi Wei,

Quoted 9 lines
Quoted 5 lines · Wei Chen

Like acpi_numa in x86 as a switch for ACPI based NUMA, we introduce device_tree_numa as a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.

The hypervisor will never use both ACPI and DT at runtime. In fact...

Yes.

Quoted 4 lines
Quoted 3 lines

Keep using bad_srat and srat_disabled functions name, because we will reuse node_covers_memory and acpi_scan_nodes code for Arm.

Quoted 8 lines

... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.

So I think we should either rename acpi_numa to something more generic (maybe fw_numa) or convert numa_off to a tri-state.

This will allow to have the code mostly common.

I will try to address them in next version.

Quoted 65 lines

Review

Quoted 26 lines

These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.

---
  xen/arch/arm/Makefile           |  1 +
  xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++
  xen/include/asm-arm/numa.h      |  2 ++
  3 files changed, 38 insertions(+)

create mode 100644 xen/arch/arm/numa_device_tree.c

xen/arch/arm/Makefile +1 -0
index 6e3fb8033e..13e1549be0 100644
@@ -36,6 +36,7 @@ obj-y += mem_access.o
  obj-y += mm.o
  obj-y += monitor.o
  obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o
  obj-y += p2m.o
  obj-y += percpu.o
  obj-y += platform.o
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 21 lines

new file mode 100644
index 0000000000..1c74ad135d

xen/arch/arm/numa_device_tree.c new file +16 -0
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Arm Architecture support layer for NUMA.
+ *
+ * Copyright (C) 2021 Arm Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License

xen/arch/arm/numa_device_tree.c line 26

Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10 -0
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+#include <xen/init.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+s8 device_tree_numa = 0;
+
+int srat_disabled(void)

You export this one and ...

xen/arch/arm/numa_device_tree.c line 31

Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1 -0
+void __init bad_srat(void)

... this one without providing in a prototype.

Quoted 9 lines

Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.

In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).

In this case, the helpers should be exported from now.

Ok.

Quoted 24 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c new file +4 -0
+{
+    printk(KERN_ERR "DT: NUMA information is not used.\n");
+    device_tree_numa = -1;
+}
xen/include/asm-arm/numa.h +0 -0
index 559b028a01..756ad82d07 100644

xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 29

Quoted 7 lines
xen/include/asm-arm/numa.h +2 -0
@@ -23,6 +23,8 @@ typedef u8 nodeid_t;
  #define NUMA_LOCAL_DISTANCE     10
  #define NUMA_REMOTE_DISTANCE    20
 
+extern s8 device_tree_numa;
+
  extern void numa_init(bool acpi_off);
  extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t

distance);

Quoted 2 lines

Cheers,

--
Julien Grall

author
#90 lore raw

RFC PATCH 22/40 xen/arm: introduce a helper to parse device tree processor node

In reply to Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

---
 xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +39 -2
index 1c74ad135d..37cc56acf3 100644
@@ -20,16 +20,53 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
 s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
 {
     return numa_off || device_tree_numa < 0;
 }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
 {
     printk(KERN_ERR "DT: NUMA information is not used.\n");
     device_tree_numa = -1;
 }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}
Signature · 1 line
2.25.1
#91 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Wei,

Quoted 37 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +5 -2
index 1c74ad135d..37cc56acf3 100644
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;

xen/arch/arm/numa_device_tree.c line 47

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +9 -0
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;

You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.

xen/arch/arm/numa_device_tree.c line 59

Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12 -0
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.

In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +3 -0
+{
+    uint32_t nid;
+

xen/arch/arm/numa_device_tree.c line 72

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

Cheers,

Signature · 1 line
Julien Grall
author
#92 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Julien,

Quoted 56 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Wei,

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 30 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +10 -2
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
Quoted 9 lines

xen/arch/arm/numa_device_tree.c line 47

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;

You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.

I copied some code from x86, and x86 is Linux style. So, yes, I should adjust them it Xen coding style. I will do it in next version.

Quoted 17 lines

xen/arch/arm/numa_device_tree.c line 59

Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12 -0
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.

Quoted 4 lines

In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.

That's make sense, I will try to address it in next version.

Quoted 20 lines

xen/arch/arm/numa_device_tree.c

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

xen/arch/arm/numa_device_tree.c line 72

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

Cheers,

--
Julien Grall

#93 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

In reply to Wei Chen
Quoted 31 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +4 -1
index 1c74ad135d..37cc56acf3 100644
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }

xen/arch/arm/numa_device_tree.c line 41

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +4 -1
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)

I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?

Quoted 22 lines
xen/arch/arm/numa_device_tree.c +21 -0
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+

xen/arch/arm/numa_device_tree.c line 72

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

Cheers,

Signature · 1 line
Julien Grall
author
#94 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Julien,

Quoted 43 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 19 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +4 -1
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }
 
Quoted 14 lines

xen/arch/arm/numa_device_tree.c line 41

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4 -1
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)

I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?

yes, in the very beginning, I want to keep device_tree_ prefix for functions that will handle dtb file. And use dtb_ prefix to replace acpi, to indicate, this function is device tree version numa implementation.

If that's not the right reason, I will unify all prefix to device_tree_ in next version. How do you think about it?

Quoted 38 lines
Quoted 16 lines
xen/arch/arm/numa_device_tree.c +16 -0
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+

xen/arch/arm/numa_device_tree.c

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

xen/arch/arm/numa_device_tree.c line 72

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

Cheers,

--
Julien Grall

#95 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 58 lines
Quoted 43 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
   xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
   1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 19 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +4 -1
@@ -20,16 +20,53 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
   s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
   {
       return numa_off || device_tree_numa < 0;
   }
 
Quoted 14 lines

xen/arch/arm/numa_device_tree.c line 41

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4 -1
-void __init bad_srat(void)
+static __init void bad_srat(void)
   {
       printk(KERN_ERR "DT: NUMA information is not used.\n");
       device_tree_numa = -1;
   }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)

I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?

Quoted 3 lines

yes, in the very beginning, I want to keep device_tree_ prefix for functions that will handle dtb file. And use dtb_ prefix to replace acpi, to indicate, this function is device tree version numa implementation.

Thanks for the clarification. The difference between "dtb" and "device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.

Quoted 3 lines

If that's not the right reason, I will unify all prefix to device_tree_ in next version. How do you think about it?

AFAICT, your parsing functions will always start with "device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".

If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".

Cheers,

Signature · 1 line
Julien Grall
author
#96 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Julien,

Quoted 78 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月23日 18:59
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 17 lines
Quoted 17 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
---
   xen/arch/arm/numa_device_tree.c | 41

+++++++++++++++++++++++++++++++--

Quoted 38 lines
Quoted 23 lines

Review

Quoted 3 lines
   1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 19 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +4 -1
@@ -20,16 +20,53 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
   s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
   {
       return numa_off || device_tree_numa < 0;
   }
 
Quoted 14 lines

xen/arch/arm/numa_device_tree.c line 41

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4 -1
-void __init bad_srat(void)
+static __init void bad_srat(void)
   {
       printk(KERN_ERR "DT: NUMA information is not used.\n");
       device_tree_numa = -1;
   }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)

I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?

Quoted 3 lines

yes, in the very beginning, I want to keep device_tree_ prefix for functions that will handle dtb file. And use dtb_ prefix to replace acpi, to indicate, this function is device tree version numa

Quoted 7 lines

implementation.

Thanks for the clarification. The difference between "dtb" and "device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.

Thanks for the clarification.

Quoted 11 lines
Quoted 3 lines

If that's not the right reason, I will unify all prefix to device_tree_ in next version. How do you think about it?

AFAICT, your parsing functions will always start with "device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".

If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".

That makes sense, I will do it.

Quoted 5 lines

Cheers,

--
Julien Grall

#97 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

In reply to Wei Chen

Hi Wei,

Quoted 17 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +0 -0
index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c line 23

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +1 -0
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>

Nothing in this file seems to depend on xen/device_tree.h. So why do you need to include it?

Quoted 52 lines
xen/arch/arm/numa_device_tree.c +38 -2
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}
Signature · 1 line
Julien Grall
author
#98 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Julien,

Quoted 26 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Wei,

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
  xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
  1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 12 lines

xen/arch/arm/numa_device_tree.c line 23

Quoted 8 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +1 -0
@@ -20,16 +20,53 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/device_tree.h>

Nothing in this file seems to depend on xen/device_tree.h. So why do you need to include it?

I remember that without this header file, device_tree_get_u32 in this patch will cause compiling failed.

Quoted 56 lines
Quoted 36 lines
xen/arch/arm/numa_device_tree.c +23 -2
+#include <asm/setup.h>
 
  s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
  {
      return numa_off || device_tree_numa < 0;
  }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
  {
      printk(KERN_ERR "DT: NUMA information is not used.\n");
      device_tree_numa = -1;
  }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+

xen/arch/arm/numa_device_tree.c

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

xen/arch/arm/numa_device_tree.c line 72

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

--
Julien Grall

#99 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 39 lines
Quoted 26 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Wei,

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
   xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
   1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 12 lines

xen/arch/arm/numa_device_tree.c line 23

Quoted 8 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +1 -0
@@ -20,16 +20,53 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/device_tree.h>

Nothing in this file seems to depend on xen/device_tree.h. So why do you need to include it?

Quoted 2 lines

I remember that without this header file, device_tree_get_u32 in this patch will cause compiling failed.

I looked at the prototype of device_tree_get_u32() and I can't find how it depends on bits from device_tree.h. Can you paste the compilation error?

Quoted 48 lines
Quoted 48 lines
Quoted 37 lines
xen/arch/arm/numa_device_tree.c +23 -2
+#include <asm/setup.h>
 
   s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
   {
       return numa_off || device_tree_numa < 0;
   }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
   {
       printk(KERN_ERR "DT: NUMA information is not used.\n");
       device_tree_numa = -1;
   }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+

xen/arch/arm/numa_device_tree.c

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

Quoted 9 lines
Quoted 9 lines

xen/arch/arm/numa_device_tree.c line 72

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

--
Julien Grall

Cheers,

Signature · 1 line
Julien Grall
author
#100 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Julien,

Quoted 57 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:44
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 19 lines
Quoted 19 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Wei,

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
---
   xen/arch/arm/numa_device_tree.c | 41

+++++++++++++++++++++++++++++++--

Quoted 4 lines
Quoted 4 lines

Review

Quoted 3 lines
   1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 10 lines
Quoted 10 lines

xen/arch/arm/numa_device_tree.c line 23

Quoted 8 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +1 -0
@@ -20,16 +20,53 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/device_tree.h>

Nothing in this file seems to depend on xen/device_tree.h. So why do

you

Quoted 3 lines
Quoted 2 lines

need to include it?

Quoted 7 lines
Quoted 1 line

I remember that without this header file, device_tree_get_u32 in this

patch

Quoted 1 line

will cause compiling failed.

I looked at the prototype of device_tree_get_u32() and I can't find how it depends on bits from device_tree.h. Can you paste the compilation error?

I tested it again, this header file should be introduced in following patches:
numa_device_tree.c: In function ‘device_tree_parse_numa_distance_map_v1’:
numa_device_tree.c:243:16: error: implicit declaration of function ‘dt_read_number’ [-Werror=implicit-function-declaration]

  243 |         from = dt_read_number(matrix, 1);
      |                ^~~~~~~~~~~~~~

numa_device_tree.c:243:16: error: nested extern declaration of ‘dt_read_number’ [-Werror=nested-externs]
cc1: all warnings being treated as errors

I will move it to another patch.

Quoted 63 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
xen/arch/arm/numa_device_tree.c +21 -2
+#include <asm/setup.h>
 
   s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
   {
       return numa_off || device_tree_numa < 0;
   }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
   {
       printk(KERN_ERR "DT: NUMA information is not used.\n");
       device_tree_numa = -1;
   }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id",

MAX_NUMNODES);

Quoted 5 lines
Quoted 5 lines

xen/arch/arm/numa_device_tree.c

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

Quoted 9 lines
Quoted 9 lines

xen/arch/arm/numa_device_tree.c line 72

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}

--
Julien Grall

Cheers,

--
Julien Grall

#101 lore raw

Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

In reply to Wei Chen
Quoted 54 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

---
 xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +21 -2
index 1c74ad135d..37cc56acf3 100644
@@ -20,16 +20,53 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
 s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
 {
     return numa_off || device_tree_numa < 0;
 }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
 {
     printk(KERN_ERR "DT: NUMA information is not used.\n");
     device_tree_numa = -1;
 }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+

xen/arch/arm/numa_device_tree.c line 64

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);

Given that this is not actually a warning (is it?) then I would move it to XENLOG_INFO

xen/arch/arm/numa_device_tree.c line 67

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +3 -0
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);

This could be XENLOG_ERR

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}
author
#102 lore raw

RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Hi Stefano,

Quoted 70 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 8:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node

Quoted 7 lines · Wei Chen

Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.

Review

Quoted 6 lines
---
 xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

Quoted 46 lines

index 1c74ad135d..37cc56acf3 100644

xen/arch/arm/numa_device_tree.c +26 -2
@@ -20,16 +20,53 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/device_tree.h>
+#include <asm/setup.h>
 
 s8 device_tree_numa = 0;
+static nodemask_t processor_nodes_parsed __initdata;
 
-int srat_disabled(void)
+static int srat_disabled(void)
 {
     return numa_off || device_tree_numa < 0;
 }
 
-void __init bad_srat(void)
+static __init void bad_srat(void)
 {
     printk(KERN_ERR "DT: NUMA information is not used.\n");
     device_tree_numa = -1;
 }
+
+/* Callback for device tree processor affinity */
+static int __init dtb_numa_processor_affinity_init(nodeid_t node)
+{
+    if ( srat_disabled() )
+        return -EINVAL;
+    else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) {
+		bad_srat();
+		return -EINVAL;
+	}
+
+    node_set(node, processor_nodes_parsed);
+
+    device_tree_numa = 1;
+    printk(KERN_INFO "DT: NUMA node %u processor parsed\n", node);
+
+    return 0;
+}
+
+/* Parse CPU NUMA node info */
+int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 10 lines

xen/arch/arm/numa_device_tree.c line 64

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+{
+    uint32_t nid;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    printk(XENLOG_WARNING "CPU on NUMA node:%u\n", nid);

Given that this is not actually a warning (is it?) then I would move it to XENLOG_INFO

OK

Quoted 7 lines

xen/arch/arm/numa_device_tree.c

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +3 -0
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

This could be XENLOG_ERR

OK

Quoted 6 lines
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return dtb_numa_processor_affinity_init(nid);
+}
author
#103 lore raw

RFC PATCH 23/40 xen/arm: introduce a helper to parse device tree memory node

In reply to Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
 xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
 1 file changed, 130 insertions(+)
xen/arch/arm/numa_device_tree.c +130 -0
index 37cc56acf3..bbe081dcd1 100644
@@ -20,11 +20,13 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
 #include <xen/device_tree.h>
 #include <asm/setup.h>
 
 s8 device_tree_numa = 0;
 static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
 static int srat_disabled(void)
 {
@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)
     return 0;
 }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used later */
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;
+
+    return 0;
+}
+
 /* Parse CPU NUMA node info */
 int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
 {
@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
 
     return dtb_numa_processor_affinity_init(nid);
 }
+
+/* Parse memory node NUMA info */
+int __init
+device_tree_parse_numa_memory_node(const void *fdt, int node,
+    const char *name, uint32_t addr_cells, uint32_t size_cells)
+{
+    uint32_t nid;
+    int ret = 0, len;
+    paddr_t addr, size;
+    const struct fdt_property *prop;
+    uint32_t idx, ranges;
+    const __be32 *addresses;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    prop = fdt_get_property(fdt, node, "reg", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "fdt: node `%s': missing `reg' property\n", name);
+        return -EINVAL;
+    }
+
+    addresses = (const __be32 *)prop->data;
+    ranges = len / (sizeof(__be32)* (addr_cells + size_cells));
+    for ( idx = 0; idx < ranges; idx++ )
+    {
+        device_tree_get_reg(&addresses, addr_cells, size_cells, &addr, &size);
+        /* Skip zero size ranges */
+        if ( !size )
+            continue;
+
+        ret = dtb_numa_memory_affinity_init(nid, addr, size);
+        if ( ret ) {
+            printk(XENLOG_WARNING
+                   "NUMA: process range#%d addr = %lx size=%lx failed!\n",
+                   idx, addr, size);
+            return -EINVAL;
+        }
+    }
+
+    if ( idx == 0 )
+    {
+        printk(XENLOG_ERR
+               "bad property in memory node, idx=%d ret=%d\n", idx, ret);
+        return -EINVAL;
+    }
+
+    return 0;
+}
Signature · 1 line
2.25.1
#104 lore raw

Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 31 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
  xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
  1 file changed, 130 insertions(+)
xen/arch/arm/numa_device_tree.c +2 -0
index 37cc56acf3..bbe081dcd1 100644
@@ -20,11 +20,13 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
  #include <xen/device_tree.h>
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
  static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
  static int srat_disabled(void)
  {

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4 -0
@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)
      return 0;
  }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{

The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?

Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used later */
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {

Xen coding style is using:

if ( ... )
{

Note that I may not comment on all the occurents, so please check the other places.

Quoted 50 lines
xen/arch/arm/numa_device_tree.c +46 -0
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;
+
+    return 0;
+}
+
  /* Parse CPU NUMA node info */
  int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
  {

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +5 -0
@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
 
      return dtb_numa_processor_affinity_init(nid);
  }
+
+/* Parse memory node NUMA info */
+int __init
+device_tree_parse_numa_memory_node(const void *fdt, int node,
+    const char *name, uint32_t addr_cells, uint32_t size_cells)

This is pretty much a copy of process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?

Quoted 26 lines
xen/arch/arm/numa_device_tree.c +26 -0
+{
+    uint32_t nid;
+    int ret = 0, len;
+    paddr_t addr, size;
+    const struct fdt_property *prop;
+    uint32_t idx, ranges;
+    const __be32 *addresses;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n", nid);
+        return -EINVAL;
+    }
+
+    prop = fdt_get_property(fdt, node, "reg", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "fdt: node `%s': missing `reg' property\n", name);
+        return -EINVAL;
+    }
+
+    addresses = (const __be32 *)prop->data;
+    ranges = len / (sizeof(__be32)* (addr_cells + size_cells));
+    for ( idx = 0; idx < ranges; idx++ )

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 188

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    {
+        device_tree_get_reg(&addresses, addr_cells, size_cells, &addr, &size);
+        /* Skip zero size ranges */
+        if ( !size )
+            continue;
+
+        ret = dtb_numa_memory_affinity_init(nid, addr, size);
+        if ( ret ) {
+            printk(XENLOG_WARNING
+                   "NUMA: process range#%d addr = %lx size=%lx failed!\n",

s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size are paddr_t.

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +4 -0
+                   idx, addr, size);
+            return -EINVAL;
+        }
+    }

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 202

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+
+    if ( idx == 0 )
+    {
+        printk(XENLOG_ERR
+               "bad property in memory node, idx=%d ret=%d\n", idx, ret);
+        return -EINVAL;
+    }
+
+    return 0;
+}

Cheers,

Signature · 1 line
Julien Grall
author
#105 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Julien,

Quoted 48 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 7 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

Review

Quoted 6 lines
---
  xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
  1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
  #include <xen/device_tree.h>
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
  static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
  static int srat_disabled(void)
  {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -0
      return 0;
  }
 
Quoted 7 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{

The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?

In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.

But, yes, maybe we can abstract some common code to functions, that can be reused in two functions. I would try it in next version.

Quoted 26 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +7 -0
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used

later */

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
Quoted 8 lines

Xen coding style is using:

if ( ... )
{

Note that I may not comment on all the occurents, so please check the other places.

Ok.

Quoted 53 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7 -0
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const

Quoted 43 lines
xen/arch/arm/numa_device_tree.c +39 -0
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;
+
+    return 0;
+}
+
  /* Parse CPU NUMA node info */
  int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
  {
@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const

void *fdt, int node)

Quoted 1 line
Quoted 11 lines

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152

Quoted 7 lines
      return dtb_numa_processor_affinity_init(nid);

}

xen/arch/arm/numa_device_tree.c +5 -0
+
+/* Parse memory node NUMA info */
+int __init
+device_tree_parse_numa_memory_node(const void *fdt, int node,
+    const char *name, uint32_t addr_cells, uint32_t size_cells)

This is pretty much a copy of process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?

I had tried to parse NUMA id in process_memory_node or in early_scan_node. But I found, this change would make NUMA init code in different places. And I prefer a unify NUMA init entry. Because it may be good for maintenance. When we want to disable NUMA, we just need to disable it in one place. But you're right, we still can do some code abstraction. I will do it in next version.

Quoted 38 lines

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12 -0
+{
+    uint32_t nid;
+    int ret = 0, len;
+    paddr_t addr, size;
+    const struct fdt_property *prop;
+    uint32_t idx, ranges;
+    const __be32 *addresses;
+
+    nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+    if ( nid >= MAX_NUMNODES )
+    {
+        printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",

nid);

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+        return -EINVAL;
+    }
+
+    prop = fdt_get_property(fdt, node, "reg", &len);
+    if ( !prop )
+    {

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        printk(XENLOG_WARNING
+               "fdt: node `%s': missing `reg' property\n", name);
+        return -EINVAL;
+    }
+
+    addresses = (const __be32 *)prop->data;
+    ranges = len / (sizeof(__be32)* (addr_cells + size_cells));
+    for ( idx = 0; idx < ranges; idx++ )
+    {
+        device_tree_get_reg(&addresses, addr_cells, size_cells, &addr,

&size);

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+        /* Skip zero size ranges */
+        if ( !size )
+            continue;
+
+        ret = dtb_numa_memory_affinity_init(nid, addr, size);
+        if ( ret ) {
+            printk(XENLOG_WARNING
+                   "NUMA: process range#%d addr = %lx size=%lx
Quoted 5 lines

failed!\n",

s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size are paddr_t.

OK

Quoted 21 lines

xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)

Quoted 9 lines
xen/arch/arm/numa_device_tree.c +9 -0
+                   idx, addr, size);
+            return -EINVAL;
+        }
+    }
+
+    if ( idx == 0 )
+    {
+        printk(XENLOG_ERR
+               "bad property in memory node, idx=%d ret=%d\n", idx,

ret);

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 131

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        return -EINVAL;
+    }
+
+    return 0;
+}

Cheers,

--
Julien Grall

#106 lore raw

Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 56 lines
Quoted 48 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 7 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

Review

Quoted 6 lines
---
   xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
   1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
   #include <xen/device_tree.h>
   #include <asm/setup.h>
 
   s8 device_tree_numa = 0;
   static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
   static int srat_disabled(void)
   {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -0
       return 0;
   }
 
Quoted 7 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{

The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?

Quoted 4 lines

In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.

It think you can get away from distinguishing between ACPI and DT in that helper:
* ma->flags & ACPI_SRAT_MEM_HOTPLUGGABLE could be replace by an
argument indicating whether the region is hotpluggable (this would always be false for DT)
* Access to memblk_hotplug can be stubbed (in the future we may want
to consider memory hotplug even on Arm).

Do you still have the "if else" version? If so can you post it?

Cheers,

Signature · 1 line
Julien Grall
author
#107 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Julien,

Quoted 86 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 16:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 1 line · Wei Chen

Hi Julien,

Hi Wei,

Quoted 16 lines
Quoted 16 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 4 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
   xen/arch/arm/numa_device_tree.c | 130

++++++++++++++++++++++++++++++++

Quoted 34 lines
Quoted 26 lines

Review

Quoted 3 lines
   1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
   #include <xen/init.h>
   #include <xen/nodemask.h>
   #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
   #include <xen/device_tree.h>
   #include <asm/setup.h>
 
   s8 device_tree_numa = 0;
   static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
   static int srat_disabled(void)
   {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -0
       return 0;
   }
 
Quoted 7 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{

The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?

Quoted 4 lines

In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.

It think you can get away from distinguishing between ACPI and DT in that helper:
* ma->flags & ACPI_SRAT_MEM_HOTPLUGGABLE could be replace by an
argument indicating whether the region is hotpluggable (this would always be false for DT)
* Access to memblk_hotplug can be stubbed (in the future we may want
to consider memory hotplug even on Arm).

Quoted 2 lines

Do you still have the "if else" version? If so can you post it?

I just tried to do that in draft process, because I was not satisfied with the changes, I haven't saved them as a patch.

I think your suggestions are worth to try again, I will do it in next version.

Quoted 4 lines

Cheers,

--
Julien Grall

#108 lore raw

Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

In reply to Wei Chen
Quoted 93 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
 xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
 1 file changed, 130 insertions(+)
xen/arch/arm/numa_device_tree.c +61 -0
index 37cc56acf3..bbe081dcd1 100644
@@ -20,11 +20,13 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
 #include <xen/device_tree.h>
 #include <asm/setup.h>
 
 s8 device_tree_numa = 0;
 static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
 static int srat_disabled(void)
 {
@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)
     return 0;
 }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used later */
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single NUMA node?

Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

author
#109 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Stefano,

Quoted 122 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月28日 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 7 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

Review

Quoted 6 lines
---
 xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
 1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
 #include <xen/init.h>
 #include <xen/nodemask.h>
 #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
 #include <xen/device_tree.h>
 #include <asm/setup.h>
 
 s8 device_tree_numa = 0;
 static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
 static int srat_disabled(void)
 {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11 -0
     return 0;
 }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used

later */

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

Quoted 25 lines
xen/arch/arm/numa_device_tree.c +25 -0
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single NUMA node?

Quoted 3 lines

Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too? If yes, I think this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into a neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here. Anyway, Thanks for pointing this, I will look into the latest Linux implementation.

Cheers,
Wei Chen

#110 lore raw

Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 122 lines · Wei Chen
Quoted 122 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月28日 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 7 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

Review

Quoted 6 lines
---
  xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++
  1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
  #include <xen/device_tree.h>
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
  static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
  static int srat_disabled(void)
  {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11 -0
      return 0;
  }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used

later */

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

Quoted 25 lines
xen/arch/arm/numa_device_tree.c +25 -0
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single NUMA node?

Quoted 6 lines
Quoted 3 lines

Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too?

I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.

Quoted 6 lines

If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into a neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here.

I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?

If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).

Cheers,

Signature · 1 line
Julien Grall
author
#111 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Julien,

Quoted 159 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:34
To: Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021��8��28�� 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 9 lines
Quoted 9 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 4 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
  xen/arch/arm/numa_device_tree.c | 130

++++++++++++++++++++++++++++++++

Quoted 62 lines
Quoted 62 lines

Review

Quoted 3 lines
  1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
  #include <xen/device_tree.h>
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
  static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
  static int srat_disabled(void)
  {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11 -0
      return 0;
  }
 
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
+    if ( srat_disabled() )
+        return -EINVAL;
+

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be used

later */

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage */;
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning here.
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

Quoted 10 lines
Quoted 10 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+               node, start, end,
+               node_memblk_range[i].start, node_memblk_range[i].end);
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-

%"PRIx64")\n",

Quoted 30 lines
Quoted 30 lines
Quoted 15 lines
xen/arch/arm/numa_device_tree.c +15 -0
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start, node_memblk_range[i].end);
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single NUMA node?

Quoted 6 lines
Quoted 3 lines

Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too?

I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.

Quoted 6 lines

If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into a neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here.

I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?

Quoted 6 lines

If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).

Yes, I agree.

Quoted 4 lines

Cheers,

--
Julien Grall

author
#112 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Julien, Stefano, Jan

Quoted 183 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 21:58
To: Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Julien,

Quoted 12 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:34
To: Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Wei,

Quoted 9 lines
Quoted 5 lines · Wei Chen
Quoted 5 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021��8��28�� 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];

[email protected];

Quoted 2 lines
Quoted 2 lines

[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to

parse

Quoted 40 lines
Quoted 7 lines
Quoted 7 lines

device tree memory node

Quoted 4 lines · Wei Chen

Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.

---
  xen/arch/arm/numa_device_tree.c | 130

++++++++++++++++++++++++++++++++

Quoted 26 lines
Quoted 26 lines

Review

Quoted 3 lines
  1 file changed, 130 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c static int __init

Quoted 18 lines

index 37cc56acf3..bbe081dcd1 100644

xen/arch/arm/numa_device_tree.c +2 -0
@@ -20,11 +20,13 @@
  #include <xen/init.h>
  #include <xen/nodemask.h>
  #include <xen/numa.h>
+#include <xen/libfdt/libfdt.h>
  #include <xen/device_tree.h>
  #include <asm/setup.h>
 
  s8 device_tree_numa = 0;
  static nodemask_t processor_nodes_parsed __initdata;
+static nodemask_t memory_nodes_parsed __initdata;
 
  static int srat_disabled(void)
  {
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -0
      return 0;
  }
 
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS

\n");

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be

used

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

later */

Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage

*/;

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning

here.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2 -0
+               node, start, end,
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 11 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-

%"PRIx64")\n",

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2 -0
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single

NUMA

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

node?

Looking at the DT bindings and Linux implementation, it seems

possible.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI

allow

Quoted 8 lines
Quoted 1 line

non-contiguous ranges of memory for a single NUMA node too?

I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.

Quoted 3 lines

If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into

a

Quoted 8 lines
Quoted 3 lines

neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here.

I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?

Quoted 9 lines
Quoted 6 lines

If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).

Yes, I agree.

I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.

This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.

To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?

Or could you give me suggestions that we can use some simple ways to fix it?

Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]

[1] https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c
[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3] https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c

Quoted 4 lines
Quoted 4 lines

Cheers,

--
Julien Grall

#113 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 133 lines · Wei Chen
Quoted 113 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Review static int __init

Quoted 1 line
(quoted diff) +0 -0
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0 -0
      return 0;
  }
 
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger NR_NODE_MEMBLKS

\n");

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be

used

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

later */

Quoted 2 lines
(quoted diff) +2 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later usage

*/;

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning

here.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+         * This memblk will be merged with conflicted memblk later.
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-

%"PRIx64")\n",

Quoted 2 lines
(quoted diff) +2 -0
+               node, start, end,
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 11 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8 -0
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn off.
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-

%"PRIx64")\n",

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +2 -0
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a single

NUMA

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

node?

Looking at the DT bindings and Linux implementation, it seems

possible.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI

allow

Quoted 8 lines
Quoted 1 line

non-contiguous ranges of memory for a single NUMA node too?

I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.

Quoted 3 lines

If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into

a

Quoted 8 lines
Quoted 3 lines

neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here.

I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?

Quoted 9 lines
Quoted 6 lines

If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).

Yes, I agree.

I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.

This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.

Quoted 6 lines

To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?

Or could you give me suggestions that we can use some simple ways to fix it?

It looks like that we would have to replace all the node->start / node->end checks with node_memblk_range checks. There are a few of them in valid_numa_range, conflicting_memblks, cutoff_node, nodes_cover_memory. It wouldn't be trivial.

Although I do think that non-contiguous memory for NUMA nodes is important to support, the patch series is already 40 patches. I don't think it is a good idea to add other significant changes to it. I wouldn't upgrade the x86 NUMA driver now. If we can't find a better way, we can proceed as you are doing in this version, with the known gap that we can't deal with non-contigious memory for NUMA nodes, and fix it with a follow-up series later. In that case we would want to have an explicit check for non-contiguous memory for NUMA nodes in dtb_numa_memory_affinity_init and error out if found.

Quoted 6 lines

Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]

[1] https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c
[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3] https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c

In general, I like the idea of sharing code as much as possible between architectures (x86, ARM, etc.) and between DT/ACPI because it makes the code maintainance easier and one might even gain certain features for free.

However, the excercise of sharing code shouldn't take significant additional efforts. In fact, it should decrease the overall effort:
instead of writing new code one just take existing code and move it to common.

In this instance, I think it would be good to be able to share the NUMA initialization code between x86/ARM and ACPI/DT if it doesn't cause extra efforts.

Here the extra effort that my previous comment might cause doesn't derive from x86/ARM or DT/ACPI code sharing. It derives from the fact that our existing code doesn't deal with non-contigous memory for NUMA nodes unfortunately. That is something we need to find a way to cope with anyway.

author
#114 lore raw

RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Hi Stefano,

Quoted 168 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月9日 6:32
To: Wei Chen <[email protected]>
Cc: Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node

Quoted 5 lines · Wei Chen
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Review static int __init

Quoted 1 line
(quoted diff) +0 -0
@@ -55,6 +57,79 @@ static int __init

dtb_numa_processor_affinity_init(nodeid_t node)

Quoted 3 lines
(quoted diff) +0 -0
      return 0;
  }
 
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8 -0
+/* Callback for parsing of the memory regions affinity */
+static int __init dtb_numa_memory_affinity_init(nodeid_t node,
+                                paddr_t start, paddr_t size)
+{
+    struct node *nd;
+    paddr_t end;
+    int i;
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review static int __init · line 75

Quoted 8 lines
(quoted diff) +8 -0
+    if ( srat_disabled() )
+        return -EINVAL;
+
+    end = start + size;
+    if ( num_node_memblks >= NR_NODE_MEMBLKS )
+    {
+        dprintk(XENLOG_WARNING,
+                "Too many numa entry, try bigger

NR_NODE_MEMBLKS

Quoted 10 lines
Quoted 10 lines

\n");

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Review static int __init · line 80

Quoted 5 lines
(quoted diff) +5 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+    /* It is fine to add this area to the nodes data it will be

used

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

later */

Quoted 2 lines
(quoted diff) +2 -0
+    i = conflicting_memblks(start, end);
+    /* No conflicting memory block, we can save it for later

usage

Quoted 9 lines
Quoted 9 lines

*/;

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Review static int __init · line 88

Quoted 6 lines
(quoted diff) +6 -0
+    if ( i < 0 )
+        goto save_memblk;
+
+    if ( memblk_nodeid[i] == node ) {
+        /*
+         * Overlaps with other memblk in the same node, warning

here.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +1 -0
+         * This memblk will be merged with conflicted memblk

later.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4 -0
+         */
+        printk(XENLOG_WARNING
+               "DT: NUMA NODE %u (%"PRIx64
+               "-%"PRIx64") overlaps with itself (%"PRIx64"-
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

%"PRIx64")\n",

Quoted 2 lines
(quoted diff) +2 -0
+               node, start, end,
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4 -0
+    } else {
+        /*
+         * Conflict with memblk in other node, this is an error.
+         * The NUMA information is invalid, NUMA will be turn

off.

Quoted 26 lines
Quoted 26 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Review static int __init · line 103

Quoted 4 lines
(quoted diff) +4 -0
+         */
+        printk(XENLOG_ERR
+               "DT: NUMA NODE %u (%"PRIx64"-%"
+               PRIx64") overlaps with NODE %u (%"PRIx64"-

%"PRIx64")\n",

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +2 -0
+               node, start, end, memblk_nodeid[i],
+               node_memblk_range[i].start,

node_memblk_range[i].end);

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
(quoted diff) +18 -0
+        bad_srat();
+        return -EINVAL;
+    }
+
+save_memblk:
+    nd = &nodes[node];
+    if ( !node_test_and_set(node, memory_nodes_parsed) ) {
+        nd->start = start;
+        nd->end = end;
+    } else {
+        if ( start < nd->start )
+            nd->start = start;
+        if ( nd->end < end )
+            nd->end = end;
+    }
+
+    printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n",
+           node, start, end);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review static int __init · line 128

Quoted 5 lines
(quoted diff) +5 -0
+
+    node_memblk_range[num_node_memblks].start = start;
+    node_memblk_range[num_node_memblks].end = end;
+    memblk_nodeid[num_node_memblks] = node;
+    num_node_memblks++;

Is it possible to have non-contigous ranges of memory for a

single

Quoted 6 lines
Quoted 6 lines

NUMA

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

node?

Looking at the DT bindings and Linux implementation, it seems

possible.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Here, it seems that node_memblk_range/memblk_nodeid could handle

it,

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

but nodes couldn't.

Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI

allow

Quoted 7 lines
Quoted 1 line

non-contiguous ranges of memory for a single NUMA node too?

I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.

Quoted 2 lines

If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to

merge

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init

into

Quoted 10 lines
Quoted 10 lines

a

Quoted 9 lines
Quoted 3 lines

neutral function. So we can fix them at the same time.

If not, maybe we have to keep the diversity for dtb and ACPI here.

I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?

If so, then I disagree with that. It is fine to have code that

supports

Quoted 1 line
Quoted 1 line
Quoted 1 line

more than what a firmware table supports. The main benefit is less

code

Quoted 1 line
Quoted 1 line
Quoted 1 line

and therefore less long term maintenance (with the current solution

we

Quoted 1 line
Quoted 1 line
Quoted 1 line

would need to check both the ACPI and DT implementation if there is

a

Quoted 16 lines
Quoted 5 lines
Quoted 2 lines

bug in one).

Yes, I agree.

I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.

This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.

Quoted 6 lines

To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?

Or could you give me suggestions that we can use some simple ways to fix it?

It looks like that we would have to replace all the node->start / node->end checks with node_memblk_range checks. There are a few of them in valid_numa_range, conflicting_memblks, cutoff_node, nodes_cover_memory. It wouldn't be trivial.

Quoted 10 lines

Although I do think that non-contiguous memory for NUMA nodes is important to support, the patch series is already 40 patches. I don't think it is a good idea to add other significant changes to it. I wouldn't upgrade the x86 NUMA driver now. If we can't find a better way, we can proceed as you are doing in this version, with the known gap that we can't deal with non-contigious memory for NUMA nodes, and fix it with a follow-up series later. In that case we would want to have an explicit check for non-contiguous memory for NUMA nodes in dtb_numa_memory_affinity_init and error out if found.

Yes, I think this may be a more appropriate method at present. I would add some code to do explicit check and give warning/error.

Quoted 25 lines
Quoted 5 lines

Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]

[1]

https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c

Quoted 2 lines

[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3]

https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c

In general, I like the idea of sharing code as much as possible between architectures (x86, ARM, etc.) and between DT/ACPI because it makes the code maintainance easier and one might even gain certain features for free.

However, the excercise of sharing code shouldn't take significant additional efforts. In fact, it should decrease the overall effort:
instead of writing new code one just take existing code and move it to common.

In this instance, I think it would be good to be able to share the NUMA initialization code between x86/ARM and ACPI/DT if it doesn't cause extra efforts.

Quoted 5 lines

Here the extra effort that my previous comment might cause doesn't derive from x86/ARM or DT/ACPI code sharing. It derives from the fact that our existing code doesn't deal with non-contigous memory for NUMA nodes unfortunately. That is something we need to find a way to cope with anyway.

Yes. I posted above links didn't mean to create diversity for Arm/x86. Though, I'm not sure exactly why Linux does this. But I think for Xen, we still can try to share code for Arm/x86 and DT/ACPI.

Cheers,
Wei Chen

author
#115 lore raw

RFC PATCH 24/40 xen/arm: introduce a helper to parse device tree NUMA distance map

In reply to Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
xen/arch/arm/numa_device_tree.c +67 -0
index bbe081dcd1..6e0d1d3d9f 100644
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
 
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d = %d\n",
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n",
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);
+    }
+
+    return 0;
+}
Signature · 1 line
2.25.1
#116 lore raw

Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Wei,

Quoted 2 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a

s/introduce/introduces/

Quoted 13 lines

new helper to parse this distance map.

---
  xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
  1 file changed, 67 insertions(+)
xen/arch/arm/numa_device_tree.c +0 -0
index bbe081dcd1..6e0d1d3d9f 100644
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
 

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +8 -0
      return 0;
  }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;

entry_count and i should be unsigned. len unfortunately can't because fdt_get_property expects a signed int.

Quoted 11 lines
xen/arch/arm/numa_device_tree.c +11 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )

I understand that entry_count may be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244

Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13 -0
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

You can use dt_next_cell() which will update the pointer for you.

Quoted 16 lines
xen/arch/arm/numa_device_tree.c +15 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d = %d\n",
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);
+    }
+
+    return 0;
+}

Cheers,

Signature · 1 line
Julien Grall
author
#117 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Julien,

Quoted 10 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:56
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Quoted 7 lines

Hi Wei,

Quoted 2 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a

s/introduce/introduces/

OK

Quoted 16 lines

Review

Quoted 9 lines

new helper to parse this distance map.

---
  xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
  1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,

int node,

Quoted 1 line
Quoted 14 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210

Quoted 10 lines
      return 0;

}

xen/arch/arm/numa_device_tree.c +8 -0
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;

entry_count and i should be unsigned. len unfortunately can't because fdt_get_property expects a signed int.

OK

Quoted 17 lines
Quoted 17 lines
xen/arch/arm/numa_device_tree.c +17 -0
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
Quoted 9 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231

Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4 -0
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )

I understand that entry_count may be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".

From the fdt_get_property's comment, when prop is NULL, the len can be negative. But, yes, prop==NULL check will handled before this code. negative len will not reach here. I am ok to change it to "== 0"

Quoted 7 lines
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7 -0
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
Quoted 9 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

You can use dt_next_cell() which will update the pointer for you.

Thanks

Quoted 33 lines

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269

Quoted 11 lines
xen/arch/arm/numa_device_tree.c +10 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);
+    }
+
+    return 0;
+}

Cheers,

--
Julien Grall

#118 lore raw

Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

In reply to Wei Chen
Quoted 70 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
xen/arch/arm/numa_device_tree.c +53 -0
index bbe081dcd1..6e0d1d3d9f 100644
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
 
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d = %d\n",
+                   from, to, distance);

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n",
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);

I am a bit unsure about this last 2 lines: why calling numa_set_distance in the opposite direction only when to > from? Wouldn't it be OK to always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

author
#119 lore raw

RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Hi Stefano,

Quoted 95 lines

-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map

Review

Quoted 10 lines · Wei Chen

A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.

---
 xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
(quoted diff) +0 -0

b/xen/arch/arm/numa_device_tree.c

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,

Quoted 4 lines

index bbe081dcd1..6e0d1d3d9f 100644

xen/arch/arm/numa_device_tree.c +0 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,

int node,

Quoted 45 lines
xen/arch/arm/numa_device_tree.c +42 -0
     return 0;
 }
+
+/* Parse NUMA distance map v1 */
+int __init
+device_tree_parse_numa_distance_map_v1(const void *fdt, int node)
+{
+    const struct fdt_property *prop;
+    const __be32 *matrix;
+    int entry_count, len, i;
+
+    printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+    prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_WARNING
+               "NUMA: No distance-matrix property in distance-map\n");
+
+        return -EINVAL;
+    }
+
+    if ( len % sizeof(uint32_t) != 0 )
+    {
+        printk(XENLOG_WARNING
+               "distance-matrix in node is not a multiple of u32\n");
+        return -EINVAL;
+    }
+
+    entry_count = len / sizeof(uint32_t);
+    if ( entry_count <= 0 )
+    {
+        printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+        return -EINVAL;
+    }
+
+    matrix = (const __be32 *)prop->data;
+    for ( i = 0; i + 2 < entry_count; i += 3 )
+    {
+        uint32_t from, to, distance;
+
+        from = dt_read_number(matrix, 1);
+        matrix++;

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10 -0
+        to = dt_read_number(matrix, 1);
+        matrix++;
+        distance = dt_read_number(matrix, 1);
+        matrix++;
+
+        if ( (from == to && distance != NUMA_LOCAL_DISTANCE) ||
+            (from != to && distance <= NUMA_LOCAL_DISTANCE) )
+        {
+            printk(XENLOG_WARNING
+                   "Invalid nodes' distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,

Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5 -0
+                   from, to, distance);
+            return -EINVAL;
+        }
+
+        printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d

= %d\n",

xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265

Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6 -0
+               from, to, distance);
+        numa_set_distance(from, to, distance);
+
+        /* Set default distance of node B->A same as A->B */
+        if (to > from)
+             numa_set_distance(to, from, distance);
Quoted 9 lines

I am a bit unsure about this last 2 lines: why calling numa_set_distance in the opposite direction only when to > from? Wouldn't it be OK to always do both:

numa_set_distance(from, to, distance);
numa_set_distance(to, from, distance);

?

I borrowed this code from Linux, but here is my understanding:

First, I read some notes in Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.

Here is an example of distance-map node in DTB:
Sample#1, full list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 0  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 0  40>,
					   <2 1  20>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 0  20>,
					   <3 1  40>,
					   <3 2  20>,
					   <3 3  10>;
		};

Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1 0 20>. But, numa_set_distance(1, 0, 20) will be call twice.

Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:

		distance-map {
			 compatible = "numa-distance-map-v1";
			 distance-matrix = <0 0  10>,
					   <0 1  20>,
					   <0 2  40>,
					   <0 3  20>,
					   <1 1  10>,
					   <1 2  20>,
					   <1 3  40>,
					   <2 2  10>,
					   <2 3  20>,
					   <3 3  10>;
		};

There is not any "from > to" entry in the map. But using this partial map still can set all distances for all pairs. And numa_set_distance(1, 0, 20) will be only once.

Quoted 5 lines

But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).

Oh, I should see this comment first, then I will not post above comment : )

Quoted 11 lines

So my suggestion is to parse it as follows:

- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1

- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;

I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.

Cheers,
Wei Chen