Add device tree based NUMA support to Arm64
cover Add device tree based NUMA support to Arm64
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system. Like TX2, some Hisilicon chips and the Ampere Altra.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
Hongda Deng (2):
xen/arm: return default DMA bit width when platform is not set xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
Wei Chen (38):
tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf xen/arm: Print a 64-bit number in hex from early uart xen/x86: Initialize memnodemapsize while faking NUMA node
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API
xen/x86: Move NUMA memory node map functions to common xen/x86: Move numa_add_cpu_node to common
xen/x86: Move NR_NODE_MEMBLKS macro to common
xen/x86: Move NUMA nodes and memory block ranges to common xen/x86: Move numa_initmem_init to common
xen/arm: introduce numa_set_node for Arm
xen/arm: set NUMA nodes max number to 64 by default xen/x86: move NUMA API from x86 header to common header xen/arm: Create a fake NUMA node to use common code xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
xen: fdt: Introduce a helper to check fdt node type xen/arm: implement node distance helpers for Arm64 xen/arm: introduce device_tree_numa as a switch for device tree NUMA xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: Add boot and secondary CPU to NUMA system xen/arm: build CPU NUMA node map while creating cpu_logical_map xen/x86: decouple nodes_cover_memory with E820 map xen/arm: implement Arm arch helpers Arm to get memory map info
xen: move NUMA memory and CPU parsed nodemasks to common xen/x86: move nodes_cover_memory to common
xen/x86: make acpi_scan_nodes to be neutral
xen: export bad_srat and srat_disabled to extern
xen: move numa_scan_nodes from x86 to common
xen: enable numa_scan_nodes for device tree based NUMA xen/arm: keep guest still be NUMA unware
xen: introduce an arch helper to do NUMA init failed fallback xen/arm: enable device tree based NUMA in system init xen/x86: move numa_setup to common to support NUMA switch in command
line
xen/x86: move dump_numa info hotkey to common
tools/libs/util/libxlu_pci.c | 3 +- xen/arch/arm/Kconfig | 10 + xen/arch/arm/Makefile | 2 + xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 17 +- xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 162 +++++++++ xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++ xen/arch/arm/platform.c | 4 +- xen/arch/arm/setup.c | 14 + xen/arch/arm/smpboot.c | 37 +- xen/arch/x86/Kconfig | 2 +- xen/arch/x86/numa.c | 421 +---------------------- xen/arch/x86/srat.c | 147 +------- xen/common/Kconfig | 3 + xen/common/Makefile | 1 + xen/common/libfdt/fdt_ro.c | 15 + xen/common/numa.c | 588 ++++++++++++++++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/numa.h | 33 ++ xen/include/asm-arm/setup.h | 6 + xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/numa.h | 65 +--- xen/include/asm-x86/setup.h | 1 - xen/include/xen/libfdt/libfdt.h | 25 ++ xen/include/xen/nodemask.h | 2 + xen/include/xen/numa.h | 80 +++++ 31 files changed, 1325 insertions(+), 659 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.c
Signature · 1 line
2.25.1
RFC PATCH 1/40 tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
| libxlu_pci.c: In function 'xlu_pci_parse_bdf': | libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~ | libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~
--- tools/libs/util/libxlu_pci.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
tools/libs/util/libxlu_pci.c +1 -2
@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) {const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
Quoted 4 lines · Wei Chen
| libxlu_pci.c: In function 'xlu_pci_parse_bdf': | libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~
I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is
pci->domain = domain;
pci->bus = bus;
pci->dev = dev;
pci->func = func;Quoted 10 lines
| libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this function [-Werror=maybe-uninitialized]
Quoted 8 lines
| 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this function [-Werror=maybe-uninitialized] | 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~
I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar if(). May I ask what code base these were observed with? Is the change needed at all anymore?
Quoted 13 lines
tools/libs/util/libxlu_pci.c +1
-1@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) {const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33
Quoted 7 lines
tools/libs/util/libxlu_pci.c +0
-1@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Also - which compiler did you encounter this with?
Finally please don't forget to Cc maintainers.
Jan
RE: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
Hi Jan,
Quoted 16 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:50
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized forxlu_pci_parse_bdfQuoted 2 lines · Wei Chen
|
libxlu_pci.c: In function 'xlu_pci_parse_bdf':
| libxlu_pci.c:32:18: error: 'func' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 2 lines
| 32 | pcidev->func = func; | | ~~~~~~~~~~~~~^~~~~~
Quoted 8 lines
I'm afraid I can't spot such an assignment in the file (nor the two similar ones further down). All I can see is
pci->domain = domain; pci->bus = bus; pci->dev = dev; pci->func = func;
Sorry, I forgot to update my commit log with the latest code base. I revert this change in my current code, I can't reproduce it. I'm not sure if it's because I upgraded my build environment. Give me sometime, if I can reproduce it I will update the commit log in next version. If it's no longer needed, I will remove this patch from this series.
Quoted 18 lines
Quoted 4 lines
| libxlu_pci.c:51:29: note: 'func' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~~ | libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 6 lines
| 31 | pcidev->dev = dev; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:24: note: 'dev' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in thisfunction [-Werror=maybe-uninitialized]
Quoted 6 lines
| 30 | pcidev->bus = bus; | | ~~~~~~~~~~~~^~~~~ | libxlu_pci.c:51:19: note: 'bus' was declared here | 51 | unsigned dom, bus, dev, func, vslot = 0; | | ^~~ | libxlu_pci.c:78:26: error: 'dom' may be used uninitialized in this
Quoted 8 lines
function [-Werror=maybe-uninitialized]
Quoted 2 lines
| 78 | if ( dom & ~0xff ) | | ~~~~^~~~~~~I'm afraid I also can't spot a variable named "dom", nor a sufficiently similar
if(). May I ask what code base these were observed with? Is the change needed at all anymore?
same as above.
Quoted 24 lines
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char
Quoted 3 lines
tools/libs/util/libxlu_pci.c +0
-0@@ -15,7 +15,7 @@ static int parse_bdf(libxl_device_pci *pci, const char*str, const char **endp)
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp)
Quoted 8 lines
tools/libs/util/libxlu_pci.c +1
-1{const char *ptr = str;unsigned int colons =0 ; -unsigned int domain, bus, dev, func; +unsigned int domain =0 , bus =0 , dev =0 , func =0 ;int n;/* Count occurrences of ':' to detrmine presence/absence of the 'domain' */
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char
Quoted 1 line
tools/libs/util/libxlu_pci.c +0
-0@@ -28,7 +28,6 @@ static int parse_bdf(libxl_device_pci *pci, const char*str, const char **endp)
tools/libs/util/libxlu_pci.c static int parse_bdf(libxl_device_pci *pci, const char *str, const char **endp) · line 33
Quoted 8 lines
tools/libs/util/libxlu_pci.c +0
-1ptr = str;switch (colons) {case 1 : - domain =0 ;if (sscanf (ptr,"%x:%x.%n" , &bus, &dev, &n) !=2 )return ERROR_INVAL ;break ;
Quoted 4 lines
Also - which compiler did you encounter this with?
Finally please don't forget to Cc maintainers.
If this patch still needed, I will do in next version.
Quoted 1 line
Jan
RFC PATCH 2/40 xen/arm: Print a 64-bit number in hex from early uart
In reply to Wei ChenCurrent putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +5 -4
@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART. * x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8 + mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble at a time */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart
Hi Wei,
Quoted 10 lines · Wei Chen
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -
Quoted 4 lines
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
Signed-off-by Wei Chen <[email protected]>
Quoted 9 lines
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +0
-0
xen/arch/arm/arm64/head.S puts: · line 865
Quoted 5 lines
xen/arch/arm/arm64/head.S +1
-1@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART.
As you modify the line, can you take the opportunity to write:
/*
* Print a 64-bit...
And also drop the second sentence as it the code has not been PL011 specific for quite a while now.
Quoted 8 lines
xen/arch/arm/arm64/head.S +1
-1* x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8
xen/arch/arm/arm64/head.S puts: · line 879
Quoted 10 lines
xen/arch/arm/arm64/head.S +3
-2+ mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble at a time */
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart
Hi Julien,
Quoted 46 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:05
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uartHi Wei,
Quoted 10 lines · Wei Chen
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -Quoted 4 lines
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
Signed-off-by Wei Chen <[email protected]>Acked-by Julien Grall <[email protected]>Quoted 9 lines
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)xen/arch/arm/arm64/head.S +0
-0xen/arch/arm/arm64/head.S puts: · line 865
Quoted 5 lines
xen/arch/arm/arm64/head.S +1
-1@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. Specific to the PL011 UART.As you modify the line, can you take the opportunity to write:
Quoted 6 lines
/*
* Print a 64-bit...And also drop the second sentence as it the code has not been PL011 specific for quite a while now.
Ok, I will do it in next version
Quoted 24 lines
Quoted 7 lines
xen/arch/arm/arm64/head.S +1
-1* x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8xen/arch/arm/arm64/head.S puts:
Quoted 10 lines
xen/arch/arm/arm64/head.S +3
-2+ mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble ata time */
Quoted 1 line
Cheers,
--
Julien Grall
RFC PATCH 3/40 xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Wei ChenWhen system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of _memnodemap. Xen hadn't done it, and Xen should assert in phys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.
In this patch, we set memnodemapsize to ARRAY_SIZE(_memnodemap) to fix it.
--- xen/arch/x86/numa.c | 2 ++ 1 file changed, 2 insertions(+)
xen/arch/x86/numa.c +2 -0
@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); +nodes_clear (node_online_map);node_set_online (0 );for ( i =0 ; i < nr_cpu_ids; i++ )
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
Quoted 10 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of
_memnodemap. Xen hadn't done it, and Xen should assert inphys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.
How about we promote VIRTUAL_BUG_ON() to expand to at least ASSERT()?
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 7 lines
xen/arch/x86/numa.c +1
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);
But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Jan
RE: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
Hi Jan,
Quoted 14 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA nodeQuoted 4 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
Quoted 9 lines
Quoted 6 lines
In this case the memory node map doesn't need to be allocated from boot pages. But we should set the memnodemapsize to the array size of
_memnodemap. Xen hadn't done it, and Xen should assert inphys_to_nid. But because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace ASSERT, this bug will not be triggered.How about we promote
VIRTUAL_BUG_ON()to expand to at leastASSERT()?
That would be good. Frankly, we discovered this because we used ASSERT in Arm and then noticed that x86 was using VIRTUAL_BUG_ON.
Quoted 4 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 3 lines
xen/arch/x86/numa.c +0
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)
Quoted 8 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 4 lines
xen/arch/x86/numa.c +1
-0/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in acpi_scan_nodes, but acpi_scan_nodes doesn't reset memnodemap when it failed. I think maybe we can add:
memnodemap[0] = 0;
memnodemapsize = 1;How do you think about it?
Quoted 1 line
Jan
Re: [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node
Quoted 17 lines · Wei Chen
Quoted 8 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:33xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -270,6 +270,8 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 8 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 273
Quoted 4 lines
xen/arch/x86/numa.c +1
-0/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap);But this doesn't reflect reality then, does it? We'd rather want to set the size to 1, I would think.
Quoted 6 lines
Yes, you're right. Actually, we just only used 1 slot. But furthermore, memnodemap[0] may be set in
acpi_scan_nodes, butacpi_scan_nodesdoesn't reset memnodemap when it failed. I think maybe we can add:memnodemap[0] = 0; memnodemapsize = 1;How do you think about it?
Well, yes, if data may have been put there, then resetting of course makes sense.
Jan
RFC PATCH 4/40 xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenFrom: Hongda Deng <[email protected]>
In current code, arch_get_dma_bitsize will return 32 when platorm or platform->dma_bitsize is not set. It's not resonable, for Arm, we don't require to reserve DMA memory. So we set dma_bitsize always be 0. In NO-NUMA system, arch_get_dma_bitsize will not be invoked, so dma_bitsize will not be overrided by this function. But in NUMA system, once the online nodes are greater than 1, this function will be invoked. The dma_bitsize will be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware. Only platform setting can override dma_bitsize. So in this patch, we return default dma_bitsize, when platform and platorm->dma_bitsize are not set.
--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)
xen/arch/arm/platform.c +3 -1
@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;static bool __init platform_is_compatible (const struct platform_desc *plat) {
@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize :32 ; +return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize + : dma_bitsize; }/*
xen/common/page_alloc.c +1 -1
@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 7 lines · Wei Chen
xen/arch/arm/platform.c +1
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;
This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;
... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.
Jan
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Hi Jan,
Quoted 14 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setxen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
Quoted 7 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 4 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
Ok, I will place it to a header.
Quoted 13 lines
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 9 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Sorry, I am not very clear about this comment.
Quoted 4 lines
Also I'm afraid the description does not make clear why this variable is what you want to use. Connected to this is the question why you need to consume it on Arm in the first place, when x86 never had the need.
Different Arm platforms may have different DMA bitsize. So in my previous thought, If Arm platform doesn't provide any DMA bitsize info, I will return the system DMA bitsize (dma_bitsize) in arch_get_dma_bitsize. But your comment made me think again. My current change may have fallen into a logical loophole. dma_bitsize as a high level variable, it's value depends on xen boot command line or arch_get_dma_bitsize. I can't use it for arch_get_dma_bitsize's input.
So I think, in next version, I will discard the changes of dma_bitsize, just keep the changes in arch_get_dma_bitsize to return 0, when platform hasn't specify the DMA bitsize. Just like:
unsigned int arch_get_dma_bitsize(void)
{
- return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+ return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize
+ : 0;
}
Thanks,
Wei Chen
Quoted 1 line
Jan
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Quoted 32 lines · Wei Chen
Quoted 8 lines
From: Jan Beulich <[email protected]>
Sent: 2021年8月11日 18:54xen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 3 lines · Wei Chen
xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
Quoted 7 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[]; · line 30
Quoted 4 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;This is a no-go: Declarations need to live in a header which the producer and all consumers include. Else ...
Ok, I will place it to a header.
Quoted 13 lines
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 230
Quoted 9 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;... a change here (of e.g. the type) will go unnoticed by the compiler, and the consumer of the variable may no longer work correctly.
Quoted 1 line
Sorry, I am not very clear about this comment.
I've merely been trying to explain _why_ the declaration needs to be in a header.
Jan
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
In reply to Wei ChenHi,
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,
s/resonable/reasonable/
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize() is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 40 lines
Only
platform setting can overridedma_bitsize. So in this patch, we return defaultdma_bitsize, when platform andplatorm->dma_bitsizeare not set.Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)xen/arch/arm/platform.c +3
-1@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[], _eplatform[];/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize;static bool __init platform_is_compatible (const struct platform_desc *plat) {@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize :32 ; +return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize + : dma_bitsize; }/* xen/common/page_alloc.c +0
-0
xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Hi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setHi,
Quoted 8 lines
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 7 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.
Quoted 9 lines
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?
Quoted 5 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit?
In my opinion, arbitrary to give 32 on AArch64 doesn't make sense. But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from platform->dma_bitsize. In this case, return 0 will also cause some issue.
Quoted 3 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
For Dom0, we squash the patch#5 into this patch?
Quoted 3 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...
if ( !dma_bitsize && (num_online_nodes() > 1) )
dma_bitsize = arch_get_dma_bitsize();Quoted 58 lines
xen/arch/arm/platform.c extern const struct platform_desc _splatform[],
Quoted 17 lines
Only
platform setting can overridedma_bitsize. So in this patch, we return defaultdma_bitsize, when platform andplatorm->dma_bitsizeare not set.Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/platform.c | 4 +++- xen/common/page_alloc.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)xen/arch/arm/platform.c +0
-0@@ -27,6 +27,7 @@ extern const struct platform_desc _splatform[],_eplatform[];
xen/arch/arm/platform.c extern const struct platform_desc _splatform[], _eplatform[];
Quoted 6 lines
xen/arch/arm/platform.c +1
-0/* Pointer to the current platform description */ static const struct platform_desc *platform; +extern unsigned int dma_bitsize; static bool __init platform_is_compatible(const struct platform_desc*plat)
xen/arch/arm/platform.c bool platform_device_is_blacklisted(const struct
Quoted 2 lines
xen/arch/arm/platform.c +0
-0{@@ -157,7 +158,8 @@ bool platform_device_is_blacklisted(const struct
dt_device_node*node)Quoted 17 lines
xen/arch/arm/platform.c +1
-1unsigned int arch_get_dma_bitsize (void ) { -return ( platform && platform->dma_bitsize ) ? platform- dma_bitsize :32 ; + return ( platform && platform->dma_bitsize ) ? platform-
dma_bitsizexen/arch/arm/platform.c +1
-0+ : dma_bitsize; }/* xen/common/page_alloc.c +0
-0xen/common/page_alloc.c static bool __read_mostly scrub_debug; · line 233
Quoted 8 lines
xen/common/page_alloc.c +1
-1@@ -227,7 +227,7 @@ static bool __read_mostly scrub_debug;* Bit width of the DMA heap -- used to override NUMA-node-first. * allocation strategy, which can otherwise exhaust low memory. */ -static unsigned int dma_bitsize; +unsigned int dma_bitsize;integer_param ("dma_bits" , dma_bitsize);/* Offlined page list, protected by heap_lock. */ Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 35 lines
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setHi,
Quoted 8 lines
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 13 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by every devices.Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.
Quoted 7 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may not have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
Quoted 10 lines
Quoted 7 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.
We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.
Quoted 10 lines
But as you mentioned, if Xen is running on a platform with DMA limitation, but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from
platform->dma_bitsize. In this case, return 0 will also cause some issue.Quoted 3 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for now).
For Dom0, we squash the patch#5 into this patch?
Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...
Quoted 10 lines
Quoted 4 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();
... Why do you think we need this check on Arm when NUMA is enabled?
We can discuss how to remove it once this is answered.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Hi Julien,
Quoted 59 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 7 lines
Quoted 7 lines
when platform is not set
Hi,
Quoted 2 lines · Wei Chen
From: Hongda Deng <[email protected]>
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 6 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by everydevices.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.Quoted 6 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They may
Quoted 8 lines
not
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )
Anyway, keep the arch_get_dma_bitsize as original maybe the best approach for now.
Quoted 22 lines
Quoted 10 lines
Quoted 7 lines
Anyway, there are plenty of platform out that have devices which can't DMA into memory above 32-bit. On RPI, this is even lower (30-bit).
So I would be cautious to change the default limit.
How about return 0 when platform doesn't specify the limit? In my opinion, arbitrary to give 32 on AArch64 doesn't make sense.
We have to care about the common use-case. We added this restriction a few years ago because we had a few reports of users using only 32-bit DMA capable device.
Quoted 1 line
But as you mentioned, if Xen is running on a platform with DMA
limitation,
Quoted 6 lines
but we have not listed this platform in Xen supported list, Xen cannot get DMA limit from
platform->dma_bitsize. In this case, return 0 will also cause some issue.Quoted 2 lines
At the moment, the only place on Arm where we need DMA-able memory is for dom0. This is allocated at boot and can't change afterwards (for
Quoted 7 lines
now).
Quoted 3 lines
Quoted 1 line
For Dom0, we squash the patch#5 into this patch?
Let me answer with another question. Why should we modify the Arm code rather than the common code? IOW...
As I answered above, let's keep arch_get_dma_bitsize as original then we don't need to modify Arm code.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 8 lines
Quoted 5 lines
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();... Why do you think we need this check on Arm when NUMA is enabled?
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 68 lines
Quoted 59 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 7 lines
Quoted 7 lines
when platform is not set
Hi,
Quoted 2 lines · Wei Chen
From: Hongda Deng <[email protected]>
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Ok
Quoted 6 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We need to be able to allocate some DMA-able memory that can be used by everydevices.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against our hardware design. We don't have that kind of restriction on hardware.Quoted 6 lines
Quoted 3 lines
What do you mean by "hardware design"? Are you referring to the server you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They mayQuoted 8 lines
not
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged). What you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
Quoted 3 lines
OK, I understand now. Let's keep 32-bit as default value, but even in this case, how about DMA-16 devices? Although these devices are very rare, they still exist : )
I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.
My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).
If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
So I would explore to remove the NUMA check for drop the DMA zone. FAOD, both suggestion are for Arm only. For x86, they need to be kept.
Quoted 8 lines
Quoted 5 lines
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();... Why do you think we need this check on Arm when NUMA is enabled?
Quoted 7 lines
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this check.
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
How about replacing the second part of the check with a new helper arch_have_default_dma_zone() (or a different name)?
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set
Hi Julien,
Quoted 120 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:18
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not setQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:20
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 30 lines
Quoted 30 lines
when platform is not set
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:28
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 04/40] xen/arm: return default DMA bitwidth
Quoted 13 lines
Quoted 12 lines
when platform is not set
Hi,
Quoted 4 lines · Wei Chen
From: Hongda Deng <[email protected]>
In current code,
arch_get_dma_bitsizewill return 32 when platorm orplatform->dma_bitsizeis not set. It's not resonable, for Arm,s/resonable/reasonable/
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Ok
Quoted 5 lines
Quoted 3 lines
we don't require to reserve DMA memory. So we set
dma_bitsizealways be 0. In NO-NUMA system,arch_get_dma_bitsizewill not be invoked, sodma_bitsizewill not be overrided by this function.
arch_get_dma_bitsize()is also used to allocate dom0 memory. We needto
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
be able to allocate some DMA-able memory that can be used by every
devices.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
But in NUMA
system, once the online nodes are greater than 1, this function will be invoked. Thedma_bitsizewill be limited to 32. That means, only first 4GB memory can be used for DMA. But that's against ourhardware
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
design. We don't have that kind of restriction on hardware.
What do you mean by "hardware design"? Are you referring to the
server
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
you boot Xen on?
Yes. I will change it to some neutral words. something like:
"But that could not reflect some hardware's real DMA ability. They maynot
Quoted 1 line
have kind of restriction on hardware." ?
The thing is DMA ability is not about the platform itself. It is more about the devices (this could just be a PCI card you just plugged).
What
Quoted 5 lines
Quoted 3 lines
you seem to suggest is no-one will ever plug such card on your platform. Is that correct?
OK, I understand now. Let's keep 32-bit as default value, but even in
this
Quoted 1 line
case, how about DMA-16 devices? Although these devices are very rare,
they
Quoted 1 line
still exist : )
I haven't heard anyone reporting issues with them on Xen on Arm. So I assume that either it works or no-one is using them.
My main point is we need to care about the common use case. 32-bit DMA device is still a thing and caused trouble to some of our users (e.g. NXP).
If tomorrow, someone report issue with 16-bit DMA device, then we can consider our options how to handle.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I would explore to remove the NUMA check for drop the DMA zone.
FAOD,
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
both suggestion are for Arm only. For x86, they need to be kept.
Without introducing new flag, such as
lowmem_for_dma, it's a little hard to skip the numa node check. Unless we crudely add #ifdef ARCH to common code, which is not what we want to see ...if ( !dma_bitsize && (num_online_nodes() > 1) ) dma_bitsize = arch_get_dma_bitsize();Quoted 5 lines
Quoted 2 lines
... Why do you think we need this check on Arm when NUMA is enabled?
I didn't think Arm needs, what I said is introduce a flag to disable this check for Arm or other Architectures that they don't need this
check.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
We can discuss how to remove it once this is answered.
I think we can start to discuss it.
How about replacing the second part of the check with a new helper
arch_have_default_dma_zone()(or a different name)?
This seems like a method worth trying, I will try to follow this way to compose next version.
Quoted 5 lines
Cheers,
--
Julien Grall
RFC PATCH 5/40 xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
In reply to Wei ChenFrom: Hongda Deng <[email protected]>
In previous patch, we make arch_get_dma_bitsize return 0 when dma_bitsize and platform->dma_bitsize are not set. But this will affect lowmem_bitsize in allocate_memory_11 for domain0. Because this function depends lowmem_bitsize to allocate memory below 4GB.
In current code, when arch_get_dma_bitsize return 0, lowmem_bitsize will be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.
This behavior is inconsistent with what allocate_memory_11 done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before set lowmem_bitsize. Avoid setting lowmem_bitsize to 0 by mistake.
--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +10 -1
@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ()); +unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/ + lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
Hi,
I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.
Quoted 13 lines · Wei Chen
From: Hongda Deng <[email protected]>
In previous patch, we make
arch_get_dma_bitsizereturn 0 whendma_bitsizeandplatform->dma_bitsizeare not set. But this will affectlowmem_bitsizeinallocate_memory_11for domain0. Because this function dependslowmem_bitsizeto allocate memory below 4GB.In current code, when
arch_get_dma_bitsizereturn 0,lowmem_bitsizewill be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.
Quoted 4 lines
This behavior is inconsistent with what
allocate_memory_11done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before setlowmem_bitsize. Avoid settinglowmem_bitsizeto 0 by mistake.
In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.
Quoted 16 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +0
-1@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain *d,int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ());
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277
Quoted 10 lines
xen/arch/arm/domain_build.c +8
-0+unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/
The coding style for comments is:
/*
* A
* B
*/
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282
Quoted 6 lines
xen/arch/arm/domain_build.c +2
-0+ lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
Hi Julien,
Quoted 9 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:32
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 05/40] xen/arm: Fixlowmem_bitsizewhenarch_get_dma_bitsizereturn 0
Quoted 5 lines
Hi,
I guess this patch may be dropped after my comment on patch #4. I will comment just on the process.
Ok
Quoted 50 lines · Wei Chen
Quoted 13 lines
From: Hongda Deng <[email protected]>
In previous patch, we make
arch_get_dma_bitsizereturn 0 whendma_bitsizeandplatform->dma_bitsizeare not set. But this will affectlowmem_bitsizeinallocate_memory_11for domain0. Because this function dependslowmem_bitsizeto allocate memory below 4GB.In current code, when
arch_get_dma_bitsizereturn 0,lowmem_bitsizewill be set to 0. In this case, we will get "No bank has been allocated below 0-bit." message while allocating domain0 memory. And the lowmem will be set to false.Quoted 4 lines
This behavior is inconsistent with what
allocate_memory_11done before, and doesn't meet this functions requirements. So we check arch_get_dma_bitsize's return value before setlowmem_bitsize. Avoid settinglowmem_bitsizeto 0 by mistake.In general, we want to avoid breaking bisection within a series. This means that this patch should be before patch #4.
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain
Quoted 12 lines
Signed-off-by Wei Chen <[email protected]>Signed-off-by Hongda Deng <[email protected]>--- xen/arch/arm/domain_build.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +0
-0@@ -265,9 +265,18 @@ static void __init allocate_memory_11(struct domain*d,
Quoted 4 lines
xen/arch/arm/domain_build.c +0
-1int i;bool lowmem = true; -unsigned int lowmem_bitsize =min (32U ,arch_get_dma_bitsize ());xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 277
Quoted 10 lines
xen/arch/arm/domain_build.c +8
-0+unsigned int lowmem_bitsize =arch_get_dma_bitsize ();unsigned int bits; +/* +When dma_bitsize and platform->dma_bitsize are not set, +arch_get_dma_bitsize will return 0. That means this system +doesn't need to reserve memory for DMA. But in order to +meet above requirements, we still need to try to allocate +memory below 4GB for Dom0. +*/
Quoted 7 lines
The coding style for comments is:
/*
* A
* B
*/
I will fix it.
Quoted 11 lines
xen/arch/arm/domain_build.c static void __init allocate_memory_11(struct domain *d, · line 282
Quoted 6 lines
xen/arch/arm/domain_build.c +2
-0+ lowmem_bitsize = lowmem_bitsize ?min (32U , lowmem_bitsize) :32U ; +/* * TODO: Implement memory bank allocation when DOM0 is not direct * mapped Cheers,
--
Julien Grall
RFC PATCH 6/40 xen: decouple NUMA from ACPI in Kconfig
In reply to Wei ChenIn current Xen code only implments x86 ACPI-based NUMA support. So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.
In this patch series, we introduced device tree based NUMA for Arm. That means we will have two NUMA implemetations, so in this patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.
--- xen/arch/x86/Kconfig | 2 +- xen/common/Kconfig | 3 +++ xen/drivers/acpi/Kconfig | 3 ++- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-x86/config.h | 1 - 5 files changed, 7 insertions(+), 4 deletions(-)
xen/arch/x86/Kconfig +1 -1
@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMA config ARCH_DEFCONFIG string
xen/common/Kconfig +3 -0
@@ -67,6 +67,9 @@ config MEM_ACCESS config NEEDS_LIBELF bool +config NUMA + bool + menu "Speculative hardening" config SPECULATIVE_HARDEN_ARRAY
xen/drivers/acpi/Kconfig +2 -1
@@ -5,5 +5,6 @@ config ACPI config ACPI_LEGACY_TABLES_LOOKUP bool -config NUMA +config ACPI_NUMA bool + select NUMA
xen/drivers/acpi/Makefile +1 -1
@@ -3,7 +3,7 @@ obj-y += utilities/ obj-$(CONFIG_X86) += apei/ obj-bin-y += tables.init.o -obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_ACPI_NUMA) += numa.o obj-y += osl.o obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
xen/include/asm-x86/config.h +0 -1
@@ -31,7 +31,6 @@/* Intel P4 currently has largest cache line (L2 line size is 128 bytes). */ #define CONFIG_X86_L1_CACHE_SHIFT 7 -#define CONFIG_ACPI_NUMA 1#define CONFIG_ACPI_SRAT 1#define CONFIG_ACPI_CSTATE 1
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
xen/arch/x86/Kconfig config X86 · line 27
Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1
-1@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMA
We try to keep this alphabetically sorted, so please move up the replacement line. Then
Jan
RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
Hi Jan,
Quoted 20 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月12日 23:37
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI inKconfigxen/arch/x86/Kconfig config X86 · line 27
Quoted 8 lines · Wei Chen
xen/arch/x86/Kconfig +1
-1@@ -24,7 +24,7 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA + select ACPI_NUMAWe try to keep this alphabetically sorted, so please move up the replacement line. Then
Quoted 2 lines
Reviewed-by Jan Beulich <[email protected]>
Thanks, I will do it in next version.
Quoted 1 line
Jan
Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
In reply to Wei ChenHi Wei,
Quoted 1 line · Wei Chen
In current Xen code only implments x86 ACPI-based NUMA support.
s/implments/implements/
Quoted 5 lines
So in Xen
Kconfigsystem, NUMA equals toACPI_NUMA. x86 selects NUMA by default, andCONFIG_ACPI_NUMAis hardcode inconfig.h.In this patch series, we introduced device tree based NUMA for Arm.
The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:
"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".
Quoted 1 line
That means we will have two NUMA implemetations, so in this
s/implemetations/implementations/
Quoted 2 lines
patch we decouple NUMA from ACPI based NUMA in
Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig
Hi Julien,
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月13日 0:55
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI inKconfig
Quoted 6 lines
Hi Wei,
Quoted 1 line · Wei Chen
In current Xen code only implments x86 ACPI-based NUMA support.
s/implments/implements/
Got it.
Quoted 10 lines
Quoted 6 lines
So in Xen
Kconfigsystem, NUMA equals toACPI_NUMA. x86 selects NUMA by default, andCONFIG_ACPI_NUMAis hardcode inconfig.h.In this patch series, we introduced device tree based NUMA for Arm.
The concept of patch series only applies to the ML. Once checked-in this is only a series of commit. So I would write:
Quoted 7 lines
"In a follow-up patch, we will introduce support for NUMA using the Device-Tree".
Quoted 1 line
That means we will have two NUMA implemetations, so in this
s/implemetations/implementations/
Thanks, I will update the commit log in next version.
Quoted 7 lines
Quoted 2 lines
patch we decouple NUMA from ACPI based NUMA in
Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.Cheers,
--
Julien Grall
RFC PATCH 7/40 xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenOnly Arm64 supports NUMA, the CONFIG_NUMA could not be enabled for Arm32. Even in Arm64, users still can disable the CONFIG_NUMA through Kconfig option. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.
--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA ) +/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables:
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.
What do you mean by "could not be enabled"?
Quoted 15 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h line 8
Quoted 4 lines
xen/include/asm-arm/numa.h +1
-0@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )
NIT: We tend to use #ifndef rather than #if !defined(...)
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables:
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Hi Julien,
Quoted 9 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 7 lines
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA. In this case, this Kconfig option could not be enabled on Arm32.
Quoted 17 lines
Quoted 17 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 12 lines
Quoted 12 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIHi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.
... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@
Quoted 28 lines
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien Grall
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Hi Julien,
Quoted 41 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 4 lines
Quoted 4 lines
NUMA API
Hi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 6 lines
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make CONFIG_NUMA enabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed. I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use? And how can we test this code? Before this patch, I had checked Linux, and found that OF_NUMA only selected by Arm64 not Arm32. But if you feel the need to add to arm32, I have no problem with that.
Quoted 51 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Even in Arm64, users still can disable
theCONFIG_NUMAthroughKconfigoption. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +0
-0@@ -5,6 +5,8 @@Quoted 28 lines
Quoted 6 lines
xen/include/asm-arm/numa.h line 8
Quoted 3 lines
xen/include/asm-arm/numa.h +1
-0typedef u8 nodeid_t ; +#if !defined(CONFIG_NUMA )NIT: We tend to use #ifndef rather than #if !defined(...)
OK, I will change related changes in this series.
Quoted 19 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +1
-0+/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 34
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: Cheers,
--
Julien GrallCheers,
--
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 45 lines
Quoted 45 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 4 lines
Quoted 4 lines
NUMA API
Hi Wei,
Quoted 8 lines
Quoted 5 lines · Wei Chen
Quoted 2 lines
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And that's fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.
Quoted 7 lines
Quoted 3 lines
... you continue to say "couldn't be enabled" without clarifying whether this mean that the build will break or this was just not tested because you don't have any platform.
Ok, I understand your concern. Yes, my words would lead to mis-understanding. If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.
When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Technically you already added the code because arch/arm/ is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...
And how can we test this code?
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Hi Julien,
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 26 lines
Quoted 26 lines
NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 8 lines
Quoted 8 lines
NUMA API
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And
that's
Quoted 6 lines
Quoted 6 lines
fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying
whether
Quoted 3 lines
Quoted 2 lines
this mean that the build will break or this was just not tested because you don't have any platform.
Quoted 9 lines
Quoted 1 line
Ok, I understand your concern. Yes, my words would lead to mis-
understanding.
Quoted 2 lines
If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
It could not build because I have not tried to enable device_tree_numa option for Arm32 but enabled NUMA for arm32.
I have tested it again, yes, simple enable device_tree_numa and NUMA for arm32 can build a image successfully.
So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work? And I will remove "could not enable for Arm32" from commit log.
Quoted 12 lines
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Quoted 8 lines
Technically you already added the code because
arch/arm/is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...And how can we test this code?
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.
Quoted 5 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 81 lines
Quoted 68 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 26 lines
Quoted 26 lines
NUMA API
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:34
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keepfake
Quoted 8 lines
Quoted 8 lines
NUMA API
Hi Wei,
Quoted 2 lines · Wei Chen
Only Arm64 supports NUMA, the
CONFIG_NUMAcould not be enabled for Arm32.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What do you mean by "could not be enabled"?
I have not seen any Arm32 hardware support NUMA, so I think we don't need to support Arm32 NUMA.
I understand that there may not be 32-bit platform with NUMA. And
that's
Quoted 6 lines
Quoted 6 lines
fine stating that in the commit message. However...
Quoted 2 lines
In this case, this
Kconfig
option could not be enabled on Arm32.... you continue to say "couldn't be enabled" without clarifying
whether
Quoted 3 lines
Quoted 2 lines
this mean that the build will break or this was just not tested because you don't have any platform.
Quoted 9 lines
Quoted 1 line
Ok, I understand your concern. Yes, my words would lead to mis-
understanding.
Quoted 2 lines
If we make
CONFIG_NUMAenabled in Arm32, it need Arm32 to implement some code to support NUMA common code. Otherwise the Arm32 build will failed.When I skimmed through the series, most of the code seems to be either in common, arm (bitness neutral). So I am not quite too sure why it would not build. Do you have more details?
It could not build because I have not tried to enable
device_tree_numaoption for Arm32 but enabled NUMA for arm32.
Quoted 6 lines
I have tested it again, yes, simple enable
device_tree_numaand NUMA for arm32 can build a image successfully.So, I think it's OK to enable this on Arm32, and I will do it in next version. But, can we still keep these FAKE APIs? If user don't want to enable NUMA they still can make Xen work?
Yes, we still need to keep the FAKE APIs. I was only commenting about the wording in the commit message.
Quoted 21 lines
And I will remove "could not
enable for Arm32" from commit log.Quoted 18 lines
Quoted 11 lines
I have not tried to implement those code for Arm32. And I found there is no Arm32 machine support NUMA, so I wanted Arm32 to use fake NUMA API as before.
Quoted 4 lines
To put it differently, the code for NUMA looks bitness neutral. So I cannot really what what prevent us to potentially use it on Arm 32-bit.
Yes, you're right, it's neutral. But do we really need to add code to an ARCH that it may never use?
Technically you already added the code because
arch/arm/is common between arm32 and arm64. My only ask is to not make the new config depends on arm64. If you only build test it that fine because...And how can we test this code?
Quoted 7 lines
Quoted 2 lines
I don't expect any of the code to be an issue on arm32 as the code should mostly be arch neutral.
I mean, we don't have Arm32 NUMA machine to test, I don't know the code works well on Arm32 NUMA or not. I only can verify them on non-NUMA arm32, and make sure this code will not break existed machines.
I understood you don't have any Arm32 NUMA machine. But I don't see the lack of testing as an issue because the code doesn't look to contain bits that may rely on arm64. So there are very limited reasons for the code to break on arm32.
If we really want to test it, then it should be feasible to fake the NUMA node in the DT. However, I don't expect you to do it.
Cheers,
Signature · 1 line
Julien Grall
RFC PATCH 8/40 xen/x86: Move NUMA memory node map functions to common
In reply to Wei ChenIn the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from arch/x86 to common.
--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-) create mode 100644 xen/common/numa.c
xen/arch/x86/numa.c +0 -114
@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };
@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, -int numnodes,int shift,nodeid_t *nodeids) -{ -unsigned long spdx, epdx; -int i, res =-1 ; - -memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ; - size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the -* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {
xen/common/Makefile +1 -0
@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.o obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
xen/common/numa.c new file +131 -0
131 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -29
@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -#define VIRTUAL_BUG_ON (x)extern void numa_add_cpu (int cpu);extern void numa_init_array (void );
@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -/* Simple perfect hash to map pdx to node numbers */ -extern int memnode_shift; -extern unsigned long memnodemapsize; -extern u8 *memnodemap; - -struct node_data { -unsigned long node_start_pfn; -unsigned long node_spanned_pages; -}; - -extern struct node_data node_data[]; - -static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) -{ -nodeid_t nid; -VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); - nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); -return nid; -} - -#define NODE_DATA (nid) (&(node_data[nid])) - -#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) -#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) -#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ - NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node);void srat_parse_regions (u64 addr);
xen/include/xen/numa.h +35 -0
@@ -18,4 +18,39 @@ (((d)->vcpu !=NULL && (d)->vcpu[0 ] !=NULL ) \ ?vcpu_to_node ((d)->vcpu[0 ]) :NUMA_NO_NODE ) +/* The following content can be used when NUMA feature is enabled */ +#if defined(CONFIG_NUMA ) + +/* Simple perfect hash to map pdx to node numbers */ +extern int memnode_shift; +extern unsigned long memnodemapsize; +extern u8 *memnodemap; +extern typeof(*memnodemap) _memnodemap[64]; + +struct node_data { +unsigned long node_start_pfn; +unsigned long node_spanned_pages; +}; + +extern struct node_data node_data[]; +#define VIRTUAL_BUG_ON (x) + +static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) +{ +nodeid_t nid; +VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); + nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +return nid; +} + +#define NODE_DATA (nid) (&(node_data[nid])) + +#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) +#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) +#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ + NODE_DATA(nid)->node_spanned_pages) + +#endif /* CONFIG_NUMA */ +#endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common
Hi Wei,
Quoted 4 lines · Wei Chen
In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from
arch/x86to common.
I would add "No functional changes intended" to make clear this patch is only moving code.
Quoted 148 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-)create mode 100644
xen/common/numa.cxen/arch/x86/numa.c +0
-114@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, -int numnodes,int shift,nodeid_t *nodeids) -{ -unsigned long spdx, epdx; -int i, res =-1 ; - -memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ; - size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the -* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {xen/common/Makefile +0
-0
xen/common/Makefile obj-y += wait.o · line 57
Quoted 4 lines
xen/common/Makefile +1
-0@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.o
AFAICT, the Makefile is listing the file in alphabetical order. So please add numa.o in the correct position.
Quoted 9 lines
xen/common/Makefile +0
-0obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma lzo unlzo unlz4 unzstd earlycpio,$(n).init.o)xen/common/numa.c new file +0
-0
xen/common/numa.c line 10
Quoted 10 lines
xen/common/numa.c new file +10
-0@@ -0,0 +1,131 @@ +/* +* Generic VM initialization for x86-64 NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/mm.h> +#include <xen/string.h> +#include <xen/init.h> +#include <xen/ctype.h>
You don't seem to use any helpers./types directly defined by at least this header...
xen/common/numa.c line 13
Quoted 3 lines
xen/common/numa.c new file +3
-0+#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/time.h>
... this one and ...
xen/common/numa.c line 14
Quoted 1 line
xen/common/numa.c new file +1
-0+#include <xen/smp.h>
... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.
xen/common/numa.c line 16
Quoted 2 lines
xen/common/numa.c new file +2
-0+#include <xen/pfn.h> +#include <xen/sched.h>
Please sort the includes in alphabetical order.
Quoted 167 lines
xen/common/numa.c new file +115
-0+ +struct node_data node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +int memnode_shift; +typeof(*memnodemap) _memnodemap[64]; +unsigned long memnodemapsize; +u8 *memnodemap; + +/* +* Given a shift value, try to populate memnodemap[] +* Returns : +* 1 if OK +* 0 if memnodmap[] too small (of shift too small) +* -1 if node overlap or lost ram (shift too big) +*/ +static int __init populate_memnodemap (const struct node *nodes, +int numnodes,int shift,nodeid_t *nodeids) +{ +unsigned long spdx, epdx; +int i, res =-1 ; + +memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; +if ( (epdx >> shift) >= memnodemapsize ) +return 0 ; +do { +if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) +return -1 ; + +if ( !nodeids ) + memnodemap[spdx >> shift] = i; +else + memnodemap[spdx >> shift] = nodeids[i]; + + spdx += (1UL << shift); + }while ( spdx < epdx ); + res =1 ; + } + +return res; +} + +static int __init allocate_cachealigned_memnodemap (void ) +{ +unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); +unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); + + memnodemap =mfn_to_virt (mfn); + mfn <<=PAGE_SHIFT ; + size <<=PAGE_SHIFT ; +printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , + mfn, mfn + size); + memnodemapsize = size /sizeof (*memnodemap); + +return 0 ; +} + +/* +* The LSB of all start and end addresses in the node map is the value of the +* maximum possible shift. +*/ +static int __init extract_lsb_from_nodes (const struct node *nodes, +int numnodes) +{ +int i, nodes_used =0 ; +unsigned long spdx, epdx; +unsigned long bitfield =0 , memtop =0 ; + +for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; + bitfield |= spdx; + nodes_used++; +if ( epdx > memtop ) + memtop = epdx; + } +if ( nodes_used <=1 ) + i =BITS_PER_LONG -1 ; +else + i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); + memnodemapsize = (memtop >> i) +1 ; +return i; +} + +int __init compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids) +{ +int shift; + + shift =extract_lsb_from_nodes (nodes, numnodes); +if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) + memnodemap = _memnodemap; +else if (allocate_cachealigned_memnodemap () ) +return -1 ; +printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); + +if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) + { +printk (KERN_INFO "Your memory is not aligned you need to " +"rebuild your hypervisor with a bigger NODEMAPSIZE " +"shift=%d\n" , shift); +return -1 ; + } + +return shift; +}xen/include/asm-x86/numa.h +0
-29@@ -26,7 +26,6 @@ extern int compute_hash_shift(struct node *nodes, int numnodes,extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -#define VIRTUAL_BUG_ON (x)extern void numa_add_cpu (int cpu);extern void numa_init_array (void );@@ -47,34 +46,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -/* Simple perfect hash to map pdx to node numbers */ -extern int memnode_shift; -extern unsigned long memnodemapsize; -extern u8 *memnodemap; - -struct node_data { -unsigned long node_start_pfn; -unsigned long node_spanned_pages; -}; - -extern struct node_data node_data[]; - -static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) -{ -nodeid_t nid; -VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); - nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); -return nid; -} - -#define NODE_DATA (nid) (&(node_data[nid])) - -#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) -#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) -#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ - NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node);void srat_parse_regions (u64 addr);xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h line 22
Quoted 5 lines
xen/include/xen/numa.h +2
-0@@ -18,4 +18,39 @@ (((d)->vcpu !=NULL && (d)->vcpu[0 ] !=NULL ) \ ?vcpu_to_node ((d)->vcpu[0 ]) :NUMA_NO_NODE ) +/* The following content can be used when NUMA feature is enabled */ +#if defined(CONFIG_NUMA )
Please use #ifdef CONFIG_NUMA
xen/include/xen/numa.h line 28
Quoted 6 lines
xen/include/xen/numa.h +6
-0+ +/* Simple perfect hash to map pdx to node numbers */ +extern int memnode_shift; +extern unsigned long memnodemapsize; +extern u8 *memnodemap; +extern typeof(*memnodemap) _memnodemap[64];
AFAICT, this will be turned static against in a follow-up patch. Can this be avoided?
Quoted 19 lines
xen/include/xen/numa.h +18
-0+ +struct node_data { +unsigned long node_start_pfn; +unsigned long node_spanned_pages; +}; + +extern struct node_data node_data[]; +#define VIRTUAL_BUG_ON (x) + +static inline __attribute__((pure))nodeid_t phys_to_nid (paddr_t addr) +{ +nodeid_t nid; +VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); + nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +return nid; +} +
xen/include/xen/numa.h line 56
Quoted 10 lines
xen/include/xen/numa.h +9
-0+#define NODE_DATA (nid) (&(node_data[nid])) + +#define node_start_pfn (nid) (NODE_DATA(nid)->node_start_pfn) +#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages) +#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ + NODE_DATA(nid)->node_spanned_pages) + +#endif /* CONFIG_NUMA */ +#endif /* _XEN_NUMA_H */
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common
Hi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:47
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to commonHi Wei,
Quoted 8 lines
Quoted 4 lines · Wei Chen
In the later patches we will add NUMA support to Arm. Arm NUMA support will follow current memory node map management as x86. So this part of code can be common, in this case, we move this part of code from
arch/x86to common.I would add "No functional changes intended" to make clear this patch is only moving code.
Ok, I will do it.
Quoted 151 lines
Quoted 34 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 114 -------------------------------- xen/common/Makefile | 1 + xen/common/numa.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 29 -------- xen/include/xen/numa.h | 35 ++++++++++ 5 files changed, 167 insertions(+), 143 deletions(-)create mode 100644
xen/common/numa.cxen/arch/x86/numa.c +0
-8@@ -29,14 +29,6 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =NUMA_NO_NODE };@@ -58,112 +50,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; }xen/arch/x86/numa.c int srat_disabled(void) · line 69
Quoted 10 lines
xen/arch/x86/numa.c +0
-9-/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, - int numnodes, int shift, nodeid_t*nodeids)
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 5 lines
xen/arch/x86/numa.c +0
-5-{ -unsigned long spdx, epdx; -int i, res =-1 ; - - memset(memnodemap, NUMA_NO_NODE, memnodemapsize *
sizeof(*memnodemap));Quoted 32 lines
xen/arch/x86/numa.c +0
-32-for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the
Quoted 57 lines
xen/arch/x86/numa.c +0
-50-* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -}/* initialize NODE_DATA given nodeid and start/end */ void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) {xen/common/Makefile +0
-0
Quoted 9 lines
xen/common/Makefile obj-y += wait.o · line 57
Quoted 5 lines
xen/common/Makefile +1
-0@@ -54,6 +54,7 @@ obj-y += wait.o obj-bin-y += warning.init.o obj-$(CONFIG_XENOPROF) += xenoprof.o obj-y += xmalloc_tlsf.o +obj-$(CONFIG_NUMA) += numa.oAFAICT, the
Makefileis listing the file in alphabetical order. So please add numa.o in the correct position.
Thanks for the reminder, I will fix it.
Quoted 30 lines
xen/common/Makefile obj-y += wait.o · line 58
Quoted 2 lines
xen/common/Makefile +0
-0obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzmalzo unlzo unlz4 unzstd earlycpio,$(n).init.o)
Quoted 7 lines
xen/common/numa.c new file +0
-0xen/common/numa.c line 10
Quoted 10 lines
xen/common/numa.c new file +10
-0@@ -0,0 +1,131 @@ +/* +* Generic VM initialization for x86-64 NUMA setups. +* Copyright 2002,2003 Andi Kleen, SuSE Labs. +* Adapted for Xen: Ryan Harper <[email protected]> +*/ + +#include <xen/mm.h> +#include <xen/string.h> +#include <xen/init.h> +#include <xen/ctype.h> You don't seem to use any helpers./types directly defined by at least this header...
xen/common/numa.c line 13
Quoted 3 lines
xen/common/numa.c new file +3
-0+#include <xen/nodemask.h> +#include <xen/numa.h> +#include <xen/time.h> ... this one and ...
Quoted 6 lines
xen/common/numa.c line 14
Quoted 1 line
xen/common/numa.c new file +1
-0+#include <xen/smp.h> ... this one. Can you check the list of headers and introduce the minimum? If the dependency is required by another headers, then I think that dependency should be moved in the header requiring it.
I will check it in next version. If it isn't needed, I will remove it.
Quoted 5 lines
xen/common/numa.c line 16
Quoted 2 lines
xen/common/numa.c new file +2
-0+#include <xen/pfn.h> +#include <xen/sched.h> Please sort the includes in alphabetical order.
OK
Quoted 180 lines
Quoted 8 lines
xen/common/numa.c new file +8
-0+ +struct node_data node_data[MAX_NUMNODES ]; + +/* Mapping from pdx to node id */ +int memnode_shift; +typeof(*memnodemap) _memnodemap[64]; +unsigned long memnodemapsize; +u8 *memnodemap;xen/common/numa.c
Quoted 10 lines
xen/common/numa.c new file +10
-0+ +/* +* Given a shift value, try to populate memnodemap[] +* Returns : +* 1 if OK +* 0 if memnodmap[] too small (of shift too small) +* -1 if node overlap or lost ram (shift too big) +*/ +static int __init populate_memnodemap (const struct node *nodes, + int numnodes, int shift, nodeid_t*nodeids)
xen/common/numa.c
Quoted 5 lines
xen/common/numa.c new file +5
-0+{ +unsigned long spdx, epdx; +int i, res =-1 ; + + memset(memnodemap, NUMA_NO_NODE, memnodemapsize *
sizeof(*memnodemap));Quoted 32 lines
xen/common/numa.c new file +32
-0+for ( i =0 ; i < numnodes; i++ ) + { + spdx =paddr_to_pdx (nodes[i].start); + epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; +if ( spdx >= epdx ) +continue ; +if ( (epdx >> shift) >= memnodemapsize ) +return 0 ; +do { +if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) +return -1 ; + +if ( !nodeids ) + memnodemap[spdx >> shift] = i; +else + memnodemap[spdx >> shift] = nodeids[i]; + + spdx += (1UL << shift); + }while ( spdx < epdx ); + res =1 ; + } + +return res; +} + +static int __init allocate_cachealigned_memnodemap (void ) +{ +unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); +unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); + + memnodemap =mfn_to_virt (mfn); + mfn <<=PAGE_SHIFT ;xen/common/numa.c
Quoted 10 lines
xen/common/numa.c new file +10
-0+ size <<=PAGE_SHIFT ; +printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , + mfn, mfn + size); + memnodemapsize = size /sizeof (*memnodemap); + +return 0 ; +} + +/* +* The LSB of all start and end addresses in the node map is the value of the
110 lines not shown · raw message
Quoted 10 lines
8 lines not shown · raw message
36 lines not shown · raw message
Part of this message is not shown here. View the raw message.
RFC PATCH 9/40 xen/x86: Move numa_add_cpu_node to common
In reply to Wei ChenThis function will be reused by Arm later, so we move it from arch/x86 to common. But we keep cpu_to_node and node_to_cpumask to x86 header file. Because cpu_to_node and node_to_cpumask have different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
--- xen/arch/x86/numa.c | 9 --------- xen/common/numa.c | 11 +++++++++++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 ++ 4 files changed, 13 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +0 -9
@@ -29,16 +29,12 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { - [0 ...NR_CPUS -1 ] =NUMA_NO_NODE -};/* * Keep BIOS's CPU2node information, should not be used for memory allocaion */ nodeid_t apicid_to_node[MAX_LOCAL_APIC ] = { [0 ...MAX_LOCAL_APIC -1 ] =NUMA_NO_NODE }; -cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ;nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };
@@ -167,11 +163,6 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) (u64)end_pfn << PAGE_SHIFT); } -void numa_add_cpu (int cpu) -{ -cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;
xen/common/numa.c +11 -0
@@ -23,6 +23,12 @@ typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap; +nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { + [0 ...NR_CPUS -1 ] =NUMA_NO_NODE +}; + +cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +/* * Given a shift value, try to populate memnodemap[] * Returns :
@@ -129,3 +135,8 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; } + +void numa_add_cpu (int cpu) +{ +cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +}
xen/include/asm-x86/numa.h +0 -1
@@ -27,7 +27,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_add_cpu (int cpu);extern void numa_init_array (void );extern bool numa_off;
xen/include/xen/numa.h +2 -0
@@ -51,6 +51,8 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) +extern void numa_add_cpu (int cpu); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common
Hi Wei,
Quoted 6 lines · Wei Chen
This function will be reused by Arm later, so we move it from
arch/x86to common. But we keepcpu_to_nodeandnode_to_cpumaskto x86 header file. Becausecpu_to_nodeandnode_to_cpumaskhave different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node to common
Hi Julien,
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月24日 1:54
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 09/40] xen/x86: Movenuma_add_cpu_nodeto commonHi Wei,
Quoted 6 lines · Wei Chen
This function will be reused by Arm later, so we move it from
arch/x86to common. But we keepcpu_to_nodeandnode_to_cpumaskto x86 header file. Becausecpu_to_nodeandnode_to_cpumaskhave different implementation for x86 and Arm. We will move them to common header file when we change the Arm implementation in later patches.
Quoted 4 lines
AFAICT, the Arm helpers are gated by !CONFIG_NUMA and the ones in common code will be gated by
CONFIG_NUMA. So I am not quite too understand why they can't be moved now. Can you clarify it?
Yes, you're right. After we had introduced !CONFIG_NUMA, we can move node_to_cpumask and cpu_to_node in this patch too. I will fix it in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
RFC PATCH 10/40 xen/x86: Move NR_NODE_MEMBLKS macro to common
In reply to Wei ChenNot only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number. So we move it from x86 ACPI header file to common NUMA header file.
--- xen/include/asm-x86/acpi.h | 1 - xen/include/xen/numa.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)
xen/include/asm-x86/acpi.h +0 -1
@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;extern s8 acpi_numa;extern int acpi_scan_nodes (u64 start,u64 end); -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)
xen/include/xen/numa.h +1 -0
@@ -11,6 +11,7 @@#define NUMA_NO_DISTANCE 0xFF#define MAX_NUMNODES (1 << NODES_SHIFT) +#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 10/40] xen/x86: Move NR_NODE_MEMBLKS macro to common
Hi Wei,
Quoted 2 lines · Wei Chen
Not only x86 ACPI need this macro. Device tree based NUMA also needs this macro to present max memory block number.
AFAICT, a memory range described in DT cannot be split across multiple nodes. So I think we want to define NR_NODE_MEMBLKS as NR_MEM_BANKS.
Quoted 27 lines
So we move it from x86 ACPI header file to common NUMA header file.
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/acpi.h | 1 - xen/include/xen/numa.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)xen/include/asm-x86/acpi.h +0
-1@@ -103,7 +103,6 @@ extern unsigned long acpi_wakeup_address;extern s8 acpi_numa;extern int acpi_scan_nodes (u64 start,u64 end); -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h line 17
Quoted 7 lines
xen/include/xen/numa.h +1
-0@@ -11,6 +11,7 @@#define NUMA_NO_DISTANCE 0xFF#define MAX_NUMNODES (1 << NODES_SHIFT) +#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)#define vcpu_to_node (v) (cpu_to_node((v)->processor))
Cheers,
Signature · 1 line
Julien Grall
RFC PATCH 11/40 xen/x86: Move NUMA nodes and memory block ranges to common
In reply to Wei ChenThese data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)
xen/arch/x86/srat.c +0 -50
@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;
@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)
@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;
xen/common/numa.c +51 -0
63 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -8
@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );
xen/include/xen/numa.h +15 -0
@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; + +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common
Hi Wei,
Quoted 189 lines · Wei Chen
These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)xen/arch/x86/srat.c +0
-50@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;@@ -37,9 +36,6 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;xen/common/numa.c +51
-0@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +struct node nodes[MAX_NUMNODES ]__initdata ; +int num_node_memblks; +struct node node_memblk_range[NR_NODE_MEMBLKS ]; +nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; } +int valid_numa_range (u64 start,u64 end,nodeid_t node) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; + +if (nd->start <= start && nd->end >= end && + memblk_nodeid[i] == node) +return 1 ; + } + +return 0 ; +} + +int __init conflicting_memblks (u64 start,u64 end) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; +if (nd->start == nd->end) +continue ; +if (nd->end > start && nd->start < end) +return i; +if (nd->end == end && nd->start == start) +return i; + } +return -1 ; +} + +void __init cutoff_node (int i,u64 start,u64 end) +{ +struct node *nd = &nodes[i]; +if (nd->start < start) { + nd->start = start; +if (nd->end < nd->start) + nd->start = nd->end; + } +if (nd->end > end) { + nd->end = end; +if (nd->start > nd->end) + nd->start = nd->end; + } +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);xen/include/asm-x86/numa.h +0
-8@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );xen/include/xen/numa.h +0
-0@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64
Quoted 10 lines
xen/include/xen/numa.h +8
-0extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];
I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for acpi_numa_memory_affinity_init() and dtb_numa_memory_affinity_init().
There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in printk() and the hotplug bits.
So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 74
Quoted 11 lines
xen/include/xen/numa.h +7
-0+ +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
[1] https://lore.kernel.org/xen-devel/[email protected]/
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to common
Hi Julien,
Quoted 219 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 1:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges to commonHi Wei,
xen/arch/x86/srat.c static struct pxm2node __read_mostly
Quoted 26 lines · Wei Chen
These data structures and functions are used to create the mapping between node and memory blocks. In device tree based NUMA, we will reuse these data structures and functions, so we move this part of code from x86 to common.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 50 ------------------------------------- xen/common/numa.c | 51 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/numa.h | 8 ------ xen/include/xen/numa.h | 15 +++++++++++ 4 files changed, 66 insertions(+), 58 deletions(-)xen/arch/x86/srat.c +0
-1@@ -26,7 +26,6 @@ static struct acpi_table_slit *__read_mostly acpi_slit;static nodemask_t memory_nodes_parsed__initdata ;static nodemask_t processor_nodes_parsed__initdata ; -static struct node nodes[MAX_NUMNODES ]__initdata ;struct pxm2node {unsigned pxm;@@ -37,9 +36,6 @@ static struct pxm2node __read_mostlypxm2node[MAX_NUMNODES] =
xen/common/numa.c int __init compute_hash_shift(struct node *nodes,
Quoted 79 lines
xen/arch/x86/srat.c +0
-49static unsigned node_to_pxm (nodeid_t n); -static int num_node_memblks; -static struct node node_memblk_range[NR_NODE_MEMBLKS ]; -static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);static inline bool node_found (unsigned idx,unsigned pxm)@@ -104,52 +100,6 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; - -if (nd->start <= start && nd->end >= end && - memblk_nodeid[i] == node) -return 1 ; - } - -return 0 ; -} - -static __init int conflicting_memblks (u64 start,u64 end) -{ -int i; - -for (i =0 ; i < num_node_memblks; i++) { -struct node *nd = &node_memblk_range[i]; -if (nd->start == nd->end) -continue ; -if (nd->end > start && nd->start < end) -return i; -if (nd->end == end && nd->start == start) -return i; - } -return -1 ; -} - -static __init void cutoff_node (int i,u64 start,u64 end) -{ -struct node *nd = &nodes[i]; -if (nd->start < start) { - nd->start = start; -if (nd->end < nd->start) - nd->start = nd->end; - } -if (nd->end > end) { - nd->end = end; -if (nd->start > nd->end) - nd->start = nd->end; - } -} -static __init void bad_srat (void ) {int i;xen/common/numa.c +5
-0@@ -29,6 +29,11 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; +struct node nodes[MAX_NUMNODES ]__initdata ; +int num_node_memblks; +struct node node_memblk_range[NR_NODE_MEMBLKS ]; +nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -136,6 +141,52 @@ int __init compute_hash_shift(struct node *nodes,int numnodes,
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t
Quoted 83 lines
xen/common/numa.c +46
-0return shift; } +int valid_numa_range (u64 start,u64 end,nodeid_t node) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; + +if (nd->start <= start && nd->end >= end && + memblk_nodeid[i] == node) +return 1 ; + } + +return 0 ; +} + +int __init conflicting_memblks (u64 start,u64 end) +{ +int i; + +for (i =0 ; i < num_node_memblks; i++) { +struct node *nd = &node_memblk_range[i]; +if (nd->start == nd->end) +continue ; +if (nd->end > start && nd->start < end) +return i; +if (nd->end == end && nd->start == start) +return i; + } +return -1 ; +} + +void __init cutoff_node (int i,u64 start,u64 end) +{ +struct node *nd = &nodes[i]; +if (nd->start < start) { + nd->start = start; +if (nd->end < nd->start) + nd->start = nd->end; + } +if (nd->end > end) { + nd->end = end; +if (nd->start > nd->end) + nd->start = nd->end; + } +} +void numa_add_cpu (int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]);xen/include/asm-x86/numa.h +0
-8@@ -17,12 +17,6 @@ extern cpumask_t node_to_cpumask[];#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node]))#define node_to_cpumask (node) (node_to_cpumask[node]) -struct node { -u64 start,end; -}; - -extern int compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids);extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))@@ -45,8 +39,6 @@ static inline void clear_node_cpumask(int cpu)cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );xen/include/xen/numa.h +0
-0@@ -54,6 +54,21 @@ static inline __attribute__((pure)) nodeid_t
phys_to_nid(paddr_t addr)xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) · line 64
Quoted 11 lines
xen/include/xen/numa.h +8
-0extern void numa_add_cpu (int cpu); +struct node { +u64 start,end; +}; + +extern struct node nodes[MAX_NUMNODES ]; +extern int num_node_memblks; +extern struct node node_memblk_range[NR_NODE_MEMBLKS ]; +extern nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];I am not overly happy that the 4 ariables above are now exported. Looking at the code, they are (only?) used in arch specific code for
acpi_numa_memory_affinity_init()anddtb_numa_memory_affinity_init().
Quoted 8 lines
There bits touching the variables looks quite similar between the two functions. The main differences seems to be the messages in
printk()and the hotplug bits.So I think we should attempt to abstract the code. IIRC, we discussed some of the way to abstract when Vijay Kilari attempted to add support for NUMA (see [1]). It might be worth to have a look and see if you can re-use some of the ideas.
Ok, I will look at that thread. If it's useful, I will do it in next version.
Quoted 19 lines
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
Quoted 12 lines
xen/include/xen/numa.h +7
-0+ +extern int compute_hash_shift (struct node *nodes,int numnodes, +nodeid_t *nodeids); +extern int conflicting_memblks (u64 start,u64 end); +extern void cutoff_node (int i,u64 start,u64 end); +extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ [1]
https://lore.kernel.org/xen-devel/1500378106-2620-1-git-send-email-
[email protected]/--
Julien Grall
RFC PATCH 12/40 xen/x86: Move numa_initmem_init to common
In reply to Wei ChenThis function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:setup_node_bootmem, numa_init_array and numa_emulation.
As numa_initmem_init has been moved to common, _memnodemap is not used cross files. We can restore _memnodemap to static.
--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)
xen/arch/x86/numa.c +0 -118
@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )
@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; - -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1 cpu->node -mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ -if (hweight64 (sz) >1 ) - { -u64 x =1 ; -while ( (x <<1 ) < sz ) - x <<=1 ; -if ( x < sz/2 ) -printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); - sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); -for ( i =0 ; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; -if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; -printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , - i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >>20 ); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); -if ( memnode_shift <0 ) - { - memnode_shift =0 ; -printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); -return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) -{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ; -#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - -printk (KERN_INFO "%s\n" , - numa_off ?"NUMA turned off" :"No NUMA configuration found" ); - -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;
xen/common/numa.c +121 -1
142 lines not on this page · raw message
xen/include/asm-x86/numa.h +0 -5
@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );
xen/include/asm-x86/setup.h +0 -1
@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );
xen/include/xen/numa.h +7 -1
@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;
@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern void numa_init_array (void ); +extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
Hi Wei,
Quoted 8 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.
As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Quoted 151 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-118@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; - -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1 cpu->node -mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ -if (hweight64 (sz) >1 ) - { -u64 x =1 ; -while ( (x <<1 ) < sz ) - x <<=1 ; -if ( x < sz/2 ) -printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); - sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); -for ( i =0 ; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; -if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; -printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , - i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >>20 ); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); -if ( memnode_shift <0 ) - { - memnode_shift =0 ; -printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); -return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) -{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ; -#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - -printk (KERN_INFO "%s\n" , - numa_off ?"NUMA turned off" :"No NUMA configuration found" ); - -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-0
xen/common/numa.c line 16
Quoted 3 lines
xen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h>
NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
Quoted 21 lines
xen/common/numa.c +4
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ]; +bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns :
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 6 lines
xen/common/numa.c +3
-0@@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)
From an abstract PoV, start and end should be paddr_t. This should be done on a separate patch though.
Quoted 26 lines
xen/common/numa.c +26
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; + +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1 cpu->node +mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ;
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 10 lines
xen/common/numa.c +10
-0+numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} + +#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)
Here, this should be either "unsigned long" or ideally "mfn_t". Although, if you use "unsigned long", you will need to...
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 4 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;
... cast "(end_pfn - start_pfn)" to uin64_t or use pfn_to_paddr().
Quoted 42 lines
xen/common/numa.c +42
-0+ +/* Kludge needed for the hash function */ +if (hweight64 (sz) >1 ) + { +u64 x =1 ; +while ( (x <<1 ) < sz ) + x <<=1 ; +if ( x < sz/2 ) +printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); + sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); +for ( i =0 ; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; +if ( i == numa_fake -1 ) + sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; +printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , + i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >>20 ); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); +if ( memnode_shift <0 ) + { + memnode_shift =0 ; +printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); +return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + + return 0; +} +#endif + +void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) +{
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 10 lines
xen/common/numa.c +10
-0+int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )
(u64)v << PAGE_SHIFT should be switched to use pfn_to_paddr() or mfn_to_paddr() if you decide to make start_pfn and end_pfn typesafe.
xen/common/numa.c void numa_add_cpu(int cpu) · line 300
Quoted 9 lines
xen/common/numa.c +9
-0+return ; +#endif + +printk (KERN_INFO "%s\n" , + numa_off ?"NUMA turned off" :"No NUMA configuration found" ); + +printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , + (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT );
Same remark here. PRIx64 would also have to be switched to PRIpaddr.
Quoted 63 lines
xen/common/numa.c +13
-0+/* setup dummy node covering all memory */ + memnode_shift =BITS_PER_LONG -1 ; + memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); + +nodes_clear (node_online_map); +node_set_online (0 ); +for ( i =0 ; i < nr_cpu_ids; i++ ) +numa_set_node (i,0 ); +cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); +setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ); +}xen/include/asm-x86/numa.h +0
-5@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );xen/include/asm-x86/setup.h +0
-1@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );xen/include/xen/numa.h +0
-1@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node);
xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80
Quoted 10 lines
xen/include/xen/numa.h +7
-0+extern void numa_init_array (void ); +extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
Hi Julien,
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Movenuma_initmem_initto commonHi Wei,
Quoted 5 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.
Quoted 7 lines
Quoted 3 lines
As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Yes, thanks!
Quoted 155 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-11@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c int srat_disabled(void) · line 68
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 21 lines
xen/arch/x86/numa.c +0
-21-mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ];xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ - if ( hweight64(sz) > 1 ) - { - u64 x = 1; - while ( (x << 1) < sz ) - x <<= 1; - if ( x < sz/2 ) - printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11- sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); - for ( i = 0; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; - if ( i == numa_fake - 1 ) - sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; - printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >> 20); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); - if ( memnode_shift < 0 ) - { - memnode_shift = 0; - printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11-return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 6 lines
xen/arch/x86/numa.c +0
-6-{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - - printk(KERN_INFO "%s\n", - numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 24 lines
xen/arch/x86/numa.c +0
-18- -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-1--- a/xen/common/numa.c
Quoted 9 lines
xen/common/numa.c line 16
Quoted 5 lines
+++
b/xen/common/numa.cxen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
got it
Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];
Quoted 16 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 12 lines
xen/common/numa.c +5
-0+bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)From an abstract PoV, start and end should be
paddr_t. This should be
done on a separate patch though.
Ok.
Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 13 lines
xen/common/numa.c +13
-0+mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ; +numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} +
Quoted 9 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 5 lines
xen/common/numa.c +5
-0+#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)Here, this should be either "unsigned long" or ideally "
mfn_t". Although, if you use "unsigned long", you will need to...
Do we need a separate patch to do it?
Quoted 7 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 4 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;... cast "(
end_pfn-start_pfn)" touin64_tor usepfn_to_paddr().
Ok
Quoted 45 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 9 lines
xen/common/numa.c +9
-0+ +/* Kludge needed for the hash function */ + if ( hweight64(sz) > 1 ) + { + u64 x = 1; + while ( (x << 1) < sz ) + x <<= 1; + if ( x < sz/2 ) + printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+ sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); + for ( i = 0; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; + if ( i == numa_fake - 1 ) + sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; + printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >> 20); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); + if ( memnode_shift < 0 ) + { + memnode_shift = 0; + printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + +return 0 ; +} +#endif + +void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)
Quoted 15 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 11 lines
xen/common/numa.c +11
-0+{ +int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )(u64)v <<
PAGE_SHIFTshould be switched to usepfn_to_paddr()ormfn_to_paddr()if you decide to makestart_pfnandend_pfntypesafe.
Still need a separate patch to change it before move?
Quoted 6 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 5 lines
xen/common/numa.c +5
-0+return ; +#endif + + printk(KERN_INFO "%s\n", + numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 7 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 300
Quoted 4 lines
xen/common/numa.c +4
-0+ +printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , + (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT );Same remark here. PRIx64 would also have to be switched to PRIpaddr.
Hmm, It seems I'd better to use a separate patch to do PRIpaddr clean up before move code.
Quoted 79 lines
Quoted 59 lines
xen/common/numa.c +13
-0+/* setup dummy node covering all memory */ + memnode_shift =BITS_PER_LONG -1 ; + memnodemap = _memnodemap; + memnodemapsize =ARRAY_SIZE (_memnodemap); + +nodes_clear (node_online_map); +node_set_online (0 ); +for ( i =0 ; i < nr_cpu_ids; i++ ) +numa_set_node (i,0 ); +cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); +setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ); +}xen/include/asm-x86/numa.h +0
-5@@ -21,16 +21,11 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT)) -extern void numa_init_array (void ); -extern bool numa_off; - -extern int srat_disabled (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );xen/include/asm-x86/setup.h +0
-1@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );xen/include/xen/numa.h +0
-1@@ -26,7 +26,6 @@extern int memnode_shift;extern unsigned long memnodemapsize;extern u8 *memnodemap; -extern typeof(*memnodemap) _memnodemap[64];struct node_data {unsigned long node_start_pfn;xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 72
Quoted 5 lines
xen/include/xen/numa.h +2
-0@@ -69,6 +68,13 @@ extern int conflicting_memblks(u64 start, u64 end);extern void cutoff_node (int i,u64 start,u64 end);extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern void numa_init_array (void ); +extern void numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn);xen/include/xen/numa.h extern int conflicting_memblks(u64 start, u64 end); · line 80
Quoted 9 lines
xen/include/xen/numa.h +5
-0+extern bool numa_off; +extern int numa_fake; + +extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */ Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init to common
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 270 lines
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 12/40] xen/x86: Movenuma_initmem_initto commonHi Wei,
Quoted 5 lines · Wei Chen
This function can be reused by Arm device tree based NUMA support. So we move it from x86 to common, as well as its related variables and functions:
setup_node_bootmem,numa_init_arrayandnuma_emulation.Quoted 7 lines
Quoted 3 lines
As
numa_initmem_inithas been moved to common,_memnodemapis not used cross files. We can restore_memnodemapto static.As we discussed on a previous patch, we should try to avoid this kind of dance. I can help to find a split that would achieve that.
Yes, thanks!
Quoted 155 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 118 ---------------------------------- xen/common/numa.c | 122 +++++++++++++++++++++++++++++++++++- xen/include/asm-x86/numa.h | 5 -- xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 8 ++- 5 files changed, 128 insertions(+), 126 deletions(-)xen/arch/x86/numa.c +0
-11@@ -38,7 +38,6 @@ nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; -bool numa_off;s8 acpi_numa =0 ;int srat_disabled (void )@@ -46,123 +45,6 @@ int srat_disabled(void)return numa_off || acpi_numa <0 ; } -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c int srat_disabled(void) · line 68
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 21 lines
xen/arch/x86/numa.c +0
-21-mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ];xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; - -/* Kludge needed for the hash function */ - if ( hweight64(sz) > 1 ) - { - u64 x = 1; - while ( (x << 1) < sz ) - x <<= 1; - if ( x < sz/2 ) - printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11- sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); - for ( i = 0; i < numa_fake; i++ ) - { - nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; - if ( i == numa_fake - 1 ) - sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; - printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10- i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >> 20); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); - if ( memnode_shift < 0 ) - { - memnode_shift = 0; - printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/arch/x86/numa.c int srat_disabled(void)
Quoted 11 lines
xen/arch/x86/numa.c +0
-11-return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 6 lines
xen/arch/x86/numa.c +0
-6-{ -int i; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ;xen/arch/x86/numa.c int srat_disabled(void)
Quoted 10 lines
xen/arch/x86/numa.c +0
-10-#endif - -#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) -return ; -#endif - - printk(KERN_INFO "%s\n", - numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 24 lines
xen/arch/x86/numa.c +0
-18- -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; - memnodemapsize =ARRAY_SIZE (_memnodemap); - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); -} -void numa_set_node (int cpu,nodeid_t node) { cpu_to_node[cpu] = node;xen/common/numa.c +0
-1--- a/xen/common/numa.cQuoted 9 lines
xen/common/numa.c line 16
Quoted 5 lines
+++
b/xen/common/numa.cxen/common/numa.c +0
-0@@ -14,12 +14,13 @@#include <xen/smp.h> #include <xen/pfn.h> #include <xen/sched.h> NIT: We tend to add a newline betwen <xen/...> headers and <asm/...> headers.
got it
Quoted 15 lines
Quoted 15 lines
xen/common/numa.c +2
-1+#include <asm/acpi.h> struct node_data node_data[MAX_NUMNODES ];/* Mapping from pdx to node id */ int memnode_shift; -typeof(*memnodemap) _memnodemap[64]; +static typeof(*memnodemap) _memnodemap[64];unsigned long memnodemapsize;u8 *memnodemap;@@ -34,6 +35,8 @@ int num_node_memblks;struct node node_memblk_range[NR_NODE_MEMBLKS ];nodeid_t memblk_nodeid[NR_NODE_MEMBLKS ];Quoted 16 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 199
Quoted 12 lines
xen/common/numa.c +5
-0+bool numa_off; +/* * Given a shift value, try to populate memnodemap[] * Returns : @@ -191,3 +194,120 @@ void numa_add_cpu(int cpu) {cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); } + +/* initialize NODE_DATA given nodeid and start/end */ +void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end)From an abstract PoV, start and end should be
paddr_t. This should be
done on a separate patch though.Ok.
Quoted 32 lines
Quoted 8 lines
xen/common/numa.c +8
-0+{ +unsigned long start_pfn, end_pfn; + + start_pfn = start >>PAGE_SHIFT ; + end_pfn = end >>PAGE_SHIFT ; + +NODE_DATA (nodeid)->node_start_pfn = start_pfn; +NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ +node_set_online (nodeid); +} + +void __init numa_init_array (void ) +{ +int rr, i; + +/* There are unfortunately some poorly designed mainboards around +that only connect memory to a single CPU. This breaks the 1:1
cpu->nodeQuoted 13 lines
xen/common/numa.c +13
-0+mapping. To avoid this fill in the mapping for all possible +CPUs, as the number of CPUs is not known yet. +We round robin the existing nodes. */ + rr =first_node (node_online_map); +for ( i =0 ; i < nr_cpu_ids; i++ ) + { +if ( cpu_to_node[i] !=NUMA_NO_NODE ) +continue ; +numa_set_node (i, rr); + rr =cycle_node (rr, node_online_map); + } +} +Quoted 9 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 235
Quoted 5 lines
xen/common/numa.c +5
-0+#ifdef CONFIG_NUMA_EMU +int numa_fake__initdata =0 ; + +/* Numa emulation */ +static int __init numa_emulation (u64 start_pfn,u64 end_pfn)Here, this should be either "unsigned long" or ideally "
mfn_t". Although, if you use "unsigned long", you will need to...
Quoted 1 line
Do we need a separate patch to do it?
I would prefer if the cleanups are done separately as this makes easier to review code movement.
Quoted 72 lines
Quoted 8 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 239
Quoted 5 lines
xen/common/numa.c +4
-0+{ +int i; +struct node nodes[MAX_NUMNODES ]; +u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake;... cast "(
end_pfn-start_pfn)" touin64_tor usepfn_to_paddr().Ok
Quoted 45 lines
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 9 lines
xen/common/numa.c +9
-0+ +/* Kludge needed for the hash function */ + if ( hweight64(sz) > 1 ) + { + u64 x = 1; + while ( (x << 1) < sz ) + x <<= 1; + if ( x < sz/2 ) + printk(KERN_ERR "Numa emulation unbalanced. Complain tomaintainer\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+ sz = x; + } + +memset (&nodes,0 ,sizeof (nodes)); + for ( i = 0; i < numa_fake; i++ ) + { + nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; + if ( i == numa_fake - 1 ) + sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; + nodes[i].end = nodes[i].start + sz; + printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 10 lines
xen/common/numa.c +10
-0+ i, + nodes[i].start, nodes[i].end, + (nodes[i].end - nodes[i].start) >> 20); +node_set_online (i); + } + memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); + if ( memnode_shift < 0 ) + { + memnode_shift = 0; + printk(KERN_ERR "No NUMA hash function found. Emulationdisabled.\n");
xen/common/numa.c void numa_add_cpu(int cpu)
Quoted 11 lines
xen/common/numa.c +11
-0+return -1 ; + } +for_each_online_node ( i ) + setup_node_bootmem(i, nodes[i].start, nodes[i].end); +numa_init_array (); + +return 0 ; +} +#endif + +void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 15 lines
xen/common/numa.c void numa_add_cpu(int cpu) · line 291
Quoted 11 lines
xen/common/numa.c +11
-0+{ +int i; + +#ifdef CONFIG_NUMA_EMU +if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) +return ; +#endif + +#ifdef CONFIG_ACPI_NUMA +if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , + (u64 )end_pfn <<PAGE_SHIFT ) )(u64)v <<
PAGE_SHIFTshould be switched to usepfn_to_paddr()ormfn_to_paddr()if you decide to makestart_pfnandend_pfntypesafe.
Quoted 1 line
Still need a separate patch to change it before move?
Yes.
Cheers,
Signature · 1 line
Julien Grall
RFC PATCH 13/40 xen/arm: introduce numa_set_node for Arm
In reply to Wei ChenThis API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed, node_online_map will be cleared and set node#0 online. So we use node_online_map to prevent to set a CPU to an offline node.
--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 xen/arch/arm/numa.c
xen/arch/arm/Makefile +1 -0
@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o
xen/arch/arm/numa.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid) +{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}
xen/include/asm-arm/numa.h +2 -0
@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_set_node (x, y) do { } while (0) +#endif #endif /* __ARCH_ARM_NUMA_H */
xen/include/asm-x86/numa.h +0 -1
@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
xen/include/xen/numa.h +1 -0
@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
Hi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.
IHMO numa_set_node() should behave exactly the same way on x86 and Arm because this is going to be used by the common code.
From the commit message, I don't quite understand why the check is necessary on Arm but not on x86. Can you clarify it?
Quoted 50 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)create mode 100644
xen/arch/arm/numa.cxen/arch/arm/Makefile +1
-0@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa.c new file +21
-0@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h>
xen/arch/arm/numa.c line 31
Quoted 10 lines
xen/arch/arm/numa.c new file +10
-0+#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid) +{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}
I think numa_set_node() will want to be implemented in common code.
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30
Quoted 9 lines
xen/include/asm-arm/numa.h +1
-0@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_set_node (x, y) do { } while (0)
I would define it in xen/numa.h so other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.
Quoted 22 lines
xen/include/asm-arm/numa.h +1
-0+#endif #endif /* __ARCH_ARM_NUMA_H */ xen/include/asm-x86/numa.h +0
-1@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);xen/include/xen/numa.h +0
-0
xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node); · line 76
Quoted 7 lines
xen/include/xen/numa.h +1
-0@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);extern void numa_init_array (void );extern void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn); +extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
Hi Julien,
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.
Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?
Yes, in patch#27, in smpboot.c, dt_smp_init_cpus function. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invoke numa_set_node to create CPU-NODE map. But in our testing, we found when NUMA init failed, numa_set_node still can set CPU to a offline or invalid NODE. So we're using node_online_map to prevent this behavior. Otherwise we have to check node_online_map everywhere before we call numa_set_node.
x86 actually is doing the same way, but it handles node_online_map check out of numa_set_node:
57 void __init init_cpu_to_node(void)
58 {
59 unsigned int i;
60 nodeid_t node;
61
62 for ( i = 0; i < nr_cpu_ids; i++ )
63 {
64 u32 apicid = x86_cpu_to_apicid[i];
65 if ( apicid == BAD_APICID )
66 continue;
67 node = apicid < MAX_LOCAL_APIC ? apicid_to_node[apicid] : NUMA_NO_NODE;
68 if ( node == NUMA_NO_NODE || !node_online(node) )
69 node = 0;
70 numa_set_node(i, node);
71 }
72 }
Quoted 53 lines
Quoted 53 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa.c | 31 +++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 1 + 5 files changed, 35 insertions(+), 1 deletion(-)create mode 100644
xen/arch/arm/numa.cxen/arch/arm/Makefile +1
-0@@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o +obj-$(CONFIG_NUMA) += numa.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa.c new file +24
-0@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +void numa_set_node (int cpu,nodeid_t nid)
Quoted 9 lines
xen/arch/arm/numa.c line 31
Quoted 7 lines
xen/arch/arm/numa.c new file +7
-0+{ +if ( nid >=MAX_NUMNODES || + !nodemask_test (nid, &node_online_map) ) + nid =0 ; + + cpu_to_node[cpu] = nid; +}I think
numa_set_node()will want to be implemented in common code.
See my above comment. If x86 is ok, I think yes, we can do it in common code.
Quoted 8 lines
Quoted 8 lines
xen/include/asm-arm/numa.h +0
-0@@ -27,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20)
Quoted 5 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 30
Quoted 1 line
xen/include/asm-arm/numa.h +1
-0+#define numa_set_node (x, y) do { } while (0)I would define it in
xen/numa.hso other arch can take advantage ot it. Also, please use a static inline helper so the arguments are evaluated.
Ok
Quoted 36 lines
xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end,
Quoted 21 lines
xen/include/asm-arm/numa.h +1
-0+#endif #endif /* __ARCH_ARM_NUMA_H */ xen/include/asm-x86/numa.h +0
-1@@ -22,7 +22,6 @@ extern nodeid_t pxm_to_node(unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))extern int srat_disabled (void ); -extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);xen/include/xen/numa.h +0
-0@@ -70,6 +70,7 @@ extern int valid_numa_range(u64 start, u64 end,
nodeid_tnode);xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, · line 71
Quoted 3 lines
xen/include/xen/numa.h +0
-0extern void numa_init_array (void ); extern void numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn);xen/include/xen/numa.h extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
Quoted 5 lines
xen/include/xen/numa.h +1
-0+extern void numa_set_node (int cpu,nodeid_t node);extern bool numa_off;extern int numa_fake;Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 26 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?
Quoted 8 lines
Yes, in patch#27, in
smpboot.c,dt_smp_init_cpusfunction. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invokenuma_set_nodeto create CPU-NODE map. But in our testing, we found when NUMA init failed,numa_set_nodestill can set CPU to a offline or invalid NODE. So we're usingnode_online_mapto prevent this behavior. Otherwise we have to checknode_online_mapeverywhere before we callnuma_set_node.
What do you mean by invalid NODE? Is it 0xFF (NUMA_NO_NODE)?
Quoted 3 lines
x86 actually is doing the same way, but it handles
node_online_mapcheck out ofnuma_set_node:
Right...
Quoted 5 lines
Quoted 2 lines
I think
numa_set_node()will want to be implemented in common code.See my above comment. If x86 is ok, I think yes, we can do it in common code.
... on x86, this check is performed outside of numa_set_node() for one caller whereas on Arm you are adding it in numa_set_node().
For example, numa_set_node() can be called with NUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.
This will change the behavior of users to cpu_to_node() later on (such as XEN_SYSCTL_cputopoinfo).
NUMA is not something architecture specific, so I dont't think the implementation should differ here.
In this case, I think numa_set_node() shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm
Hi Julien,
Quoted 51 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:24
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 26 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 18:37
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 13/40] xen/arm: introducenuma_set_nodefor ArmHi Wei,
Quoted 6 lines · Wei Chen
This API is used to set one CPU to a NUMA node. If the system configure NUMA off or system initialize NUMA failed, the online NUMA node would set to only node#0. This will be done in following patches. When NUMA turn off or init failed,
node_online_mapwill be cleared and set node#0 online. So we usenode_online_mapto prevent to set a CPU to an offline node.Quoted 6 lines
IHMO
numa_set_node()should behave exactly the same way on x86 and Arm because this is going to be used by the common code.From the commit message, I don't quite understand why the check is
necessary on Arm but not on x86. Can you clarify it?Quoted 8 lines
Yes, in patch#27, in
smpboot.c,dt_smp_init_cpusfunction. We will parse CPU numa-node-id from dtb CPU node. If we get a valid node ID for one CPU, we will invokenuma_set_nodeto create CPU-NODE map. But in our testing, we found when NUMA init failed,numa_set_nodestill can set CPU to a offline or invalid NODE. So we're usingnode_online_mapto prevent this behavior. Otherwise we have to checknode_online_mapeverywhere before we callnuma_set_node.
Quoted 1 line
What do you mean by invalid NODE? Is it
0xFF(NUMA_NO_NODE)?
No, I mean some wrong content in device tree. For example, if the dtb set a wrong numa-node-id in CPU dt-node.
Quoted 20 lines
Quoted 4 lines
x86 actually is doing the same way, but it handles
node_online_mapcheck out ofnuma_set_node:Right...
Quoted 5 lines
Quoted 2 lines
I think
numa_set_node()will want to be implemented in common code.See my above comment. If x86 is ok, I think yes, we can do it in common code.
... on x86, this check is performed outside of
numa_set_node()for one caller whereas on Arm you are adding it innuma_set_node().For example,
numa_set_node()can be called withNUMA_NO_NODE. On x86, we would set cpu_to_node[] to that value. However, if I am not mistaken, on Arm we would set the value to 0.
Quoted 9 lines
This will change the behavior of users to
cpu_to_node()later on (such asXEN_SYSCTL_cputopoinfo).NUMA is not something architecture specific, so I dont't think the implementation should differ here.
In this case, I think
numa_set_node()shouldn't check if the node is valid. Instead, the caller should take care of it if it is important.
Yes, I agree. I will change it in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
RFC PATCH 14/40 xen/arm: set NUMA nodes max number to 64 by default
In reply to Wei ChenToday's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.
--- xen/include/asm-arm/numa.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +9 -1
@@ -5,7 +5,15 @@typedef u8 nodeid_t ; -#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ +#define NODES_SHIFT 6 + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
Hi Wei,
Quoted 13 lines · Wei Chen
Today's Arm64 systems can reach or exceed 16 NUMA nodes, so we set the number to 64 to match with x86.
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h line 13
Quoted 10 lines
xen/include/asm-arm/numa.h +6
-1@@ -5,7 +5,15 @@typedef u8 nodeid_t ; -#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/
Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to xen/numa.h.
xen/include/asm-arm/numa.h line 19
Quoted 7 lines
xen/include/asm-arm/numa.h +3
-0+#define NODES_SHIFT 6 + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
Quoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0
-0@@ -5,7 +5,15 @@typedef u8 nodeid_t ;
Quoted 11 lines
xen/include/asm-arm/numa.h line 13
Quoted 7 lines
xen/include/asm-arm/numa.h +6
-1-#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to
xen/numa.h.
To be honest - if this gets moved, please at least consider making it a proper Kconfig setting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...
Jan
RE: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default
Hi Jan, Julien,
Quoted 29 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月25日 21:36
To: Julien Grall <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by defaultQuoted 7 lines · Julien Grall
Quoted 6 lines · Wei Chen
xen/include/asm-arm/numa.h +0
-0@@ -5,7 +5,15 @@typedef u8 nodeid_t ;Quoted 11 lines
xen/include/asm-arm/numa.h line 13
Quoted 7 lines
xen/include/asm-arm/numa.h +6
-1-#if !defined(CONFIG_NUMA ) +#if defined(CONFIG_NUMA ) + +/* +* Same as x86, we set the max number of NUMA nodes to 64 and +* set the number of NUMA memory block number to 128. +*/ Such comment can rot easily if x86 decides to bump there values. But given the value is the same, I think it would make sense to move the define to
xen/numa.h.
Quoted 5 lines
To be honest - if this gets moved, please at least consider making it a proper
Kconfigsetting. Just as much a the number of CPUs can be configured, the number of nodes should be possible to choose by the build manager. Of course - if it's not too much trouble ...
Ok
Quoted 1 line
Jan
RFC PATCH 15/40 xen/x86: move NUMA API from x86 header to common header
In reply to Wei ChenAs all functions and macros that depends on these NUMA API, like clear_node_cpumask, have been moved to common header file. In this case, we can move NUMA API from x86 header file to common header file, and will not trigger the symbols not found error when functions in arch NUMA header file depends on the symbols in common NUMA header file.
--- xen/include/asm-x86/numa.h | 13 ------------- xen/include/xen/numa.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 13 deletions(-)
xen/include/asm-x86/numa.h +0 -13
@@ -9,14 +9,6 @@ typedef u8 nodeid_t;extern int srat_rev; -extern nodeid_t cpu_to_node[NR_CPUS ]; -extern cpumask_t node_to_cpumask[]; - -#define cpu_to_node (cpu) (cpu_to_node[cpu]) -#define parent_node (node) (node) -#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node])) -#define node_to_cpumask (node) (node_to_cpumask[node]) -extern nodeid_t pxm_to_node (unsigned int pxm);#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -28,11 +20,6 @@ extern void srat_detect_node(int cpu);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void ); -static inline void clear_node_cpumask (int cpu) -{ -cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); -} -void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );
xen/include/xen/numa.h +13 -0
@@ -22,6 +22,14 @@/* The following content can be used when NUMA feature is enabled */ #if defined(CONFIG_NUMA ) +extern nodeid_t cpu_to_node[NR_CPUS ]; +extern cpumask_t node_to_cpumask[]; + +#define cpu_to_node (cpu) (cpu_to_node[cpu]) +#define parent_node (node) (node) +#define node_to_first_cpu (node) (__ffs(node_to_cpumask[node])) +#define node_to_cpumask (node) (node_to_cpumask[node]) +/* Simple perfect hash to map pdx to node numbers */ extern int memnode_shift;extern unsigned long memnodemapsize;
@@ -76,6 +84,11 @@ extern int numa_fake;extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +static inline void clear_node_cpumask (int cpu) +{ +cpumask_clear_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); +} +#endif /* CONFIG_NUMA */ #endif /* _XEN_NUMA_H */
Signature · 1 line
2.25.1
RFC PATCH 16/40 xen/arm: Create a fake NUMA node to use common code
In reply to Wei ChenWhen CONFIG_NUMA is enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, like cpu_to_node, as single node NUMA system.
So in this patch, we introduce a numa_init function for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.
--- xen/arch/arm/numa.c | 53 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/setup.c | 8 ++++++ xen/include/asm-arm/numa.h | 11 ++++++++ 3 files changed, 72 insertions(+)
xen/arch/arm/numa.c +53 -0
@@ -20,6 +20,8 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/pfn.h> +#include <asm/setup.h> void numa_set_node (int cpu,nodeid_t nid) {
@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } + +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" ); +/* Arm NUMA has not been implemented until this patch */ + numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ; + +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + } + +/* +* Find the minimal and maximum address of RAM, NUMA will +* build a memory to node mapping table for the whole range. +*/ + ram_start = bootinfo.mem.bank[0 ].start; + ram_size = bootinfo.mem.bank[0 ].size; + ram_end = ram_start + ram_size; +for ( idx =1 ; idx < bootinfo.mem.nr_banks; idx++ ) + { +paddr_t bank_start = bootinfo.mem.bank[idx].start; +paddr_t bank_size = bootinfo.mem.bank[idx].size; +paddr_t bank_end = bank_start + bank_size; + + ram_size = ram_size + bank_size; + ram_start =min (ram_start, bank_start); + ram_end =max (ram_end, bank_end); + } + +numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end)); +return ; +}
xen/arch/arm/setup.c +8 -0
@@ -30,6 +30,7 @@#include <xen/init.h> #include <xen/irq.h> #include <xen/mm.h> +#include <xen/numa.h> #include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h>
@@ -874,6 +875,13 @@ void __init start_xen(unsigned long boot_phys_offset,/* Parse the ACPI tables for possible boot-time configuration */ acpi_boot_table_init (); +/* +* Try to initialize NUMA system, if failed, the system will +* fallback to uniform system which means system has only 1 +* NUMA node. +*/ +numa_init (acpi_disabled); +end_boot_allocator ();/*
xen/include/asm-arm/numa.h +11 -0
@@ -13,6 +13,16 @@ typedef u8 nodeid_t;*/ #define NODES_SHIFT 6 +extern void numa_init (bool acpi_off); + +/* +* Temporary for fake NUMA node, when CPU, memory and distance +* matrix will be read from DTB or ACPI SRAT. The following +* symbols will be removed. +*/ +extern mfn_t first_valid_mfn; +#define __node_distance (a, b) (20) +#else /* Fake one node for now. See also node_online_map. */
@@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_init (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code
Quoted 35 lines · Wei Chen
When
CONFIG_NUMAis enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, likecpu_to_node, as single node NUMA system.So in this patch, we introduce a
numa_initfunction for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 53 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/setup.c | 8 ++++++ xen/include/asm-arm/numa.h | 11 ++++++++ 3 files changed, 72 insertions(+)xen/arch/arm/numa.c +2
-0@@ -20,6 +20,8 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/pfn.h> +#include <asm/setup.h> void numa_set_node (int cpu,nodeid_t nid) {@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; }
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" );
NIT: please align
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44
Quoted 1 line
xen/arch/arm/numa.c +1
-0+/* Arm NUMA has not been implemented until this patch */
"Arm NUMA is not implemented yet"
Quoted 8 lines
xen/arch/arm/numa.c +8
-0+ numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ;
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + }
I just want to note that this works because MAX_NUMNODES is 1. If MAX_NUMNODES was > 1 then it would be wrong to set node_to_cpumask, node_spanned_pages and node_start_pfn for all nodes to the same values.
It might be worth writing something about it in the in-code comment.
Quoted 4 lines
xen/arch/arm/numa.c +4
-0+/* +* Find the minimal and maximum address of RAM, NUMA will +* build a memory to node mapping table for the whole range. +*/
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 77
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ ram_start = bootinfo.mem.bank[0 ].start; + ram_size = bootinfo.mem.bank[0 ].size; + ram_end = ram_start + ram_size; +for ( idx =1 ; idx < bootinfo.mem.nr_banks; idx++ ) + { +paddr_t bank_start = bootinfo.mem.bank[idx].start; +paddr_t bank_size = bootinfo.mem.bank[idx].size; +paddr_t bank_end = bank_start + bank_size; + + ram_size = ram_size + bank_size;
ram_size is updated but not utilized
Quoted 65 lines
xen/arch/arm/numa.c +7
-0+ ram_start =min (ram_start, bank_start); + ram_end =max (ram_end, bank_end); + } + +numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end)); +return ; +}xen/arch/arm/setup.c +8
-0@@ -30,6 +30,7 @@#include <xen/init.h> #include <xen/irq.h> #include <xen/mm.h> +#include <xen/numa.h> #include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> @@ -874,6 +875,13 @@ void __init start_xen(unsigned long boot_phys_offset,/* Parse the ACPI tables for possible boot-time configuration */ acpi_boot_table_init (); +/* +* Try to initialize NUMA system, if failed, the system will +* fallback to uniform system which means system has only 1 +* NUMA node. +*/ +numa_init (acpi_disabled); +end_boot_allocator ();/* xen/include/asm-arm/numa.h +11
-0@@ -13,6 +13,16 @@ typedef u8 nodeid_t;*/ #define NODES_SHIFT 6 +extern void numa_init (bool acpi_off); + +/* +* Temporary for fake NUMA node, when CPU, memory and distance +* matrix will be read from DTB or ACPI SRAT. The following +* symbols will be removed. +*/ +extern mfn_t first_valid_mfn; +#define __node_distance (a, b) (20) +#else /* Fake one node for now. See also node_online_map. */ @@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_init (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif Signature · 1 line
2.25.1
RE: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code
Hi Stefano,
Quoted 43 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:10
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common codeQuoted 33 lines · Wei Chen
When
CONFIG_NUMAis enabled for Arm, Xen will switch to use common NUMA API instead of previous fake NUMA API. Before we parse NUMA information from device tree or ACPI SRAT table, we need to init the NUMA related variables, likecpu_to_node, as single node NUMA system.So in this patch, we introduce a
numa_initfunction for to initialize these data structures as all resources belongs to node#0. This will make the new API returns the same values as the fake API has done.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 53 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/setup.c | 8 ++++++ xen/include/asm-arm/numa.h | 11 ++++++++ 3 files changed, 72 insertions(+)xen/arch/arm/numa.c +2
-0@@ -20,6 +20,8 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/pfn.h> +#include <asm/setup.h> void numa_set_node (int cpu,nodeid_t nid) {@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid)
Quoted 15 lines
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43
Quoted 12 lines
xen/arch/arm/numa.c +10
-0cpu_to_node[cpu] = nid; } + +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" );NIT: please align
OK
Quoted 5 lines
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44
Quoted 2 lines
xen/arch/arm/numa.c +1
-0+/* Arm NUMA has not been implemented until this patch */ "Arm NUMA is not implemented yet"
OK
Quoted 24 lines
Quoted 9 lines
xen/arch/arm/numa.c +8
-0+ numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ;xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + }I just want to note that this works because
MAX_NUMNODESis 1. IfMAX_NUMNODESwas > 1 then it would be wrong to setnode_to_cpumask,node_spanned_pagesandnode_start_pfnfor all nodes to the same values.
Quoted 2 lines
It might be worth writing something about it in the in-code comment.
OK, I will do it.
Quoted 16 lines
Quoted 5 lines
xen/arch/arm/numa.c +4
-0+/* +* Find the minimal and maximum address of RAM, NUMA will +* build a memory to node mapping table for the whole range. +*/ xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 77
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ ram_start = bootinfo.mem.bank[0 ].start; + ram_size = bootinfo.mem.bank[0 ].size; + ram_end = ram_start + ram_size; +for ( idx =1 ; idx < bootinfo.mem.nr_banks; idx++ ) + { +paddr_t bank_start = bootinfo.mem.bank[idx].start; +paddr_t bank_size = bootinfo.mem.bank[idx].size; +paddr_t bank_end = bank_start + bank_size; + + ram_size = ram_size + bank_size;
Quoted 2 lines
ram_sizeis updated but not utilized
Ok, I will remove it.
Quoted 67 lines
xen/arch/arm/setup.c void __init start_xen(unsigned long
Quoted 21 lines
xen/arch/arm/numa.c +7
-0+ ram_start =min (ram_start, bank_start); + ram_end =max (ram_end, bank_end); + } + +numa_initmem_init (PFN_UP (ram_start),PFN_DOWN (ram_end)); +return ; +}xen/arch/arm/setup.c +1
-0@@ -30,6 +30,7 @@#include <xen/init.h> #include <xen/irq.h> #include <xen/mm.h> +#include <xen/numa.h> #include <xen/param.h> #include <xen/softirq.h> #include <xen/keyhandler.h> @@ -874,6 +875,13 @@ void __init start_xen(unsigned long
boot_phys_offset,Quoted 45 lines
xen/arch/arm/setup.c +7
-0/* Parse the ACPI tables for possible boot-time configuration */ acpi_boot_table_init (); +/* +* Try to initialize NUMA system, if failed, the system will +* fallback to uniform system which means system has only 1 +* NUMA node. +*/ +numa_init (acpi_disabled); +end_boot_allocator ();/* xen/include/asm-arm/numa.h +11
-1@@ -13,6 +13,16 @@ typedef u8 nodeid_t;*/ #define NODES_SHIFT 6 +extern void numa_init (bool acpi_off); + +/* +* Temporary for fake NUMA node, when CPU, memory and distance +* matrix will be read from DTB or ACPI SRAT. The following +* symbols will be removed. +*/ +extern mfn_t first_valid_mfn; +#define __node_distance (a, b) (20) +#else /* Fake one node for now. See also node_online_map. */ @@ -35,6 +45,7 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#define numa_init (x) do { } while (0)#define numa_set_node (x, y) do { } while (0)#endif --2.25.1
Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code
In reply to Stefano StabelliniQuoted 47 lines · Stefano Stabellini
Quoted 4 lines · Wei Chen
xen/arch/arm/numa.c +0
-0@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; }xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" );NIT: please align
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44
Quoted 1 line
xen/arch/arm/numa.c +1
-0+/* Arm NUMA has not been implemented until this patch */ "Arm NUMA is not implemented yet"
Quoted 8 lines
xen/arch/arm/numa.c +8
-0+ numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ;xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + }I just want to note that this works because
MAX_NUMNODESis 1. IfMAX_NUMNODESwas > 1 then it would be wrong to setnode_to_cpumask,node_spanned_pagesandnode_start_pfnfor all nodes to the same values.
Quoted 1 line
It might be worth writing something about it in the in-code comment.
Plus perhaps BUILD_BUG_ON(MAX_NUMNODES != 1), so the issue is actually noticed at build time once the constant gets changed?
Jan
RE: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code
Hi Jan,
Quoted 58 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年8月27日 14:18
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common codeQuoted 47 lines · Stefano Stabellini
Quoted 4 lines · Wei Chen
xen/arch/arm/numa.c +0
-0@@ -29,3 +31,54 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; }xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 43
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +void __init numa_init (bool acpi_off) +{ +uint32_t idx; +paddr_t ram_start = ~0 ; +paddr_t ram_size =0 ; +paddr_t ram_end =0 ; + +printk (XENLOG_WARNING +"NUMA has not been supported yet, NUMA off!\n" );NIT: please align
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 44
Quoted 1 line
xen/arch/arm/numa.c +1
-0+/* Arm NUMA has not been implemented until this patch */ "Arm NUMA is not implemented yet"
Quoted 8 lines
xen/arch/arm/numa.c +8
-0+ numa_off = true; + +/* +* Set all cpu_to_node mapping to 0, this will make cpu_to_node +* function return 0 as previous fake cpu_to_node API. +*/ +for ( idx =0 ; idx <NR_CPUS ; idx++ ) + cpu_to_node[idx] =0 ;xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 10 lines
xen/arch/arm/numa.c +10
-0+ +/* +* Make node_to_cpumask, node_spanned_pages and node_start_pfn +* return as previous fake APIs. +*/ +for ( idx =0 ; idx <MAX_NUMNODES ; idx++ ) { + node_to_cpumask[idx] = cpu_online_map; +node_spanned_pages (idx) = (max_page -mfn_x (first_valid_mfn)); +node_start_pfn (idx) = (mfn_x (first_valid_mfn)); + }I just want to note that this works because
MAX_NUMNODESis 1. IfMAX_NUMNODESwas > 1 then it would be wrong to setnode_to_cpumask,node_spanned_pagesandnode_start_pfnfor all nodes to the same values.
Quoted 5 lines
Quoted 1 line
It might be worth writing something about it in the in-code comment.
Plus perhaps
BUILD_BUG_ON(MAX_NUMNODES != 1), so the issue is actually noticed at build time once the constant gets changed?
It would be better. I will use it in next version.
Quoted 1 line
Jan
RFC PATCH 17/40 xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
In reply to Wei ChenWe need a Kconfig option to distinguish with ACPI based NUMA. So we introduce the new Kconfig option:DEVICE_TREE_NUMA in this patch for Arm64.
--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)
xen/arch/arm/Kconfig +10 -0
@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xen is an alternative to device tree on ARM64. +config DEVICE_TREE_NUMA + bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design used in + multiprocessing, where the memory access time depends on the memory + location relative to the processor. + config GICV3 bool "GICv3 driver" depends on ARM_64 && !NEW_VGIC
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Hi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 4 lines
xen/arch/arm/Kconfig +1
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xen is an alternative to device tree on ARM64. +config DEVICE_TREE_NUMA
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI in the future).
xen/arch/arm/Kconfig config ACPI · line 45
Quoted 9 lines
xen/arch/arm/Kconfig +9
-0+ bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design used in + multiprocessing, where the memory access time depends on the memory + location relative to the processor. +
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Hi Julien,
Quoted 35 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0xen/arch/arm/Kconfig config ACPI · line 33
Quoted 1 line
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xenis
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
Quoted 7 lines
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI in the future).
How about we select DT_NUMA default on Arm64. And DT_NUMA select NUMA like what we have done in patch#6 in x86? And remove the description?
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
Quoted 17 lines
xen/arch/arm/Kconfig config ACPI · line 37
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ bool "NUMA (Non-Uniform Memory Access) Support (UNSUPPORTED)" ifUNSUPPORTED
xen/arch/arm/Kconfig config ACPI
Quoted 5 lines
xen/arch/arm/Kconfig +5
-0+ depends on ARM_64 + select NUMA + ---help--- + + Non-Uniform Memory Access (NUMA) is a computer memory design usedin
xen/arch/arm/Kconfig config ACPI · line 43
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+ multiprocessing, where the memory access time depends on thememory
xen/arch/arm/Kconfig config ACPI · line 45
Quoted 2 lines
xen/arch/arm/Kconfig +2
-0+ location relative to the processor. +Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 39 lines
Quoted 39 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 14 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/arch/arm/Kconfig +0
-0xen/arch/arm/Kconfig config ACPI · line 33
Quoted 1 line
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) support for Xenis
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
Quoted 6 lines
Quoted 3 lines
So we should have a generic config that will then select DT (and ACPI in the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?
I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Hi Julien,
Quoted 56 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 21 lines
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 5 lines
Quoted 5 lines
is
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAQuoted 8 lines
Quoted 8 lines
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already have CONFIG_NUMA in common layer. So we need to add another one for Arm like CONFIG_ARM_NUMA?
And in this option, we can select CONFIG_DEVICE_TREE_NUMA automatically if device tree is enabled. If CONFIG_ACPI is enabled, we will select CONFIG_ACPI_NUMA too (in the future)
In Xen code, DT_NUMA and ACPI_NUMA code can co-exist, Xen will check the system ACPI support status to decide to use DT_NUMA or ACPI_NUMA?
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 78 lines
Quoted 57 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 21 lines
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 5 lines
Quoted 5 lines
is
Quoted 3 lines
xen/arch/arm/Kconfig +1
-0an alternative to device tree on ARM64. +config DEVICE_TREE_NUMAQuoted 8 lines
Quoted 8 lines
The name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the choice to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Quoted 7 lines
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already haveCONFIG_NUMAin common layer. So we need to add another one for Arm likeCONFIG_ARM_NUMA?
I think so.
Quoted 5 lines
And in this option, we can select
CONFIG_DEVICE_TREE_NUMAautomatically if device tree is enabled. IfCONFIG_ACPIis enabled, we will selectCONFIG_ACPI_NUMAtoo (in the future)
In Xen code,DT_NUMAandACPI_NUMAcode can co-exist, Xen
Distributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.
Quoted 2 lines
will check the system ACPI support status to decide to use
DT_NUMAorACPI_NUMA?
Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.
Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
Hi Julien,
Quoted 118 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 19:29
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 15 lines
Quoted 15 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:41
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: IntroduceDEVICE_TREE_NUMAKconfigfor arm64Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月19日 21:38
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 17/40] xen/arm: Introduce
DEVICE_TREE_NUMAQuoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Kconfigfor arm64Hi,
Quoted 9 lines · Wei Chen
We need a
Kconfigoption to distinguish with ACPI based NUMA. So we introduce the newKconfigoption:DEVICE_TREE_NUMAin this patch for Arm64.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)Quoted 6 lines
Quoted 6 lines
xen/arch/arm/Kconfig config ACPI · line 33
Quoted 6 lines
xen/arch/arm/Kconfig +0
-0@@ -33,6 +33,16 @@ config ACPI Advanced Configuration and Power Interface (ACPI) supportfor Xen
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
xen/arch/arm/Kconfig +0
-0an alternative to device tree on ARM64.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/Kconfig config ACPI · line 36
Quoted 1 line
xen/arch/arm/Kconfig +1
-0+config DEVICE_TREE_NUMAThe name suggests that NUMA should only be enabled for Device-Tree... But the description looks generic.
However, I think the user should only have the choice to say whether they want NUMA to be enabled or not. We should not give them the
choice
Quoted 25 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
to enable/disable the parsing for DT/ACPI.
So we should have a generic config that will then select DT (and ACPI
in
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
the future).
How about we select
DT_NUMAdefault on Arm64. AndDT_NUMAselect NUMA like what we have done in patch#6 in x86? And remove the description?I would rather not make NUMA supported by default on Arm64. Instead, we should go throught the same process as other new features and gate it behind UNSUPPORTED until it is mature enough.
Ok. I agree with this.
Quoted 8 lines
Quoted 4 lines
If we make generic NUMA as a selectable option, and depends on NUMA to select DT or ACPI NUMA. It seems to be quite different from the existing logic?
I am a bit confused. You added just logic to select NUMA from ACPI, right? So are you talking about a different logic?
Quoted 7 lines
No, I didn't want a different one. I thought you wanted it that way. Obviously, I mis-understanded your comments.
Can I understand your previous comments like following:
1. We should have a generic config that will then select DT and ACPI:
Because we already haveCONFIG_NUMAin common layer. So we need to add another one for Arm likeCONFIG_ARM_NUMA?I think so.
Quoted 5 lines
And in this option, we can select
CONFIG_DEVICE_TREE_NUMAautomatically if device tree is enabled. IfCONFIG_ACPIis enabled, we will selectCONFIG_ACPI_NUMAtoo (in the future)
In Xen code,DT_NUMAandACPI_NUMAcode can co-exist, XenDistributions should not have to build a different Xen for DT and ACPI. So it is more they *must* co-exist.
Quoted 2 lines
will check the system ACPI support status to decide to use
DT_NUMAorACPI_NUMA?
Quoted 7 lines
Yes. A user should only have to say "I want to use NUMA". This is Xen to figure out whether we need to compile the support for DT and/or ACPI.
Once we have support for APCI, it doesn't make a lot of sense for the users to say "I want to compile with DT and ACPI but I only want NUMA when using DT".
I am glad we are now in the same page. Ok, I will change the Kconfig like this in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
RFC PATCH 18/40 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenEFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prvent early_scan_node to parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h>
@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )
xen/arch/arm/efi/efi-boot.h +0 -25
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions.
xen/include/asm-arm/setup.h +6 -0
@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt); +#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif +#endif /* * Local variables:
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Hi Wei,
Quoted 9 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
Quoted 6 lines
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto
s/prvent/prevent/
Quoted 25 lines
parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h>
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340
Quoted 6 lines
xen/arch/arm/bootfdt.c +2
-1@@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI,
"If the system boot". Although, I would suggest to write:
"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344
Quoted 4 lines
xen/arch/arm/bootfdt.c +4
-0+* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) )
arch_efi_enabled() is going to be less expensive than device_tree_node_matches(). So I would suggest to re-order the operands.
Quoted 46 lines
xen/arch/arm/bootfdt.c +0
-0rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem); else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )xen/arch/arm/efi/efi-boot.h +0
-25@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/asm-arm/setup.h +0
-0
xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130
Quoted 8 lines
xen/include/asm-arm/setup.h +5
-0@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32 address_cells,u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt); +#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif
I would prefer if we introduce CONFIG_EFI that would stub efi_enabled for architecture not supporting EFI.
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Hi Julien,
Quoted 21 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:35
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIHi Wei,
Quoted 9 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
Quoted 9 lines
Quoted 6 lines
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodetos/prvent/prevent/
Ok
Quoted 28 lines
Quoted 28 lines
parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;
Quoted 9 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 340
Quoted 3 lines
xen/arch/arm/bootfdt.c +2
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, "If the system boot". Although, I would suggest to write:
"If Xen has been booted via UEFI, the memory banks will already be populated. So we should skip the parsing."
Yes, that would be better. I will change it in next version.
Quoted 8 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 344
Quoted 4 lines
xen/arch/arm/bootfdt.c +4
-0+* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) )
arch_efi_enabled()is going to be less expensive thandevice_tree_node_matches(). So I would suggest to re-order the operands.
yes.
Quoted 52 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 346
Quoted 2 lines
xen/arch/arm/bootfdt.c +0
-0rc = process_memory_node(fdt, node, name, depth, address_cells, size_cells,&bootinfo.mem);
xen/arch/arm/efi/efi-boot.h EFI_STATUS __init
Quoted 6 lines
xen/arch/arm/bootfdt.c +0
-0else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )xen/arch/arm/efi/efi-boot.h +0
-0@@ -197,33 +197,8 @@ EFI_STATUS __initfdt_add_uefi_nodes(EFI_SYSTEM_TABLE
*sys_table,xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32
Quoted 38 lines
xen/arch/arm/efi/efi-boot.h +0
-25int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/asm-arm/setup.h +0
-0@@ -123,6 +123,12 @@ void device_tree_get_reg(const __be32 **cell, u32
address_cells,Quoted 3 lines
xen/include/asm-arm/setup.h +0
-0u32 device_tree_get_u32 (const void *fdt,int node,const char *prop_name,u32 dflt);
Quoted 9 lines
xen/include/asm-arm/setup.h void device_tree_get_reg(const __be32 **cell, u32 address_cells, · line 130
Quoted 5 lines
xen/include/asm-arm/setup.h +5
-0+#if defined(CONFIG_ARM_64 ) +#define arch_efi_enabled (x) efi_enabled(x) +#else +#define arch_efi_enabled (x) (0) +#endif I would prefer if we introduce
CONFIG_EFIthat would stubefi_enabledfor architecture not supporting EFI.
Yes, that's a good idea.
I also feel a little awkward with the current arch helpers.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenQuoted 43 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto parse memory nodes in EFI boot.As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347
Quoted 10 lines
xen/arch/arm/bootfdt.c +6
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )
If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?
I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?
Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Hi Stefano,
Quoted 44 lines · Stefano Stabellini
Quoted 43 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto parse memory nodes in EFI boot.As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;
Quoted 15 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347
Quoted 10 lines
xen/arch/arm/bootfdt.c +6
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?
Yes. AFAIK, when booting using EFI, the Device-Tree may not contain all the reserved regions. Furthermore, we are still too early to know whether we boot using ACPI and DT.
Quoted 3 lines
I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?
I looked at Linux when I first reviewed this patch because I was wondering what happens if the DT and UEFI map disagrees.
Linux and Xen are the same after this patch:
1) The memory map is coming from UEFI map
2) NUMA ID is coming from the DT
The commit that introduced the change in Linux is:
commit 500899c2cc3e3f06140373b587a69d30650f2d9d
Author: Ard Biesheuvel <[email protected]> Date: Fri Apr 8 15:50:23 2016 -0700
efi: ARM/arm64: ignore DT memory nodes instead of removing them
There are two problems with the UEFI stub DT memory node removal routine:
- it deletes nodes as it traverses the tree, which happens to work but is not supported, as deletion invalidates the node iterator;
- deleting memory nodes entirely may discard annotations in the form of additional properties on the nodes.
Since the discovery of DT memory nodes occurs strictly before the UEFI init sequence, we can simply clear the memblock memory table before parsing the UEFI memory map. This way, it is no longer necessary to remove the nodes, so we can remove that logic from the stub as well.
Cheers,
Signature · 1 line
Julien Grall
Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Quoted 70 lines · Julien Grall
Hi Stefano,
Quoted 51 lines · Stefano Stabellini
Quoted 50 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto parse memory nodes in EFI boot.As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +7
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; - if ( device_tree_node_matches(fdt, node, "memory") ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. + */ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc = process_memory_node(fdt, node, name, depth,Quoted 8 lines
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt,
Quoted 3 lines
xen/arch/arm/bootfdt.c +0
-0address_cells, size_cells, &bootinfo.mem); else if ( depth == 1 && !dt_node_cmp(name, "reserved-memory") )If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?
Yes. AFAIK, when booting using EFI, the Device-Tree may not contain all the reserved regions. Furthermore, we are still too early to know whether we boot using ACPI and DT.
Quoted 3 lines
I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?
Quoted 8 lines
I looked at Linux when I first reviewed this patch because I was wondering what happens if the DT and UEFI map disagrees.
Linux and Xen are the same after this patch:
1) The memory map is coming from UEFI map
2) NUMA ID is coming from the DTThe commit that introduced the change in Linux is:
[...]
Thanks both you and Wei for the investigation :-)
RE: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Stefano StabelliniHi Stefano,
Quoted 66 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:25
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIQuoted 42 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prventearly_scan_nodeto parse memory nodes in EFI boot.As EFI APIs only can be used in Arm64, so we introduced a wrapper in header file to prevent #ifdef CONFIG_ARM_64/32 in code block.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/asm-arm/setup.h | 6 ++++++ 3 files changed, 13 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -335,7 +336,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 346
Quoted 10 lines
xen/arch/arm/bootfdt.c +6
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If system boot from EFI, bootinfo.mem has been set by EFI, +* so we don't need to parse memory node from DTB. +*/ +if (device_tree_node_matches (fdt, node,"memory" ) && + !arch_efi_enabled (EFI_BOOT ) ) rc = process_memory_node(fdt, node, name, depth, address_cells, size_cells,&bootinfo.mem);
xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 347
Quoted 1 line
xen/arch/arm/bootfdt.c +0
-0else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )
Quoted 6 lines
If we are going to use the device tree info for the numa nodes (and related memory) does it make sense to still rely on the EFI tables for the memory map?
I wonder if we should just use device tree for memory and ignore EFI instead. Do you know what Linux does in this regard?
We don't use device tree for memory map on EFI boot. We just reply on device tree to provide memory NUMA node info. Because EFI system table doesn't contain this kind of data.
I have a quick look into Linux. Linux efi stub has a update_fdt function in:drivers/firmware/efi/libstub/fdt.c. In this function, efi stub only delete reserve memory nodes. Usable memory nodes will not be touched in this function.
Before Linux efi_init, early_init_dt_scan will be invoked. This means, if the efi stub doesn't remove the normal memory nodes, these nodes will be found and added to memblock by early_init_dt_add_memory_arch();
Later, in efi_init, sytemtable->memory_description will also be parsed, the memory block will be added to memblock by early_init_dt_add_memory_arch. So the duplicated memory nodes will be merged/ignored in memblock_add.
In Linux NUMA init, if ACPI is off, the NUMA info will be parsed from DT.
So I think in EFI boot, we use system table to parse memory, but use DT to parse NUMA info. It doesn't seem particularly strange : )
Cheers,
Wei Chen
RFC PATCH 19/40 xen: fdt: Introduce a helper to check fdt node type
In reply to Wei ChenIn later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce fdt_node_check_type in this patch to avoid redundant code in subsequent patches.
--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
xen/common/libfdt/fdt_ro.c +15 -0
@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,return 1 ; } +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type)) +return 0 ; +else +return 1 ; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {
xen/include/xen/libfdt/libfdt.h +25 -0
@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a 'device_type' +* property with the given string as one of its elements, it returns non-zero +* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag +* -FDT_ERR_BADMAGIC, +* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value * @fdt: pointer to the device tree blob
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type
Hi Wei,
Quoted 9 lines · Wei Chen
In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce
fdt_node_check_typein this patch to avoid redundant code in subsequent patches.Signed-off-by Wei Chen <[email protected]>--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++
This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.
Quoted 10 lines
2 files changed, 40 insertions(+)xen/common/libfdt/fdt_ro.c +0
-0@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, int nodeoffset,return 1 ; }
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557
Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10
-0+int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type))
The "device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?
Quoted 35 lines
xen/common/libfdt/fdt_ro.c +5
-0+return 0 ; + else + return 1; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {xen/include/xen/libfdt/libfdt.h +18
-0@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a 'device_type' +* property with the given string as one of its elements, it returns non-zero +* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a BEGIN_NODE tag +* -FDT_ERR_BADMAGIC,
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829
Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +7
-0+* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value * @fdt: pointer to the device tree blob
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type
Hi Julien,
Quoted 18 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:39
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node typeHi Wei,
Quoted 5 lines · Wei Chen
In later patches, we will parse CPU and memory NUMA information from device tree. FDT is using device type property to indicate CPU nodes and memory nodes. So we introduce
fdt_node_check_typein this patch to avoid redundant code in subsequent patches.
Quoted 9 lines
Quoted 4 lines
Signed-off-by Wei Chen <[email protected]>--- xen/common/libfdt/fdt_ro.c | 15 +++++++++++++++ xen/include/xen/libfdt/libfdt.h | 25 +++++++++++++++++++++++++This is meant to be a verbatim copy of libfdt. So I am not entirely in favor of adding a new function therefore without been upstreamed to libfdt first.
Oh, if we need to upstream this change in libfdt. I think I'd better to remove this change in libfdt. Because we can implement type checking in other place, and I don't want to introduce a dependency on external repo upstream in this series.
Quoted 11 lines
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int
Quoted 7 lines
2 files changed, 40 insertions(+)xen/common/libfdt/fdt_ro.c +0
-0@@ -545,6 +545,21 @@ int fdt_node_check_compatible(const void *fdt, intnodeoffset,
Quoted 3 lines
xen/common/libfdt/fdt_ro.c +0
-0return 1 ; }
Quoted 15 lines
xen/common/libfdt/fdt_ro.c int fdt_node_check_compatible(const void *fdt, int nodeoffset, · line 557
Quoted 10 lines
xen/common/libfdt/fdt_ro.c +10
-0+int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type) +{ +const void *prop; +int len; + + prop =fdt_getprop (fdt, nodeoffset,"device_type" , &len); +if (!prop) +return len; +if (fdt_stringlist_contains (prop, len, type))The "
device_type" is not a list of string. So I am a bit confused why you are using this helper. Shouldn't we simply check that the property value and type matches?
Yes, I think you're right. This function was based on the modification of fdt_node_check_compatible, and I forgot to replace fdt_stringlist_contains. And, as I reply above, we can simply the check. And I will implement it out of libfdt.
Quoted 56 lines
Review
Quoted 9 lines
xen/common/libfdt/fdt_ro.c +5
-0+return 0 ; + else + return 1; +} +int fdt_node_offset_by_compatible (const void *fdt,int startoffset,const char *compatible) {(quoted diff) +0
-0
b/xen/include/xen/libfdt/libfdt.hxen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt,
Quoted 4 lines
index
7c75688a39..7e4930dbcd100644xen/include/xen/libfdt/libfdt.h +0
-0@@ -799,6 +799,31 @@ int fdt_node_offset_by_phandle(const void *fdt,
uint32_tphandle);xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 11 lines
xen/include/xen/libfdt/libfdt.h +8
-0int fdt_node_check_compatible (const void *fdt,int nodeoffset,const char *compatible); +/** +* fdt_node_check_type: check a node's device_type property +* @fdt: pointer to the device tree blob +* @nodeoffset: offset of a tree node +* @type: string to match against +* +* +* fdt_node_check_type() returns 0 if the given node contains a '
device_type'xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, · line 810
Quoted 1 line
xen/include/xen/libfdt/libfdt.h +1
-0+* property with the given string as one of its elements, it returns non-zero
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 8 lines
xen/include/xen/libfdt/libfdt.h +8
-0+* otherwise, or on error. +* +* returns: +* 0, if the node has a 'device_type' property listing the given string +* 1, if the node has a 'device_type' property, but it does not list +* the given string +* -FDT_ERR_NOTFOUND, if the given node has no 'device_type' property +* -FDT_ERR_BADOFFSET, if nodeoffset does not refer to a
BEGIN_NODEtagxen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle);
Quoted 10 lines
xen/include/xen/libfdt/libfdt.h +8
-0+* -FDT_ERR_BADMAGIC, +* -FDT_ERR_BADVERSION, +* -FDT_ERR_BADSTATE, +* -FDT_ERR_BADSTRUCTURE, standard meanings +*/ +int fdt_node_check_type (const void *fdt,int nodeoffset, +const char *type); +/** * fdt_node_offset_by_compatible - find nodes with a given 'compatible' value
xen/include/xen/libfdt/libfdt.h int fdt_node_offset_by_phandle(const void *fdt, uint32_t phandle); · line 829
Quoted 2 lines
xen/include/xen/libfdt/libfdt.h +0
-0* @fdt: pointer to the device tree blob Cheers,
--
Julien Grall
RFC PATCH 20/40 xen/arm: implement node distance helpers for Arm64
In reply to Wei ChenIn current Xen code, __node_distance is a fake API, it always returns NUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide a set_node_distance API to set the distance for any two nodes in this patch.
--- xen/arch/arm/numa.c | 44 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 12 ++++++++++- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 +- 4 files changed, 56 insertions(+), 3 deletions(-)
xen/arch/arm/numa.c +44 -0
@@ -23,6 +23,11 @@#include <xen/pfn.h> #include <asm/setup.h> +static uint8_t __read_mostly +node_distance_map[MAX_NUMNODES ][MAX_NUMNODES ] = { + {NUMA_REMOTE_DISTANCE } +}; +void numa_set_node (int cpu,nodeid_t nid) { if ( nid >= MAX_NUMNODES ||
@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } +void __init numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance) +{ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) + { +printk (KERN_WARNING +"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n" , + from, to, distance); +return ; + } + +/* NUMA defines 0xff as an unreachable node and 0-9 are undefined */ +if ( distance >=NUMA_NO_DISTANCE || + (distance >=NUMA_DISTANCE_UDF_MIN && + distance <=NUMA_DISTANCE_UDF_MAX ) || + (from == to && distance !=NUMA_LOCAL_DISTANCE ) ) + { +printk (KERN_WARNING +"Invalid NUMA node distance, from:%d to:%d distance=%d\n" , + from, to, distance); +return ; + } + + node_distance_map[from][to] = distance; +} + +uint8_t __node_distance (nodeid_t from,nodeid_t to) +{ +/* +* Check whether the nodes are in the matrix range. +* When any node is out of range, except from and to nodes are the +* same, we treat them as unreachable (return 0xFF) +*/ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) +return from == to ?NUMA_LOCAL_DISTANCE :NUMA_NO_DISTANCE ; + +return node_distance_map[from][to]; +} +EXPORT_SYMBOL (__node_distance); +void __init numa_init (bool acpi_off) {uint32_t idx;
xen/include/asm-arm/numa.h +11 -1
@@ -12,8 +12,19 @@ typedef u8 nodeid_t;* set the number of NUMA memory block number to 128. */ #define NODES_SHIFT 6 +/* +* In ACPI spec, 0-9 are the reserved values for node distance, +* 10 indicates local node distance, 20 indicates remote node +* distance. Set node distance map in device tree will follow +* the ACPI's definition. +*/ +#define NUMA_DISTANCE_UDF_MIN 0 +#define NUMA_DISTANCE_UDF_MAX 9 +#define NUMA_LOCAL_DISTANCE 10 +#define NUMA_REMOTE_DISTANCE 20extern void numa_init (bool acpi_off); +extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);/* * Temporary for fake NUMA node, when CPU, memory and distance
@@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);* symbols will be removed. */ extern mfn_t first_valid_mfn; -#define __node_distance (a, b) (20)#else
xen/include/asm-x86/numa.h +0 -1
@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );void srat_parse_regions (u64 addr); -extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );#endif
xen/include/xen/numa.h +1 -1
@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) - +extern u8 __node_distance (nodeid_t a,nodeid_t b);extern void numa_add_cpu (int cpu);struct node {
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64
Quoted 31 lines · Wei Chen
In current Xen code,
__node_distanceis a fake API, it always returnsNUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide aset_node_distanceAPI to set the distance for any two nodes in this patch.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 44 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 12 ++++++++++- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 +- 4 files changed, 56 insertions(+), 3 deletions(-)xen/arch/arm/numa.c +5
-0@@ -23,6 +23,11 @@#include <xen/pfn.h> #include <asm/setup.h> +static uint8_t __read_mostly +node_distance_map[MAX_NUMNODES ][MAX_NUMNODES ] = { + {NUMA_REMOTE_DISTANCE } +}; +void numa_set_node (int cpu,nodeid_t nid) { if ( nid >= MAX_NUMNODES ||
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 46
Quoted 10 lines
xen/arch/arm/numa.c +7
-0@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } +void __init numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance) +{ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) + { +printk (KERN_WARNING +"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n" , + from, to, distance);
NIT: please align. Example:
printk(KERN_WARNING
"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",
Also please use PRIu32 for uint32_t. Probably should use PRIu8 for nodeids.
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 58
Quoted 12 lines
xen/arch/arm/numa.c +12
-0+return ; + } + +/* NUMA defines 0xff as an unreachable node and 0-9 are undefined */ +if ( distance >=NUMA_NO_DISTANCE || + (distance >=NUMA_DISTANCE_UDF_MIN && + distance <=NUMA_DISTANCE_UDF_MAX ) || + (from == to && distance !=NUMA_LOCAL_DISTANCE ) ) + { +printk (KERN_WARNING +"Invalid NUMA node distance, from:%d to:%d distance=%d\n" , + from, to, distance);
NIT: please align
Also you used %u before for nodeids, which is better because from and to are unsigned. Distance should be uint32_t.
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 4 lines
xen/arch/arm/numa.c +4
-0+return ; + } + + node_distance_map[from][to] = distance;
Shouldn't we also be setting:
node_distance_map[to][from] = distance;
?
Quoted 79 lines
xen/arch/arm/numa.c +16
-0+} + +uint8_t __node_distance (nodeid_t from,nodeid_t to) +{ +/* +* Check whether the nodes are in the matrix range. +* When any node is out of range, except from and to nodes are the +* same, we treat them as unreachable (return 0xFF) +*/ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) +return from == to ?NUMA_LOCAL_DISTANCE :NUMA_NO_DISTANCE ; + +return node_distance_map[from][to]; +} +EXPORT_SYMBOL (__node_distance); +void __init numa_init (bool acpi_off) {uint32_t idx;xen/include/asm-arm/numa.h +11
-1@@ -12,8 +12,19 @@ typedef u8 nodeid_t;* set the number of NUMA memory block number to 128. */ #define NODES_SHIFT 6 +/* +* In ACPI spec, 0-9 are the reserved values for node distance, +* 10 indicates local node distance, 20 indicates remote node +* distance. Set node distance map in device tree will follow +* the ACPI's definition. +*/ +#define NUMA_DISTANCE_UDF_MIN 0 +#define NUMA_DISTANCE_UDF_MAX 9 +#define NUMA_LOCAL_DISTANCE 10 +#define NUMA_REMOTE_DISTANCE 20extern void numa_init (bool acpi_off); +extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);/* * Temporary for fake NUMA node, when CPU, memory and distance @@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);* symbols will be removed. */ extern mfn_t first_valid_mfn; -#define __node_distance (a, b) (20)#else xen/include/asm-x86/numa.h +0
-1@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );void srat_parse_regions (u64 addr); -extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );#endif xen/include/xen/numa.h +1
-1@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) - +extern u8 __node_distance (nodeid_t a,nodeid_t b);extern void numa_add_cpu (int cpu);struct node {Signature · 1 line
2.25.1
RE: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64
Hi Stefano,
Quoted 53 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 7:52
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64Quoted 31 lines · Wei Chen
In current Xen code,
__node_distanceis a fake API, it always returnsNUMA_REMOTE_DISTANCE(20). Now we use a matrix to record the distance between any two nodes. Accordingly, we provide aset_node_distanceAPI to set the distance for any two nodes in this patch.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa.c | 44 ++++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 12 ++++++++++- xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 2 +- 4 files changed, 56 insertions(+), 3 deletions(-)xen/arch/arm/numa.c +5
-0@@ -23,6 +23,11 @@#include <xen/pfn.h> #include <asm/setup.h> +static uint8_t __read_mostly +node_distance_map[MAX_NUMNODES ][MAX_NUMNODES ] = { + {NUMA_REMOTE_DISTANCE } +}; +void numa_set_node (int cpu,nodeid_t nid) { if ( nid >= MAX_NUMNODES ||xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 40
Quoted 4 lines
xen/arch/arm/numa.c +1
-0@@ -32,6 +37,45 @@ void numa_set_node(int cpu, nodeid_t nid) cpu_to_node[cpu] = nid; } +void __init numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance)
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 46
Quoted 6 lines
xen/arch/arm/numa.c +6
-0+{ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) + { +printk (KERN_WARNING +"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n" , + from, to, distance);
Quoted 8 lines
NIT: please align. Example:
printk(KERN_WARNING
"NUMA nodes are out of matrix, from=%u to=%u distance=%u\n",Also please use PRIu32 for
uint32_t. Probably should use PRIu8 for nodeids.
OK
Quoted 15 lines
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid)
Quoted 5 lines
xen/arch/arm/numa.c +4
-0+return ; + } + +/* NUMA defines 0xff as an unreachable node and 0-9 are undefined */
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 58
Quoted 8 lines
xen/arch/arm/numa.c +8
-0+if ( distance >=NUMA_NO_DISTANCE || + (distance >=NUMA_DISTANCE_UDF_MIN && + distance <=NUMA_DISTANCE_UDF_MAX ) || + (from == to && distance !=NUMA_LOCAL_DISTANCE ) ) + { +printk (KERN_WARNING +"Invalid NUMA node distance, from:%d to:%d distance=%d\n" , + from, to, distance);
Quoted 5 lines
NIT: please align
Also you used %u before for nodeids, which is better because from and to are unsigned. Distance should be
uint32_t.
OK
Quoted 6 lines
xen/arch/arm/numa.c void numa_set_node(int cpu, nodeid_t nid) · line 62
Quoted 5 lines
xen/arch/arm/numa.c +4
-0+return ; + } + + node_distance_map[from][to] = distance;
Quoted 6 lines
Shouldn't we also be setting:
node_distance_map[to][from] = distance;?
No, we want numa_set_distance behavior is single. "node_distance_map[to][from] = distance" is handled in caller.
Quoted 82 lines
Quoted 31 lines
xen/arch/arm/numa.c +16
-0+} + +uint8_t __node_distance (nodeid_t from,nodeid_t to) +{ +/* +* Check whether the nodes are in the matrix range. +* When any node is out of range, except from and to nodes are the +* same, we treat them as unreachable (return 0xFF) +*/ +if ( from >=MAX_NUMNODES || to >=MAX_NUMNODES ) +return from == to ?NUMA_LOCAL_DISTANCE :NUMA_NO_DISTANCE ; + +return node_distance_map[from][to]; +} +EXPORT_SYMBOL (__node_distance); +void __init numa_init (bool acpi_off) {uint32_t idx;xen/include/asm-arm/numa.h +3
-0@@ -12,8 +12,19 @@ typedef u8 nodeid_t;* set the number of NUMA memory block number to 128. */ #define NODES_SHIFT 6 +/* +* In ACPI spec, 0-9 are the reserved values for node distance, +* 10 indicates local node distance, 20 indicates remote node xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 27
Quoted 10 lines
xen/include/asm-arm/numa.h +8
-0+* distance. Set node distance map in device tree will follow +* the ACPI's definition. +*/ +#define NUMA_DISTANCE_UDF_MIN 0 +#define NUMA_DISTANCE_UDF_MAX 9 +#define NUMA_LOCAL_DISTANCE 10 +#define NUMA_REMOTE_DISTANCE 20extern void numa_init (bool acpi_off); +extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
xen/include/xen/numa.h static inline __attribute__((pure)) nodeid_t
Quoted 28 lines
xen/include/asm-arm/numa.h +0
-1/* * Temporary for fake NUMA node, when CPU, memory and distance @@ -21,7 +32,6 @@ extern void numa_init(bool acpi_off);* symbols will be removed. */ extern mfn_t first_valid_mfn; -#define __node_distance (a, b) (20)#else xen/include/asm-x86/numa.h +0
-1@@ -21,7 +21,6 @@ extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );void srat_parse_regions (u64 addr); -extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );#endif xen/include/xen/numa.h +0
-0@@ -58,7 +58,7 @@ static inline __attribute__((pure)) nodeid_t
phys_to_nid(paddr_t addr)Quoted 11 lines
xen/include/xen/numa.h +1
-2#define node_spanned_pages (nid) (NODE_DATA(nid)->node_spanned_pages)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) - +extern u8 __node_distance (nodeid_t a,nodeid_t b);extern void numa_add_cpu (int cpu);struct node { --2.25.1
RFC PATCH 21/40 xen/arm: introduce device_tree_numa as a switch for device tree NUMA
In reply to Wei ChenLike acpi_numa in x86 as a switch for ACPI based NUMA, we introduce device_tree_numa as a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.
Keep using bad_srat and srat_disabled functions name, because we will reuse node_covers_memory and acpi_scan_nodes code for Arm. These functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.
--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+) create mode 100644 xen/arch/arm/numa_device_tree.c
xen/arch/arm/Makefile +1 -0
@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o
xen/arch/arm/numa_device_tree.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void ) +{ +return numa_off || device_tree_numa <0 ; +} + +void __init bad_srat (void ) +{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}
xen/include/asm-arm/numa.h +2 -0
@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA
Hi Wei,
Quoted 5 lines · Wei Chen
Like
acpi_numain x86 as a switch for ACPI based NUMA, we introducedevice_tree_numaas a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.
The hypervisor will never use both ACPI and DT at runtime. In fact...
Quoted 3 lines
Keep using
bad_sratandsrat_disabledfunctions name, because we will reusenode_covers_memoryandacpi_scan_nodescode for Arm.
... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.
So I think we should either rename acpi_numa to something more generic (maybe fw_numa) or convert numa_off to a tri-state.
This will allow to have the code mostly common.
Quoted 47 lines
These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+)create mode 100644
xen/arch/arm/numa_device_tree.cxen/arch/arm/Makefile +1
-0@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.oxen/arch/arm/numa_device_tree.c new file +16
-0@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License
xen/arch/arm/numa_device_tree.c line 26
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10
-0+* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void )
You export this one and ...
xen/arch/arm/numa_device_tree.c line 31
Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1
-0+void __init bad_srat (void )
... this one without providing in a prototype.
Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.
In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).
In this case, the helpers should be exported from now.
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +4
-0+{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}xen/include/asm-arm/numa.h +0
-0
xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 30
Quoted 8 lines
xen/include/asm-arm/numa.h +2
-0@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off);extern void numa_set_distance (nodeid_t from,nodeid_t to,uint32_t distance);
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA
Hi Julien,
Quoted 11 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 1:45
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 21/40] xen/arm: introducedevice_tree_numaas a switch for device tree NUMAHi Wei,
Quoted 9 lines
Quoted 5 lines · Wei Chen
Like
acpi_numain x86 as a switch for ACPI based NUMA, we introducedevice_tree_numaas a switch for Arm device tree based NUMA. When NUMA information in device tree is invalid, this switch will be set to -1, then NUMA support for Arm will be disabled, even if user set numa_off=0.The hypervisor will never use both ACPI and DT at runtime. In fact...
Yes.
Quoted 4 lines
Quoted 3 lines
Keep using
bad_sratandsrat_disabledfunctions name, because we will reusenode_covers_memoryandacpi_scan_nodescode for Arm.
Quoted 8 lines
... given that both functions will be called from the common code, it will be a lot more difficult to add ACPI afterwards.
So I think we should either rename
acpi_numato something more generic (maybefw_numa) or convertnuma_offto a tri-state.This will allow to have the code mostly common.
I will try to address them in next version.
Quoted 65 lines
Review
Quoted 26 lines
These
functions are using these two API names. And, as device tree can be treated as one kind of static resource table. So we keep these two function names.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/numa_device_tree.c | 35 +++++++++++++++++++++++++++++++++ xen/include/asm-arm/numa.h | 2 ++ 3 files changed, 38 insertions(+)create mode 100644
xen/arch/arm/numa_device_tree.cxen/arch/arm/Makefile +1
-0@@ -36,6 +36,7 @@ obj-y += mem_access.o obj-y += mm.o obj-y += monitor.o obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_DEVICE_TREE_NUMA) += numa_device_tree.o obj-y += p2m.o obj-y += percpu.o obj-y += platform.o(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 21 lines
new file mode 100644
index 0000000000..1c74ad135dxen/arch/arm/numa_device_tree.c new file +16
-0@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Arm Architecture support layer for NUMA. +* +* Copyright (C) 2021 Arm Ltd +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License xen/arch/arm/numa_device_tree.c line 26
Quoted 10 lines
xen/arch/arm/numa_device_tree.c new file +10
-0+* along with this program. If not, see <http://www.gnu.org/licenses/>. +* +*/ +#include <xen/init.h> +#include <xen/nodemask.h> +#include <xen/numa.h> + +s8 device_tree_numa =0 ; + +int srat_disabled (void )You export this one and ...
xen/arch/arm/numa_device_tree.c line 31
Quoted 1 line
xen/arch/arm/numa_device_tree.c new file +1
-0+void __init bad_srat (void )... this one without providing in a prototype.
Quoted 9 lines
Looking at the rest of the series... they will be turned static in the next patch (#21) but then re-exported in patch #33.
In general, we should refrain to modify code that was added in the same patch unless it is not possible for split reason (e.g code clean-up and then code movement).
In this case, the helpers should be exported from now.
Ok.
Quoted 24 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c new file +4
-0+{ +printk (KERN_ERR "DT: NUMA information is not used.\n" ); + device_tree_numa =-1 ; +}xen/include/asm-arm/numa.h +0
-0xen/include/asm-arm/numa.h typedef u8 nodeid_t; · line 29
Quoted 7 lines
xen/include/asm-arm/numa.h +2
-0@@ -23,6 +23,8 @@ typedef u8 nodeid_t;#define NUMA_LOCAL_DISTANCE 10#define NUMA_REMOTE_DISTANCE 20 +extern s8 device_tree_numa; +extern void numa_init (bool acpi_off); extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_tdistance);
Quoted 2 lines
Cheers,
--
Julien Grall
RFC PATCH 22/40 xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenProcessor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)
xen/arch/arm/numa_device_tree.c +39 -2
@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Wei,
Quoted 37 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +5
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ;
xen/arch/arm/numa_device_tree.c line 47
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +9
-0} + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ;
You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.
xen/arch/arm/numa_device_tree.c line 59
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+ } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node)
AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.
In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +3
-0+{ +uint32_t nid; +
xen/arch/arm/numa_device_tree.c line 72
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Julien,
Quoted 56 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:10
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 30 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +10
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () )
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 47
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+return -EINVAL ; + else if ( node == NUMA_NO_NODE || node >= MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ;You seem to have a mix of soft and hard tab in this file. Is there a lot of the code that was directly copied from Linux? If not, then the file should be using Xen coding style.
I copied some code from x86, and x86 is Linux style. So, yes, I should adjust them it Xen coding style. I will do it in next version.
Quoted 17 lines
xen/arch/arm/numa_device_tree.c line 59
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+ } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node)AFAICT, you are going to turn this helper static in a follow-up patch. This is a bad practice. Instead, the function should be static from the beginning. If it is not possible, then you should re-order the code.
Quoted 4 lines
In this case, I think you can add the boilerplate to parse the NUMA information (patch #25) here and then extend it in each patch.
That's make sense, I will try to address it in next version.
Quoted 20 lines
xen/arch/arm/numa_device_tree.c
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenQuoted 31 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }
xen/arch/arm/numa_device_tree.c line 41
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)
I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 22 lines
xen/arch/arm/numa_device_tree.c +21
-0+{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; +
xen/arch/arm/numa_device_tree.c line 72
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Julien,
Quoted 43 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }
Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
yes, in the very beginning, I want to keep device_tree_ prefix for functions that will handle dtb file. And use dtb_ prefix to replace acpi, to indicate, this function is device tree version numa implementation.
If that's not the right reason, I will unify all prefix to device_tree_ in next version. How do you think about it?
Quoted 38 lines
Quoted 16 lines
xen/arch/arm/numa_device_tree.c +16
-0+{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 58 lines
Quoted 43 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 3 lines
yes, in the very beginning, I want to keep
device_tree_prefix for functions that will handle dtb file. And usedtb_prefix to replace acpi, to indicate, this function is device tree version numa implementation.
Thanks for the clarification. The difference between "dtb" and "device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.
Quoted 3 lines
If that's not the right reason, I will unify all prefix to
device_tree_in next version. How do you think about it?
AFAICT, your parsing functions will always start with "device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".
If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Julien,
Quoted 78 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月23日 18:59
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 17 lines
Quoted 17 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:11
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41+++++++++++++++++++++++++++++++--
Quoted 38 lines
Quoted 23 lines
Review
Quoted 3 lines
1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 19 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +4
-1@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; }Quoted 14 lines
xen/arch/arm/numa_device_tree.c line 41
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +4
-1-void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node)I forgot to answer. It seems odd that some of the function names start with dtb_* while other starts device_tree_*. Any particular reason for that difference of naming?
Quoted 3 lines
yes, in the very beginning, I want to keep
device_tree_prefix for functions that will handle dtb file. And usedtb_prefix to replace acpi, to indicate, this function is device tree version numa
Quoted 7 lines
implementation.
Thanks for the clarification. The difference between "dtb" and "
device_tree" is quite subttle: the former refers to the binary while the latter refers to the format. Most of the readers are likely to infer they mean the same. So I think this will bring more confusion.
Thanks for the clarification.
Quoted 11 lines
Quoted 3 lines
If that's not the right reason, I will unify all prefix to
device_tree_in next version. How do you think about it?AFAICT, your parsing functions will always start with "
device_tree_parse_". I would prefer if the set replacing the ACPI helpers start with "device_tree_".If you are concern with the length of the function name, then I would suggest to prefix all the functions with "fdt" (We are dealing with the flattened DT after all) or "dt".
That makes sense, I will do it.
Quoted 5 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenHi Wei,
Quoted 17 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +0
-0
xen/arch/arm/numa_device_tree.c line 23
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h>
Nothing in this file seems to depend on xen/device_tree.h. So why do you need to include it?
Quoted 52 lines
xen/arch/arm/numa_device_tree.c +38
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Julien,
Quoted 26 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.c
Quoted 12 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why do you need to include it?
I remember that without this header file, device_tree_get_u32 in this patch will cause compiling failed.
Quoted 56 lines
Quoted 36 lines
xen/arch/arm/numa_device_tree.c +23
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 39 lines
Quoted 26 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 12 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why do you need to include it?
Quoted 2 lines
I remember that without this header file,
device_tree_get_u32in this patch will cause compiling failed.
I looked at the prototype of device_tree_get_u32() and I can't find how it depends on bits from device_tree.h. Can you paste the compilation error?
Quoted 48 lines
Quoted 48 lines
Quoted 37 lines
xen/arch/arm/numa_device_tree.c +23
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} +xen/arch/arm/numa_device_tree.c
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien Grall
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Julien,
Quoted 57 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 16:44
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 19 lines
Quoted 19 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月20日 2:13
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeHi Wei,
Quoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41+++++++++++++++++++++++++++++++--
Quoted 4 lines
Quoted 4 lines
Review
Quoted 3 lines
1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 10 lines
Quoted 10 lines
xen/arch/arm/numa_device_tree.c line 23
Quoted 8 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +1
-0@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> Nothing in this file seems to depend on
xen/device_tree.h. So why doyou
Quoted 3 lines
Quoted 2 lines
need to include it?
Quoted 7 lines
Quoted 1 line
I remember that without this header file,
device_tree_get_u32in thispatch
Quoted 1 line
will cause compiling failed.
I looked at the prototype of
device_tree_get_u32()and I can't find how it depends on bits fromdevice_tree.h. Can you paste the compilation error?
I tested it again, this header file should be introduced in following patches:numa_device_tree.c: In function ‘device_tree_parse_numa_distance_map_v1’:
numa_device_tree.c:243:16: error: implicit declaration of function ‘dt_read_number’ [-Werror=implicit-function-declaration]
243 | from = dt_read_number(matrix, 1);
| ^~~~~~~~~~~~~~numa_device_tree.c:243:16: error: nested extern declaration of ‘dt_read_number’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
I will move it to another patch.
Quoted 63 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
xen/arch/arm/numa_device_tree.c +21
-2+#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ;Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid = device_tree_get_u32(fdt, node, "numa-node-id",
MAX_NUMNODES);Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa_device_tree.c
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 9 lines
Quoted 9 lines
xen/arch/arm/numa_device_tree.c line 72
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}--
Julien GrallCheers,
--
Julien Grall
Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
In reply to Wei ChenQuoted 54 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)xen/arch/arm/numa_device_tree.c +21
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); +
xen/arch/arm/numa_device_tree.c line 64
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) +{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid);
Given that this is not actually a warning (is it?) then I would move it to XENLOG_INFO
xen/arch/arm/numa_device_tree.c line 67
Quoted 3 lines
xen/arch/arm/numa_device_tree.c +3
-0+if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid);
This could be XENLOG_ERR
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
RE: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node
Hi Stefano,
Quoted 70 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月27日 8:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor nodeQuoted 7 lines · Wei Chen
Processor NUMA ID information is stored in device tree's processor node as "numa-node-id". We need a new helper to parse this ID from processor node. If we get this ID from processor node, this ID's validity still need to be checked. Once we got a invalid NUMA ID from any processor node, the device tree will be marked as NUMA information invalid.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 41 +++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cQuoted 46 lines
index
1c74ad135d..37cc56acf3100644xen/arch/arm/numa_device_tree.c +26
-2@@ -20,16 +20,53 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/device_tree.h> +#include <asm/setup.h> s8 device_tree_numa =0 ; +static nodemask_t processor_nodes_parsed__initdata ; -int srat_disabled (void ) +static int srat_disabled (void ) {return numa_off || device_tree_numa <0 ; } -void __init bad_srat (void ) +static __init void bad_srat (void ) {printk (KERN_ERR "DT: NUMA information is not used.\n" ); device_tree_numa =-1 ; } + +/* Callback for device tree processor affinity */ +static int __init dtb_numa_processor_affinity_init (nodeid_t node) +{ +if (srat_disabled () ) +return -EINVAL ; +else if ( node ==NUMA_NO_NODE || node >=MAX_NUMNODES ) { +bad_srat (); +return -EINVAL ; + } + +node_set (node, processor_nodes_parsed); + + device_tree_numa =1 ; +printk (KERN_INFO "DT: NUMA node %u processor parsed\n" , node); + +return 0 ; +} + +/* Parse CPU NUMA node info */ +int __init device_tree_parse_numa_cpu_node (const void *fdt,int node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c line 64
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+{ +uint32_t nid; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +printk (XENLOG_WARNING "CPU on NUMA node:%u\n" , nid);Given that this is not actually a warning (is it?) then I would move it to
XENLOG_INFO
OK
Quoted 7 lines
xen/arch/arm/numa_device_tree.c
Quoted 3 lines
xen/arch/arm/numa_device_tree.c +3
-0+if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
This could be
XENLOG_ERR
OK
Quoted 6 lines
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return dtb_numa_processor_affinity_init (nid); +}
RFC PATCH 23/40 xen/arm: introduce a helper to parse device tree memory node
In reply to Wei ChenMemory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)
xen/arch/arm/numa_device_tree.c +130 -0
@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */ + i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) { +/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 +"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , + node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {
@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)return dtb_numa_processor_affinity_init (nid); } + +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells) +{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ ) + { +device_tree_get_reg (&addresses, addr_cells, size_cells, &addr, &size); +/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); +if ( ret ) { +printk (XENLOG_WARNING +"NUMA: process range#%d addr = %lx size=%lx failed!\n" , + idx, addr, size); +return -EINVAL ; + } + } + +if ( idx ==0 ) + { +printk (XENLOG_ERR +"bad property in memory node, idx=%d ret=%d\n" , idx, ret); +return -EINVAL ; + } + +return 0 ; +}
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Wei,
Quoted 31 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +4
-0@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{
The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat ();
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */ + i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) {
Xen coding style is using:
if ( ... )
{
Note that I may not comment on all the occurents, so please check the other places.
Quoted 50 lines
xen/arch/arm/numa_device_tree.c +46
-0+/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 +"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , + node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +5
-0@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)return dtb_numa_processor_affinity_init (nid); } + +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells)
This is pretty much a copy of process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?
Quoted 26 lines
xen/arch/arm/numa_device_tree.c +26
-0+{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { +printk (XENLOG_WARNING "Node id %u exceeds maximum value\n" , nid); +return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ )
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 188
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ { +device_tree_get_reg (&addresses, addr_cells, size_cells, &addr, &size); +/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); +if ( ret ) { + printk(XENLOG_WARNING + "NUMA: process range#%d addr = %lx size=%lx failed!\n",
s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size are paddr_t.
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +4
-0+ idx, addr, size); +return -EINVAL ; + } + }
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 202
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ +if ( idx ==0 ) + { +printk (XENLOG_ERR +"bad property in memory node, idx=%d ret=%d\n" , idx, ret); +return -EINVAL ; + } + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Julien,
Quoted 48 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }
Quoted 7 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.
But, yes, maybe we can abstract some common code to functions, that can be reused in two functions. I would try it in next version.
Quoted 26 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +7
-0+struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; +xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 86
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) {
Quoted 8 lines
Xen coding style is using:
if ( ... )
{Note that I may not comment on all the occurents, so please check the other places.
Ok.
Quoted 53 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const
Quoted 43 lines
xen/arch/arm/numa_device_tree.c +39
-0+ node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++; + +return 0 ; +} +/* Parse CPU NUMA node info */ int __init device_tree_parse_numa_cpu_node (const void *fdt,int node) {@@ -70,3 +145,58 @@ int __init device_tree_parse_numa_cpu_node(constvoid *fdt, int node)
Quoted 1 line
Quoted 11 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node) · line 152
Quoted 7 lines
return dtb_numa_processor_affinity_init(nid);}
xen/arch/arm/numa_device_tree.c +5
-0+ +/* Parse memory node NUMA info */ +int __init +device_tree_parse_numa_memory_node (const void *fdt,int node, +const char *name,uint32_t addr_cells,uint32_t size_cells)This is pretty much a copy of
process_memory_node(). Can we consider to collect the NUMA ID from there? If not, can we at least abstract the code?
I had tried to parse NUMA id in process_memory_node or in early_scan_node. But I found, this change would make NUMA init code in different places. And I prefer a unify NUMA init entry. Because it may be good for maintenance. When we want to disable NUMA, we just need to disable it in one place. But you're right, we still can do some code abstraction. I will do it in next version.
Quoted 38 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 12 lines
xen/arch/arm/numa_device_tree.c +12
-0+{ +uint32_t nid; +int ret =0 , len; +paddr_t addr, size; +const struct fdt_property *prop; +uint32_t idx, ranges; +const __be32 *addresses; + + nid =device_tree_get_u32 (fdt, node,"numa-node-id" ,MAX_NUMNODES ); +if ( nid >=MAX_NUMNODES ) + { + printk(XENLOG_WARNING "Node id %u exceeds maximum value\n",nid);
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+return -EINVAL ; + } + + prop =fdt_get_property (fdt, node,"reg" , &len); +if ( !prop ) + {xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+printk (XENLOG_WARNING +"fdt: node `%s': missing `reg' property\n" , name); +return -EINVAL ; + } + + addresses = (const __be32 *)prop->data; + ranges = len / (sizeof (__be32)* (addr_cells + size_cells)); +for ( idx =0 ; idx < ranges; idx++ ) + { + device_tree_get_reg(&addresses, addr_cells, size_cells, &addr,&size);
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+/* Skip zero size ranges */ +if ( !size ) +continue ; + + ret =dtb_numa_memory_affinity_init (nid, addr, size); + if ( ret ) { + printk(XENLOG_WARNING + "NUMA: process range#%d addr = %lx size=%lx
Quoted 5 lines
failed!\n",
s/%d/%u/ as idx is an unsigned int
s/%lx/%"PRI_paddr"/ as addr and size arepaddr_t.
OK
Quoted 21 lines
xen/arch/arm/numa_device_tree.c int __init device_tree_parse_numa_cpu_node(const void *fdt, int node)
Quoted 9 lines
xen/arch/arm/numa_device_tree.c +9
-0+ idx, addr, size); +return -EINVAL ; + } + } + +if ( idx ==0 ) + { + printk(XENLOG_ERR + "bad property in memory node, idx=%d ret=%d\n", idx,ret);
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 131
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +5
-0+return -EINVAL ; + } + +return 0 ; +}Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Quoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 56 lines
Quoted 48 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }Quoted 7 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
Quoted 4 lines
In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.
It think you can get away from distinguishing between ACPI and DT in that helper:
* ma->flags & ACPI_SRAT_MEM_HOTPLUGGABLE could be replace by an
argument indicating whether the region is hotpluggable (this would always be false for DT)
* Access to memblk_hotplug can be stubbed (in the future we may want
to consider memory hotplug even on Arm).
Do you still have the "if else" version? If so can you post it?
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Julien,
Quoted 86 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月26日 16:22
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeQuoted 1 line · Wei Chen
Hi Julien,
Hi Wei,
Quoted 16 lines
Quoted 16 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:49
To: Wei Chen <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 4 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130++++++++++++++++++++++++++++++++
Quoted 34 lines
Quoted 26 lines
Review
Quoted 3 lines
1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }Quoted 7 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 63
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{The implementation of this function is quite similar ot the ACPI version. Can this be abstracted?
Quoted 4 lines
In my draft, I had tried to merge ACPI and DTB versions in one function. I introduced a number of "if else" to distinguish ACPI from DTB, especially ACPI hotplug. The function seems very messy. Not enough benefits to make up for the mess, so I gave up.
It think you can get away from distinguishing between ACPI and DT in that helper:
*ma->flags&ACPI_SRAT_MEM_HOTPLUGGABLEcould be replace by an
argument indicating whether the region is hotpluggable (this would always be false for DT)
* Access tomemblk_hotplugcan be stubbed (in the future we may want
to consider memory hotplug even on Arm).
Quoted 2 lines
Do you still have the "if else" version? If so can you post it?
I just tried to do that in draft process, because I was not satisfied with the changes, I haven't saved them as a patch.
I think your suggestions are worth to try again, I will do it in next version.
Quoted 4 lines
Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
In reply to Wei ChenQuoted 93 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)xen/arch/arm/numa_device_tree.c +61
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init dtb_numa_processor_affinity_init(nodeid_t node)return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */ + i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) { +/* +* Overlaps with other memblk in the same node, warning here. +* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 +"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , + node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end )
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;
Is it possible to have non-contigous ranges of memory for a single NUMA node?
Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Stefano,
Quoted 122 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月28日 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeQuoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11
-0return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; +xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + + if ( memblk_nodeid[i] == node ) { + /* + * Overlaps with other memblk in the same node, warning here. + * This memblk will be merged with conflicted memblk later. + */ + printk(XENLOG_WARNING + "DT: NUMA NODE %u (%"PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
Quoted 25 lines
xen/arch/arm/numa_device_tree.c +25
-0+ node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end )xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a single NUMA node?
Quoted 3 lines
Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too? If yes, I think this will affect x86 ACPI NUMA too. In next version, we plan to merge dtb_numa_memory_affinity_init and acpi_numa_memory_affinity_init into a neutral function. So we can fix them at the same time.
If not, maybe we have to keep the diversity for dtb and ACPI here. Anyway, Thanks for pointing this, I will look into the latest Linux implementation.
Cheers,
Wei Chen
Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Wei,
Quoted 122 lines · Wei Chen
Quoted 122 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月28日 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeQuoted 7 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Review
Quoted 6 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11
-0return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; +xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + + if ( memblk_nodeid[i] == node ) { + /* + * Overlaps with other memblk in the same node, warning here. + * This memblk will be merged with conflicted memblk later. + */ + printk(XENLOG_WARNING + "DT: NUMA NODE %u (%"PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
Quoted 25 lines
xen/arch/arm/numa_device_tree.c +25
-0+ node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + }else { +/* +* Conflict with memblk in other node, this is an error. +* The NUMA information is invalid, NUMA will be turn off. +*/ +printk (XENLOG_ERR +"DT: NUMA NODE %u (%" PRIx64"-%" + PRIx64") overlaps with NODE %u (%" PRIx64"-%" PRIx64")\n" , + node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end )xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a single NUMA node?
Quoted 6 lines
Quoted 3 lines
Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too?
I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.
Quoted 6 lines
If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to mergedtb_numa_memory_affinity_initandacpi_numa_memory_affinity_initinto a neutral function. So we can fix them at the same time.If not, maybe we have to keep the diversity for dtb and ACPI here.
I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?
If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Julien,
Quoted 159 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:34
To: Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021��8��28�� 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 9 lines
Quoted 9 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeQuoted 4 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130++++++++++++++++++++++++++++++++
Quoted 62 lines
Quoted 62 lines
Review
Quoted 3 lines
1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 14 lines
xen/arch/arm/numa_device_tree.c +11
-0return 0 ; } +/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; + +if (srat_disabled () ) +return -EINVAL ; +xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used later */
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */ ; +if ( i <0 ) + goto save_memblk; + + if ( memblk_nodeid[i] == node ) { + /* + * Overlaps with other memblk in the same node, warning here. + * This memblk will be merged with conflicted memblk later. + */ + printk(XENLOG_WARNING + "DT: NUMA NODE %u (%"PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
Quoted 10 lines
Quoted 10 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ node, start, end, + node_memblk_range[i].start, node_memblk_range[i].end); + } else { + /* + * Conflict with memblk in other node, this is an error. + * The NUMA information is invalid, NUMA will be turn off. + */ + printk(XENLOG_ERR + "DT: NUMA NODE %u (%"PRIx64"-%" + PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
Quoted 30 lines
Quoted 30 lines
Quoted 15 lines
xen/arch/arm/numa_device_tree.c +15
-0+ node, start, end, memblk_nodeid[i], + node_memblk_range[i].start, node_memblk_range[i].end); +bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end )xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end); + + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a single NUMA node?
Quoted 6 lines
Quoted 3 lines
Looking at the DT bindings and Linux implementation, it seems possible. Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI allow non-contiguous ranges of memory for a single NUMA node too?
I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.
Quoted 6 lines
If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to mergedtb_numa_memory_affinity_initandacpi_numa_memory_affinity_initinto a neutral function. So we can fix them at the same time.If not, maybe we have to keep the diversity for dtb and ACPI here.
I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?
Quoted 6 lines
If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).
Yes, I agree.
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Julien, Stefano, Jan
Quoted 183 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年8月28日 21:58
To: Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Julien,
Quoted 12 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月28日 18:34
To: Wei Chen <[email protected]>; Stefano Stabellini <[email protected]>
Cc: [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeHi Wei,
Quoted 9 lines
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021��8��28�� 9:06
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
[email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 23/40] xen/arm: introduce a helper toparse
Quoted 40 lines
Quoted 7 lines
Quoted 7 lines
device tree memory node
Quoted 4 lines · Wei Chen
Memory blocks' NUMA ID information is stored in device tree's memory nodes as "numa-node-id". We need a new helper to parse and verify this ID from memory nodes.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In order to support memory affinity in later use, the valid memory ranges and NUMA ID will be saved to tables.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 130++++++++++++++++++++++++++++++++
Quoted 26 lines
Quoted 26 lines
Review
Quoted 3 lines
1 file changed, 130 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c static int __init
Quoted 18 lines
index
37cc56acf3..bbe081dcd1100644xen/arch/arm/numa_device_tree.c +2
-0@@ -20,11 +20,13 @@#include <xen/init.h> #include <xen/nodemask.h> #include <xen/numa.h> +#include <xen/libfdt/libfdt.h> #include <xen/device_tree.h> #include <asm/setup.h> s8 device_tree_numa =0 ;static nodemask_t processor_nodes_parsed__initdata ; +static nodemask_t memory_nodes_parsed__initdata ;static int srat_disabled (void ) {@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; +Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { + dprintk(XENLOG_WARNING, + "Too many numa entry, try bigger NR_NODE_MEMBLKS\n");
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
later */
Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2
-0+ i =conflicting_memblks (start, end); +/* No conflicting memory block, we can save it for later usage */;
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) { +/* +* Overlaps with other memblk in the same node, warning here.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2
-0+ node, start, end, + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 11 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ } else { + /* + * Conflict with memblk in other node, this is an error. + * The NUMA information is invalid, NUMA will be turn off. + */ + printk(XENLOG_ERR + "DT: NUMA NODE %u (%"PRIx64"-%" + PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
xen/arch/arm/numa_device_tree.c +2
-0+ node, start, end, memblk_nodeid[i], + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18
-0+bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a single
NUMA
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
node?
Looking at the DT bindings and Linux implementation, it seems
possible.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI
allow
Quoted 8 lines
Quoted 1 line
non-contiguous ranges of memory for a single NUMA node too?
I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.
Quoted 3 lines
If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to mergedtb_numa_memory_affinity_initandacpi_numa_memory_affinity_initintoa
Quoted 8 lines
Quoted 3 lines
neutral function. So we can fix them at the same time.
If not, maybe we have to keep the diversity for dtb and ACPI here.
I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?
Quoted 9 lines
Quoted 6 lines
If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).
Yes, I agree.
I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.
This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.
To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?
Or could you give me suggestions that we can use some simple ways to fix it?
Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]
[1] https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c
[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3] https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c
Quoted 4 lines
Quoted 4 lines
Cheers,
--
Julien Grall
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Quoted 133 lines · Wei Chen
Quoted 113 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Review static int __init
Quoted 1 line
(quoted diff) +0
-0@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
xen/arch/arm/numa_device_tree.c +0
-0return 0 ; }Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init (nodeid_t node, +paddr_t start,paddr_t size) +{ +struct node *nd; +paddr_t end; +int i; +Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+if (srat_disabled () ) +return -EINVAL ; + + end = start + size; +if ( num_node_memblks >=NR_NODE_MEMBLKS ) + { + dprintk(XENLOG_WARNING, + "Too many numa entry, try bigger NR_NODE_MEMBLKS\n");
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+bad_srat (); +return -EINVAL ; + } + +/* It is fine to add this area to the nodes data it will be used
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
later */
Quoted 2 lines
(quoted diff) +2
-0+ i = conflicting_memblks(start, end); + /* No conflicting memory block, we can save it for later usage*/;
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+if ( i <0 ) + goto save_memblk; + +if ( memblk_nodeid[i] == node ) { +/* +* Overlaps with other memblk in the same node, warning here.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+* This memblk will be merged with conflicted memblk later. +*/ +printk (XENLOG_WARNING +"DT: NUMA NODE %u (%" PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
Quoted 2 lines
(quoted diff) +2
-0+ node, start, end, + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 11 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node)
Quoted 8 lines
xen/arch/arm/numa_device_tree.c +8
-0+ } else { + /* + * Conflict with memblk in other node, this is an error. + * The NUMA information is invalid, NUMA will be turn off. + */ + printk(XENLOG_ERR + "DT: NUMA NODE %u (%"PRIx64"-%" + PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +2
-0+ node, start, end, memblk_nodeid[i], + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
xen/arch/arm/numa_device_tree.c +18
-0+bad_srat (); +return -EINVAL ; + } + +save_memblk: + nd = &nodes[node]; +if ( !node_test_and_set (node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + }else { +if ( start < nd->start ) + nd->start = start; +if ( nd->end < end ) + nd->end = end; + } + +printk (XENLOG_INFO "DT: NUMA node %u %" PRIx64"-%" PRIx64"\n" , + node, start, end);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/numa_device_tree.c static int __init dtb_numa_processor_affinity_init(nodeid_t node) · line 128
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a single
NUMA
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
node?
Looking at the DT bindings and Linux implementation, it seems
possible.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Here, it seems that node_memblk_range/memblk_nodeid could handle it, but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI
allow
Quoted 8 lines
Quoted 1 line
non-contiguous ranges of memory for a single NUMA node too?
I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.
Quoted 3 lines
If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan to mergedtb_numa_memory_affinity_initandacpi_numa_memory_affinity_initintoa
Quoted 8 lines
Quoted 3 lines
neutral function. So we can fix them at the same time.
If not, maybe we have to keep the diversity for dtb and ACPI here.
I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?
Quoted 9 lines
Quoted 6 lines
If so, then I disagree with that. It is fine to have code that supports more than what a firmware table supports. The main benefit is less code and therefore less long term maintenance (with the current solution we would need to check both the ACPI and DT implementation if there is a bug in one).
Yes, I agree.
I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.
This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.
Quoted 6 lines
To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?
Or could you give me suggestions that we can use some simple ways to fix it?
It looks like that we would have to replace all the node->start / node->end checks with node_memblk_range checks. There are a few of them in valid_numa_range, conflicting_memblks, cutoff_node, nodes_cover_memory. It wouldn't be trivial.
Although I do think that non-contiguous memory for NUMA nodes is important to support, the patch series is already 40 patches. I don't think it is a good idea to add other significant changes to it. I wouldn't upgrade the x86 NUMA driver now. If we can't find a better way, we can proceed as you are doing in this version, with the known gap that we can't deal with non-contigious memory for NUMA nodes, and fix it with a follow-up series later. In that case we would want to have an explicit check for non-contiguous memory for NUMA nodes in dtb_numa_memory_affinity_init and error out if found.
Quoted 6 lines
Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]
[1] https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c
[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3] https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c
In general, I like the idea of sharing code as much as possible between architectures (x86, ARM, etc.) and between DT/ACPI because it makes the code maintainance easier and one might even gain certain features for free.
However, the excercise of sharing code shouldn't take significant additional efforts. In fact, it should decrease the overall effort:
instead of writing new code one just take existing code and move it to common.
In this instance, I think it would be good to be able to share the NUMA initialization code between x86/ARM and ACPI/DT if it doesn't cause extra efforts.
Here the extra effort that my previous comment might cause doesn't derive from x86/ARM or DT/ACPI code sharing. It derives from the fact that our existing code doesn't deal with non-contigous memory for NUMA nodes unfortunately. That is something we need to find a way to cope with anyway.
RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node
Hi Stefano,
Quoted 168 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月9日 6:32
To: Wei Chen <[email protected]>
Cc: Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]; Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: RE: [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory nodeQuoted 5 lines · Wei Chen
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Review static int __init
Quoted 1 line
(quoted diff) +0
-0@@ -55,6 +57,79 @@ static int __init
dtb_numa_processor_affinity_init(nodeid_t node)Quoted 3 lines
(quoted diff) +0
-0return 0; }Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8
-0+/* Callback for parsing of the memory regions affinity */ +static int __init dtb_numa_memory_affinity_init(nodeid_t node, + paddr_t start, paddr_t size) +{ + struct node *nd; + paddr_t end; + int i; +Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review static int __init · line 75
Quoted 8 lines
(quoted diff) +8
-0+ if ( srat_disabled() ) + return -EINVAL; + + end = start + size; + if ( num_node_memblks >= NR_NODE_MEMBLKS ) + { + dprintk(XENLOG_WARNING, + "Too many numa entry, try bigger
NR_NODE_MEMBLKSQuoted 10 lines
Quoted 10 lines
\n");
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Review static int __init · line 80
Quoted 5 lines
(quoted diff) +5
-0+ bad_srat(); + return -EINVAL; + } + + /* It is fine to add this area to the nodes data it will beused
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
later */
Quoted 2 lines
(quoted diff) +2
-0+ i = conflicting_memblks(start, end); + /* No conflicting memory block, we can save it for laterusage
Quoted 9 lines
Quoted 9 lines
*/;
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Review static int __init · line 88
Quoted 6 lines
(quoted diff) +6
-0+ if ( i < 0 ) + goto save_memblk; + + if ( memblk_nodeid[i] == node ) { + /* + * Overlaps with other memblk in the same node, warninghere.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +1
-0+ * This memblk will be merged with conflicted memblklater.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+ */ + printk(XENLOG_WARNING + "DT: NUMA NODE %u (%"PRIx64 + "-%"PRIx64") overlaps with itself (%"PRIx64"-Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
%"PRIx64")\n",
Quoted 2 lines
(quoted diff) +2
-0+ node, start, end, + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+ } else { + /* + * Conflict with memblk in other node, this is an error. + * The NUMA information is invalid, NUMA will be turnoff.
Quoted 26 lines
Quoted 26 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Review static int __init · line 103
Quoted 4 lines
(quoted diff) +4
-0+ */ + printk(XENLOG_ERR + "DT: NUMA NODE %u (%"PRIx64"-%" + PRIx64") overlaps with NODE %u (%"PRIx64"-%"PRIx64")\n",
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +2
-0+ node, start, end, memblk_nodeid[i], + node_memblk_range[i].start,node_memblk_range[i].end);
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
(quoted diff) +18
-0+ bad_srat(); + return -EINVAL; + } + +save_memblk: + nd = &nodes[node]; + if ( !node_test_and_set(node, memory_nodes_parsed) ) { + nd->start = start; + nd->end = end; + } else { + if ( start < nd->start ) + nd->start = start; + if ( nd->end < end ) + nd->end = end; + } + + printk(XENLOG_INFO "DT: NUMA node %u %"PRIx64"-%"PRIx64"\n", + node, start, end);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review static int __init · line 128
Quoted 5 lines
(quoted diff) +5
-0+ + node_memblk_range[num_node_memblks].start = start; + node_memblk_range[num_node_memblks].end = end; + memblk_nodeid[num_node_memblks] = node; + num_node_memblks++;Is it possible to have non-contigous ranges of memory for a
single
Quoted 6 lines
Quoted 6 lines
NUMA
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
node?
Looking at the DT bindings and Linux implementation, it seems
possible.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Here, it seems that node_memblk_range/memblk_nodeid could handle
it,
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
but nodes couldn't.
Yes, you're right. I copied this code for x86 ACPI NUMA. Does ACPI
allow
Quoted 7 lines
Quoted 1 line
non-contiguous ranges of memory for a single NUMA node too?
I couldn't find any restriction for ACPI. Although, I only briefly looked at the spec.
Quoted 2 lines
If yes, I think
this will affect x86 ACPI NUMA too. In next version, we plan tomerge
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
dtb_numa_memory_affinity_initandacpi_numa_memory_affinity_initinto
Quoted 10 lines
Quoted 10 lines
a
Quoted 9 lines
Quoted 3 lines
neutral function. So we can fix them at the same time.
If not, maybe we have to keep the diversity for dtb and ACPI here.
I am not entirely sure what you mean. Are you saying if ACPI doesn't allow non-contiguous ranges of memory, then we should keep the implementation separated?
If so, then I disagree with that. It is fine to have code that
supports
Quoted 1 line
Quoted 1 line
Quoted 1 line
more than what a firmware table supports. The main benefit is less
code
Quoted 1 line
Quoted 1 line
Quoted 1 line
and therefore less long term maintenance (with the current solution
we
Quoted 1 line
Quoted 1 line
Quoted 1 line
would need to check both the ACPI and DT implementation if there is
a
Quoted 16 lines
Quoted 5 lines
Quoted 2 lines
bug in one).
Yes, I agree.
I am looking for some methods to address this comment. Current "nodes" has not considered the situation of memory addresses of different NUMA nodes can be interleaved.
This code exists in x86 NUMA implementation. I think it may be based on one early version of Linux x86 NUMA implementation. In recent Linux code, both ACPI/numa/srat.c[1] and x86 NUMA code[2] are not using "nodes" to record NUMA memory address boundary. They don't depend on "nodes" to do sanity check.
Quoted 6 lines
To fix it, we'd better to upgrade the x86 NUMA driver. It will make a great affect for Xen-x86. And I think it might out of this series scope. Can we create another thread to discuss about it?
Or could you give me suggestions that we can use some simple ways to fix it?
It looks like that we would have to replace all the
node->start/node->endchecks withnode_memblk_rangechecks. There are a few of them invalid_numa_range,conflicting_memblks,cutoff_node,nodes_cover_memory. It wouldn't be trivial.
Quoted 10 lines
Although I do think that non-contiguous memory for NUMA nodes is important to support, the patch series is already 40 patches. I don't think it is a good idea to add other significant changes to it. I wouldn't upgrade the x86 NUMA driver now. If we can't find a better way, we can proceed as you are doing in this version, with the known gap that we can't deal with non-contigious memory for NUMA nodes, and fix it with a follow-up series later. In that case we would want to have an explicit check for non-contiguous memory for NUMA nodes in
dtb_numa_memory_affinity_initand error out if found.
Yes, I think this may be a more appropriate method at present. I would add some code to do explicit check and give warning/error.
Quoted 25 lines
Quoted 5 lines
Also, on Linux, NUMA implementations for x86 are different from Arm64 and RISC-V implementations.[3]
[1]
https://github.com/torvalds/linux/blob/master/drivers/acpi/numa/srat.c
Quoted 2 lines
[2] https://github.com/torvalds/linux/blob/master/arch/x86/mm/numa.c
[3]https://github.com/torvalds/linux/blob/master/drivers/base/arch_numa.c
In general, I like the idea of sharing code as much as possible between architectures (x86, ARM, etc.) and between DT/ACPI because it makes the code maintainance easier and one might even gain certain features for free.
However, the excercise of sharing code shouldn't take significant additional efforts. In fact, it should decrease the overall effort:
instead of writing new code one just take existing code and move it to common.In this instance, I think it would be good to be able to share the NUMA initialization code between x86/ARM and ACPI/DT if it doesn't cause extra efforts.
Quoted 5 lines
Here the extra effort that my previous comment might cause doesn't derive from x86/ARM or DT/ACPI code sharing. It derives from the fact that our existing code doesn't deal with non-contigous memory for NUMA nodes unfortunately. That is something we need to find a way to cope with anyway.
Yes. I posted above links didn't mean to create diversity for Arm/x86. Though, I'm not sure exactly why Linux does this. But I think for Xen, we still can try to share code for Arm/x86 and DT/ACPI.
Cheers,
Wei Chen
RFC PATCH 24/40 xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Wei ChenA NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
xen/arch/arm/numa_device_tree.c +67 -0
@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i; + +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++; + to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + +if ( (from == to && distance !=NUMA_LOCAL_DISTANCE ) || + (from != to && distance <=NUMA_LOCAL_DISTANCE ) ) + { +printk (XENLOG_WARNING +"Invalid nodes' distance from node#%d to node#%d = %d\n" , + from, to, distance); +return -EINVAL ; + } + +printk (XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n" , + from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}
Signature · 1 line
2.25.1
Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Wei,
Quoted 2 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a
s/introduce/introduces/
Quoted 13 lines
new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +8
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;
entry_count and i should be unsigned. len unfortunately can't because fdt_get_property expects a signed int.
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +11
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + }
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 )
I understand that entry_count may be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244
Quoted 13 lines
xen/arch/arm/numa_device_tree.c +13
-0+ { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;
You can use dt_next_cell() which will update the pointer for you.
Quoted 16 lines
xen/arch/arm/numa_device_tree.c +15
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + +if ( (from == to && distance !=NUMA_LOCAL_DISTANCE ) || + (from != to && distance <=NUMA_LOCAL_DISTANCE ) ) + { +printk (XENLOG_WARNING +"Invalid nodes' distance from node#%d to node#%d = %d\n" , + from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}
Cheers,
Signature · 1 line
Julien Grall
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Julien,
Quoted 10 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2021年8月25日 21:56
To: Wei Chen <[email protected]>; [email protected]; [email protected]
Cc: Bertrand Marquis <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Quoted 7 lines
Hi Wei,
Quoted 2 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a
s/introduce/introduces/
OK
Quoted 16 lines
Review
Quoted 9 lines
new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,int node,
Quoted 1 line
Quoted 14 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 210
Quoted 10 lines
return 0;}
xen/arch/arm/numa_device_tree.c +8
-0+ +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i;
entry_countand i should be unsigned. len unfortunately can't becausefdt_get_propertyexpects a signed int.
OK
Quoted 17 lines
Quoted 17 lines
xen/arch/arm/numa_device_tree.c +17
-0+ +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ;
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 231
Quoted 4 lines
xen/arch/arm/numa_device_tree.c +4
-0+ } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 )I understand that
entry_countmay be 0. But I can't see how it can be negative as the property len cannot be (even if it is a signed type). So I think this wants to be "== 0".
From the fdt_get_property's comment, when prop is NULL, the len can be negative. But, yes, prop==NULL check will handled before this code. negative len will not reach here. I am ok to change it to "== 0"
Quoted 7 lines
Quoted 7 lines
xen/arch/arm/numa_device_tree.c +7
-0+ { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data;
Quoted 9 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 244
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;You can use
dt_next_cell()which will update the pointer for you.
Thanks
Quoted 33 lines
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 269
Quoted 11 lines
xen/arch/arm/numa_device_tree.c +10
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance); + } + +return 0 ; +}Cheers,
--
Julien Grall
Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
In reply to Wei ChenQuoted 70 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)xen/arch/arm/numa_device_tree.c +53
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt, int node,return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i; + +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++; + to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + +if ( (from == to && distance !=NUMA_LOCAL_DISTANCE ) || + (from != to && distance <=NUMA_LOCAL_DISTANCE ) ) + { +printk (XENLOG_WARNING +"Invalid nodes' distance from node#%d to node#%d = %d\n" , + from, to, distance);
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+return -EINVAL ; + } + +printk (XENLOG_INFO "NUMA: distance from node#%d to node#%d = %d\n" , + from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);
I am a bit unsure about this last 2 lines: why calling numa_set_distance in the opposite direction only when to > from? Wouldn't it be OK to always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);
?
But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).
So my suggestion is to parse it as follows:
- call numa_set_distance just once from
device_tree_parse_numa_distance_map_v1
- in numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;
RE: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map
Hi Stefano,
Quoted 95 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年8月31日 8:48
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance mapReview
Quoted 10 lines · Wei Chen
A NUMA aware device tree will provide a "distance-map" node to describe distance between any two nodes. This patch introduce a new helper to parse this distance map.
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/numa_device_tree.c | 67 +++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)(quoted diff) +0
-0
b/xen/arch/arm/numa_device_tree.cxen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt,
Quoted 4 lines
index
bbe081dcd1..6e0d1d3d9f100644xen/arch/arm/numa_device_tree.c +0
-0@@ -200,3 +200,70 @@ device_tree_parse_numa_memory_node(const void *fdt,int node,
Quoted 45 lines
xen/arch/arm/numa_device_tree.c +42
-0return 0 ; } + +/* Parse NUMA distance map v1 */ +int __init +device_tree_parse_numa_distance_map_v1 (const void *fdt,int node) +{ +const struct fdt_property *prop; +const __be32 *matrix; +int entry_count, len, i; + +printk (XENLOG_INFO "NUMA: parsing numa-distance-map\n" ); + + prop =fdt_get_property (fdt, node,"distance-matrix" , &len); +if ( !prop ) + { +printk (XENLOG_WARNING +"NUMA: No distance-matrix property in distance-map\n" ); + +return -EINVAL ; + } + +if ( len %sizeof (uint32_t ) !=0 ) + { +printk (XENLOG_WARNING +"distance-matrix in node is not a multiple of u32\n" ); +return -EINVAL ; + } + + entry_count = len /sizeof (uint32_t ); +if ( entry_count <=0 ) + { +printk (XENLOG_WARNING "NUMA: Invalid distance-matrix\n" ); + +return -EINVAL ; + } + + matrix = (const __be32 *)prop->data; +for ( i =0 ; i +2 < entry_count; i +=3 ) + { +uint32_t from, to, distance; + + from =dt_read_number (matrix,1 ); + matrix++;xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 10 lines
xen/arch/arm/numa_device_tree.c +10
-0+ to =dt_read_number (matrix,1 ); + matrix++; + distance =dt_read_number (matrix,1 ); + matrix++; + + if ( (from == to && distance != NUMA_LOCAL_DISTANCE) || + (from != to && distance <= NUMA_LOCAL_DISTANCE) ) + { + printk(XENLOG_WARNING + "Invalid nodes' distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node,
Quoted 5 lines
xen/arch/arm/numa_device_tree.c +5
-0+ from, to, distance); +return -EINVAL ; + } + + printk(XENLOG_INFO "NUMA: distance from node#%d to node#%d= %d\n",
xen/arch/arm/numa_device_tree.c device_tree_parse_numa_memory_node(const void *fdt, int node, · line 265
Quoted 6 lines
xen/arch/arm/numa_device_tree.c +6
-0+ from, to, distance); +numa_set_distance (from, to, distance); + +/* Set default distance of node B->A same as A->B */ +if (to > from) +numa_set_distance (to, from, distance);
Quoted 9 lines
I am a bit unsure about this last 2 lines: why calling
numa_set_distancein the opposite direction only when to > from? Wouldn't it be OK to always do both:
numa_set_distance(from, to, distance);numa_set_distance(to, from, distance);?
I borrowed this code from Linux, but here is my understanding:
First, I read some notes in Documentation/devicetree/bindings/numa.txt
1. Each entry represents distance from first node to second node. The distances are equal in either direction.
2. distance-matrix should have entries in lexicographical ascending order of nodes.
Here is an example of distance-map node in DTB:
Sample#1, full list:
distance-map {
compatible = "numa-distance-map-v1";
distance-matrix = <0 0 10>,
<0 1 20>,
<0 2 40>,
<0 3 20>,
<1 0 20>,
<1 1 10>,
<1 2 20>,
<1 3 40>,
<2 0 40>,
<2 1 20>,
<2 2 10>,
<2 3 20>,
<3 0 20>,
<3 1 40>,
<3 2 20>,
<3 3 10>;
};Call numa_set_distance when "to > from" will prevent Xen to call numa_set_distance(0, 1, 20) again when it's setting distance for <1 0 20>. But, numa_set_distance(1, 0, 20) will be call twice.
Normally, distance-map node will be optimized in following sample#2, all redundant entries are removed:
Sample#2, partial list:
distance-map {
compatible = "numa-distance-map-v1";
distance-matrix = <0 0 10>,
<0 1 20>,
<0 2 40>,
<0 3 20>,
<1 1 10>,
<1 2 20>,
<1 3 40>,
<2 2 10>,
<2 3 20>,
<3 3 10>;
};There is not any "from > to" entry in the map. But using this partial map still can set all distances for all pairs. And numa_set_distance(1, 0, 20) will be only once.
Quoted 5 lines
But in any case, I have a different suggestion. The binding states that "distances are equal in either direction". Also it has an example where only one direction is expressed unfortunately (at the end of the document).
Oh, I should see this comment first, then I will not post above comment : )
Quoted 11 lines
So my suggestion is to parse it as follows:
- call
numa_set_distancejust once from
device_tree_parse_numa_distance_map_v1- in
numa_set_distance:
- set node_distance_map[from][to] = distance;
- check node_distance_map[to][from]
- if unset, node_distance_map[to][from] = distance;
- if already set to the same value, return success;
- if already set to a different value, return error;
I don't really like this implementation. I want the behavior of numa_set_distance just like the function name, do not include implicit operations. Otherwise, except the user read this function implementation before he use it, he probably doesn't know this function has done so many things.
Cheers,
Wei Chen