Add device tree based NUMA support to Arm
cover Add device tree based NUMA support to Arm
Xen memory allocation and scheduler modules are NUMA aware. But actually, on x86 has implemented the architecture APIs to support NUMA. Arm was providing a set of fake architecture APIs to make it compatible with NUMA awared memory allocation and scheduler.
Arm system was working well as a single node NUMA system with these fake APIs, because we didn't have multiple nodes NUMA system on Arm. But in recent years, more and more Arm devices support multiple nodes NUMA system.
So now we have a new problem. When Xen is running on these Arm devices, Xen still treat them as single node SMP systems. The NUMA affinity capability of Xen memory allocation and scheduler becomes meaningless. Because they rely on input data that does not reflect real NUMA layout.
Xen still think the access time for all of the memory is the same for all CPUs. However, Xen may allocate memory to a VM from different NUMA nodes with different access speeds. This difference can be amplified in workloads inside VM, causing performance instability and timeouts.
So in this patch series, we implement a set of NUMA API to use device tree to describe the NUMA layout. We reuse most of the code of x86 NUMA to create and maintain the mapping between memory and CPU, create the matrix between any two NUMA nodes. Except ACPI and some x86 specified code, we have moved other code to common. In next stage, when we implement ACPI based NUMA for Arm64, we may move the ACPI NUMA code to common too, but in current stage, we keep it as x86 only.
This patch serires has been tested and booted well on one Arm64 NUMA machine and one HPE x86 NUMA machine.
---
@Julien, about numa=noacpi option, I haven't remove it in this patch series. I had tried, and this option is not easy to remove. So I'm not going to think about doing that in this series
rfc -> v1:
1. Re-order the whole patch series to avoid temporary code
2. Add detection of discontinous node memory range
3. Fix typos in commit messages and code comments.
4. For variables that are used in common code, we don't covert them to external any more. Instead, we export helpers to provide variable access.
5. Revert memnodemap[0] to 0 when NUMA init failed. Change memnodemapsize from ARRAY_SIZE(_memnodemap) to 1 to reflect reality.
6. Use arch_has_default_dmazone for page_alloc.c instead of changing code inside Arm
7. Keep Kconfig options alphabetically sorted.
8. Replace #if !defined by #ifndef
9. Use paddr_t for addresses in NUMA node structures and function parameters
10. Use fw_numa to replace acpi_numa for neutrality
11. Change BIOS to Firmware in print message.
12. Promote VIRTUAL_BUG_ON to ASSERT
13. Introduce CONFIG_EFI to stub API for non-EFI architecture
14. Use EFI stub API to replace arch helper for efi_enabled
15. Use NR_MEM_BANKS for Arm's NR_NODE_MEMBLKS
16. Change matrix map default value from NUMA_REMOTE_DISTANCE to 0
17. Remove check in numa_set_node.
18. Follow the x86's method of adding CPU to NUMA
19. Use fdt prefix for all device tree NUMA parser's API
20. Check un-matched bi-direction distance in matrix map
21. Remove unless fdt type check function
22. Update doc to remove numa x86 spcific
23. Introduce Arm generic NUMA Kconfig option
Wei Chen (37):
xen/arm: Print a 64-bit number in hex from early uart
xen: introduce a Kconfig option to configure NUMA nodes number xen/x86: Initialize memnodemapsize while faking NUMA node
xen: introduce an arch helper for default dma zone status
xen: decouple NUMA from ACPI in Kconfig
xen/arm: use !CONFIG_NUMA to keep fake NUMA API xen/x86: use paddr_t for addresses in NUMA node structure xen/x86: add detection of discontinous node memory range xen/x86: introduce two helpers to access memory hotplug end xen/x86: use helpers to access/update mem_hotplug xen/x86: abstract neutral code from acpi_numa_memory_affinity_init xen/x86: decouple nodes_cover_memory from E820 map xen/x86: decouple processor_nodes_parsed from acpi numa functions xen/x86: use name fw_numa to replace acpi_numa
xen/x86: rename acpi_scan_nodes to numa_scan_nodes xen/x86: export srat_bad to external
xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
xen: move NUMA common code from x86 to common
xen/x86: promote VIRTUAL_BUG_ON to ASSERT in
xen: introduce CONFIG_EFI to stub API for non-EFI architecture xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS xen/arm: implement node distance helpers for Arm xen/arm: implement two arch helpers to get memory map info xen/arm: implement bad_srat for Arm NUMA initialization xen/arm: build NUMA cpu_to_node map in dt_smp_init_cpus xen/arm: Add boot and secondary CPU to NUMA system xen/arm: stub memory hotplug access helpers for Arm xen/arm: introduce a helper to parse device tree processor node xen/arm: introduce a helper to parse device tree memory node xen/arm: introduce a helper to parse device tree NUMA distance map xen/arm: unified entry to parse all NUMA data from device tree xen/arm: keep guest still be NUMA unware
xen/arm: enable device tree based NUMA in system init xen/arm: use CONFIG_NUMA to gate node_online_map in smpboot xen/arm: Provide Kconfig options for Arm to enable NUMA
docs: update numa command line to support Arm
docs/misc/xen-command-line.pandoc | 2 +- xen/arch/Kconfig | 11 + xen/arch/arm/Kconfig | 12 + xen/arch/arm/Makefile | 4 +- xen/arch/arm/arm64/head.S | 9 +- xen/arch/arm/bootfdt.c | 8 +- xen/arch/arm/domain_build.c | 6 + xen/arch/arm/efi/efi-boot.h | 25 -- xen/arch/arm/numa.c | 155 ++++++++++ xen/arch/arm/numa_device_tree.c | 274 ++++++++++++++++++ xen/arch/arm/setup.c | 12 + xen/arch/arm/smpboot.c | 39 ++- xen/arch/x86/Kconfig | 3 +- xen/arch/x86/numa.c | 449 ++--------------------------- xen/arch/x86/setup.c | 2 +- xen/arch/x86/srat.c | 232 ++------------- xen/common/Kconfig | 14 + xen/common/Makefile | 2 + xen/common/numa.c | 450 ++++++++++++++++++++++++++++++ xen/common/numa_srat.c | 264 ++++++++++++++++++ xen/common/page_alloc.c | 2 +- xen/drivers/acpi/Kconfig | 3 +- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-arm/mm.h | 10 + xen/include/asm-arm/numa.h | 50 ++++ xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/config.h | 1 - xen/include/asm-x86/mm.h | 10 + xen/include/asm-x86/numa.h | 65 +---- xen/include/asm-x86/setup.h | 1 - xen/include/xen/efi.h | 11 + xen/include/xen/numa.h | 94 ++++++- 32 files changed, 1470 insertions(+), 756 deletions(-) create mode 100644 xen/arch/arm/numa.c create mode 100644 xen/arch/arm/numa_device_tree.c create mode 100644 xen/common/numa.c create mode 100644 xen/common/numa_srat.c
Signature · 1 line
2.25.1
PATCH 1/37 xen/arm: Print a 64-bit number in hex from early uart
Current putn function that is using for early print only can print low 32-bit of AArch64 register. This will lose some important messages while debugging with early console. For example:
(XEN) Bringing up CPU5
- CPU 0000000100000100 booting -
Will be truncated to
(XEN) Bringing up CPU5
- CPU 00000100 booting -
In this patch, we increased the print loops and shift bits to make putn print 64-bit number.
--- xen/arch/arm/arm64/head.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
xen/arch/arm/arm64/head.S +5 -4
@@ -862,17 +862,18 @@ puts: ret ENDPROC(puts) -/* Print a 32-bit number in hex. Specific to the PL011 UART. +/* Print a 64-bit number in hex. * x0: Number to print. * x23: Early UART base address * Clobbers x0-x3 */ +#define PRINT_MASK 0xf000000000000000 putn: adr x1, hex - mov x3, #8 + mov x3, #16 1: early_uart_ready x23, 2 - and x2, x0, #0xf0000000 /* Mask off the top nybble */ - lsr x2, x2, #28 + and x2, x0, #PRINT_MASK /* Mask off the top nybble */ + lsr x2, x2, #60 ldrb w2, [x1, x2] /* Convert to a char */ early_uart_transmit x23, w2 lsl x0, x0, #4 /* Roll it through one nybble at a time */
Signature · 1 line
2.25.1
PATCH 2/37 xen: introduce a Kconfig option to configure NUMA nodes number
In reply to Wei ChenCurrent NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.
So in this patch, we introduce this new Kconfig option for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, this Kconfig option only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 as MAX_NUMNODES.
--- xen/arch/Kconfig | 11 +++++++++++ xen/include/asm-x86/numa.h | 2 -- xen/include/xen/numa.h | 10 +++++----- 3 files changed, 16 insertions(+), 7 deletions(-)
xen/arch/Kconfig +11 -0
@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading or similar technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095 + default "64" + depends on NUMA + help + Controls the build-time size of various arrays and bitmaps + associated with multiple-nodes management. It is the upper bound of + the number of NUMA nodes the scheduler, memory allocation and other + NUMA-aware components can handle.
xen/include/asm-x86/numa.h +0 -2
@@ -3,8 +3,6 @@#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;extern int srat_rev;
xen/include/xen/numa.h +5 -5
@@ -3,14 +3,14 @@#include <asm/numa.h> -#ifndef NODES_SHIFT -#define NODES_SHIFT 0 -#endif -#define NUMA_NO_NODE 0xFF#define NUMA_NO_DISTANCE 0xFF -#define MAX_NUMNODES (1 << NODES_SHIFT) +#ifdef CONFIG_NR_NUMA_NODES +#define MAX_NUMNODES CONFIG_NR_NUMA_NODES +#else +#define MAX_NUMNODES 1 +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor))
Signature · 1 line
2.25.1
Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number
Quoted 10 lines · Wei Chen
Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.
So in this patch, we introduce this new
Kconfigoption for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, thisKconfigoption only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 asMAX_NUMNODES.
This is OK but I think you should also mention in the commit message that you are taking the opportunity to remove NODES_SHIFT because it is currently unused.
With that:
Quoted 66 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/Kconfig | 11 +++++++++++ xen/include/asm-x86/numa.h | 2 -- xen/include/xen/numa.h | 10 +++++----- 3 files changed, 16 insertions(+), 7 deletions(-)xen/arch/Kconfig +11
-0@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading or similar technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095 + default "64" + depends on NUMA + help + Controls the build-time size of various arrays and bitmaps + associated with multiple-nodes management. It is the upper bound of + the number of NUMA nodes the scheduler, memory allocation and other + NUMA-aware components can handle.xen/include/asm-x86/numa.h +0
-2@@ -3,8 +3,6 @@#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;extern int srat_rev;xen/include/xen/numa.h +5
-5@@ -3,14 +3,14 @@#include <asm/numa.h> -#ifndef NODES_SHIFT -#define NODES_SHIFT 0 -#endif -#define NUMA_NO_NODE 0xFF#define NUMA_NO_DISTANCE 0xFF -#define MAX_NUMNODES (1 << NODES_SHIFT) +#ifdef CONFIG_NR_NUMA_NODES +#define MAX_NUMNODES CONFIG_NR_NUMA_NODES +#else +#define MAX_NUMNODES 1 +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor))Signature · 1 line
2.25.1
RE: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number
Hi Stefano,
Quoted 21 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 7:45
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 02/37] xen: introduce aKconfigoption to configure NUMA nodes numberQuoted 10 lines · Wei Chen
Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.
So in this patch, we introduce this new
Kconfigoption for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, thisKconfigoption only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 asMAX_NUMNODES.
Quoted 9 lines
This is OK but I think you should also mention in the commit message that you are taking the opportunity to remove
NODES_SHIFTbecause it is currently unused.With that:
Reviewed-by Stefano Stabellini <[email protected]>
Thanks, I will update it in next version.
Quoted 69 lines
xen/arch/Kconfig config NR_CPUS · line 17
Quoted 13 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/Kconfig | 11 +++++++++++ xen/include/asm-x86/numa.h | 2 -- xen/include/xen/numa.h | 10 +++++----- 3 files changed, 16 insertions(+), 7 deletions(-)xen/arch/Kconfig +0
-0@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading orsimilar
xen/arch/Kconfig config NR_CPUS
Quoted 11 lines
xen/arch/Kconfig +9
-0technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095 + default "64" + depends on NUMA + help + Controls the build-time size of various arrays and bitmaps + associated with multiple-nodes management. It is the upper boundof
xen/arch/Kconfig config NR_CPUS · line 29
Quoted 1 line
xen/arch/Kconfig +1
-0+ the number of NUMA nodes the scheduler, memory allocation andother
Quoted 41 lines
xen/arch/Kconfig +1
-0+ NUMA-aware components can handle.xen/include/asm-x86/numa.h +0
-2@@ -3,8 +3,6 @@#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;extern int srat_rev;xen/include/xen/numa.h +5
-6@@ -3,14 +3,14 @@#include <asm/numa.h> -#ifndef NODES_SHIFT -#define NODES_SHIFT 0 -#endif -#define NUMA_NO_NODE 0xFF#define NUMA_NO_DISTANCE 0xFF -#define MAX_NUMNODES (1 << NODES_SHIFT) +#ifdef CONFIG_NR_NUMA_NODES +#define MAX_NUMNODES CONFIG_NR_NUMA_NODES +#else +#define MAX_NUMNODES 1 +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor)) --2.25.1
Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number
In reply to Wei ChenQuoted 10 lines · Wei Chen
Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.
So in this patch, we introduce this new
Kconfigoption for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, thisKconfigoption only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 asMAX_NUMNODES.
Do you really mean administrators here? To me command line options are for administrators, but build decisions are usually taken by build managers of distros.
xen/arch/Kconfig config NR_CPUS · line 23
Quoted 10 lines
xen/arch/Kconfig +4
-0@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading or similar technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095
How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...
xen/include/asm-x86/numa.h line 6
Quoted 9 lines
xen/include/asm-x86/numa.h +0
-2@@ -3,8 +3,6 @@#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;
... this.
Jan
RE: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number
Hi Jan,
Quoted 46 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:56
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 02/37] xen: introduce aKconfigoption to configure NUMA nodes numberQuoted 10 lines · Wei Chen
Current NUMA nodes number is a hardcode configuration. This configuration is difficult for an administrator to change unless changing the code.
So in this patch, we introduce this new
Kconfigoption for administrators to change NUMA nodes number conveniently. Also considering that not all architectures support NUMA, thisKconfigoption only can be visible on NUMA enabled architectures. Non-NUMA supported architectures can still use 1 asMAX_NUMNODES.Do you really mean administrators here? To me command line options are for administrators, but build decisions are usually taken by build managers of distros.
xen/arch/Kconfig config NR_CPUS · line 17
Quoted 4 lines
xen/arch/Kconfig +0
-0@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading orsimilar
xen/arch/Kconfig config NR_CPUS · line 23
Quoted 6 lines
xen/arch/Kconfig +4
-0technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...
Quoted 4 lines
xen/include/asm-x86/numa.h +0
-0@@ -3,8 +3,6 @@
Quoted 8 lines
xen/include/asm-x86/numa.h line 6
Quoted 5 lines
xen/include/asm-x86/numa.h +0
-2#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;... this.
you're right, we use u8 as nodeid_t. 4095 for node number in this option is not reasonable. Maybe a 255 upper bound is good?
Quoted 1 line
Jan
Re: [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number
Quoted 34 lines · Wei Chen
Quoted 25 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:56xen/arch/Kconfig config NR_CPUS · line 17
Quoted 4 lines · Wei Chen
xen/arch/Kconfig +0
-0@@ -17,3 +17,14 @@ config NR_CPUS For CPU cores which support Simultaneous Multi-Threading orsimilar
xen/arch/Kconfig config NR_CPUS · line 23
Quoted 6 lines
xen/arch/Kconfig +4
-0technologies, this the number of logical threads which Xen will support. + +config NR_NUMA_NODES + int "Maximum number of NUMA nodes supported" + range 1 4095How was this upper bound established? Seeing 4095 is the limit of the number of CPUs, do we really expect a CPU per node on such huge systems? And did you check that whichever involved data types and structures are actually suitable? I'm thinking e.g. of things like ...
Quoted 4 lines
xen/include/asm-x86/numa.h +0
-0@@ -3,8 +3,6 @@Quoted 8 lines
xen/include/asm-x86/numa.h line 6
Quoted 5 lines
xen/include/asm-x86/numa.h +0
-2#include <xen/cpumask.h> -#define NODES_SHIFT 6 -typedef u8 nodeid_t ;... this.
Quoted 2 lines
you're right, we use u8 as
nodeid_t. 4095 for node number in this option is not reasonable. Maybe a 255 upper bound is good?
I think it is, yes, but you will want to properly check.
Jan
PATCH 3/37 xen/x86: Initialize memnodemapsize while faking NUMA node
In reply to Wei ChenWhen system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages, it will use the _memnodemap directly. But memnodemapsize hasn't been set. Xen should assert in phys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.
Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.
--- xen/arch/x86/numa.c | 4 ++++ 1 file changed, 4 insertions(+)
xen/arch/x86/numa.c +4 -0
@@ -270,6 +270,10 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap; +/* Dummy node only uses 1 slot in reality */ + memnodemap[0 ] =0 ; + memnodemapsize =1 ; +nodes_clear (node_online_map);node_set_online (0 );for ( i =0 ; i < nr_cpu_ids; i++ )
Signature · 1 line
2.25.1
Re: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node
Quoted 9 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
In this case the memory node map doesn't need to be allocated from boot pages, it will use the
_memnodemapdirectly. But memnodemapsize hasn't been set. Xen should assert inphys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.
Somehow and A got lost here, which I'll add back while committing.
Quoted 5 lines
Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.
Signed-off-by Wei Chen <[email protected]>
RE: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node
Quoted 16 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 16:57
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA nodeQuoted 4 lines · Wei Chen
When system turns NUMA off or system lacks of NUMA support, Xen will fake a NUMA node to make system works as a single node NUMA system.
Quoted 8 lines
Quoted 5 lines
In this case the memory node map doesn't need to be allocated from boot pages, it will use the
_memnodemapdirectly. But memnodemapsize hasn't been set. Xen should assert inphys_to_nid. Because x86 was using an empty macro "VIRTUAL_BUG_ON" to replace SSERT, this bug will not be triggered on x86.Somehow and A got lost here, which I'll add back while committing.
Thanks!
Quoted 7 lines
Quoted 5 lines
Actually, Xen will only use 1 slot of memnodemap in this case. So we set memnodemap[0] to 0 and memnodemapsize to 1 in this patch to fix it.
Signed-off-by Wei Chen <[email protected]>Acked-by Jan Beulich <[email protected]>
PATCH 4/37 xen: introduce an arch helper for default dma zone status
In reply to Wei ChenIn current code, when Xen is running in a multiple nodes NUMA system, it will set dma_bitsize in end_boot_allocator to reserve some low address memory for DMA.
There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an arch_have_default_dmazone helper for arch to determine that it need to set dma_bitsize for reserve DMA allocations or not.
--- xen/arch/x86/numa.c | 5 +++++ xen/common/page_alloc.c | 2 +- xen/include/asm-arm/numa.h | 5 +++++ xen/include/asm-x86/numa.h | 1 + 4 files changed, 12 insertions(+), 1 deletion(-)
xen/arch/x86/numa.c +5 -0
@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void ) +{ +return (num_online_nodes () >1 ) ?1 :0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();
xen/common/page_alloc.c +1 -1
@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ; -if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();printk ("Domain heap initialised" );
xen/include/asm-arm/numa.h +5 -0
@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +static inline unsigned int arch_have_default_dmazone (void ) +{ +return 0 ; +} +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables:
xen/include/asm-x86/numa.h +1 -0
@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void ); +unsigned int arch_have_default_dmazone (void );#endif
Signature · 1 line
2.25.1
Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Quoted 5 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase
^ the ^Because
Quoted 2 lines
on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86
^,
Quoted 3 lines
prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges.
^ of memory
Quoted 7 lines
The allocations from these memory ranges would happen only after exhausting all other nodes' memory.
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.
^ needs
Quoted 14 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 5 +++++ xen/common/page_alloc.c | 2 +- xen/include/asm-arm/numa.h | 5 +++++ xen/include/asm-x86/numa.h | 1 + 4 files changed, 12 insertions(+), 1 deletion(-)xen/arch/x86/numa.c +0
-0
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 374
Quoted 4 lines
xen/arch/x86/numa.c +1
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void )
Can this function return bool?
Also, can it be a static inline?
Quoted 50 lines
xen/arch/x86/numa.c +4
-0+{ +return (num_online_nodes () >1 ) ?1 :0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();xen/common/page_alloc.c +1
-1@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ; -if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();printk ("Domain heap initialised" );xen/include/asm-arm/numa.h +5
-0@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +static inline unsigned int arch_have_default_dmazone (void ) +{ +return 0 ; +} +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: xen/include/asm-x86/numa.h +1
-0@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end, nodeid_t node);void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void ); +unsigned int arch_have_default_dmazone (void );#endif Signature · 1 line
2.25.1
RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Hi Stefano,
Quoted 26 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 7:56
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone statusQuoted 5 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase
^ the ^Because
Quoted 2 lines
on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86
^,Quoted 3 lines
prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges.^ of memory
Quoted 9 lines
Quoted 7 lines
The allocations from these memory ranges would happen only after exhausting all other nodes' memory.
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.^ needs
I will fix above typos in next version.
Quoted 17 lines
Quoted 17 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 5 +++++ xen/common/page_alloc.c | 2 +- xen/include/asm-arm/numa.h | 5 +++++ xen/include/asm-x86/numa.h | 1 + 4 files changed, 12 insertions(+), 1 deletion(-)xen/arch/x86/numa.c +0
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); }
Quoted 5 lines
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 374
Quoted 1 line
xen/arch/x86/numa.c +1
-0+unsigned int arch_have_default_dmazone (void )Can this function return bool?
Also, can it be a static inline?
Yes, bool would be better. I will place a static inline in asm/numa.h. Because arm will have another static inline implementation.
Quoted 52 lines
xen/include/asm-x86/numa.h extern int valid_numa_range(u64 start, u64 end,
Quoted 42 lines
xen/arch/x86/numa.c +4
-0+{ +return (num_online_nodes () >1 ) ?1 :0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();xen/common/page_alloc.c +1
-1@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ; -if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();printk ("Domain heap initialised" );xen/include/asm-arm/numa.h +5
-0@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +static inline unsigned int arch_have_default_dmazone (void ) +{ +return 0 ; +} +#endif /* __ARCH_ARM_NUMA_H */ /* * Local variables: xen/include/asm-x86/numa.h +0
-0@@ -78,5 +78,6 @@ extern int valid_numa_range(u64 start, u64 end,
nodeid_tnode);Quoted 9 lines
xen/include/asm-x86/numa.h +1
-1void srat_parse_regions (u64 addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void ); +unsigned int arch_have_default_dmazone (void );#endif --2.25.1
Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
In reply to Wei ChenI realize this series has been pending for a long time, but I don't recall any indication that it would have been dropped. Hence as a first try, a few comments on this relatively simple change. I'm sorry it to have taken so long to get to it.
Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.
Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.
How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377
Quoted 10 lines
xen/arch/x86/numa.c +4
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void ) +{ +return (num_online_nodes () >1 ) ?1 :0 ; +}
According to the expression and ...
xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893
Quoted 9 lines
xen/common/page_alloc.c +1
-1@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ; -if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();
... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 31
Quoted 10 lines
xen/include/asm-arm/numa.h +4
-0@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +static inline unsigned int arch_have_default_dmazone (void ) +{ +return 0 ; +}
... like this one, x86'es couldn't be inline as well. If indeed it can't be, making it a macro may still be better (and avoid a further comment regarding the lack of __init).
Jan
RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Hi Jan,
Quoted 9 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Quoted 5 lines
I realize this series has been pending for a long time, but I don't recall any indication that it would have been dropped. Hence as a first try, a few comments on this relatively simple change. I'm sorry it to have taken so long to get to it.
Thanks for reviewing this series and lift it up. We are still working on this series and will send a new version soon.
Quoted 19 lines · Wei Chen
Quoted 13 lines
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.
Quoted 6 lines
How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the dma_bitsize. Like RPi platform, it sets its dma_bitsize to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some previous discussions are placed here [1].
Quoted 20 lines
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377
Quoted 11 lines
xen/arch/x86/numa.c +4
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void ) +{ +return (num_online_nodes () >1 ) ?1 :0 ; +}According to the expression and ...
Quoted 6 lines
xen/common/page_alloc.c +0
-0@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ;
Quoted 9 lines
xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893
Quoted 3 lines
xen/common/page_alloc.c +1
-1-if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...
I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return ( num_online_nodes() > 1 ) ? 1 : 0;"?
Quoted 6 lines
Quoted 6 lines
xen/include/asm-arm/numa.h +0
-0@@ -25,6 +25,11 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20)
Quoted 8 lines
xen/include/asm-arm/numa.h extern mfn_t first_valid_mfn; · line 31
Quoted 4 lines
xen/include/asm-arm/numa.h +4
-0+static inline unsigned int arch_have_default_dmazone (void ) +{ +return 0 ; +}... like this one, x86'es couldn't be inline as well. If indeed it can't be, making it a macro may still be better (and avoid a further comment regarding the lack of
__init).
Ok, that would be better, I will do it in next version.
Quoted 2 lines
Jan
[1] https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00772.html
Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Quoted 29 lines · Wei Chen
Quoted 22 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.Quoted 6 lines
How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 6 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets itsdma_bitsizeto 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint ondma_bitsize. And some previous discussions are placed here [1].
I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.
Quoted 29 lines
Quoted 19 lines
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377
Quoted 10 lines
xen/arch/x86/numa.c +4
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void ) +{ +return (num_online_nodes () >1 ) ?1 :0 ; +}According to the expression and ...
Quoted 6 lines
xen/common/page_alloc.c +0
-0@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ;Quoted 9 lines
xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893
Quoted 3 lines
xen/common/page_alloc.c +1
-1-if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...
Quoted 4 lines
I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return (num_online_nodes()> 1 ) ? 1 : 0;"?
Yes. Even without the use of bool such an expression is a more complicated form of
return num_online_nodes() > 1;
where we'd prefer to use the simpler variant for being easier to read / follow.
Jan
RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Hi Jan,
Quoted 46 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone statusQuoted 29 lines · Wei Chen
Quoted 22 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.Quoted 6 lines
How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 6 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets itsdma_bitsizeto 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint ondma_bitsize. And some previous discussions are placed here [1].
Quoted 6 lines
I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting
dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.
I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of dma_bitsize indicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?
Quoted 35 lines
Quoted 30 lines
Quoted 20 lines
xen/arch/x86/numa.c unsigned int __init arch_get_dma_bitsize(void) · line 377
Quoted 11 lines
xen/arch/x86/numa.c +4
-0@@ -371,6 +371,11 @@ unsigned int __init arch_get_dma_bitsize(void) + PAGE_SHIFT, 32); } +unsigned int arch_have_default_dmazone (void ) +{ +return (num_online_nodes () >1 ) ?1 :0 ; +}According to the expression and ...
Quoted 6 lines
xen/common/page_alloc.c +0
-0@@ -1889,7 +1889,7 @@ void __init end_boot_allocator(void) } nr_bootmem_regions =0 ;Quoted 9 lines
xen/common/page_alloc.c void __init end_boot_allocator(void) · line 1893
Quoted 3 lines
xen/common/page_alloc.c +1
-1-if ( !dma_bitsize && (num_online_nodes () >1 ) ) +if ( !dma_bitsize &&arch_have_default_dmazone () ) dma_bitsize =arch_get_dma_bitsize ();... the use site, you mean the function to return boolean. Please indicate so by making it have a return type of "bool". Independent of that you don't need a conditional expression above, nor (malformed) use of parentheses. I further wonder whether ...
Quoted 4 lines
I will fix them in next version. But I am not very clear about this comment "of that you don't need a conditional expression above", The "above" indicates this line:
"return (num_online_nodes()> 1 ) ? 1 : 0;"?
Quoted 8 lines
Yes. Even without the use of bool such an expression is a more complicated form of
return num_online_nodes() > 1;where we'd prefer to use the simpler variant for being easier to read / follow.
Thanks for clarification, I will fix it.
Quoted 6 lines
Jan
Quoted 1 line
[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00772.html
Quoted 2 lines
Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Quoted 46 lines · Wei Chen
Quoted 39 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16Quoted 28 lines · Wei Chen
Quoted 21 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.Quoted 6 lines
How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 6 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets itsdma_bitsizeto 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint ondma_bitsize. And some previous discussions are placed here [1].Quoted 6 lines
I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting
dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.
Quoted 7 lines
I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of
dma_bitsizeindicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?
I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...
Jan
RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Hi Jan,
Quoted 65 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone statusQuoted 32 lines · Wei Chen
Quoted 32 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16Quoted 28 lines · Wei Chen
Quoted 27 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets its
dma_bitsizeQuoted 11 lines
Quoted 10 lines
Quoted 3 lines
to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on
dma_bitsize. And some previous discussions are placed here [1].I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting
dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.Quoted 7 lines
I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of
dma_bitsizeindicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?
Quoted 10 lines
I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...
I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?
Thanks,
Wei Chen
Quoted 1 line
Jan
Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Quoted 78 lines · Wei Chen
Hi Jan,
Quoted 65 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone statusQuoted 32 lines · Wei Chen
Quoted 32 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16Quoted 28 lines · Wei Chen
Quoted 27 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 4 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch to determine that it need to setdma_bitsizefor reserve DMA allocations or not.How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets its
dma_bitsizeQuoted 11 lines
Quoted 10 lines
Quoted 3 lines
to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on
dma_bitsize. And some previous discussions are placed here [1].I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting
dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need of such a range to perform I/O to _some_ devices.Quoted 7 lines
I am sorry that my last reply hasn't given you more clues. On Arm, only Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of
dma_bitsizeindicated. I think these operations meet your above Dom0 special address range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is that answers your earlier question?Quoted 10 lines
I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-like, due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...
Quoted 4 lines
I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?
Extending the description is my primary request, or else we may end up having the same discussion every time you submit a new version. As to improving the situation such that preferably per-arch customization wouldn't be necessary - that's perhaps better to be thought about by Arm folks. Otoh, as said, an x86 build with CONFIG_PV=n could probably leverage the new hook to actually not trigger reservation.
Jan
RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status
Hi Jan,
Quoted 102 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 15:50
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone statusQuoted 9 lines · Wei Chen
Hi Jan,
Quoted 7 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 22:16
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 04/37] xen: introduce an arch helper for defaultdma
Quoted 23 lines
Quoted 23 lines
zone status
Quoted 20 lines · Wei Chen
Quoted 20 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 16:16Quoted 16 lines · Wei Chen
Quoted 16 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:11Quoted 13 lines · Wei Chen
In current code, when Xen is running in a multiple nodes NUMA system, it will set
dma_bitsizeinend_boot_allocatorto reserve some low address memory for DMA.There are some x86 implications in current implementation. Becuase on x86, memory starts from 0. On a multiple nodes NUMA system, if a single node contains the majority or all of the DMA memory. x86 prefer to give out memory from non-local allocations rather than exhausting the DMA memory ranges. Hence x86 use
dma_bitsizeto set aside some largely arbitrary amount memory for DMA memory ranges. The allocations from these memory ranges would happen only after exhausting all other nodes' memory.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
But the implications are not shared across all architectures. For example, Arm doesn't have these implications. So in this patch, we introduce an
arch_have_default_dmazonehelper for arch todetermine
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
that it need to set
dma_bitsizefor reserve DMA allocations or not.How would Arm guarantee availability of memory below a certain boundary for limited-capability devices? Or is there no need because there's an assumption that I/O for such devices would always pass through an IOMMU, lifting address size restrictions? (I guess in a !PV build on x86 we could also get rid of such a reservation.)
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
On Arm, we still can have some devices with limited DMA capability. And we also don't force all such devices to use IOMMU. This devices will affect the
dma_bitsize. Like RPi platform, it sets its
dma_bitsizeQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
to 30. But in multiple NUMA nodes system, Arm doesn't have a default DMA zone. Multiple nodes is not a constraint on
dma_bitsize. Andsome
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
previous discussions are placed here [1].
I'm afraid that doesn't give me more clues. For example, in the mail being replied to there I find "That means, only first 4GB memory can be used for DMA." Yet that's not an implication from setting
dma_bitsize. DMA is fine to occur to any address. The special address range is being held back in case in particular Dom0 is in need ofsuch
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
a range to perform I/O to _some_ devices.
I am sorry that my last reply hasn't given you more clues. On Arm,
only
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0, we're trying to allocate memory under 4GB or in the range of
dma_bitsizeQuoted 1 line
Quoted 1 line
Quoted 1 line
indicated. I think these operations meet your above Dom0 special
address
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
range description. As we have already allocated memory for DMA, so I think we don't need a DMA zone in page allocation. I am not sure is
that
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
answers your earlier question?
I view all of this as flawed, or as a workaround at best. Xen shouldn't make assumptions on what Dom0 may need. Instead Dom0 should make arrangements such that it can do I/O to/from all devices of interest. This may involve arranging for address restricted buffers. And for this to be possible, Xen would need to have available some suitable memory. I understand this is complicated by the fact that despite being HVM-
like,
Quoted 9 lines
Quoted 4 lines
due to the lack of an IOMMU in front of certain devices address restrictions on Dom0 address space alone (i.e. without any Xen involvement) won't help ...
I agree with you that the current implementation is probably the best kind of workaround. Do you have some suggestions for this patch to address above comments? Or should I just need to modify the commit log to contain some of our above discussions?
Quoted 7 lines
Extending the description is my primary request, or else we may end up having the same discussion every time you submit a new version. As to improving the situation such that preferably per-arch customization wouldn't be necessary - that's perhaps better to be thought about by Arm folks. Otoh, as said, an x86 build with
CONFIG_PV=ncould probably leverage the new hook to actually not trigger reservation.
Ok, I will try to extend the description to avoid the same discussion in next version.
Quoted 1 line
Jan
PATCH 5/37 xen: decouple NUMA from ACPI in Kconfig
In reply to Wei ChenIn current Xen code only implements x86 ACPI-based NUMA support. So in Xen Kconfig system, NUMA equals to ACPI_NUMA. x86 selects NUMA by default, and CONFIG_ACPI_NUMA is hardcode in config.h.
In a follow-up patch, we will introduce support for NUMA using the device tree. That means we will have two NUMA implementations, so in this patch we decouple NUMA from ACPI based NUMA in Kconfig. Make NUMA as a common feature, that device tree based NUMA also can select it.
--- xen/arch/x86/Kconfig | 2 +- xen/common/Kconfig | 3 +++ xen/drivers/acpi/Kconfig | 3 ++- xen/drivers/acpi/Makefile | 2 +- xen/include/asm-x86/config.h | 1 - 5 files changed, 7 insertions(+), 4 deletions(-)
xen/arch/x86/Kconfig +1 -1
@@ -6,6 +6,7 @@ config X86 def_bool y select ACPI select ACPI_LEGACY_TABLES_LOOKUP + select ACPI_NUMA select ALTERNATIVE_CALL select ARCH_SUPPORTS_INT128 select CORE_PARKING
@@ -25,7 +26,6 @@ config X86 select HAS_UBSAN select HAS_VPCI if HVM select NEEDS_LIBELF - select NUMA config ARCH_DEFCONFIG string
xen/common/Kconfig +3 -0
@@ -70,6 +70,9 @@ config MEM_ACCESS config NEEDS_LIBELF bool +config NUMA + bool + config STATIC_MEMORY bool "Static Allocation Support (UNSUPPORTED)" if UNSUPPORTED depends on ARM
xen/drivers/acpi/Kconfig +2 -1
@@ -5,5 +5,6 @@ config ACPI config ACPI_LEGACY_TABLES_LOOKUP bool -config NUMA +config ACPI_NUMA bool + select NUMA
xen/drivers/acpi/Makefile +1 -1
@@ -3,7 +3,7 @@ obj-y += utilities/ obj-$(CONFIG_X86) += apei/ obj-bin-y += tables.init.o -obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_ACPI_NUMA) += numa.o obj-y += osl.o obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
xen/include/asm-x86/config.h +0 -1
@@ -31,7 +31,6 @@/* Intel P4 currently has largest cache line (L2 line size is 128 bytes). */ #define CONFIG_X86_L1_CACHE_SHIFT 7 -#define CONFIG_ACPI_NUMA 1#define CONFIG_ACPI_SRAT 1#define CONFIG_ACPI_CSTATE 1
Signature · 1 line
2.25.1
PATCH 6/37 xen/arm: use !CONFIG_NUMA to keep fake NUMA API
In reply to Wei ChenWe have introduced CONFIG_NUMA in previous patch. And this option is enabled only on x86 in current stage. In a follow up patch, we will enable this option for Arm. But we still want users can disable the CONFIG_NUMA through Kconfig. In this case, keep current fake NUMA API, will make Arm code still can work with NUMA aware memory allocation and scheduler.
--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)
xen/include/asm-arm/numa.h +4 -0
@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#ifndef CONFIG_NUMA +/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)
@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +static inline unsigned int arch_have_default_dmazone (void ) {return 0 ;
Signature · 1 line
2.25.1
Re: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 1 line · Wei Chen
We have introduced
CONFIG_NUMAin previous patch. And this
^ a
Quoted 1 line
option is enabled only on x86 in current stage. In a follow
^ at the
Quoted 2 lines
up patch, we will enable this option for Arm. But we still want users can disable the
CONFIG_NUMAthroughKconfig. In
^ to be able to disable CONFIG_NUMA via Kconfig.
Quoted 1 line
this case, keep current fake NUMA API, will make Arm code
^ the
Quoted 1 line
still can work with NUMA aware memory allocation and scheduler.
^ able to work
Quoted 2 lines
Signed-off-by Wei Chen <[email protected]>
With the small grammar fixes:
Quoted 29 lines
--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +4
-0@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#ifndef CONFIG_NUMA +/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +static inline unsigned int arch_have_default_dmazone (void ) {return 0 ;Signature · 1 line
2.25.1
RE: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API
Quoted 25 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:05
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA APIQuoted 1 line · Wei Chen
We have introduced
CONFIG_NUMAin previous patch. And this^ aQuoted 1 line
option is enabled only on x86 in current stage. In a follow
^ at the
Quoted 2 lines
up patch, we will enable this option for Arm. But we still want users can disable the
CONFIG_NUMAthroughKconfig. In^ to be able to disable
CONFIG_NUMAviaKconfig.Quoted 1 line
this case, keep current fake NUMA API, will make Arm code
^ the
Quoted 1 line
still can work with NUMA aware memory allocation and scheduler.
Quoted 10 lines
^ able to work
Quoted 2 lines
Signed-off-by Wei Chen <[email protected]>With the small grammar fixes:
Reviewed-by Stefano Stabellini <[email protected]>
Thanks, I will fix them in next version.
Quoted 29 lines
Quoted 29 lines
--- xen/include/asm-arm/numa.h | 4 ++++ 1 file changed, 4 insertions(+)xen/include/asm-arm/numa.h +4
-1@@ -5,6 +5,8 @@typedef u8 nodeid_t ; +#ifndef CONFIG_NUMA +/* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0#define node_to_cpumask (node) (cpu_online_map)@@ -25,6 +27,8 @@ extern mfn_t first_valid_mfn;#define node_start_pfn (nid) (mfn_x(first_valid_mfn))#define __node_distance (a, b) (20) +#endif +static inline unsigned int arch_have_default_dmazone (void ) {return 0 ; --2.25.1
PATCH 7/37 xen/x86: use paddr_t for addresses in NUMA node structure
In reply to Wei ChenNUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to paddr_t. And use pfn_to_paddr and paddr_to_pfn to replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.
--- xen/arch/x86/numa.c | 32 +++++++++++++++++--------------- xen/arch/x86/srat.c | 26 +++++++++++++------------- xen/include/asm-x86/numa.h | 8 ++++---- 3 files changed, 34 insertions(+), 32 deletions(-)
xen/arch/x86/numa.c +17 -15
@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; }/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ +void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end) +{unsigned long start_pfn, end_pfn; - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; + start_pfn =paddr_to_pfn (start); + end_pfn =paddr_to_pfn (end);NODE_DATA (nodeid)->node_start_pfn = start_pfn;NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;
@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn) {int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;/* Kludge needed for the hash function */ if (hweight64 (sz) >1 )
@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)memset (&nodes,0 ,sizeof (nodes));for ( i =0 ; i < numa_fake; i++ ) { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; + nodes[i].start =pfn_to_paddr (start_pfn) + i*sz;if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; + sz =pfn_to_paddr (end_pfn) - nodes[i].start; nodes[i].end = nodes[i].start + sz; printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n", i,
@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) {int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn); +#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;#endif printk (KERN_INFO "%s\n" , numa_off ?"NUMA turned off" :"No NUMA configuration found" ); -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , + start, end);/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap;
@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)for ( i =0 ; i < nr_cpu_ids; i++ )numa_set_node (i,0 );cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +setup_node_bootmem (0 , start, end); }void numa_add_cpu (int cpu)
xen/arch/x86/srat.c +13 -13
@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;
@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)return 0 ; } -static __init int conflicting_memblks (u64 start,u64 end) +static __init int conflicting_memblks (paddr_t start,paddr_t end) {int i;
@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)return -1 ; } -static __init void cutoff_node (int i,u64 start,u64 end) +static __init void cutoff_node (int i,paddr_t start,paddr_t end) {struct node *nd = &nodes[i];if (nd->start < start) {
@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) { -u64 start, end; +paddr_t start, end;unsigned pxm;nodeid_t node;int i;
@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != !test_bit (i, memblk_hotplug); -printk ("%sSRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , +printk ("%sSRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) {
@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) } }else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with PXM %u (%" PRIx64"-%" PRIx64")\n" , +"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , pxm, start, end,node_to_pxm (memblk_nodeid[i]), node_memblk_range[i].start, node_memblk_range[i].end);bad_srat ();
@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->end = end; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIx64"-%" PRIx64"%s\n" , +printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , node, pxm, start, end, ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" );
@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)for (i =0 ; i < e820.nr_map; i++) {int j, found; -unsigned long long start, end; +paddr_t start, end;if (e820.map[i].type !=E820_RAM ) {continue ;
@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)if (start < end) {printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); +"%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; } }
@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,return 0 ; } -void __init srat_parse_regions (u64 addr) +void __init srat_parse_regions (paddr_t addr) {u64 mask;unsigned int i;
@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);acpi_table_parse_srat (ACPI_SRAT_TYPE_MEMORY_AFFINITY , srat_parse_region,0 );
@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init acpi_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;
@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " "BIOS Bug or mis-configured hardware?\n" , i);
xen/include/asm-x86/numa.h +4 -4
@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end; }; extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +extern void setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );
@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); -void srat_parse_regions (u64 addr); +void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void );
Signature · 1 line
2.25.1
Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
Quoted 36 lines · Wei Chen
NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to
paddr_t. And usepfn_to_paddrandpaddr_to_pfnto replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 32 +++++++++++++++++--------------- xen/arch/x86/srat.c | 26 +++++++++++++------------- xen/include/asm-x86/numa.h | 8 ++++---- 3 files changed, 34 insertions(+), 32 deletions(-)xen/arch/x86/numa.c +4
-4@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; }/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ +void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end) +{unsigned long start_pfn, end_pfn; - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; + start_pfn =paddr_to_pfn (start); + end_pfn =paddr_to_pfn (end);NODE_DATA (nodeid)->node_start_pfn = start_pfn;NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;
xen/arch/x86/numa.c void __init numa_init_array(void) · line 205
Quoted 6 lines
xen/arch/x86/numa.c +2
-1@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn)
Why not changing numa_emulation to take paddr_t too?
Quoted 21 lines
xen/arch/x86/numa.c +3
-3{int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;/* Kludge needed for the hash function */ if (hweight64 (sz) >1 )@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)memset (&nodes,0 ,sizeof (nodes));for ( i =0 ; i < numa_fake; i++ ) { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; + nodes[i].start =pfn_to_paddr (start_pfn) + i*sz;if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; + sz =pfn_to_paddr (end_pfn) - nodes[i].start; nodes[i].end = nodes[i].start + sz; printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n", i,
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 250
Quoted 1 line
xen/arch/x86/numa.c +0
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn)
same here
Quoted 198 lines
xen/arch/x86/numa.c +8
-7{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn); +#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;#endif printk (KERN_INFO "%s\n" , numa_off ?"NUMA turned off" :"No NUMA configuration found" ); -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , + start, end);/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap;@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)for ( i =0 ; i < nr_cpu_ids; i++ )numa_set_node (i,0 );cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +setup_node_bootmem (0 , start, end); }void numa_add_cpu (int cpu)xen/arch/x86/srat.c +13
-13@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)return 0 ; } -static __init int conflicting_memblks (u64 start,u64 end) +static __init int conflicting_memblks (paddr_t start,paddr_t end) {int i;@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)return -1 ; } -static __init void cutoff_node (int i,u64 start,u64 end) +static __init void cutoff_node (int i,paddr_t start,paddr_t end) {struct node *nd = &nodes[i];if (nd->start < start) {@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) { -u64 start, end; +paddr_t start, end;unsigned pxm;nodeid_t node;int i;@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != !test_bit (i, memblk_hotplug); -printk ("%sSRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , +printk ("%sSRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) {@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) } }else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with PXM %u (%" PRIx64"-%" PRIx64")\n" , +"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , pxm, start, end,node_to_pxm (memblk_nodeid[i]), node_memblk_range[i].start, node_memblk_range[i].end);bad_srat ();@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->end = end; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIx64"-%" PRIx64"%s\n" , +printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , node, pxm, start, end, ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" );@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)for (i =0 ; i < e820.nr_map; i++) {int j, found; -unsigned long long start, end; +paddr_t start, end;if (e820.map[i].type !=E820_RAM ) {continue ;@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)if (start < end) {printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); +"%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; } }@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,return 0 ; } -void __init srat_parse_regions (u64 addr) +void __init srat_parse_regions (paddr_t addr) {u64 mask;unsigned int i;@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);acpi_table_parse_srat (ACPI_SRAT_TYPE_MEMORY_AFFINITY , srat_parse_region,0 );@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init acpi_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " "BIOS Bug or mis-configured hardware?\n" , i);xen/include/asm-x86/numa.h +4
-4@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end; }; extern int compute_hash_shift(struct node *nodes, int numnodes,@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +extern void setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); -void srat_parse_regions (u64 addr); +void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void );Signature · 1 line
2.25.1
Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
You forgot to add the x86 maintainers in CC to all the patches touching x86 code in this series. Adding them now but you should probably resend.
Quoted 272 lines · Stefano Stabellini
Quoted 36 lines · Wei Chen
NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to
paddr_t. And usepfn_to_paddrandpaddr_to_pfnto replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 32 +++++++++++++++++--------------- xen/arch/x86/srat.c | 26 +++++++++++++------------- xen/include/asm-x86/numa.h | 8 ++++---- 3 files changed, 34 insertions(+), 32 deletions(-)xen/arch/x86/numa.c +4
-4@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,return shift; }/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ +void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end) +{unsigned long start_pfn, end_pfn; - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; + start_pfn =paddr_to_pfn (start); + end_pfn =paddr_to_pfn (end);NODE_DATA (nodeid)->node_start_pfn = start_pfn;NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c void __init numa_init_array(void) · line 205
Quoted 6 lines
xen/arch/x86/numa.c +2
-1@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn)Why not changing
numa_emulationto takepaddr_ttoo?Quoted 21 lines
xen/arch/x86/numa.c +3
-3{int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;/* Kludge needed for the hash function */ if (hweight64 (sz) >1 )@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)memset (&nodes,0 ,sizeof (nodes));for ( i =0 ; i < numa_fake; i++ ) { - nodes[i].start = (start_pfn<<PAGE_SHIFT ) + i*sz; + nodes[i].start =pfn_to_paddr (start_pfn) + i*sz;if ( i == numa_fake -1 ) - sz = (end_pfn<<PAGE_SHIFT ) - nodes[i].start; + sz =pfn_to_paddr (end_pfn) - nodes[i].start; nodes[i].end = nodes[i].start + sz; printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n", i,xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 250
Quoted 1 line
xen/arch/x86/numa.c +0
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn)same here
Quoted 198 lines
xen/arch/x86/numa.c +8
-7{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn); +#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;#endif printk (KERN_INFO "%s\n" , numa_off ?"NUMA turned off" :"No NUMA configuration found" ); -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , + start, end);/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap;@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)for ( i =0 ; i < nr_cpu_ids; i++ )numa_set_node (i,0 );cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +setup_node_bootmem (0 , start, end); }void numa_add_cpu (int cpu)xen/arch/x86/srat.c +13
-13@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)return node; } -int valid_numa_range (u64 start,u64 end,nodeid_t node) +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)return 0 ; } -static __init int conflicting_memblks (u64 start,u64 end) +static __init int conflicting_memblks (paddr_t start,paddr_t end) {int i;@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start, u64 end)return -1 ; } -static __init void cutoff_node (int i,u64 start,u64 end) +static __init void cutoff_node (int i,paddr_t start,paddr_t end) {struct node *nd = &nodes[i];if (nd->start < start) {@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) { -u64 start, end; +paddr_t start, end;unsigned pxm;nodeid_t node;int i;@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != !test_bit (i, memblk_hotplug); -printk ("%sSRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with itself (%" PRIx64"-%" PRIx64")\n" , +printk ("%sSRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) {@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) } }else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIx64"-%" PRIx64") overlaps with PXM %u (%" PRIx64"-%" PRIx64")\n" , +"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , pxm, start, end,node_to_pxm (memblk_nodeid[i]), node_memblk_range[i].start, node_memblk_range[i].end);bad_srat ();@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->end = end; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIx64"-%" PRIx64"%s\n" , +printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , node, pxm, start, end, ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" );@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)for (i =0 ; i < e820.nr_map; i++) {int j, found; -unsigned long long start, end; +paddr_t start, end;if (e820.map[i].type !=E820_RAM ) {continue ;@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)if (start < end) {printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); +"%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; } }@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,return 0 ; } -void __init srat_parse_regions (u64 addr) +void __init srat_parse_regions (paddr_t addr) {u64 mask;unsigned int i;@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);acpi_table_parse_srat (ACPI_SRAT_TYPE_MEMORY_AFFINITY , srat_parse_region,0 );@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (u64 start,u64 end) +int __init acpi_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " "BIOS Bug or mis-configured hardware?\n" , i);xen/include/asm-x86/numa.h +4
-4@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end; }; extern int compute_hash_shift(struct node *nodes, int numnodes,@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +extern void setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end);extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); -void srat_parse_regions (u64 addr); +void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void );Signature · 1 line
2.25.1
RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
Quoted 13 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:14
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: usepaddr_tfor addresses in NUMA node structure
Quoted 3 lines
You forgot to add the x86 maintainers in CC to all the patches touching x86 code in this series. Adding them now but you should probably resend.
I am very sorry about it. I realized the problem when I pressed Enter. I had wanted to repost it at that time, but I didn't know whether these patches will be turned into spamming...
I will resend this series ASAP with some changes to address your comments.
Quoted 23 lines · Stefano Stabellini
Quoted 15 lines · Wei Chen
Quoted 14 lines
NUMA node structure "struct node" is using u64 as node memory range. In order to make other architectures can reuse this NUMA node relative code, we replace the u64 to
paddr_t. And usepfn_to_paddrandpaddr_to_pfnto replace explicit shift operations. The relate PRIx64 in print messages have been replaced by PRIpaddr at the same time.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 32 +++++++++++++++++--------------- xen/arch/x86/srat.c | 26 +++++++++++++------------- xen/include/asm-x86/numa.h | 8 ++++---- 3 files changed, 34 insertions(+), 32 deletions(-)Quoted 5 lines
xen/arch/x86/numa.c int __init compute_hash_shift(struct node *nodes,
Quoted 5 lines
xen/arch/x86/numa.c +0
-0@@ -165,12 +165,12 @@ int __init compute_hash_shift(struct node *nodes,int numnodes,
Quoted 27 lines
Quoted 6 lines
xen/arch/x86/numa.c int __init compute_hash_shift(struct node *nodes, int numnodes,
Quoted 6 lines
xen/arch/x86/numa.c +1
-2return shift; }/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end) -{ +void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start,
paddr_tend)Quoted 20 lines
Quoted 11 lines
xen/arch/x86/numa.c +3
-2+{unsigned long start_pfn, end_pfn; - start_pfn = start >>PAGE_SHIFT ; - end_pfn = end >>PAGE_SHIFT ; + start_pfn =paddr_to_pfn (start); + end_pfn =paddr_to_pfn (end);NODE_DATA (nodeid)->node_start_pfn = start_pfn;NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn;xen/arch/x86/numa.c void __init numa_init_array(void) · line 205
Quoted 6 lines
xen/arch/x86/numa.c +2
-1@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn)Why not changing
numa_emulationto takepaddr_ttoo?
numa_emulation parameter is pfn, it's not address. I have discussed with Julien in RFC about pfn. He suggested to use mfn_t or unsigned long for pfn. Comparing to mfn_t, use unsigned long brings less changes.
Quoted 249 lines
Quoted 10 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn,
Quoted 10 lines
xen/arch/x86/numa.c +1
-1{int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;/* Kludge needed for the hash function */ if (hweight64 (sz) >1 )@@ -221,9 +222,9 @@ static int __init numa_emulation(u64 start_pfn,u64
end_pfn)Quoted 10 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
Quoted 10 lines
xen/arch/x86/numa.c +2
-2memset (&nodes,0 ,sizeof (nodes)); for ( i = 0; i < numa_fake; i++ ) { - nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz; + nodes[i].start = pfn_to_paddr(start_pfn) + i*sz; if ( i == numa_fake - 1 ) - sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start; + sz = pfn_to_paddr(end_pfn) - nodes[i].start; nodes[i].end = nodes[i].start + sz; printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64"(%"PRIu64"MB)\n",
Quoted 2 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn,
Quoted 2 lines
xen/arch/x86/numa.c +0
-0i,@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,u64
end_pfn)Quoted 1 line
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, · line 250
Quoted 1 line
xen/arch/x86/numa.c +0
-0void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 25 lines
same here
Quoted 21 lines
xen/arch/x86/numa.c +5
-2{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn); +#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;#endif printk(KERN_INFO "%s\n", numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 5 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
Quoted 5 lines
xen/arch/x86/numa.c +1
-3-printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); + printk(KERN_INFO "Faking a node at %016"PRIpaddr"-%016"PRIpaddr"\n",
Quoted 5 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 5 lines
xen/arch/x86/numa.c +1
-0+ start, end);/* setup dummy node covering all memory */ memnode_shift =BITS_PER_LONG -1 ; memnodemap = _memnodemap;@@ -279,8 +282,7 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 17 lines
Quoted 17 lines
xen/arch/x86/numa.c +1
-2for ( i =0 ; i < nr_cpu_ids; i++ )numa_set_node (i,0 );cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +setup_node_bootmem (0 , start, end); }void numa_add_cpu (int cpu)xen/arch/x86/srat.c +0
-0@@ -104,7 +104,7 @@ nodeid_t setup_node(unsigned pxm)return node; }Quoted 6 lines
xen/arch/x86/srat.c int valid_numa_range(u64 start, u64 end, nodeid_t
Quoted 6 lines
xen/arch/x86/srat.c +1
-1-int valid_numa_range (u64 start,u64 end,nodeid_t node) +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;@@ -119,7 +119,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_tnode)
Quoted 9 lines
xen/arch/x86/srat.c static __init int conflicting_memblks(u64 start,
Quoted 9 lines
xen/arch/x86/srat.c +1
-1return 0 ; } -static __init int conflicting_memblks (u64 start,u64 end) +static __init int conflicting_memblks (paddr_t start,paddr_t end) {int i;@@ -135,7 +135,7 @@ static __init int conflicting_memblks(u64 start,u64 end)
Quoted 9 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct
Quoted 9 lines
xen/arch/x86/srat.c +1
-1return -1 ; } -static __init void cutoff_node (int i,u64 start,u64 end) +static __init void cutoff_node (int i,paddr_t start,paddr_t end) {struct node *nd = &nodes[i];if (nd->start < start) {@@ -275,7 +275,7 @@ acpi_numa_processor_affinity_init(const struct
acpi_srat_cpu_affinity*pa)Quoted 2 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct · line 276
Quoted 2 lines
xen/arch/x86/srat.c +0
-0void __init acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity*ma)
Quoted 7 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 7 lines
xen/arch/x86/srat.c +1
-1{ -u64 start, end; +paddr_t start, end;unsigned pxm;nodeid_t node;int i;@@ -318,7 +318,7 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 4 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 4 lines
xen/arch/x86/srat.c +0
-1bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != !test_bit (i, memblk_hotplug); - printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps withitself (%"PRIx64"-%"PRIx64")\n",
Quoted 1 line
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 321
Quoted 1 line
xen/arch/x86/srat.c +1
-0+ printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps withitself (%"PRIpaddr"-%"PRIpaddr")\n",
Quoted 4 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 4 lines
xen/arch/x86/srat.c +0
-0mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) {@@ -327,7 +327,7 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 4 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 4 lines
xen/arch/x86/srat.c +0
-1} }else { printk(KERN_ERR - "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps withPXM %u (%"PRIx64"-%"PRIx64")\n",
Quoted 1 line
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 330
Quoted 1 line
xen/arch/x86/srat.c +1
-0+ "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps withPXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
Quoted 4 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 4 lines
xen/arch/x86/srat.c +0
-0pxm, start, end,node_to_pxm (memblk_nodeid[i]), node_memblk_range[i].start, node_memblk_range[i].end);bad_srat ();@@ -346,7 +346,7 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 19 lines
Quoted 19 lines
xen/arch/x86/srat.c +2
-2nd->end = end; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIx64"-%" PRIx64"%s\n" , +printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , node, pxm, start, end, ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" );@@ -369,7 +369,7 @@ static int __init nodes_cover_memory(void)for (i =0 ; i < e820.nr_map; i++) {int j, found; -unsigned long long start, end; +paddr_t start, end;if (e820.map[i].type !=E820_RAM ) {continue ;@@ -396,7 +396,7 @@ static int __init nodes_cover_memory(void)Quoted 8 lines
xen/arch/x86/srat.c static int __init srat_parse_region(struct
Quoted 8 lines
xen/arch/x86/srat.c +1
-1if (start < end) {printk (KERN_ERR "SRAT: No PXM for e820 range: " -"%016Lx - %016Lx\n" , start, end); +"%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; } }@@ -432,7 +432,7 @@ static int __init srat_parse_region(struct
acpi_subtable_header*header,Quoted 17 lines
Quoted 17 lines
xen/arch/x86/srat.c +2
-2return 0 ; } -void __init srat_parse_regions (u64 addr) +void __init srat_parse_regions (paddr_t addr) {u64 mask;unsigned int i;@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);acpi_table_parse_srat (ACPI_SRAT_TYPE_MEMORY_AFFINITY , srat_parse_region,0 );Quoted 4 lines
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 459
Quoted 4 lines
xen/arch/x86/srat.c +0
-0@@ -457,7 +457,7 @@ void __init srat_parse_regions(u64 addr) }/* Use the information discovered above to actually set up the nodes. */
Quoted 26 lines
Quoted 26 lines
xen/arch/x86/srat.c +2
-2-int __init acpi_scan_nodes (u64 start,u64 end) +int __init acpi_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " "BIOS Bug or mis-configured hardware?\n" , i);xen/include/asm-x86/numa.h +1
-1@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end; };Quoted 7 lines
xen/include/asm-x86/numa.h extern void numa_set_node(int cpu, nodeid_t node); · line 39
Quoted 7 lines
xen/include/asm-x86/numa.h +1
-1extern int compute_hash_shift(struct node *nodes, int numnodes,@@ -36,7 +36,7 @@ extern void numa_set_node(int cpu, nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu); -extern void setup_node_bootmem (nodeid_t nodeid,u64 start,u64 end); +extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start,
paddr_tend);Quoted 4 lines
xen/include/asm-x86/numa.h static inline __attribute__((pure)) nodeid_t
Quoted 4 lines
xen/include/asm-x86/numa.h +0
-0extern nodeid_t apicid_to_node[];extern void init_cpu_to_node (void );@@ -73,9 +73,9 @@ static inline __attribute__((pure)) nodeid_t
phys_to_nid(paddr_t addr)Quoted 5 lines
xen/include/asm-x86/numa.h static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
Quoted 5 lines
xen/include/asm-x86/numa.h +1
-1#define node_end_pfn (nid) (NODE_DATA(nid)->node_start_pfn + \ NODE_DATA(nid)->node_spanned_pages) -extern int valid_numa_range (u64 start,u64 end,nodeid_t node); +extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_tnode);
Quoted 10 lines
Quoted 9 lines
xen/include/asm-x86/numa.h +1
-2-void srat_parse_regions (u64 addr); +void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void ); --2.25.1
Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
In reply to Wei Chenxen/arch/x86/numa.c void __init numa_init_array(void) · line 209
Quoted 12 lines · Wei Chen
xen/arch/x86/numa.c +3
-2@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn) {int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;
Nit: Please convert to uint64_t (and alike) whenever you touch a line anyway that uses being-phased-out types.
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261
Quoted 13 lines
xen/arch/x86/numa.c +3
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) {int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn);
Nit: Would be slightly neater if these were the initializers of the variables.
Quoted 5 lines
xen/arch/x86/numa.c +1
-2#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 272
Quoted 10 lines
xen/arch/x86/numa.c +2
-3#endif printk (KERN_INFO "%s\n" , numa_off ?"NUMA turned off" :"No NUMA configuration found" ); -printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , + start, end);
When switching to PRIpaddr I suppose you did look up what that one expands to? IOW - please drop the 016 from here.
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444
Quoted 6 lines
xen/arch/x86/srat.c +1
-1@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);
I don't see the need for a cast here.
xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 493
Quoted 7 lines
xen/arch/x86/srat.c +1
-1@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )
Please take the opportunity and add the missing blank line between declarations and statements.
xen/include/asm-x86/numa.h extern cpumask_t node_to_cpumask[]; · line 19
Quoted 8 lines
xen/include/asm-x86/numa.h +1
-1@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end;
Please take the opportunity and add the missing blank after the comma.
Jan
RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
Hi Jan,
Quoted 13 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: usepaddr_tfor addresses in NUMA node structureQuoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -201,11 +201,12 @@ void __init numa_init_array(void)static int numa_fake__initdata =0 ;
Quoted 13 lines
xen/arch/x86/numa.c void __init numa_init_array(void) · line 209
Quoted 9 lines
xen/arch/x86/numa.c +3
-2/* Numa emulation */ -static int __init numa_emulation (u64 start_pfn,u64 end_pfn) +static int __init numa_emulation (unsigned long start_pfn, +unsigned long end_pfn) {int i;struct node nodes[MAX_NUMNODES ]; -u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT ) / numa_fake; +u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake;Nit: Please convert to
uint64_t(and alike) whenever you touch a line anyway that uses being-phased-out types.
Ok, I will do it.
Quoted 13 lines
Review static int __init numa_emulation(u64 start_pfn,
Quoted 1 line
(quoted diff) +0
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,u64
end_pfn)Review static int __init numa_emulation(u64 start_pfn, · line 250
Quoted 1 line
(quoted diff) +0
-0void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 9 lines
xen/arch/x86/numa.c +1
-0{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif
Quoted 5 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+ start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn);Nit: Would be slightly neater if these were the initializers of the variables.
But if this function returns in numa_fake && !numa_emulation, will the two pfn_to_paddr operations be waste?
Quoted 12 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
Quoted 10 lines
xen/arch/x86/numa.c +1
-2#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes ((u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ) ) +if ( !numa_off && !acpi_scan_nodes (start, end) )return ;#endif printk(KERN_INFO "%s\n", numa_off ? "NUMA turned off" : "No NUMA configurationfound");
Quoted 1 line
Quoted 8 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 272
Quoted 5 lines
xen/arch/x86/numa.c +2
-3-printk (KERN_INFO "Faking a node at %016" PRIx64"-%016" PRIx64"\n" , - (u64 )start_pfn <<PAGE_SHIFT , - (u64 )end_pfn <<PAGE_SHIFT ); +printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , + start, end);When switching to PRIpaddr I suppose you did look up what that one expands to? IOW - please drop the 016 from here.
Oh, yes, I forgot to drop the duplicated 016. I would do it.
Quoted 10 lines
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444
Quoted 7 lines
xen/arch/x86/srat.c +1
-1@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);I don't see the need for a cast here.
current addr type has been changed to paddr_t, but pdx_init_mask accept parameter type is u64. I know paddr_t is a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?
Quoted 11 lines
xen/arch/x86/srat.c int __init acpi_scan_nodes(u64 start, u64 end) · line 493
Quoted 7 lines
xen/arch/x86/srat.c +1
-1@@ -489,7 +489,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)/* Finally register nodes */ for_each_node_mask (i, all_nodes_parsed) { -u64 size = nodes[i].end - nodes[i].start; +paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )Please take the opportunity and add the missing blank line between declarations and statements.
Ok
Quoted 11 lines
xen/include/asm-x86/numa.h extern cpumask_t node_to_cpumask[]; · line 19
Quoted 8 lines
xen/include/asm-x86/numa.h +1
-1@@ -16,7 +16,7 @@ extern cpumask_t node_to_cpumask[];#define node_to_cpumask (node) (node_to_cpumask[node])struct node { -u64 start,end; +paddr_t start,end;Please take the opportunity and add the missing blank after the comma.
Ok
Quoted 1 line
Jan
Re: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
Quoted 17 lines · Wei Chen
Quoted 17 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23Review static int __init numa_emulation(u64 start_pfn,
Quoted 1 line · Wei Chen
(quoted diff) +0
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,u64
end_pfn)Review static int __init numa_emulation(u64 start_pfn, · line 250
Quoted 1 line
(quoted diff) +0
-0void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 9 lines
xen/arch/x86/numa.c +1
-0{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif
Quoted 8 lines
Quoted 5 lines
xen/arch/x86/numa.c static int __init numa_emulation(u64 start_pfn, u64 end_pfn) · line 261
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+ start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn);Nit: Would be slightly neater if these were the initializers of the variables.
But if this function returns in
numa_fake&& !numa_emulation, will the twopfn_to_paddroperations be waste?
And what harm would that do?
Quoted 10 lines
Quoted 9 lines
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444
Quoted 6 lines
xen/arch/x86/srat.c +1
-1@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);I don't see the need for a cast here.
Quoted 5 lines
current addr type has been changed to
paddr_t, butpdx_init_maskaccept parameter type is u64. I knowpaddr_tis a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?
It may only ever be an up-cast, yet the compiler would do a widening conversion (according to the usual type conversion rules) for us anyway no matter whether there's a cast. Down-casts (in the general compiler case, i.e. considering a wider set than just gcc and clang) sometimes need making explicit to silence compiler warnings about truncation, but I've not observed any compiler warning when widening values.
Jan
RE: [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure
Hi Jan,
Quoted 30 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 15:55
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 07/37] xen/x86: usepaddr_tfor addresses in NUMA node structureQuoted 20 lines · Wei Chen
Quoted 20 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 23:23Review static int __init numa_emulation(u64 start_pfn,
Quoted 1 line · Wei Chen
(quoted diff) +0
-0@@ -249,24 +250,26 @@ static int __init numa_emulation(u64 start_pfn,u64
end_pfn)Review static int __init numa_emulation(u64 start_pfn, · line 250
Quoted 1 line
(quoted diff) +0
-0void __init numa_initmem_init(unsigned long start_pfn, unsigned long
end_pfn)Quoted 11 lines
xen/arch/x86/numa.c +3
-0{int i; +paddr_t start, end;#ifdef CONFIG_NUMA_EMU if ( numa_fake && !numa_emulation (start_pfn, end_pfn) )return ;#endif + start =pfn_to_paddr (start_pfn); + end =pfn_to_paddr (end_pfn);
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Nit: Would be slightly neater if these were the initializers of the variables.
But if this function returns in
numa_fake&& !numa_emulation, will the twopfn_to_paddroperations be waste?And what harm would that do?
Ok, two or three instructions waste in init time will not take too much harm. I will move them above with initializers.
Quoted 17 lines
Quoted 11 lines
Quoted 10 lines
xen/arch/x86/srat.c void __init srat_parse_regions(u64 addr) · line 444
Quoted 7 lines
xen/arch/x86/srat.c +1
-1@@ -441,7 +441,7 @@ void __init srat_parse_regions(u64 addr) acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) return; - srat_region_mask =pdx_init_mask (addr); + srat_region_mask =pdx_init_mask ((u64 )addr);I don't see the need for a cast here.
Quoted 5 lines
current addr type has been changed to
paddr_t, butpdx_init_maskaccept parameter type is u64. I knowpaddr_tis a typedef of u64 on Arm64/32, or unsinged long on x86. In current stage, their machine byte-lengths are the same. But in case of future changes I think an explicit case here maybe better?
Quoted 7 lines
It may only ever be an up-cast, yet the compiler would do a widening conversion (according to the usual type conversion rules) for us anyway no matter whether there's a cast. Down-casts (in the general compiler case, i.e. considering a wider set than just gcc and clang) sometimes need making explicit to silence compiler warnings about truncation, but I've not observed any compiler warning when widening values.
Ok, I will drop that cast.
Quoted 2 lines
Jan
PATCH 8/37 xen/x86: add detection of discontinous node memory range
In reply to Wei ChenOne NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
--- xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
xen/arch/x86/srat.c +36 -0
@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) pxm, pa->apic_id, node); } +/* +* Check to see if there are other nodes within this node's range. +* We just need to check full contains situation. Because overlaps +* have been checked before by conflicting_memblks. +*/ +static bool __init is_node_memory_continuous (nodeid_t nid, +paddr_t start,paddr_t end) +{ +nodeid_t i; + +struct node *nd = &nodes[nid]; +for_each_node_mask (i, memory_nodes_parsed) + { +/* Skip itself */ +if (i == nid) +continue ; + + nd = &nodes[i]; +if (start < nd->start && nd->end < end) + { +printk (KERN_ERR +"NODE %u: (%" PRIpaddr"-%" PRIpaddr") intertwine with NODE %u (%" PRIpaddr"-%" PRIpaddr")\n" , + nid, start, end, i, nd->start, nd->end); +return false; + } + } + +return true; +} +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma)
@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->start = start;if (nd->end < end) nd->end = end; + +/* Check whether this range contains memory for other nodes */ +if (!is_node_memory_continuous (node, nd->start, nd->end)) { +bad_srat (); +return ; + } } } printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",
Signature · 1 line
2.25.1
Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
CC'ing x86 maintainers
Quoted 7 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
Quoted 8 lines
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000
So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a device_type = "memory" node of a NUMA node in device tree.
Quoted 62 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)xen/arch/x86/srat.c +36
-0@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) pxm, pa->apic_id, node); } +/* +* Check to see if there are other nodes within this node's range. +* We just need to check full contains situation. Because overlaps +* have been checked before by conflicting_memblks. +*/ +static bool __init is_node_memory_continuous (nodeid_t nid, +paddr_t start,paddr_t end) +{ +nodeid_t i; + +struct node *nd = &nodes[nid]; +for_each_node_mask (i, memory_nodes_parsed) + { +/* Skip itself */ +if (i == nid) +continue ; + + nd = &nodes[i]; +if (start < nd->start && nd->end < end) + { +printk (KERN_ERR +"NODE %u: (%" PRIpaddr"-%" PRIpaddr") intertwine with NODE %u (%" PRIpaddr"-%" PRIpaddr")\n" , + nid, start, end, i, nd->start, nd->end); +return false; + } + } + +return true; +} +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma)@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->start = start;if (nd->end < end) nd->end = end; + +/* Check whether this range contains memory for other nodes */ +if (!is_node_memory_continuous (node, nd->start, nd->end)) { +bad_srat (); +return ; + } } } printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n",Signature · 1 line
2.25.1
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 34 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeCC'ing x86 maintainers
Quoted 7 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
Quoted 8 lines
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:
Quoted 10 lines
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a
device_type= "memory" node of a NUMA node in device tree.
Should I need to include/refine above message to commit log?
Quoted 69 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct
Quoted 11 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)xen/arch/x86/srat.c +0
-0@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct
acpi_srat_cpu_affinity*pa)Quoted 15 lines
xen/arch/x86/srat.c +12
-0pxm, pa->apic_id, node); } +/* +* Check to see if there are other nodes within this node's range. +* We just need to check full contains situation. Because overlaps +* have been checked before by conflicting_memblks. +*/ +static bool __init is_node_memory_continuous (nodeid_t nid, +paddr_t start,paddr_t end) +{ +nodeid_t i; + +struct node *nd = &nodes[nid]; +for_each_node_mask (i, memory_nodes_parsed)xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
Quoted 10 lines
xen/arch/x86/srat.c +10
-0+ { + /* Skip itself */ + if (i == nid) + continue; + + nd = &nodes[i]; + if (start < nd->start && nd->end < end) + { + printk(KERN_ERR + "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwinewith NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
Quoted 9 lines
xen/arch/x86/srat.c +8
-0+ nid, start, end, i, nd->start, nd->end); +return false; + } + } + +return true; +} +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 3 lines
xen/arch/x86/srat.c +0
-0void __init acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma)@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 5 lines
xen/arch/x86/srat.c +2
-0nd->start = start;if (nd->end < end) nd->end = end; + +/* Check whether this range contains memory for other nodes */
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379
Quoted 1 line
xen/arch/x86/srat.c +1
-0+if (!is_node_memory_continuous (node, nd->start, nd->end)){
Quoted 9 lines
xen/arch/x86/srat.c +3
-1+bad_srat (); +return ; + } } } printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIpaddr"-%"PRIpaddr"%s\n", --2.25.1
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 37 lines · Wei Chen
Quoted 37 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeCC'ing x86 maintainers
Quoted 7 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
Quoted 8 lines
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but also MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000
Quoted 7 lines
Quoted 5 lines
So I am not familiar with the SRAT format, but I think on ARM the check would look different: we would just look for multiple memory ranges under a
device_type= "memory" node of a NUMA node in device tree.Should I need to
include/refineabove message to commit log?
Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved? An an example:
node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node0: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node1: 0x50000000 - 0x60000000
MMIO : 0x60000000 - 0x80000000
node2: 0x80000000 - 0x90000000
I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?
Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?
node0: 0x0 - 0x10000000
MMIO : 0x10000000 - 0x20000000
node1: 0x20000000 - 0x30000000
MMIO : 0x30000000 - 0x50000000
node0: 0x50000000 - 0x60000000
Because node1 is in between the two ranges of node0?
I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Hi Stefano,
Quoted 60 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 8 lines
Quoted 8 lines
Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeCC'ing x86 maintainers
On Thu, 23 Sep 2021, Wei Chen wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is
obviously
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,
the
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM the
check
Quoted 5 lines
Quoted 4 lines
would look different: we would just look for multiple memory ranges under a
device_type= "memory" node of a NUMA node in device tree.
Quoted 6 lines
Quoted 1 line
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000
MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000
Quoted 3 lines
I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not impossible.
Quoted 4 lines
Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?
Quoted 8 lines
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.
Quoted 5 lines
I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 80 lines · Wei Chen
Quoted 60 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 8 lines
Quoted 8 lines
Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeCC'ing x86 maintainers
On Thu, 23 Sep 2021, Wei Chen wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is
obviously
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,
the
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM the
check
Quoted 5 lines
Quoted 4 lines
would look different: we would just look for multiple memory ranges under a
device_type= "memory" node of a NUMA node in device tree.Quoted 6 lines
Quoted 1 line
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000
MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000
Quoted 5 lines
Quoted 3 lines
I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not impossible.
Definitely, I have seen it before.
Quoted 12 lines
Quoted 11 lines
Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
Quoted 9 lines
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.
Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.
My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check is_node_memory_continuous doesn't seem to be able to catch them.
Quoted 7 lines
Quoted 4 lines
I am asking these questions because it is certainly possible to have multiple memory ranges for each NUMA node in device tree, either by specifying multiple ranges with a single "reg" property, or by specifying multiple memory nodes with the same numa-node-id.
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 124 lines · Stefano Stabellini
Quoted 80 lines · Wei Chen
Quoted 60 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 8 lines
Quoted 8 lines
Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeCC'ing x86 maintainers
On Thu, 23 Sep 2021, Wei Chen wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is
obviously
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine,
the
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved, but
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM the
check
Quoted 5 lines
Quoted 4 lines
would look different: we would just look for multiple memory ranges under a
device_type= "memory" node of a NUMA node in device tree.Quoted 6 lines
Quoted 1 line
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000
MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000Quoted 5 lines
Quoted 3 lines
I assume we can deal with this case simply by setting node0 memory to 0x0-0x30000000 even if there is actually something else, a device, that doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not impossible.
Definitely, I have seen it before.
Quoted 12 lines
Quoted 11 lines
Is it only other nodes' memory interleaved that cause issues? In other words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
Quoted 9 lines
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.
Quoted 7 lines
My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check
is_node_memory_continuousdoesn't seem to be able to catch them.
Looking at numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:
* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.
So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.
I would appreciate if you could add an in-code comment to explain this on top of numa_update_node_memblk.
Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Hi Stefano,
Quoted 172 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinousnode
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
memory range
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection ofdiscontinous node
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
memory range
CC'ing x86 maintainers
Quoted 1 line · Wei Chen
One NUMA node may contain several memory blocks. In current
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
code, Xen will maintain a node memory range for each node to
cover
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
all its memory blocks. But here comes the problem, in the gap
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
one node's two memory blocks, if there are some memory blocks
don't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
belong to this node (remote memory blocks). This node's memory
range
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is
obviously
Quoted 1 line
Quoted 1 line
Quoted 1 line
not very reasonable. This means current NUMA code only can
support
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
node has continous memory blocks. However, on a physical
machine,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory
blocks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for one node. NUMA initializtion will be failed and error
messages
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved,
but
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
check
Quoted 1 line
Quoted 1 line
would look different: we would just look for multiple memory
ranges
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
under a
device_type= "memory" node of a NUMA node in devicetree.
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000
MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000Quoted 1 line
Quoted 1 line
Quoted 1 line
I assume we can deal with this case simply by setting node0 memory
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
0x0-0x30000000 even if there is actually something else, a device,
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not
impossible.
Quoted 5 lines
Definitely, I have seen it before.
Quoted 1 line
Quoted 1 line
Is it only other nodes' memory interleaved that cause issues? In
other
Quoted 33 lines
Quoted 11 lines
Quoted 10 lines
words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
Quoted 9 lines
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.
Quoted 7 lines
My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check
is_node_memory_continuousdoesn't seem to be able to catch them.Looking at
numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.
Quoted 6 lines
So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.
I would appreciate if you could add an in-code comment to explain this on top of
numa_update_node_memblk.
Yes, I will do it.
Quoted 6 lines
Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.
Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1 (0000000880000000-00000008c0000000)
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 172 lines · Wei Chen
Quoted 172 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinousnode
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
memory range
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection ofdiscontinous node
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
memory range
CC'ing x86 maintainers
Quoted 1 line · Wei Chen
One NUMA node may contain several memory blocks. In current
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
code, Xen will maintain a node memory range for each node to
cover
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
all its memory blocks. But here comes the problem, in the gap
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
one node's two memory blocks, if there are some memory blocks
don't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
belong to this node (remote memory blocks). This node's memory
range
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is
obviously
Quoted 1 line
Quoted 1 line
Quoted 1 line
not very reasonable. This means current NUMA code only can
support
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
node has continous memory blocks. However, on a physical
machine,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory
blocks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for one node. NUMA initializtion will be failed and error
messages
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
will be printed when Xen detect such hardware configuration.
At least on ARM, it is not just memory that can be interleaved,
but
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on ARM
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
check
Quoted 1 line
Quoted 1 line
would look different: we would just look for multiple memory
ranges
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
under a
device_type= "memory" node of a NUMA node in devicetree.
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal with cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000
MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000Quoted 1 line
Quoted 1 line
Quoted 1 line
I assume we can deal with this case simply by setting node0 memory
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
0x0-0x30000000 even if there is actually something else, a device,
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not
impossible.
Quoted 5 lines
Definitely, I have seen it before.
Quoted 1 line
Quoted 1 line
Is it only other nodes' memory interleaved that cause issues? In
other
Quoted 33 lines
Quoted 11 lines
Quoted 10 lines
words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000
MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
Quoted 9 lines
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.Yes, I am not too worried about devices with a NUMA id property because they are less common and this series doesn't handle them at all, right? I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is made of multiple banks. I understand that this patch adds an explicit check for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they are still made of multiple discontinuous banks, like in the two example above.
Quoted 7 lines
My question is whether this patch series in its current form can handle the two cases above correctly. If so, I am wondering how it works given that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above, my question is whether it would fail explicitly or not. The new check
is_node_memory_continuousdoesn't seem to be able to catch them.Looking at
numa_update_node_memblks, it is clear that the code is meant to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.
Quoted 8 lines
Quoted 6 lines
So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.
I would appreciate if you could add an in-code comment to explain this on top of
numa_update_node_memblk.Yes, I will do it.
Thank you
Quoted 6 lines
Quoted 5 lines
Have you had a chance to test this? If not it would be fantastic if you could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between in device tree, just for the sake of testing.
Quoted 6 lines
Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1 (0000000880000000-00000008c0000000)
Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Hi Stefano,
Quoted 232 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 1:19
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 20 lines · Wei Chen
Quoted 20 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection ofdiscontinous
Quoted 11 lines
Quoted 11 lines
node
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
memory range
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 1 line
Quoted 1 line
Bertrand Marquis <[email protected]>;
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection ofdiscontinous node
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
memory range
CC'ing x86 maintainers
Quoted 1 line · Wei Chen
One NUMA node may contain several memory blocks. In
current
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
code, Xen will maintain a node memory range for each node
to
Quoted 2 lines
Quoted 2 lines
cover
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
all its memory blocks. But here comes the problem, in the
gap
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
one node's two memory blocks, if there are some memory
blocks
Quoted 2 lines
Quoted 2 lines
don't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
belong to this node (remote memory blocks). This node's
memory
Quoted 4 lines
Quoted 4 lines
range
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this
is
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
obviously
Quoted 1 line
Quoted 1 line
Quoted 1 line
not very reasonable. This means current NUMA code only can
support
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
node has continous memory blocks. However, on a physical
machine,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous
memory
Quoted 4 lines
Quoted 4 lines
blocks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for one node. NUMA initializtion will be failed and error
messages
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will be printed when Xen detect such hardware
configuration.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
At least on ARM, it is not just memory that can be
interleaved,
Quoted 10 lines
Quoted 10 lines
but
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on
ARM
Quoted 9 lines
Quoted 9 lines
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
check
Quoted 1 line
Quoted 1 line
would look different: we would just look for multiple memory
ranges
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
under a
device_type= "memory" node of a NUMA node in devicetree.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal
with
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I assume we can deal with this case simply by setting node0
memory
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
0x0-0x30000000 even if there is actually something else, a
device,
Quoted 32 lines
Quoted 32 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not
impossible.
Quoted 5 lines
Definitely, I have seen it before.
Quoted 1 line
Quoted 1 line
Is it only other nodes' memory interleaved that cause issues? In
other
Quoted 21 lines
Quoted 20 lines
Quoted 10 lines
words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.Quoted 1 line
Quoted 1 line
Quoted 1 line
Yes, I am not too worried about devices with a NUMA id property
because
Quoted 1 line
Quoted 1 line
Quoted 1 line
they are less common and this series doesn't handle them at all,
right?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is
made
Quoted 1 line
Quoted 1 line
Quoted 1 line
of multiple banks. I understand that this patch adds an explicit
check
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they
are
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still made of multiple discontinuous banks, like in the two example above.
My question is whether this patch series in its current form can
handle
Quoted 1 line
Quoted 1 line
Quoted 1 line
the two cases above correctly. If so, I am wondering how it works
given
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above,
my
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
question is whether it would fail explicitly or not. The new check
is_node_memory_continuousdoesn't seem to be able to catchthem.
Quoted 3 lines
Quoted 3 lines
Looking at
numa_update_node_memblks, it is clear that the code ismeant
Quoted 6 lines
Quoted 6 lines
to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:
* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.Quoted 8 lines
Quoted 6 lines
So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.
I would appreciate if you could add an in-code comment to explain this on top of
numa_update_node_memblk.Yes, I will do it.
Thank you
Quoted 1 line
Quoted 1 line
Have you had a chance to test this? If not it would be fantastic if
you
Quoted 3 lines
Quoted 3 lines
could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between
in
Quoted 5 lines
Quoted 1 line
device tree, just for the sake of testing.
Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.
Quoted 8 lines
Quoted 3 lines
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1(0000000880000000-00000008c0000000)
Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?
Yes, I am using a device tree setting like this:
memory@80000000 {
device_type = "memory";
reg = <0x0 0x80000000 0x0 0x80000000>;
numa-node-id = <0>;
};
memory@880000000 {
device_type = "memory";
reg = <0x8 0x80000000 0x0 0x8000000>;
numa-node-id = <0>;
};
memory@890000000 {
device_type = "memory";
reg = <0x8 0x90000000 0x0 0x8000000>;
numa-node-id = <0>;
};
memory@8A0000000 {
device_type = "memory";
reg = <0x8 0xA0000000 0x0 0x8000000>;
numa-node-id = <0>;
};
memory@8B0000000 {
device_type = "memory";
reg = <0x8 0xB0000000 0x0 0x8000000>;
numa-node-id = <0>;
};
memory@8C0000000 {
device_type = "memory";
reg = <0x8 0xC0000000 0x0 0x8000000>;
numa-node-id = <1>;
};
memory@8D0000000 {
device_type = "memory";
reg = <0x8 0xD0000000 0x0 0x8000000>;
numa-node-id = <1>;
};
memory@8E0000000 {
device_type = "memory";
reg = <0x8 0xE0000000 0x0 0x8000000>;
numa-node-id = <1>;
};
memory@8F0000000 {
device_type = "memory";
reg = <0x8 0xF0000000 0x0 0x8000000>;
numa-node-id = <1>;
};And in Xen we got the output:
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000 (XEN) SRAT: Node 0 0000000880000000-0000000888000000 (XEN) SRAT: Node 0 0000000890000000-0000000898000000 (XEN) SRAT: Node 0 00000008a0000000-00000008a8000000 (XEN) SRAT: Node 0 00000008b0000000-00000008b8000000 (XEN) SRAT: Node 1 00000008c0000000-00000008c8000000 (XEN) SRAT: Node 1 00000008d0000000-00000008d8000000 (XEN) SRAT: Node 1 00000008e0000000-00000008e8000000 (XEN) SRAT: Node 1 00000008f0000000-00000008f8000000 (XEN) NUMA: parsing numa-distance-map
(XEN) NUMA: distance: NODE#0->NODE#0:10 (XEN) NUMA: distance: NODE#0->NODE#1:20 (XEN) NUMA: distance: NODE#1->NODE#1:10 (XEN) NUMA: Using 16 for the hash shift. (XEN) Domain heap initialised
(XEN) Booting using Device Tree
Dom0 can be boot successfully, xl info got:
xl info
host : X-Dom0
release : 5.12.0
version : #20 SMP PREEMPT Wed Jul 28 13:41:28 CST 2021
machine : aarch64nr_cpus : 4max_cpu_id : 3nr_nodes : 2cores_per_socket : 1threads_per_core : 1
Xen debug console to dump numa info, we got:
(XEN) 'u' pressed -> dumping numa info (now = 13229372281010) (XEN) NODE0 start->524288 size->8617984 free->388741 (XEN) NODE1 start->9175040 size->229376 free->106460 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 237 lines · Wei Chen
Hi Stefano,
Quoted 235 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 1:19
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 20 lines · Wei Chen
Quoted 20 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:05
To: Stefano Stabellini <[email protected]>
Cc: Wei Chen <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 8 lines · Stefano Stabellini
Quoted 7 lines · Wei Chen
Quoted 7 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月25日 3:53
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected];Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]
Subject: RE: [PATCH 08/37] xen/x86: add detection ofdiscontinous
Quoted 11 lines
Quoted 11 lines
node
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
memory range
Quoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 8:26
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 1 line
Quoted 1 line
Bertrand Marquis <[email protected]>;
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
[email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection ofdiscontinous node
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
memory range
CC'ing x86 maintainers
Quoted 1 line · Wei Chen
One NUMA node may contain several memory blocks. In
current
Quoted 2 lines
Quoted 2 lines
Xen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
code, Xen will maintain a node memory range for each node
to
Quoted 2 lines
Quoted 2 lines
cover
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
all its memory blocks. But here comes the problem, in the
gap
Quoted 2 lines
Quoted 2 lines
of
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
one node's two memory blocks, if there are some memory
blocks
Quoted 2 lines
Quoted 2 lines
don't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
belong to this node (remote memory blocks). This node's
memory
Quoted 4 lines
Quoted 4 lines
range
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this
is
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
obviously
Quoted 1 line
Quoted 1 line
Quoted 1 line
not very reasonable. This means current NUMA code only can
support
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
node has continous memory blocks. However, on a physical
machine,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous
memory
Quoted 4 lines
Quoted 4 lines
blocks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
for one node. NUMA initializtion will be failed and error
messages
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
will be printed when Xen detect such hardware
configuration.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
At least on ARM, it is not just memory that can be
interleaved,
Quoted 10 lines
Quoted 10 lines
but
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
also
Quoted 8 lines
Quoted 8 lines
MMIO regions. For instance:
node0 bank0 0-0x1000000
MMIO 0x1000000-0x1002000
Hole 0x1002000-0x2000000
node0 bank1 0x2000000-0x3000000So I am not familiar with the SRAT format, but I think on
ARM
Quoted 9 lines
Quoted 9 lines
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
check
Quoted 1 line
Quoted 1 line
would look different: we would just look for multiple memory
ranges
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
under a
device_type= "memory" node of a NUMA node in devicetree.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Should I need to
include/refineabove message to commit log?Let me ask you a question first.
With the NUMA implementation of this patch series, can we deal
with
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
cases where each node has multiple memory banks, not interleaved?
Yes.
Quoted 11 lines
An an example:
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node0:0x20000000-0x30000000MMIO :0x30000000-0x50000000
node1:0x50000000-0x60000000MMIO :0x60000000-0x80000000
node2:0x80000000-0x90000000Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I assume we can deal with this case simply by setting node0
memory
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
0x0-0x30000000 even if there is actually something else, a
device,
Quoted 32 lines
Quoted 32 lines
that
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
doesn't belong to node0 in between the two node0 banks?
While this configuration is rare in SoC design, but it is not
impossible.
Quoted 5 lines
Definitely, I have seen it before.
Quoted 1 line
Quoted 1 line
Is it only other nodes' memory interleaved that cause issues? In
other
Quoted 21 lines
Quoted 20 lines
Quoted 10 lines
words, only the following is a problematic scenario?
node0:
0x0-0x10000000
MMIO :0x10000000-0x20000000
node1:0x20000000-0x30000000MMIO :0x30000000-0x50000000
node0:0x50000000-0x60000000Because node1 is in between the two ranges of node0?
But only device_type="memory" can be added to allocation. For mmio there are two cases:
1. mmio doesn't have NUMA id property.
2. mmio has NUMA id property, just like some PCIe controllers. But we don’t need to handle these kinds of MMIO devices in memory block parsing. Because we don't need to allocate memory from these mmio ranges. And for accessing, we need a NUMA-aware PCIe controller driver or a generic NUMA-aware MMIO accessing APIs.Quoted 1 line
Quoted 1 line
Quoted 1 line
Yes, I am not too worried about devices with a NUMA id property
because
Quoted 1 line
Quoted 1 line
Quoted 1 line
they are less common and this series doesn't handle them at all,
right?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I imagine they would be treated like any other device without NUMA awareness.
I am thinking about the case where the memory of each NUMA node is
made
Quoted 1 line
Quoted 1 line
Quoted 1 line
of multiple banks. I understand that this patch adds an explicit
check
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
for cases where these banks are interleaving, however there are many other cases where NUMA memory nodes are *not* interleaving but they
are
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still made of multiple discontinuous banks, like in the two example above.
My question is whether this patch series in its current form can
handle
Quoted 1 line
Quoted 1 line
Quoted 1 line
the two cases above correctly. If so, I am wondering how it works
given
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that we only have a single "start" and "size" parameter per node.
On the other hand if this series cannot handle the two cases above,
my
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
question is whether it would fail explicitly or not. The new check
is_node_memory_continuousdoesn't seem to be able to catchthem.
Quoted 3 lines
Quoted 3 lines
Looking at
numa_update_node_memblks, it is clear that the code ismeant
Quoted 6 lines
Quoted 6 lines
to increase the range of each numa node to cover even MMIO regions in between memory banks. Also see the comment at the top of the file:
* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.Quoted 8 lines
Quoted 6 lines
So if there are multiple banks for each node, start and end are stretched to cover the holes between them, and it works as long as memory banks of different NUMA nodes don't interleave.
I would appreciate if you could add an in-code comment to explain this on top of
numa_update_node_memblk.Yes, I will do it.
Thank you
Quoted 1 line
Quoted 1 line
Have you had a chance to test this? If not it would be fantastic if
you
Quoted 3 lines
Quoted 3 lines
could give it a quick test to make sure it works as intended: for instance by creating multiple memory banks for each NUMA node by splitting an real bank into two smaller banks with a hole in between
in
Quoted 8 lines
Quoted 1 line
device tree, just for the sake of testing.
Yes, I have created some fake NUMA nodes in FVP device tree to test it. The intertwine of nodes' address can be detected.
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000
(XEN) SRAT: Node 1 0000000880000000-00000008c0000000
(XEN) NODE 0: (0000000080000000-00000008d0000000) intertwine with NODE 1
Quoted 7 lines
Quoted 5 lines
(0000000880000000-00000008c0000000)
Great thanks! And what if there are multiple non-contiguous memory banks per node, but *not* intertwined. Does that all work correctly as expected?
Yes, I am using a device tree setting like this:
Perfect! Thank you!
Quoted 102 lines
memory@80000000 { device_type = "memory"; reg = <0x0 0x80000000 0x0 0x80000000>; numa-node-id = <0>; }; memory@880000000 { device_type = "memory"; reg = <0x8 0x80000000 0x0 0x8000000>; numa-node-id = <0>; }; memory@890000000 { device_type = "memory"; reg = <0x8 0x90000000 0x0 0x8000000>; numa-node-id = <0>; }; memory@8A0000000 { device_type = "memory"; reg = <0x8 0xA0000000 0x0 0x8000000>; numa-node-id = <0>; }; memory@8B0000000 { device_type = "memory"; reg = <0x8 0xB0000000 0x0 0x8000000>; numa-node-id = <0>; }; memory@8C0000000 { device_type = "memory"; reg = <0x8 0xC0000000 0x0 0x8000000>; numa-node-id = <1>; }; memory@8D0000000 { device_type = "memory"; reg = <0x8 0xD0000000 0x0 0x8000000>; numa-node-id = <1>; }; memory@8E0000000 { device_type = "memory"; reg = <0x8 0xE0000000 0x0 0x8000000>; numa-node-id = <1>; }; memory@8F0000000 { device_type = "memory"; reg = <0x8 0xF0000000 0x0 0x8000000>; numa-node-id = <1>; };And in Xen we got the output:
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 0 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) DT: NUMA node 1 processor parsed
(XEN) SRAT: Node 0 0000000080000000-00000000ff000000 (XEN) SRAT: Node 0 0000000880000000-0000000888000000 (XEN) SRAT: Node 0 0000000890000000-0000000898000000 (XEN) SRAT: Node 0 00000008a0000000-00000008a8000000 (XEN) SRAT: Node 0 00000008b0000000-00000008b8000000 (XEN) SRAT: Node 1 00000008c0000000-00000008c8000000 (XEN) SRAT: Node 1 00000008d0000000-00000008d8000000 (XEN) SRAT: Node 1 00000008e0000000-00000008e8000000 (XEN) SRAT: Node 1 00000008f0000000-00000008f8000000 (XEN) NUMA: parsing numa-distance-map
(XEN) NUMA: distance: NODE#0->NODE#0:10 (XEN) NUMA: distance: NODE#0->NODE#1:20 (XEN) NUMA: distance: NODE#1->NODE#1:10 (XEN) NUMA: Using 16 for the hash shift. (XEN) Domain heap initialised
(XEN) Booting using Device TreeDom0 can be boot successfully, xl info got:
xl info
host : X-Dom0
release : 5.12.0
version : #20 SMP PREEMPT Wed Jul 28 13:41:28 CST 2021
machine : aarch64nr_cpus: 4max_cpu_id: 3nr_nodes: 2cores_per_socket: 1threads_per_core: 1Xen debug console to dump numa info, we got:
(XEN) 'u' pressed -> dumping numa info (now = 13229372281010) (XEN) NODE0 start->524288 size->8617984 free->388741 (XEN) NODE1 start->9175040 size->229376 free->106460 (XEN) CPU0...1 -> NODE0
(XEN) CPU2...3 -> NODE1
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 262144):
(XEN) Node 0: 262144
(XEN) Node 1: 0
Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
In reply to Wei ChenQuoted 7 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
Quoted 8 lines
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
Luckily what you actually check for isn't as strict as "discontinuous":
What you're after is no interleaving of memory. A single nod can still have multiple discontiguous ranges (and that'll often be the case on x86). Please adjust description and function name accordingly.
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 280
Quoted 13 lines
xen/arch/x86/srat.c +7
-0@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) pxm, pa->apic_id, node); } +/* +* Check to see if there are other nodes within this node's range. +* We just need to check full contains situation. Because overlaps +* have been checked before by conflicting_memblks. +*/ +static bool __init is_node_memory_continuous (nodeid_t nid, +paddr_t start,paddr_t end)
This indentation style demands indenting like ...
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 282
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+{ +nodeid_t i;
... variable declarations at function scope, i.e. in a Linux-style file by a tab.
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 285
Quoted 3 lines
xen/arch/x86/srat.c +3
-0+ +struct node *nd = &nodes[nid]; +for_each_node_mask (i, memory_nodes_parsed)
Please move the blank line to be between declarations and statements.
Also please make nd pointer-to const.
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 286
Quoted 1 line
xen/arch/x86/srat.c +1
-0+ {
In a Linux-style file opening braces do not belong on their own lines.
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 295
Quoted 9 lines
xen/arch/x86/srat.c +9
-0+/* Skip itself */ +if (i == nid) +continue ; + + nd = &nodes[i]; +if (start < nd->start && nd->end < end) + { + printk(KERN_ERR + "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwine with NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
s/intertwine/interleaves/ ?
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 382
Quoted 10 lines
xen/arch/x86/srat.c +6
-0@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) nd->start = start;if (nd->end < end) nd->end = end; + +/* Check whether this range contains memory for other nodes */ +if (!is_node_memory_continuous (node, nd->start, nd->end)) { +bad_srat (); +return ; + }
I think this check would better come before nodes[] gets updated? Otoh bad_srat() will zap everything anyway ...
Jan
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Hi Jan,
Quoted 9 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 13 lines
Quoted 12 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...
Quoted 3 lines
Quoted 3 lines
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...
Quoted 6 lines
Luckily what you actually check for isn't as strict as "discontinuous":
What you're after is no interleaving of memory. A single nod can still have multiple discontiguous ranges (and that'll often be the case on x86). Please adjust description and function name accordingly.
Yes, we're checking for no interlaced memory among nodes. In one node's memory range, the memory block still can be discontinuous.
I will rename the subject to:
"add detection of interlaced memory for different nodes" And I would rename is_node_memory_continuous to:node_without_interleave_memory.
Quoted 7 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct
Quoted 3 lines
xen/arch/x86/srat.c +0
-0@@ -271,6 +271,36 @@ acpi_numa_processor_affinity_init(const struct
acpi_srat_cpu_affinity*pa)Quoted 3 lines
xen/arch/x86/srat.c +0
-0pxm, pa->apic_id, node); }
Quoted 10 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 280
Quoted 7 lines
xen/arch/x86/srat.c +7
-0+/* +* Check to see if there are other nodes within this node's range. +* We just need to check full contains situation. Because overlaps +* have been checked before by conflicting_memblks. +*/ +static bool __init is_node_memory_continuous (nodeid_t nid, +paddr_t start,paddr_t end)This indentation style demands indenting like ...
Ok.
Quoted 6 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 282
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+{ +nodeid_t i;... variable declarations at function scope, i.e. in a Linux-style file by a tab.
Quoted 7 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) · line 285
Quoted 3 lines
xen/arch/x86/srat.c +3
-0+ +struct node *nd = &nodes[nid]; +for_each_node_mask (i, memory_nodes_parsed)Please move the blank line to be between declarations and statements.
Also please make nd pointer-to const.
Ok.
Quoted 5 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct · line 286
Quoted 2 lines
xen/arch/x86/srat.c +1
-0+ {In a Linux-style file opening braces do not belong on their own lines.
Ok.
Quoted 11 lines
xen/arch/x86/srat.c acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
Quoted 9 lines
xen/arch/x86/srat.c +9
-0+/* Skip itself */ +if (i == nid) +continue ; + + nd = &nodes[i]; + if (start < nd->start && nd->end < end) + { + printk(KERN_ERR + "NODE %u: (%"PRIpaddr"-%"PRIpaddr") intertwinewith NODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
Quoted 1 line
s/intertwine/interleaves/ ?
Yes, interleaves. I will fix it.
Quoted 9 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 2 lines
xen/arch/x86/srat.c +0
-0@@ -344,6 +374,12 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 5 lines
xen/arch/x86/srat.c +2
-0nd->start = start;if (nd->end < end) nd->end = end; + +/* Check whether this range contains memory for other nodes */
Quoted 8 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379
Quoted 1 line
xen/arch/x86/srat.c +1
-0+if (!is_node_memory_continuous (node, nd->start, nd->end)){
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 382
Quoted 3 lines
xen/arch/x86/srat.c +3
-0+bad_srat (); +return ; + }I think this check would better come before nodes[] gets updated? Otoh
bad_srat()will zap everything anyway ...
Yes, when I wrote this check, I considered when the check was failed, bad_srat would make numa initialization be failed. The values in nodes[] will not take any effect. So I didn't adjust the order. But if the bad_srat will be changed in future, and nodes[] will be used in fallback progress, this will take more effort to debug. In this case, I agree with you, I will update the order in next version.
Thanks,
Wei Chen
Quoted 2 lines
Jan
Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Quoted 21 lines · Wei Chen
Quoted 16 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13Quoted 12 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...
Taking just this part of your reply (the issue continues later), may I ask that you use a consistent term throughout this single patch? Mixing "interlace" and "interleave" like you do may make people wonder whether the two are intended to express slightly different aspects. Personally, as per my suggestion, I'd prefer "interleave", but I'm not a native speaker.
Jan
RE: [PATCH 08/37] xen/x86: add detection of discontinous node memory range
Hi Jan,
Quoted 39 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 16:01
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 08/37] xen/x86: add detection of discontinous node memory rangeQuoted 21 lines · Wei Chen
Quoted 16 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月19日 0:13Quoted 12 lines · Wei Chen
One NUMA node may contain several memory blocks. In current Xen code, Xen will maintain a node memory range for each node to cover all its memory blocks. But here comes the problem, in the gap of one node's two memory blocks, if there are some memory blocks don't belong to this node (remote memory blocks). This node's memory range will be expanded to cover these remote memory blocks.
One node's memory range contains othe nodes' memory, this is obviously not very reasonable. This means current NUMA code only can support node has continous memory blocks. However, on a physical machine, the addresses of multiple nodes can be interleaved.
I will adjust above paragraph to:
... This means current NUMA code only can support node has no interlaced memory blocks. ...Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
So in this patch, we add code to detect discontinous memory blocks for one node. NUMA initializtion will be failed and error messages will be printed when Xen detect such hardware configuration.
I will adjust above paragraph to:
So in this patch, we add code to detect interleave of different nodes' memory blocks. NUMA initialization will be ...
Quoted 7 lines
Taking just this part of your reply (the issue continues later), may I ask that you use a consistent term throughout this single patch? Mixing "interlace" and "interleave" like you do may make people wonder whether the two are intended to express slightly different aspects. Personally, as per my suggestion, I'd prefer "interleave", but I'm not a native speaker.
Sorry, I am not a native speaker too, I had checked dict for interlaced before I used it. https://www.merriam-webster.com/thesaurus/interlaced
Obviously, I'm probably using it incorrectly and making it harder to understand, I will use "interleave" in my patches.
Thanks,
Wei Chen
Quoted 1 line
Jan
PATCH 9/37 xen/x86: introduce two helpers to access memory hotplug end
In reply to Wei Chenx86 provides a mem_hotplug to maintain the end of memory hotplug end address. This variable can be accessed out of mm.c. We want some code out of mm.c can be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replace mem_hotplug direct access. This will give the ability to stub these two API.
--- xen/include/asm-x86/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
xen/include/asm-x86/mm.h +10 -0
@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,extern paddr_t mem_hotplug; +static inline void mem_hotplug_update_boundary (paddr_t end) +{ + mem_hotplug = end; +} + +static inline paddr_t mem_hotplug_boundary (void ) +{ +return mem_hotplug; +} +/****************************************************************************** * With shadow pagetables, the different kinds of address start * to get get confusing.
Signature · 1 line
2.25.1
Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end
+x86 maintainers
Quoted 1 line · Wei Chen
x86 provides a
mem_hotplugto maintain the end of memory hotplug
^ variable
Quoted 2 lines
end address. This variable can be accessed out of
mm.c. We want some code out ofmm.ccan be reused by other architectures without
^ so that it can be reused
Quoted 3 lines
memory hotplug ability. So in this patch, we introduce these two helpers to replace
mem_hotplugdirect access. This will give the ability to stub these two API.
^ APIs
Quoted 29 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/include/asm-x86/mm.h +10
-0@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,extern paddr_t mem_hotplug; +static inline void mem_hotplug_update_boundary (paddr_t end) +{ + mem_hotplug = end; +} + +static inline paddr_t mem_hotplug_boundary (void ) +{ +return mem_hotplug; +} +/****************************************************************************** * With shadow pagetables, the different kinds of address start * to get get confusing. Signature · 1 line
2.25.1
Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end
Quoted 6 lines · Stefano Stabellini
+x86 maintainers
Quoted 1 line · Wei Chen
x86 provides a
mem_hotplugto maintain the end of memory hotplug^ variable
Quoted 10 lines
Quoted 2 lines
end address. This variable can be accessed out of
mm.c. We want some code out ofmm.ccan be reused by other architectures without^ so that it can be reused
Quoted 3 lines
memory hotplug ability. So in this patch, we introduce these two helpers to replace
mem_hotplugdirect access. This will give the ability to stub these two API.^ APIs
OK
Quoted 29 lines
Quoted 29 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/include/asm-x86/mm.h +10
-0@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,extern paddr_t mem_hotplug; +static inline void mem_hotplug_update_boundary (paddr_t end) +{ + mem_hotplug = end; +} + +static inline paddr_t mem_hotplug_boundary (void ) +{ +return mem_hotplug; +} +/****************************************************************************** * With shadow pagetables, the different kinds of address start * to get get confusing. Signature · 1 line
2.25.1
Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end
In reply to Wei ChenQuoted 23 lines · Wei Chen
x86 provides a
mem_hotplugto maintain the end of memory hotplug end address. This variable can be accessed out ofmm.c. We want some code out ofmm.ccan be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replacemem_hotplugdirect access. This will give the ability to stub these two API.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/include/asm-x86/mm.h +3
-0@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct page_info *page,extern paddr_t mem_hotplug; +static inline void mem_hotplug_update_boundary (paddr_t end) +{ + mem_hotplug = end;
xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page, · line 490
Quoted 10 lines
xen/include/asm-x86/mm.h +7
-0+} + +static inline paddr_t mem_hotplug_boundary (void ) +{ +return mem_hotplug; +} +/****************************************************************************** * With shadow pagetables, the different kinds of address start * to get get confusing.
Imo for this to make sense you want to also use the new functions right away in the place(s) where the direct access(es) get(s) in your way.
Jan
RE: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end
Hi Jan,
Quoted 44 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:25
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug endxen/include/asm-x86/mm.h static inline int get_page_and_type(struct
Quoted 17 lines · Wei Chen
x86 provides a
mem_hotplugto maintain the end of memory hotplug end address. This variable can be accessed out ofmm.c. We want some code out ofmm.ccan be reused by other architectures without memory hotplug ability. So in this patch, we introduce these two helpers to replacemem_hotplugdirect access. This will give the ability to stub these two API.Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-x86/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)xen/include/asm-x86/mm.h +0
-0@@ -475,6 +475,16 @@ static inline int get_page_and_type(struct
page_info*page,Quoted 4 lines
xen/include/asm-x86/mm.h +1
-0extern paddr_t mem_hotplug; +static inline void mem_hotplug_update_boundary (paddr_t end)xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page,
Quoted 10 lines
xen/include/asm-x86/mm.h +9
-0+{ + mem_hotplug = end; +} + +static inline paddr_t mem_hotplug_boundary (void ) +{ +return mem_hotplug; +} +/*************************************************************************
*****
Quoted 7 lines
xen/include/asm-x86/mm.h static inline int get_page_and_type(struct page_info *page, · line 490
Quoted 2 lines
xen/include/asm-x86/mm.h +0
-0* With shadow pagetables, the different kinds of address start * to get get confusing. Imo for this to make sense you want to also use the new functions right away in the place(s) where the direct access(es) get(s) in your way.
OK, I will merge this patch with the patch that will use these functions firstly.
Quoted 1 line
Jan
PATCH 10/37 xen/x86: use helpers to access/update mem_hotplug
In reply to Wei ChenWe want to abstract code from acpi_numa_memory_affinity_init. But mem_hotplug is coupled with x86. In this patch, we use helpers to repace mem_hotplug direct accessing. This will allow most code can be common.
--- xen/arch/x86/srat.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +2 -2
@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) memblk_nodeid[num_node_memblks] = node;if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) {__set_bit (num_node_memblks, memblk_hotplug); -if (end > mem_hotplug) - mem_hotplug = end; +if (end >mem_hotplug_boundary ()) +mem_hotplug_update_boundary (end); } num_node_memblks++; }
Signature · 1 line
2.25.1
Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug
+x86 maintainers
Quoted 3 lines · Wei Chen
We want to abstract code from
acpi_numa_memory_affinity_init. Butmem_hotplugis coupled with x86. In this patch, we use helpers to repacemem_hotplugdirect accessing. This will
^ replace
Quoted 1 line
allow most code can be common.
^ to be
I think this patch could be merged with the previous patch
Quoted 23 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)xen/arch/x86/srat.c +2
-2@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) memblk_nodeid[num_node_memblks] = node;if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) {__set_bit (num_node_memblks, memblk_hotplug); -if (end > mem_hotplug) - mem_hotplug = end; +if (end >mem_hotplug_boundary ()) +mem_hotplug_update_boundary (end); } num_node_memblks++; }Signature · 1 line
2.25.1
RE: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug
Quoted 20 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Stefano Stabellini
Sent: 2021年9月24日 8:32
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 10/37] xen/x86: use helpers to access/updatemem_hotplug+x86 maintainers
Quoted 3 lines · Wei Chen
We want to abstract code from
acpi_numa_memory_affinity_init. Butmem_hotplugis coupled with x86. In this patch, we use helpers to repacemem_hotplugdirect accessing. This will
Quoted 7 lines
^ replace
Quoted 1 line
allow most code can be common.
^ to be
I think this patch could be merged with the previous patch
Ok, I will do it, and fix above typos
Quoted 25 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 11 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)xen/arch/x86/srat.c +0
-0@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)Quoted 13 lines
xen/arch/x86/srat.c +2
-3memblk_nodeid[num_node_memblks] = node;if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) {__set_bit (num_node_memblks, memblk_hotplug); -if (end > mem_hotplug) - mem_hotplug = end; +if (end >mem_hotplug_boundary ()) +mem_hotplug_update_boundary (end); } num_node_memblks++; } --2.25.1
Re: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug
In reply to Wei Chenxen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395
Quoted 10 lines · Wei Chen
xen/arch/x86/srat.c +2
-2@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) memblk_nodeid[num_node_memblks] = node;if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) {__set_bit (num_node_memblks, memblk_hotplug); -if (end > mem_hotplug) - mem_hotplug = end; +if (end >mem_hotplug_boundary ()) +mem_hotplug_update_boundary (end);
Can the if() please be folded into mem_hotplug_update_boundary(), eliminating (at least for the purpose here) the need for the separate mem_hotplug_boundary()? As said on the previous patch, I think the two want folding.
Jan
RE: [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug
Hi Jan,
Quoted 14 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:30
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 10/37] xen/x86: use helpers to access/updatemem_hotplugxen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 3 lines · Wei Chen
xen/arch/x86/srat.c +0
-0@@ -391,8 +391,8 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)
Quoted 13 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395
Quoted 7 lines
xen/arch/x86/srat.c +2
-2memblk_nodeid[num_node_memblks] = node;if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) {__set_bit (num_node_memblks, memblk_hotplug); -if (end > mem_hotplug) - mem_hotplug = end; +if (end >mem_hotplug_boundary ()) +mem_hotplug_update_boundary (end);Can the
if()please be folded intomem_hotplug_update_boundary(), eliminating (at least for the purpose here) the need for the separatemem_hotplug_boundary()? As said on the previous patch, I think the two want folding.
Yes, these changes would be more make sense. I will do it in next version.
Quoted 1 line
Jan
PATCH 11/37 xen/x86: abstract neutral code from acpi_numa_memory_affinity_init
In reply to Wei ChenThere is some code in acpi_numa_memory_affinity_init to update node memory range and update node_memblk_range array. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.
So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.
--- xen/arch/x86/srat.c | 131 +++++++++++++++++++++---------------- xen/include/asm-x86/numa.h | 3 + 2 files changed, 77 insertions(+), 57 deletions(-)
xen/arch/x86/srat.c +74 -57
@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)return node; } +bool __init numa_memblks_available (void ) +{ +if (num_node_memblks <NR_NODE_MEMBLKS ) +return true; + +return false; +} +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;
@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,return true; } -/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ -void __init -acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +/* Neutral NUMA memory affinity init function for ACPI and DT */ +int __init numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug) { -paddr_t start, end; -unsigned pxm; -nodeid_t node; +paddr_t end = start + size;int i; -if (srat_disabled ()) -return ; -if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); -return ; - } -if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) -return ; - - start = ma->base_address; - end = start + ma->length; -/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); - -if (num_node_memblks >=NR_NODE_MEMBLKS ) - { -dprintk (XENLOG_WARNING , -"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); -bad_srat (); -return ; - } - - pxm = ma->proximity_domain; -if (srat_rev <2 ) - pxm &=0xff ; - node =setup_node (pxm); -if (node ==NUMA_NO_NODE ) { -bad_srat (); -return ; - } -/* It is fine to add this area to the nodes data it will be used later*/ +/* It is fine to add this area to the nodes data it will be used later */ i =conflicting_memblks (start, end);if (i <0 )/* everything fine */ ;else if (memblk_nodeid[i] == node) { -bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != - !test_bit (i, memblk_hotplug); +bool mismatch = !hotplug != !test_bit (i, memblk_hotplug); -printk ("%sSRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , - mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, +printk ("%sSRAT: NODE %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , + mismatch ?KERN_ERR :KERN_WARNING , node, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) { -bad_srat (); -return ; +return -1 ; } }else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , - pxm, start, end,node_to_pxm (memblk_nodeid[i]), +"SRAT: NODE %u (%" PRIpaddr"-%" PRIpaddr") overlaps with NODE %u (%" PRIpaddr"-%" PRIpaddr")\n" , + node, start, end, memblk_nodeid[i], node_memblk_range[i].start, node_memblk_range[i].end); -bad_srat (); -return ; +return -1 ; } -if (!(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE )) { + +if (!hotplug) {struct node *nd = &nodes[node];if (!node_test_and_set (node, memory_nodes_parsed)) {
@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)if (nd->end < end) nd->end = end; -/* Check whether this range contains memory for other nodes */ -if (!is_node_memory_continuous (node, nd->start, nd->end)) { -bad_srat (); -return ; - } +if (!is_node_memory_continuous (node, nd->start, nd->end)) +return -1 ; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , - node, pxm, start, end, - ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" ); + +printk (KERN_INFO "SRAT: Node %u %" PRIpaddr"-%" PRIpaddr"%s\n" , + node, start, end, hotplug ?" (hotplug)" :"" ); node_memblk_range[num_node_memblks].start = start; node_memblk_range[num_node_memblks].end = end; memblk_nodeid[num_node_memblks] = node; -if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) { +if (hotplug) {__set_bit (num_node_memblks, memblk_hotplug);if (end >mem_hotplug_boundary ())mem_hotplug_update_boundary (end); } num_node_memblks++; + +return 0 ; +} + +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ +void __init +acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +{ +unsigned pxm; +nodeid_t node; +int ret; + +if (srat_disabled ()) +return ; +if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { +bad_srat (); +return ; + } +if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) +return ; + +/* Supplement the heuristics in l1tf_calculations(). */ + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP ((ma->base_address + ma->length),PAGE_SIZE )); + +if (!numa_memblks_available ()) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return ; + } + + pxm = ma->proximity_domain; +if (srat_rev <2 ) + pxm &=0xff ; + node =setup_node (pxm); +if (node ==NUMA_NO_NODE ) { +bad_srat (); +return ; + } + + ret =numa_update_node_memblks (node, ma->base_address, ma->length, + ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ); +if (ret !=0 ) +bad_srat (); }/* Sanity check to catch more bad SRATs (they are amazingly common).
xen/include/asm-x86/numa.h +3 -0
@@ -74,6 +74,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) NODE_DATA(nid)->node_spanned_pages)extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); +extern bool numa_memblks_available (void ); +extern int numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug);void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);
Signature · 1 line
2.25.1
Re: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init
+x86 maintainers
Quoted 215 lines · Wei Chen
There is some code in
acpi_numa_memory_affinity_initto update node memory range and updatenode_memblk_rangearray. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like
node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 131 +++++++++++++++++++++---------------- xen/include/asm-x86/numa.h | 3 + 2 files changed, 77 insertions(+), 57 deletions(-)xen/arch/x86/srat.c +74
-57@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)return node; } +bool __init numa_memblks_available (void ) +{ +if (num_node_memblks <NR_NODE_MEMBLKS ) +return true; + +return false; +} +int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node) {int i;@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,return true; } -/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ -void __init -acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +/* Neutral NUMA memory affinity init function for ACPI and DT */ +int __init numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug) { -paddr_t start, end; -unsigned pxm; -nodeid_t node; +paddr_t end = start + size;int i; -if (srat_disabled ()) -return ; -if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); -return ; - } -if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) -return ; - - start = ma->base_address; - end = start + ma->length; -/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); - -if (num_node_memblks >=NR_NODE_MEMBLKS ) - { -dprintk (XENLOG_WARNING , -"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); -bad_srat (); -return ; - } - - pxm = ma->proximity_domain; -if (srat_rev <2 ) - pxm &=0xff ; - node =setup_node (pxm); -if (node ==NUMA_NO_NODE ) { -bad_srat (); -return ; - } -/* It is fine to add this area to the nodes data it will be used later*/ +/* It is fine to add this area to the nodes data it will be used later */ i =conflicting_memblks (start, end);if (i <0 )/* everything fine */ ;else if (memblk_nodeid[i] == node) { -bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != - !test_bit (i, memblk_hotplug); +bool mismatch = !hotplug != !test_bit (i, memblk_hotplug); -printk ("%sSRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , - mismatch ?KERN_ERR :KERN_WARNING , pxm, start, end, +printk ("%sSRAT: NODE %u (%" PRIpaddr"-%" PRIpaddr") overlaps with itself (%" PRIpaddr"-%" PRIpaddr")\n" , + mismatch ?KERN_ERR :KERN_WARNING , node, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) { -bad_srat (); -return ; +return -1 ; } }else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , - pxm, start, end,node_to_pxm (memblk_nodeid[i]), +"SRAT: NODE %u (%" PRIpaddr"-%" PRIpaddr") overlaps with NODE %u (%" PRIpaddr"-%" PRIpaddr")\n" , + node, start, end, memblk_nodeid[i], node_memblk_range[i].start, node_memblk_range[i].end); -bad_srat (); -return ; +return -1 ; } -if (!(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE )) { + +if (!hotplug) {struct node *nd = &nodes[node];if (!node_test_and_set (node, memory_nodes_parsed)) {@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)if (nd->end < end) nd->end = end; -/* Check whether this range contains memory for other nodes */ -if (!is_node_memory_continuous (node, nd->start, nd->end)) { -bad_srat (); -return ; - } +if (!is_node_memory_continuous (node, nd->start, nd->end)) +return -1 ; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , - node, pxm, start, end, - ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" ); + +printk (KERN_INFO "SRAT: Node %u %" PRIpaddr"-%" PRIpaddr"%s\n" , + node, start, end, hotplug ?" (hotplug)" :"" ); node_memblk_range[num_node_memblks].start = start; node_memblk_range[num_node_memblks].end = end; memblk_nodeid[num_node_memblks] = node; -if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) { +if (hotplug) {__set_bit (num_node_memblks, memblk_hotplug);if (end >mem_hotplug_boundary ())mem_hotplug_update_boundary (end); } num_node_memblks++; + +return 0 ; +} + +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ +void __init +acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +{ +unsigned pxm; +nodeid_t node; +int ret; + +if (srat_disabled ()) +return ; +if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { +bad_srat (); +return ; + } +if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) +return ; + +/* Supplement the heuristics in l1tf_calculations(). */ + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP ((ma->base_address + ma->length),PAGE_SIZE )); + +if (!numa_memblks_available ()) + { +dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); +bad_srat (); +return ; + } + + pxm = ma->proximity_domain; +if (srat_rev <2 ) + pxm &=0xff ; + node =setup_node (pxm); +if (node ==NUMA_NO_NODE ) { +bad_srat (); +return ; + } + + ret =numa_update_node_memblks (node, ma->base_address, ma->length, + ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ); +if (ret !=0 ) +bad_srat (); }/* Sanity check to catch more bad SRATs (they are amazingly common). xen/include/asm-x86/numa.h +3
-0@@ -74,6 +74,9 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr) NODE_DATA(nid)->node_spanned_pages)extern int valid_numa_range (paddr_t start,paddr_t end,nodeid_t node); +extern bool numa_memblks_available (void ); +extern int numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug);void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);Signature · 1 line
2.25.1
Re: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init
In reply to Wei ChenQuoted 11 lines · Wei Chen
There is some code in
acpi_numa_memory_affinity_initto update node memory range and updatenode_memblk_rangearray. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like
node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.Signed-off-by Wei Chen <[email protected]>
SRAT is an ACPI concept, which I assume has no meaning with DT. Hence any generically usable logic here wants, I think, separating out into a file which is not SRAT-specific (peeking ahead, specifically not a file named "numa_srat.c"). This may in turn require some more though regarding the proper split between the stuff remaining in srat.c and the stuff becoming kind of library code. In particular this may mean moving some of the static variables as well, and with them perhaps some further functions (while I did peek ahead, I didn't look closely at the later patch doing the actual movement). And it is then hard to see why the separation needs to happen in two steps - you could move the generically usable code to a new file right away.
xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 113
Quoted 13 lines
xen/arch/x86/srat.c +7
-0@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)return node; } +bool __init numa_memblks_available (void ) +{ +if (num_node_memblks <NR_NODE_MEMBLKS ) +return true; + +return false; +}
Please can you avoid expressing things in more complex than necessary ways? Here I don't see why it can't just be
bool __init numa_memblks_available(void)
{
return num_node_memblks < NR_NODE_MEMBLKS;
}xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 314
Quoted 10 lines
xen/arch/x86/srat.c +3
-3@@ -301,69 +309,35 @@ static bool __init is_node_memory_continuous(nodeid_t nid,return true; } -/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ -void __init -acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +/* Neutral NUMA memory affinity init function for ACPI and DT */ +int __init numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug)
Indentation.
Quoted 40 lines
xen/arch/x86/srat.c +2
-34{ -paddr_t start, end; -unsigned pxm; -nodeid_t node; +paddr_t end = start + size;int i; -if (srat_disabled ()) -return ; -if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); -return ; - } -if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) -return ; - - start = ma->base_address; - end = start + ma->length; -/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); - -if (num_node_memblks >=NR_NODE_MEMBLKS ) - { -dprintk (XENLOG_WARNING , -"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); -bad_srat (); -return ; - } - - pxm = ma->proximity_domain; -if (srat_rev <2 ) - pxm &=0xff ; - node =setup_node (pxm); -if (node ==NUMA_NO_NODE ) { -bad_srat (); -return ; - } -/* It is fine to add this area to the nodes data it will be used later*/ +/* It is fine to add this area to the nodes data it will be used later */ i =conflicting_memblks (start, end);
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 326
Quoted 10 lines
xen/arch/x86/srat.c +2
-4if (i <0 )/* everything fine */ ;else if (memblk_nodeid[i] == node) { -bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != - !test_bit (i, memblk_hotplug); +bool mismatch = !hotplug != !test_bit (i, memblk_hotplug); - printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n", - mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end, + printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps with itself (%"PRIpaddr"-%"PRIpaddr")\n",
Nit: Unlike PXM, which is an acronym, "node" doesn't want to be all upper case.
Also did you check that the node <-> PXM association is known to a reader of a log at this point in time?
Quoted 7 lines
xen/arch/x86/srat.c +2
-2+ mismatch ?KERN_ERR :KERN_WARNING , node, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) { -bad_srat (); -return ; +return -1 ; }
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 337
Quoted 10 lines
xen/arch/x86/srat.c +3
-4}else {printk (KERN_ERR -"SRAT: PXM %u (%" PRIpaddr"-%" PRIpaddr") overlaps with PXM %u (%" PRIpaddr"-%" PRIpaddr")\n" , - pxm, start, end,node_to_pxm (memblk_nodeid[i]), +"SRAT: NODE %u (%" PRIpaddr"-%" PRIpaddr") overlaps with NODE %u (%" PRIpaddr"-%" PRIpaddr")\n" , + node, start, end, memblk_nodeid[i], node_memblk_range[i].start, node_memblk_range[i].end); -bad_srat (); -return ; +return -1 ;
Please no -1 return values. Either a function means to return boolean, in which case it should use bool / true / false, or it means to return a proper errno-style error code.
Quoted 9 lines
xen/arch/x86/srat.c +0
-5@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)if (nd->end < end) nd->end = end; -/* Check whether this range contains memory for other nodes */ -if (!is_node_memory_continuous (node, nd->start, nd->end)) { -bad_srat (); -return ; - }
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 358
Quoted 10 lines
xen/arch/x86/srat.c +5
-3+if (!is_node_memory_continuous (node, nd->start, nd->end)) +return -1 ; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , - node, pxm, start, end, - ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" ); + +printk (KERN_INFO "SRAT: Node %u %" PRIpaddr"-%" PRIpaddr"%s\n" , + node, start, end, hotplug ?" (hotplug)" :"" );
Continuing from a comment further up: Here you remove an instance of logging the node <-> PXM association.
Quoted 24 lines
xen/arch/x86/srat.c +15
-1node_memblk_range[num_node_memblks].start = start; node_memblk_range[num_node_memblks].end = end; memblk_nodeid[num_node_memblks] = node; -if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) { +if (hotplug) {__set_bit (num_node_memblks, memblk_hotplug);if (end >mem_hotplug_boundary ())mem_hotplug_update_boundary (end); } num_node_memblks++; + +return 0 ; +} + +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */ +void __init +acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +{ +unsigned pxm; +nodeid_t node; +int ret; + +if (srat_disabled ()) +return ;
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 392
Quoted 10 lines
xen/arch/x86/srat.c +10
-0+if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { +bad_srat (); +return ; + } +if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) +return ; + +/* Supplement the heuristics in l1tf_calculations(). */ + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP ((ma->base_address + ma->length),PAGE_SIZE ));
Indentation and unnecessary pair of parentheses.
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+if (!numa_memblks_available ()) + {
For code you touch, please try to bring it into consistent style. Here the brace wants to move to the previous line, seeing that the file is using Linux style.
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 397
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" );
Here you want to fix indentation and ideally also format and grammar of the actual log message.
Quoted 5 lines
xen/arch/x86/srat.c +5
-0+bad_srat (); +return ; + } + + pxm = ma->proximity_domain;
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 412
Quoted 10 lines
xen/arch/x86/srat.c +10
-0+if (srat_rev <2 ) + pxm &=0xff ; + node =setup_node (pxm); +if (node ==NUMA_NO_NODE ) { +bad_srat (); +return ; + } + + ret =numa_update_node_memblks (node, ma->base_address, ma->length, + ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE );
Indentation again.
Jan
RE: [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init
Hi Jan,
Quoted 20 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:51
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 11/37] xen/x86: abstract neutral code fromacpi_numa_memory_affinity_initQuoted 10 lines · Wei Chen
There is some code in
acpi_numa_memory_affinity_initto update node memory range and updatenode_memblk_rangearray. This code is not ACPI specific, it can be shared by other NUMA implementation, like device tree based NUMA implementation.So in this patch, we abstract this memory range and blocks relative code to a new function. This will avoid exporting static variables like
node_memblk_range. And the PXM in neutral code print messages have been replaced by NODE, as PXM is ACPI specific.
Quoted 6 lines
Quoted 1 line
Signed-off-by Wei Chen <[email protected]>SRAT is an ACPI concept, which I assume has no meaning with DT. Hence any generically usable logic here wants, I think, separating out into a file which is not SRAT-specific (peeking ahead, specifically not a file named "
numa_srat.c"). This may in turn require some more though
When I created the file, I wanted to place non-ACPI/DT specific code in a new file. But I was confused about how to name it. I chose numa_srat.c as the file name because I thought the device tree is also a static resource table. But it seems this name is still misleading, because ACPI SRAT is well known.
Quoted 8 lines
regarding the proper split between the stuff remaining in
srat.cand the stuff becoming kind of library code. In particular this may mean moving some of the static variables as well, and with them perhaps some further functions (while I did peek ahead, I didn't look closely at the later patch doing the actual movement). And it is then hard to see why the separation needs to happen in two steps - you could move the generically usable code to a new file right away.
OK, I will reduce the steps. And I think the "new file" can be common/numa.c. Because the generically usable code are some logical functions to check numa memory blocks/ranges and update nodes, we don't need a "numa_srat.c".
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/srat.c +0
-0@@ -104,6 +104,14 @@ nodeid_t setup_node(unsigned pxm)return node; }
Quoted 10 lines
xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 113
Quoted 7 lines
xen/arch/x86/srat.c +7
-0+bool __init numa_memblks_available (void ) +{ +if (num_node_memblks <NR_NODE_MEMBLKS ) +return true; + +return false; +}Please can you avoid expressing things in more complex than necessary ways? Here I don't see why it can't just be
OK, I will simplify it.
Quoted 12 lines
bool
__initnuma_memblks_available(void){ return num_node_memblks < NR_NODE_MEMBLKS; }xen/arch/x86/srat.c static bool __init
Quoted 1 line
xen/arch/x86/srat.c +0
-0@@ -301,69 +309,35 @@ static bool __initis_node_memory_continuous(nodeid_t nid,
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,
Quoted 4 lines
xen/arch/x86/srat.c +0
-1return true; } -/* Callback for parsing of the Proximity Domain <-> Memory Area
Quoted 8 lines
mappings */
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 314
Quoted 5 lines
xen/arch/x86/srat.c +3
-2-void __init -acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +/* Neutral NUMA memory affinity init function for ACPI and DT */ +int __init numa_update_node_memblks (nodeid_t node, +paddr_t start,paddr_t size,bool hotplug)Indentation.
OK.
Quoted 50 lines
Quoted 29 lines
xen/arch/x86/srat.c +1
-24{ -paddr_t start, end; -unsigned pxm; -nodeid_t node; +paddr_t end = start + size;int i; -if (srat_disabled ()) -return ; -if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { -bad_srat (); -return ; - } -if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) -return ; - - start = ma->base_address; - end = start + ma->length; -/* Supplement the heuristics in l1tf_calculations(). */ - l1tf_safe_maddr =max (l1tf_safe_maddr,ROUNDUP (end,PAGE_SIZE )); - -if (num_node_memblks >=NR_NODE_MEMBLKS ) - { -dprintk (XENLOG_WARNING , -"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" ); -bad_srat (); -return ; - }xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,
Quoted 10 lines
xen/arch/x86/srat.c +0
-10- - pxm = ma->proximity_domain; -if (srat_rev <2 ) - pxm &=0xff ; - node =setup_node (pxm); -if (node ==NUMA_NO_NODE ) { -bad_srat (); -return ; - } -/* It is fine to add this area to the nodes data it will be used later*/
xen/arch/x86/srat.c static bool __init · line 319
Quoted 1 line
xen/arch/x86/srat.c +1
-0+/* It is fine to add this area to the nodes data it will be used later */
Quoted 8 lines
xen/arch/x86/srat.c +1
-2i =conflicting_memblks (start, end);if (i <0 )/* everything fine */ ;else if (memblk_nodeid[i] == node) { -bool mismatch = !(ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) != - !test_bit (i, memblk_hotplug); +bool mismatch = !hotplug != !test_bit (i, memblk_hotplug);
Quoted 9 lines
xen/arch/x86/srat.c static bool __init · line 351
Quoted 1 line
xen/arch/x86/srat.c +0
-1- printk("%sSRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps withitself (%"PRIpaddr"-%"PRIpaddr")\n",
xen/arch/x86/srat.c static bool __init · line 326
Quoted 2 lines
xen/arch/x86/srat.c +1
-1- mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end, + printk("%sSRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlapswith itself (%"PRIpaddr"-%"PRIpaddr")\n",
Nit: Unlike PXM, which is an acronym, "node" doesn't want to be all upper case.
OK.
Quoted 3 lines
Also did you check that the node <-> PXM association is known to a reader of a log at this point in time?
Yes, I read your comment below. The original log contains node <-> PXM mapping. Because PXM is ACPI specific, I think in neutral code, we just need node in log. But this change removed the log of node <-> PXM association, it was my mistake. I will add them back in next version. But I don't think they will stay in neutal code, it would be in ACPI specific code.
I also had wanted to keep PXM <-> node mapping in the log, so I set up PXM to node 1-1 mapping for the device tree. But then I thought it was an unnecessary burden on the device tree, so I selected to remove PXM in log.
Quoted 14 lines
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid,
Quoted 10 lines
xen/arch/x86/srat.c +2
-3+ mismatch ?KERN_ERR :KERN_WARNING , node, start, end, node_memblk_range[i].start, node_memblk_range[i].end);if (mismatch) { -bad_srat (); -return ; +return -1 ; } }else { printk(KERN_ERR - "SRAT: PXM %u (%"PRIpaddr"-%"PRIpaddr") overlaps withPXM %u (%"PRIpaddr"-%"PRIpaddr")\n",
xen/arch/x86/srat.c static bool __init · line 334
Quoted 2 lines
xen/arch/x86/srat.c +1
-1- pxm, start, end, node_to_pxm(memblk_nodeid[i]), + "SRAT: NODE %u (%"PRIpaddr"-%"PRIpaddr") overlaps withNODE %u (%"PRIpaddr"-%"PRIpaddr")\n",
Quoted 10 lines
xen/arch/x86/srat.c static bool __init is_node_memory_continuous(nodeid_t nid, · line 337
Quoted 5 lines
xen/arch/x86/srat.c +2
-2+ node, start, end, memblk_nodeid[i], node_memblk_range[i].start, node_memblk_range[i].end); -bad_srat (); -return ; +return -1 ;Please no -1 return values. Either a function means to return boolean, in which case it should use bool / true / false, or it means to return a proper errno-style error code.
Ok, I will do it.
Quoted 9 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct
Quoted 1 line
xen/arch/x86/srat.c +0
-0@@ -375,26 +349,69 @@ acpi_numa_memory_affinity_init(const struct
acpi_srat_mem_affinity*ma)xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 4 lines
xen/arch/x86/srat.c +0
-1if (nd->end < end) nd->end = end; -/* Check whether this range contains memory for other nodes */
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct · line 379
Quoted 1 line
xen/arch/x86/srat.c +0
-1-if (!is_node_memory_continuous (node, nd->start, nd->end)){
Quoted 17 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 358
Quoted 13 lines
xen/arch/x86/srat.c +5
-6-bad_srat (); -return ; - } +if (!is_node_memory_continuous (node, nd->start, nd->end)) +return -1 ; } } -printk (KERN_INFO "SRAT: Node %u PXM %u %" PRIpaddr"-%" PRIpaddr"%s\n" , - node, pxm, start, end, - ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ?" (hotplug)" :"" ); + +printk (KERN_INFO "SRAT: Node %u %" PRIpaddr"-%" PRIpaddr"%s\n" , + node, start, end, hotplug ?" (hotplug)" :"" );Continuing from a comment further up: Here you remove an instance of logging the node <-> PXM association.
see my above comments.
Quoted 29 lines
Quoted 5 lines
xen/arch/x86/srat.c +1
-1node_memblk_range[num_node_memblks].start = start; node_memblk_range[num_node_memblks].end = end; memblk_nodeid[num_node_memblks] = node; -if (ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE ) { +if (hotplug) {xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
Quoted 10 lines
xen/arch/x86/srat.c +5
-0__set_bit (num_node_memblks, memblk_hotplug);if (end >mem_hotplug_boundary ())mem_hotplug_update_boundary (end); } num_node_memblks++; + +return 0 ; +} + +/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
Quoted 13 lines
xen/arch/x86/srat.c +13
-0+void __init +acpi_numa_memory_affinity_init (const struct acpi_srat_mem_affinity *ma) +{ +unsigned pxm; +nodeid_t node; +int ret; + +if (srat_disabled ()) +return ; +if (ma->header.length !=sizeof (struct acpi_srat_mem_affinity )) { +bad_srat (); +return ; + }
Quoted 9 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 392
Quoted 6 lines
xen/arch/x86/srat.c +6
-0+if (!(ma->flags &ACPI_SRAT_MEM_ENABLED )) +return ; + +/* Supplement the heuristics in l1tf_calculations(). */ + l1tf_safe_maddr =max (l1tf_safe_maddr, +ROUNDUP ((ma->base_address + ma->length),PAGE_SIZE ));Indentation and unnecessary pair of parentheses.
OK.
Quoted 7 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 395
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+if (!numa_memblks_available ()) + {For code you touch, please try to bring it into consistent style. Here the brace wants to move to the previous line, seeing that the file is using Linux style.
OK.
Quoted 6 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 397
Quoted 2 lines
xen/arch/x86/srat.c +2
-0+dprintk (XENLOG_WARNING , +"Too many numa entry, try bigger NR_NODE_MEMBLKS \n" );Here you want to fix indentation and ideally also format and grammar of the actual log message.
I will fix it, thanks.
Quoted 9 lines
Quoted 9 lines
xen/arch/x86/srat.c +9
-0+bad_srat (); +return ; + } + + pxm = ma->proximity_domain; +if (srat_rev <2 ) + pxm &=0xff ; + node =setup_node (pxm); +if (node ==NUMA_NO_NODE ) {
Quoted 8 lines
xen/arch/x86/srat.c acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma) · line 412
Quoted 6 lines
xen/arch/x86/srat.c +6
-0+bad_srat (); +return ; + } + + ret =numa_update_node_memblks (node, ma->base_address, ma->length, + ma->flags &ACPI_SRAT_MEM_HOT_PLUGGABLE );Indentation again.
Ok.
Quoted 2 lines
Jan
PATCH 12/37 xen/x86: decouple nodes_cover_memory from E820 map
In reply to Wei ChenWe will reuse nodes_cover_memory for Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_number arch_get_memory_bank_range
Depends above two helpers, we make nodes_cover_memory become architecture independent. And the only change from an x86 perspective is the additional checks:
!start || !end
--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 11 ++++------- xen/include/asm-x86/numa.h | 3 +++ 3 files changed, 25 insertions(+), 7 deletions(-)
xen/arch/x86/numa.c +18 -0
@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)return (num_online_nodes () >1 ) ?1 :0 ; } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();
xen/arch/x86/srat.c +4 -7
@@ -419,17 +419,14 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;paddr_t start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end))continue ; - } - - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; do { found = 0;
@@ -448,7 +445,7 @@ static int __init nodes_cover_memory(void) }while (found && start < end);if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " +printk (KERN_ERR "SRAT: No NODE for memory map range: " "%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; }
xen/include/asm-x86/numa.h +3 -0
@@ -82,5 +82,8 @@ void srat_parse_regions(paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void ); +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end);#endif
Signature · 1 line
2.25.1
Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
+x86 maintainers
Quoted 97 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeDepends above two helpers, we make
nodes_cover_memorybecome architecture independent. And the only change from an x86 perspective is the additional checks:
!start || !endSigned-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 18 ++++++++++++++++++ xen/arch/x86/srat.c | 11 ++++------- xen/include/asm-x86/numa.h | 3 +++ 3 files changed, 25 insertions(+), 7 deletions(-)xen/arch/x86/numa.c +18
-0@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)return (num_online_nodes () >1 ) ?1 :0 ; } +uint32_t __init arch_meminfo_get_nr_bank (void ) +{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) { +return -1 ; + } + + *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();xen/arch/x86/srat.c +4
-7@@ -419,17 +419,14 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)static int __init nodes_cover_memory (void ) {int i; +uint32_t nr_banks =arch_meminfo_get_nr_bank (); -for (i =0 ; i < e820.nr_map; i++) { +for (i =0 ; i < nr_banks; i++) {int j, found;paddr_t start, end; -if (e820.map[i].type !=E820_RAM ) { +if (arch_meminfo_get_ram_bank_range (i, &start, &end))continue ; - } - - start = e820.map[i].addr; - end = e820.map[i].addr + e820.map[i].size; do { found = 0;@@ -448,7 +445,7 @@ static int __init nodes_cover_memory(void) }while (found && start < end);if (start < end) { -printk (KERN_ERR "SRAT: No PXM for e820 range: " +printk (KERN_ERR "SRAT: No NODE for memory map range: " "%" PRIpaddr" - %" PRIpaddr"\n" , start, end);return 0 ; }xen/include/asm-x86/numa.h +3
-0@@ -82,5 +82,8 @@ void srat_parse_regions(paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);unsigned int arch_get_dma_bitsize (void );unsigned int arch_have_default_dmazone (void ); +extern uint32_t arch_meminfo_get_nr_bank (void ); +extern int arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end);#endif Signature · 1 line
2.25.1
Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
In reply to Wei ChenQuoted 5 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_range
I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 381
Quoted 7 lines
xen/arch/x86/numa.c +1
-0@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)return (num_online_nodes () >1 ) ?1 :0 ; } +uint32_t __init arch_meminfo_get_nr_bank (void )
unsigned int (also elsewhere)
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 389
Quoted 8 lines
xen/arch/x86/numa.c +8
-0+{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) {
I see no reason for the checking of start and end.
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 391
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+return -1 ; + }
No need for braces here.
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 401
Quoted 9 lines
xen/arch/x86/numa.c +6
-0+ *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();
Judging by just the two pieces of patch context you're inserting a Linux-style code fragment in the middle of two Xen-style ones.
Various other comments given for earlier patches apply here as well.
Jan
RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
Hi Jan,
Quoted 10 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouplenodes_cover_memoryfrom E820 mapOn 23.09.2021 14:02, Wei Chen wrote:
Quoted 9 lines
Quoted 5 lines
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeI'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
In my latest changes, I have updated these two helpers to:uint32_t __init arch_meminfo_get_nr_bank(void) __init arch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.
Quoted 10 lines
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 381
Quoted 7 lines
xen/arch/x86/numa.c +1
-0@@ -378,6 +378,24 @@ unsigned int arch_have_default_dmazone(void)return (num_online_nodes () >1 ) ?1 :0 ; } +uint32_t __init arch_meminfo_get_nr_bank (void )unsigned int (also elsewhere)
OK.
Quoted 11 lines
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 389
Quoted 8 lines
xen/arch/x86/numa.c +8
-0+{ +return e820.nr_map; +} + +int __init arch_meminfo_get_ram_bank_range (uint32_t bank, +paddr_t *start,paddr_t *end) +{ +if (e820.map[bank].type !=E820_RAM || !start || !end) {I see no reason for the checking of start and end.
Yes, I have removed this checking in the latest version.
Quoted 5 lines
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 391
Quoted 2 lines
xen/arch/x86/numa.c +2
-0+return -1 ; + }No need for braces here.
Ok.
Quoted 13 lines
xen/arch/x86/numa.c unsigned int arch_have_default_dmazone(void) · line 401
Quoted 9 lines
xen/arch/x86/numa.c +6
-0+ *start = e820.map[bank].addr; + *end = e820.map[bank].addr + e820.map[bank].size; + +return 0 ; +} +static void dump_numa (unsigned char key) {s_time_t now =NOW ();Judging by just the two pieces of patch context you're inserting a Linux-style code fragment in the middle of two Xen-style ones.
Quoted 2 lines
Various other comments given for earlier patches apply here as well.
Yes, the original file is xen-style. I will change the inserted code to xen-style too. Thanks!
Quoted 1 line
Jan
Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
Quoted 10 lines · Wei Chen
Quoted 10 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59Quoted 5 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_range
Quoted 8 lines
Quoted 2 lines
I'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
In my latest changes, I have updated these two helpers to:
uint32_t__initarch_meminfo_get_nr_bank(void)__initarch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.
I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the function return "uint32_t" - see ./CODING_STYLE.
Jan
RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
Hi Jan,
Quoted 23 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouplenodes_cover_memoryfrom E820 mapQuoted 13 lines · Wei Chen
Quoted 12 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59Quoted 5 lines · Wei Chen
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeI'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
Quoted 8 lines
Quoted 5 lines
In my latest changes, I have updated these two helpers to:
uint32_t__initarch_meminfo_get_nr_bank(void)__initarch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the
How about arch_meminfo_get_nr_map/ arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?
Quoted 1 line
function return "
uint32_t" - see ./CODING_STYLE.
OK.
Quoted 2 lines
Jan
Re: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
Quoted 23 lines · Wei Chen
Quoted 23 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09Quoted 18 lines · Wei Chen
Quoted 4 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59On 23.09.2021 14:02, Wei Chen wrote:
Quoted 8 lines
Quoted 5 lines
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeI'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
In my latest changes, I have updated these two helpers to:
uint32_t__initarch_meminfo_get_nr_bank(void)__initarch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.
Quoted 6 lines
Quoted 2 lines
I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the
How about arch_meminfo_get_nr_map/
arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?
First of all I don't think you need a "get_nr" function at all, which eliminates the need to find a good name for it. The "get_range" function can easily provide back a unique indicator when the passed in index is beyond the number of regions. For this function's name, how about arch_get_memory_map() or arch_get_memory_map_block() or arch_get_memory_map_range() (in order of my personal preference)?
Jan
RE: [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map
Hi Jan,
Quoted 40 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 12/37] xen/x86: decouplenodes_cover_memoryfrom E820 mapQuoted 23 lines · Wei Chen
Quoted 23 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:09Quoted 18 lines · Wei Chen
Quoted 4 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 0:59On 23.09.2021 14:02, Wei Chen wrote:
Quoted 8 lines
Quoted 5 lines
We will reuse
nodes_cover_memoryfor Arm to check its bootmem info. So we introduce two arch helpers to get memory map's entry number and specified entry's range:
arch_get_memory_bank_numberarch_get_memory_bank_rangeI'm sorry, but personally I see no way for you to introduce the term "memory bank" into x86 code.
In my latest changes, I have updated these two helpers to:
uint32_t__initarch_meminfo_get_nr_bank(void)__initarch_meminfo_get_ram_bank_range(...)
I am sorry, I forgot to change the commit log accordingly. I will update it in next version.Quoted 6 lines
Quoted 2 lines
I'm sorry for the ambiguity of my earlier reply, but my objection was against "bank", not "memory". As an aside, you also don't want the
How about arch_meminfo_get_nr_map/
arch_meminfo_get_map_range? I am sorry, I am not very familiar with e820 map, could you give me some suggestions?
Quoted 7 lines
First of all I don't think you need a "
get_nr" function at all, which eliminates the need to find a good name for it. The "get_range" function can easily provide back a unique indicator when the passed in index is beyond the number of regions. For this function's name, how aboutarch_get_memory_map()orarch_get_memory_map_block()orarch_get_memory_map_range()(in order of my personal preference)?
Thanks, I will try to use arch_get_memory_map for x86 and arm.
Quoted 1 line
Jan
PATCH 13/37 xen/x86: decouple processor_nodes_parsed from acpi numa functions
In reply to Wei ChenXen is using processor_nodes_parsed to record parsed processor nodes from ACPI table or other firmware provided resource table. This variable is used in ACPI numa functions directly. In follow-up patchs, neutral NUMA code will be abstracted and move to other files. So in this patch, we introduce numa_set_processor_nodes_parsed helper to decouple processor_nodes_parsed from acpi numa functions.
--- xen/arch/x86/srat.c | 9 +++++++-- xen/include/asm-x86/numa.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-)
xen/arch/x86/srat.c +7 -2
@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)return node; } +void __init numa_set_processor_nodes_parsed (nodeid_t node) +{ +node_set (node, processor_nodes_parsed); +} +bool __init numa_memblks_available (void ) {if (num_node_memblks <NR_NODE_MEMBLKS )
@@ -236,7 +241,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa) } apicid_to_node[pa->apic_id] = node; -node_set (node, processor_nodes_parsed); +numa_set_processor_nodes_parsed (node); acpi_numa =1 ;if (opt_acpi_verbose)
@@ -271,7 +276,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)return ; } apicid_to_node[pa->apic_id] = node; -node_set (node, processor_nodes_parsed); +numa_set_processor_nodes_parsed (node); acpi_numa =1 ;if (opt_acpi_verbose)
xen/include/asm-x86/numa.h +1 -0
@@ -77,6 +77,7 @@ extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);extern bool numa_memblks_available (void );extern int numa_update_node_memblks (nodeid_t node,paddr_t start,paddr_t size,bool hotplug); +extern void numa_set_processor_nodes_parsed (nodeid_t node);void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);
Signature · 1 line
2.25.1
Re: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions
+x86 maintainers
Quoted 62 lines · Wei Chen
Xen is using
processor_nodes_parsedto record parsed processor nodes from ACPI table or other firmware provided resource table. This variable is used in ACPI numa functions directly. In follow-up patchs, neutral NUMA code will be abstracted and move to other files. So in this patch, we introducenuma_set_processor_nodes_parsedhelper to decoupleprocessor_nodes_parsedfrom acpi numa functions.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 9 +++++++-- xen/include/asm-x86/numa.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-)xen/arch/x86/srat.c +7
-2@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)return node; } +void __init numa_set_processor_nodes_parsed (nodeid_t node) +{ +node_set (node, processor_nodes_parsed); +} +bool __init numa_memblks_available (void ) {if (num_node_memblks <NR_NODE_MEMBLKS )@@ -236,7 +241,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa) } apicid_to_node[pa->apic_id] = node; -node_set (node, processor_nodes_parsed); +numa_set_processor_nodes_parsed (node); acpi_numa =1 ;if (opt_acpi_verbose)@@ -271,7 +276,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)return ; } apicid_to_node[pa->apic_id] = node; -node_set (node, processor_nodes_parsed); +numa_set_processor_nodes_parsed (node); acpi_numa =1 ;if (opt_acpi_verbose)xen/include/asm-x86/numa.h +1
-0@@ -77,6 +77,7 @@ extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);extern bool numa_memblks_available (void );extern int numa_update_node_memblks (nodeid_t node,paddr_t start,paddr_t size,bool hotplug); +extern void numa_set_processor_nodes_parsed (nodeid_t node);void srat_parse_regions (paddr_t addr);extern u8 __node_distance (nodeid_t a,nodeid_t b);Signature · 1 line
2.25.1
Re: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions
In reply to Wei Chenxen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 107
Quoted 7 lines · Wei Chen
xen/arch/x86/srat.c +1
-0@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)return node; } +void __init numa_set_processor_nodes_parsed (nodeid_t node)
Besides (nit) the stray blank here, earlier comments apply. The way you do the rearrangement it is close to impossible to see the actual "why" behind the changes. Even if it would make for a bigger patch, I think you want to collapse patches and move things out of srat.c code while you split out generically usable functionality. Or, if the result was deemed to large a patch to have all in one go, make the movement of individual static variables the topic of each patch, introducing accessor functions like the one here.
Jan
RE: [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions
Hi Jan,
Quoted 16 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 17:49
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 13/37] xen/x86: decoupleprocessor_nodes_parsedfrom acpi numa functionsQuoted 6 lines · Wei Chen
xen/arch/x86/srat.c +0
-0@@ -104,6 +104,11 @@ nodeid_t setup_node(unsigned pxm)return node; }
Quoted 11 lines
xen/arch/x86/srat.c nodeid_t setup_node(unsigned pxm) · line 107
Quoted 1 line
xen/arch/x86/srat.c +1
-0+void __init numa_set_processor_nodes_parsed (nodeid_t node)Besides (nit) the stray blank here, earlier comments apply. The way you do the rearrangement it is close to impossible to see the actual "why" behind the changes. Even if it would make for a bigger patch, I think you want to collapse patches and move things out of
srat.ccode while you split out generically usable functionality. Or, if the result was deemed to large a patch to have all in one go, make the movement of individual static variables the topic of each patch, introducing accessor functions like the one here.
Thanks, I will fix them in next version.
Quoted 1 line
Jan
PATCH 14/37 xen/x86: use name fw_numa to replace acpi_numa
In reply to Wei ChenXen is using acpi_numa as a switch for ACPI based NUMA. We want to use this switch logic for other firmware based NUMA implementation, like device tree based NUMA in follow-up patches. As Xen will never use both ACPI and device tree based NUMA at runtime. So I rename acpi_numa to a more generic name fw_name. This will also allow to have the code mostly common.
--- xen/arch/x86/numa.c | 6 +++--- xen/arch/x86/setup.c | 2 +- xen/arch/x86/srat.c | 10 +++++----- xen/include/asm-x86/acpi.h | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)
xen/arch/x86/numa.c +3 -3
@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };bool numa_off; -s8 acpi_numa =0 ; +s8 fw_numa =0 ;int srat_disabled (void ) { -return numa_off || acpi_numa <0 ; +return numa_off || fw_numa <0 ; }/*
@@ -315,7 +315,7 @@ static __init int numa_setup(const char *opt) else if ( !strncmp(opt,"noacpi",6) ) { numa_off = false; - acpi_numa =-1 ; + fw_numa =-1 ; }#endif else
xen/arch/x86/setup.c +1 -1
@@ -313,7 +313,7 @@ void srat_detect_node(int cpu)node_set_online (node);numa_set_node (cpu, node); -if ( opt_cpu_info && acpi_numa >0 ) +if ( opt_cpu_info && fw_numa >0 )printk ("CPU %d APIC %d -> Node %d\n" , cpu, apicid, node); }
xen/arch/x86/srat.c +5 -5
@@ -167,7 +167,7 @@ static __init void bad_srat(void) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" ); - acpi_numa =-1 ; + fw_numa =-1 ;for (i =0 ; i <MAX_LOCAL_APIC ; i++) apicid_to_node[i] =NUMA_NO_NODE ;for (i =0 ; i <ARRAY_SIZE (pxm2node); i++)
@@ -242,7 +242,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa) apicid_to_node[pa->apic_id] = node;numa_set_processor_nodes_parsed (node); - acpi_numa =1 ; + fw_numa =1 ;if (opt_acpi_verbose) printk(KERN_INFO "SRAT: PXM %u -> APIC %08x -> Node %u\n",
@@ -277,7 +277,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) } apicid_to_node[pa->apic_id] = node;numa_set_processor_nodes_parsed (node); - acpi_numa =1 ; + fw_numa =1 ;if (opt_acpi_verbose) printk(KERN_INFO "SRAT: PXM %u -> APIC %02x -> Node %u\n",
@@ -492,7 +492,7 @@ void __init srat_parse_regions(paddr_t addr)u64 mask;unsigned int i; -if (acpi_disabled || acpi_numa <0 || +if (acpi_disabled || fw_numa <0 ||acpi_table_parse (ACPI_SIG_SRAT , acpi_parse_srat))return ;
@@ -521,7 +521,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end); -if (acpi_numa <=0 ) +if (fw_numa <=0 )return -1 ;if (!nodes_cover_memory ()) {
xen/include/asm-x86/acpi.h +1 -1
@@ -101,7 +101,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; +extern s8 fw_numa;extern int acpi_scan_nodes (u64 start,u64 end);#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
Signature · 1 line
2.25.1
Re: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa
+x86 maintainers
Quoted 120 lines · Wei Chen
Xen is using
acpi_numaas a switch for ACPI based NUMA. We want to use this switch logic for other firmware based NUMA implementation, like device tree based NUMA in follow-up patches. As Xen will never use both ACPI and device tree based NUMA at runtime. So I renameacpi_numato a more generic namefw_name. This will also allow to have the code mostly common.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 6 +++--- xen/arch/x86/setup.c | 2 +- xen/arch/x86/srat.c | 10 +++++----- xen/include/asm-x86/acpi.h | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)xen/arch/x86/numa.c +3
-3@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };bool numa_off; -s8 acpi_numa =0 ; +s8 fw_numa =0 ;int srat_disabled (void ) { -return numa_off || acpi_numa <0 ; +return numa_off || fw_numa <0 ; }/* @@ -315,7 +315,7 @@ static __init int numa_setup(const char *opt) else if ( !strncmp(opt,"noacpi",6) ) { numa_off = false; - acpi_numa =-1 ; + fw_numa =-1 ; }#endif elsexen/arch/x86/setup.c +1
-1@@ -313,7 +313,7 @@ void srat_detect_node(int cpu)node_set_online (node);numa_set_node (cpu, node); -if ( opt_cpu_info && acpi_numa >0 ) +if ( opt_cpu_info && fw_numa >0 )printk ("CPU %d APIC %d -> Node %d\n" , cpu, apicid, node); }xen/arch/x86/srat.c +5
-5@@ -167,7 +167,7 @@ static __init void bad_srat(void) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" ); - acpi_numa =-1 ; + fw_numa =-1 ;for (i =0 ; i <MAX_LOCAL_APIC ; i++) apicid_to_node[i] =NUMA_NO_NODE ;for (i =0 ; i <ARRAY_SIZE (pxm2node); i++)@@ -242,7 +242,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa) apicid_to_node[pa->apic_id] = node;numa_set_processor_nodes_parsed (node); - acpi_numa =1 ; + fw_numa =1 ;if (opt_acpi_verbose) printk(KERN_INFO "SRAT: PXM %u -> APIC %08x -> Node %u\n",@@ -277,7 +277,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa) } apicid_to_node[pa->apic_id] = node;numa_set_processor_nodes_parsed (node); - acpi_numa =1 ; + fw_numa =1 ;if (opt_acpi_verbose) printk(KERN_INFO "SRAT: PXM %u -> APIC %02x -> Node %u\n",@@ -492,7 +492,7 @@ void __init srat_parse_regions(paddr_t addr)u64 mask;unsigned int i; -if (acpi_disabled || acpi_numa <0 || +if (acpi_disabled || fw_numa <0 ||acpi_table_parse (ACPI_SIG_SRAT , acpi_parse_srat))return ;@@ -521,7 +521,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)for (i =0 ; i <MAX_NUMNODES ; i++)cutoff_node (i, start, end); -if (acpi_numa <=0 ) +if (fw_numa <=0 )return -1 ;if (!nodes_cover_memory ()) {xen/include/asm-x86/acpi.h +1
-1@@ -101,7 +101,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 acpi_numa; +extern s8 fw_numa;extern int acpi_scan_nodes (u64 start,u64 end);#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)Signature · 1 line
2.25.1
Re: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa
In reply to Wei Chenxen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 54
Quoted 8 lines · Wei Chen
xen/arch/x86/numa.c +1
-1@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };bool numa_off; -s8 acpi_numa =0 ; +s8 fw_numa =0 ;
In x86 code I'd prefer this to remain "acpi_numa". If you need to access the variable from to-become-generic code, introduce an inline wrapper (possibly named numa_mode()), allowing you to do whatever you need in the DT case. It may be helpful to fold this with numa_off then, seeing e.g.
...
xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 58
Quoted 4 lines
xen/arch/x86/numa.c +1
-1int srat_disabled (void ) { -return numa_off || acpi_numa <0 ; +return numa_off || fw_numa <0 ;
... this. Actually I think the underlying enumeration could even be made generic:
enum numa_mode {
numa_off,
numa_on,
numa_acpi,
};
is, I believe, sufficient to express the present (numa_off,acpi_numa) tuple. In this context I'd like to point out that the two uses of acpi_numa in srat_parse_regions() and srat_detect_node() should likely be invocations of srat_disabled() instead, to also take numa_off into account. This would then be addressed effectively as a side effect by replacing open-coded uses as well as srat_disabled() by numa_mode() (or whichever name the new helper would gain).
Jan
RE: [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa
Hi Jan,
Quoted 39 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:13
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 14/37] xen/x86: use namefw_numato replaceacpi_numaxen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES]
Quoted 3 lines · Wei Chen
xen/arch/x86/numa.c +0
-0@@ -51,11 +51,11 @@ cpumask_t node_to_cpumask[MAX_NUMNODES]
__read_mostly;xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 54
Quoted 5 lines
xen/arch/x86/numa.c +1
-1nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } };bool numa_off; -s8 acpi_numa =0 ; +s8 fw_numa =0 ;In x86 code I'd prefer this to remain "
acpi_numa". If you need to access the variable from to-become-generic code, introduce an inline wrapper (possibly namednuma_mode()), allowing you to do whatever you need in the DT case. It may be helpful to fold this withnuma_offthen, seeing e.g.
...xen/arch/x86/numa.c cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly; · line 58
Quoted 4 lines
xen/arch/x86/numa.c +1
-1int srat_disabled (void ) { -return numa_off || acpi_numa <0 ; +return numa_off || fw_numa <0 ;... this. Actually I think the underlying enumeration could even be made generic:
enum
numa_mode{
numa_off,
numa_on,
numa_acpi,
};
Quoted 8 lines
is, I believe, sufficient to express the present (numa_off,acpi_numa) tuple. In this context I'd like to point out that the two uses of
acpi_numainsrat_parse_regions()andsrat_detect_node()should likely be invocations ofsrat_disabled()instead, to also takenuma_offinto account. This would then be addressed effectively as a side effect by replacing open-coded uses as well assrat_disabled()bynuma_mode()(or whichever name the new helper would gain).
Yes, that seems more reasonable, I will try this in next version.
Quoted 1 line
Jan
PATCH 15/37 xen/x86: rename acpi_scan_nodes to numa_scan_nodes
In reply to Wei ChenThe most code in acpi_scan_nodes can be reused by other NUMA implementation. Rename acpi_scan_nodes to a more generic name numa_scan_nodes, and replace BIOS to Firmware in print message, as BIOS is x86 specific name.
--- xen/arch/x86/numa.c | 2 +- xen/arch/x86/srat.c | 4 ++-- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
xen/arch/x86/numa.c +1 -1
@@ -261,7 +261,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) end =pfn_to_paddr (end_pfn);#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes (start, end) ) +if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif
xen/arch/x86/srat.c +2 -2
@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (paddr_t start,paddr_t end) +int __init numa_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;
@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); +"Firmware Bug or mis-configured hardware?\n" , i);setup_node_bootmem (i, nodes[i].start, nodes[i].end); }
xen/include/asm-x86/acpi.h +1 -1
@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 fw_numa; -extern int acpi_scan_nodes (u64 start,u64 end); +extern int numa_scan_nodes (u64 start,u64 end);#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;
Signature · 1 line
2.25.1
Re: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes
+x86 maintainers
Quoted 63 lines · Wei Chen
The most code in
acpi_scan_nodescan be reused by other NUMA implementation. Renameacpi_scan_nodesto a more generic namenuma_scan_nodes, and replace BIOS to Firmware in print message, as BIOS is x86 specific name.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 2 +- xen/arch/x86/srat.c | 4 ++-- xen/include/asm-x86/acpi.h | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)xen/arch/x86/numa.c +1
-1@@ -261,7 +261,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) end =pfn_to_paddr (end_pfn);#ifdef CONFIG_ACPI_NUMA -if ( !numa_off && !acpi_scan_nodes (start, end) ) +if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif xen/arch/x86/srat.c +2
-2@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (paddr_t start,paddr_t end) +int __init numa_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); +"Firmware Bug or mis-configured hardware?\n" , i);setup_node_bootmem (i, nodes[i].start, nodes[i].end); }xen/include/asm-x86/acpi.h +1
-1@@ -102,7 +102,7 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1extern s8 fw_numa; -extern int acpi_scan_nodes (u64 start,u64 end); +extern int numa_scan_nodes (u64 start,u64 end);#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)extern struct acpi_sleep_info acpi_sinfo;Signature · 1 line
2.25.1
Re: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes
In reply to Wei ChenQuoted 12 lines · Wei Chen
xen/arch/x86/srat.c +1
-1@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr) }/* Use the information discovered above to actually set up the nodes. */ -int __init acpi_scan_nodes (paddr_t start,paddr_t end) +int __init numa_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;
xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t end) · line 550
Quoted 5 lines
xen/arch/x86/srat.c +1
-1@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); +"Firmware Bug or mis-configured hardware?\n" , i);
Besides the prior theme of in-place renames not making clear why the rename is being done (by deferring to future patches then moving the code) I'm puzzled by you replacing "BIOS" but leaving "SRAT" in place.
Jan
RE: [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes
Hi Jan,
Quoted 24 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:18
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 15/37] xen/x86: renameacpi_scan_nodestonuma_scan_nodesxen/arch/x86/srat.c void __init srat_parse_regions(paddr_t addr) · line 514
Quoted 6 lines · Wei Chen
xen/arch/x86/srat.c +0
-0@@ -512,7 +512,7 @@ void __init srat_parse_regions(paddr_t addr) }/* Use the information discovered above to actually set up the nodes. */
xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t
Quoted 6 lines
xen/arch/x86/srat.c +1
-1-int __init acpi_scan_nodes (paddr_t start,paddr_t end) +int __init numa_scan_nodes (paddr_t start,paddr_t end) {int i;nodemask_t all_nodes_parsed;@@ -547,7 +547,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_tend)
Quoted 10 lines
xen/arch/x86/srat.c int __init acpi_scan_nodes(paddr_t start, paddr_t end) · line 550
Quoted 5 lines
xen/arch/x86/srat.c +1
-1paddr_t size = nodes[i].end - nodes[i].start;if ( size ==0 )printk (KERN_WARNING "SRAT: Node %u has no memory. " -"BIOS Bug or mis-configured hardware?\n" , i); +"Firmware Bug or mis-configured hardware?\n" , i);Besides the prior theme of in-place renames not making clear why the rename is being done (by deferring to future patches then moving the code) I'm puzzled by you replacing "BIOS" but leaving "SRAT" in place.
The reason is the same as I said in previous patch. At that time, I want to treat device tree as another static resource table. I would change "SRAT" to "NUMA". And I will try to merge this rename patch with the actual affected patch.
Quoted 1 line
Jan
PATCH 16/37 xen/x86: export srat_bad to external
In reply to Wei Chensrat_bad is used when NUMA initialization code scan SRAT failed. It will turn fw_numa to disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.
In this patch, we export srat_bad to external. This will allow to have the code mostly common.
--- xen/arch/x86/srat.c | 2 +- xen/include/asm-x86/numa.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
xen/arch/x86/srat.c +1 -1
@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end) } } -static __init void bad_srat (void ) +__init void bad_srat (void ) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" );
xen/include/asm-x86/numa.h +1 -0
@@ -32,6 +32,7 @@ extern bool numa_off;extern int srat_disabled (void ); +extern void bad_srat (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);
Signature · 1 line
2.25.1
Re: [PATCH 16/37] xen/x86: export srat_bad to external
+x86 maintainers
Quoted 42 lines · Wei Chen
srat_badis used when NUMA initialization code scan SRAT failed. It will turnfw_numato disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.In this patch, we export
srat_badto external. This will allow to have the code mostly common.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/srat.c | 2 +- xen/include/asm-x86/numa.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)xen/arch/x86/srat.c +1
-1@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end) } } -static __init void bad_srat (void ) +__init void bad_srat (void ) {int i;printk (KERN_ERR "SRAT: SRAT not used.\n" );xen/include/asm-x86/numa.h +1
-0@@ -32,6 +32,7 @@ extern bool numa_off;extern int srat_disabled (void ); +extern void bad_srat (void );extern void numa_set_node (int cpu,nodeid_t node);extern nodeid_t setup_node (unsigned int pxm);extern void srat_detect_node (int cpu);Signature · 1 line
2.25.1
Re: [PATCH 16/37] xen/x86: export srat_bad to external
In reply to Wei ChenQuoted 7 lines · Wei Chen
srat_badis used when NUMA initialization code scan SRAT failed. It will turnfw_numato disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.In this patch, we export
srat_badto external. This will allow to have the code mostly common.
Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.
Perhaps alongside numa_mode() (see the reply to the earlier patch) you want to have a set_numa_off() helper (name subject to improvement)?
xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start, paddr_t end) · line 166
Quoted 8 lines
xen/arch/x86/srat.c +1
-1@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start, paddr_t end) } } -static __init void bad_srat (void ) +__init void bad_srat (void )
Nit: Once again, when touching code, please take the opportunity and adjust style issues (here: __init and void want to change places).
Jan
RE: [PATCH 16/37] xen/x86: export srat_bad to external
Hi Jan,
Quoted 17 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: exportsrat_badto externalQuoted 7 lines · Wei Chen
srat_badis used when NUMA initialization code scan SRAT failed. It will turnfw_numato disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.In this patch, we export
srat_badto external. This will allow to have the code mostly common.
Quoted 8 lines
Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.
Perhaps alongside
numa_mode()(see the reply to the earlier patch) you want to have aset_numa_off()helper (name subject to improvement)?
Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use numa_fw_fault to replace srat_bad.
Quoted 4 lines
xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start,
Quoted 3 lines
xen/arch/x86/srat.c +0
-0@@ -163,7 +163,7 @@ static __init void cutoff_node(int i, paddr_t start,
paddr_tend)
Quoted 9 lines
xen/arch/x86/srat.c static __init void cutoff_node(int i, paddr_t start, paddr_t end) · line 166
Quoted 5 lines
xen/arch/x86/srat.c +1
-1} } -static __init void bad_srat (void ) +__init void bad_srat (void )Nit: Once again, when touching code, please take the opportunity and adjust style issues (here:
__initand void want to change places).
Thanks again!
Quoted 1 line
Jan
Re: [PATCH 16/37] xen/x86: export srat_bad to external
Quoted 28 lines · Wei Chen
Hi Jan,
Quoted 17 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: exportsrat_badto externalQuoted 7 lines · Wei Chen
srat_badis used when NUMA initialization code scan SRAT failed. It will turnfw_numato disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.In this patch, we export
srat_badto external. This will allow to have the code mostly common.Quoted 8 lines
Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.
Perhaps alongside
numa_mode()(see the reply to the earlier patch) you want to have aset_numa_off()helper (name subject to improvement)?
Quoted 4 lines
Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use
numa_fw_faultto replacesrat_bad.
Why not simply numa_bad() or, say, numa_disable()?
Jan
RE: [PATCH 16/37] xen/x86: export srat_bad to external
Hi Jan,
Quoted 37 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 16:37
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: exportsrat_badto externalQuoted 28 lines · Wei Chen
Hi Jan,
Quoted 25 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 16/37] xen/x86: exportsrat_badto externalQuoted 7 lines · Wei Chen
srat_badis used when NUMA initialization code scan SRAT failed. It will turnfw_numato disabled status. Its implementation depends on NUMA implementation. We want every NUMA implementation to provide this function for common initialization code.In this patch, we export
srat_badto external. This will allow to have the code mostly common.Here as well as in the subject it would help if the function name wasn't the wrong way round. I also don't see how you mean to use a function containing "srat" in its name from non-ACPI code.
Perhaps alongside
numa_mode()(see the reply to the earlier patch) you want to have aset_numa_off()helper (name subject to improvement)?
Quoted 7 lines
Quoted 4 lines
Yes, actually, I want a function to disable the numa when we encounter any error in numa initialization. And we also so want to indicate the numa off is caused by configuration error. So we had thought to use
numa_fw_faultto replacesrat_bad.Why not simply
numa_bad()or, say,numa_disable()?
numa_bad would be OK.
Quoted 1 line
Jan
PATCH 17/37 xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
In reply to Wei ChenAs we have turned numa_scan_nodes to neutral function. If we still use CONFIG_ACPI_NUMA in numa_initmem_init to gate numa_scan_nodes that doesn't make sense. As CONFIG_ACPI_NUMA will be selected by CONFIG_NUMA for x86. So in this patch, we replace CONFIG_ACPI_NUMA by CONFIG_NUMA.
--- xen/arch/x86/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/arch/x86/numa.c +1 -1
@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) start =pfn_to_paddr (start_pfn); end =pfn_to_paddr (end_pfn); -#ifdef CONFIG_ACPI_NUMA +#ifdef CONFIG_NUMA if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif
Signature · 1 line
2.25.1
Re: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
+x86 maintainers
Quoted 27 lines · Wei Chen
As we have turned
numa_scan_nodesto neutral function. If we still useCONFIG_ACPI_NUMAinnuma_initmem_initto gatenuma_scan_nodesthat doesn't make sense. AsCONFIG_ACPI_NUMAwill be selected byCONFIG_NUMAfor x86. So in this patch, we replaceCONFIG_ACPI_NUMAbyCONFIG_NUMA.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/x86/numa.c +1
-1@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) start =pfn_to_paddr (start_pfn); end =pfn_to_paddr (end_pfn); -#ifdef CONFIG_ACPI_NUMA +#ifdef CONFIG_NUMA if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif Signature · 1 line
2.25.1
Re: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
In reply to Wei ChenQuoted 16 lines · Wei Chen
As we have turned
numa_scan_nodesto neutral function. If we still useCONFIG_ACPI_NUMAinnuma_initmem_initto gatenuma_scan_nodesthat doesn't make sense. AsCONFIG_ACPI_NUMAwill be selected byCONFIG_NUMAfor x86. So in this patch, we replaceCONFIG_ACPI_NUMAbyCONFIG_NUMA.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/x86/numa.c +0
-0
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 266
Quoted 8 lines
xen/arch/x86/numa.c +1
-1@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) start =pfn_to_paddr (start_pfn); end =pfn_to_paddr (end_pfn); -#ifdef CONFIG_ACPI_NUMA +#ifdef CONFIG_NUMA if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif
Why is this a separate patch, rather than part of the one renaming the function?
Jan
RE: [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes
Hi Jan,
Quoted 30 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:26
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 17/37] xen/x86: useCONFIG_NUMAto gatenuma_scan_nodesxen/arch/x86/numa.c void __init numa_initmem_init(unsigned long
Quoted 16 lines · Wei Chen
As we have turned
numa_scan_nodesto neutral function. If we still useCONFIG_ACPI_NUMAinnuma_initmem_initto gatenuma_scan_nodesthat doesn't make sense. AsCONFIG_ACPI_NUMAwill be selected byCONFIG_NUMAfor x86. So in this patch, we replaceCONFIG_ACPI_NUMAbyCONFIG_NUMA.Signed-off-by Wei Chen <[email protected]>--- xen/arch/x86/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)xen/arch/x86/numa.c +0
-0@@ -260,7 +260,7 @@ void __init numa_initmem_init(unsigned long
start_pfn, unsigned longend_pfn)Quoted 3 lines
xen/arch/x86/numa.c +0
-0start =pfn_to_paddr (start_pfn); end =pfn_to_paddr (end_pfn);
Quoted 8 lines
xen/arch/x86/numa.c void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn) · line 266
Quoted 5 lines
xen/arch/x86/numa.c +1
-1-#ifdef CONFIG_ACPI_NUMA +#ifdef CONFIG_NUMA if ( !numa_off && !numa_scan_nodes (start, end) )return ;#endif Why is this a separate patch, rather than part of the one renaming the function?
I will merge these two patches.
Quoted 2 lines
Jan
PATCH 18/37 xen: move NUMA common code from x86 to common
In reply to Wei ChenSome common code has been decoupled and abstracted from x86 ACPI based NUMA implementation. In order to make this code can be reused by other NUMA implementation, we move this code from x86 to common folder. And this code is gated by CONFIG_NUMA, it only can be used by those architectures that NUMA is enabled. For those architectures do not support NUMA, they still can implementation NUMA stub API in asm/numa.h to make NUMA-aware components happy.
In this patch, we also remove some unused include headers.
--- xen/arch/x86/numa.c | 446 +---------------------------------- xen/arch/x86/srat.c | 253 +------------------- xen/common/Makefile | 2 + xen/common/numa.c | 450 ++++++++++++++++++++++++++++++++++++ xen/common/numa_srat.c | 264 +++++++++++++++++++++ xen/include/asm-x86/acpi.h | 4 - xen/include/asm-x86/numa.h | 68 ------ xen/include/asm-x86/setup.h | 1 - xen/include/xen/numa.h | 82 +++++++ 9 files changed, 802 insertions(+), 768 deletions(-) create mode 100644 xen/common/numa.c create mode 100644 xen/common/numa_srat.c
xen/arch/x86/numa.c +2 -444
@@ -3,24 +3,13 @@* Copyright 2002,2003 Andi Kleen, SuSE Labs. * Adapted for Xen: Ryan Harper <[email protected]> */ - -#include <xen/mm.h> -#include <xen/string.h> #include <xen/init.h> -#include <xen/ctype.h> +#include <xen/mm.h> #include <xen/nodemask.h> #include <xen/numa.h> -#include <xen/keyhandler.h> -#include <xen/param.h> -#include <xen/time.h> -#include <xen/smp.h> -#include <xen/pfn.h> -#include <asm/acpi.h> #include <xen/sched.h> -#include <xen/softirq.h> -static int numa_setup (const char *s); -custom_param ("numa" , numa_setup); +#include <asm/acpi.h> #ifndef Dprintk #define Dprintk(x...)
@@ -29,300 +18,12 @@ custom_param("numa", numa_setup);/* from proto.h */ #define round_up (x,y) ((((x)+(y))-1) & (~((y)-1))) -struct node_data node_data[MAX_NUMNODES ]; - -/* Mapping from pdx to node id */ -int memnode_shift; -static typeof(*memnodemap) _memnodemap[64]; -unsigned long memnodemapsize; -u8 *memnodemap; - -nodeid_t cpu_to_node[NR_CPUS ]__read_mostly = { - [0 ...NR_CPUS -1 ] =NUMA_NO_NODE -};/* * Keep BIOS's CPU2node information, should not be used for memory allocaion */ nodeid_t apicid_to_node[MAX_LOCAL_APIC ] = { [0 ...MAX_LOCAL_APIC -1 ] =NUMA_NO_NODE }; -cpumask_t node_to_cpumask[MAX_NUMNODES ]__read_mostly ; - -nodemask_t __read_mostly node_online_map = { { [0 ] =1UL } }; - -bool numa_off; -s8 fw_numa =0 ; - -int srat_disabled (void ) -{ -return numa_off || fw_numa <0 ; -} - -/* -* Given a shift value, try to populate memnodemap[] -* Returns : -* 1 if OK -* 0 if memnodmap[] too small (of shift too small) -* -1 if node overlap or lost ram (shift too big) -*/ -static int __init populate_memnodemap (const struct node *nodes, -int numnodes,int shift,nodeid_t *nodeids) -{ -unsigned long spdx, epdx; -int i, res =-1 ; - -memset (memnodemap,NUMA_NO_NODE , memnodemapsize *sizeof (*memnodemap)); -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; -if ( (epdx >> shift) >= memnodemapsize ) -return 0 ; -do { -if ( memnodemap[spdx >> shift] !=NUMA_NO_NODE ) -return -1 ; - -if ( !nodeids ) - memnodemap[spdx >> shift] = i; -else - memnodemap[spdx >> shift] = nodeids[i]; - - spdx += (1UL << shift); - }while ( spdx < epdx ); - res =1 ; - } - -return res; -} - -static int __init allocate_cachealigned_memnodemap (void ) -{ -unsigned long size =PFN_UP (memnodemapsize *sizeof (*memnodemap)); -unsigned long mfn =mfn_x (alloc_boot_pages (size,1 )); - - memnodemap =mfn_to_virt (mfn); - mfn <<=PAGE_SHIFT ; - size <<=PAGE_SHIFT ; -printk (KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n" , - mfn, mfn + size); - memnodemapsize = size /sizeof (*memnodemap); - -return 0 ; -} - -/* -* The LSB of all start and end addresses in the node map is the value of the -* maximum possible shift. -*/ -static int __init extract_lsb_from_nodes (const struct node *nodes, -int numnodes) -{ -int i, nodes_used =0 ; -unsigned long spdx, epdx; -unsigned long bitfield =0 , memtop =0 ; - -for ( i =0 ; i < numnodes; i++ ) - { - spdx =paddr_to_pdx (nodes[i].start); - epdx =paddr_to_pdx (nodes[i].end -1 ) +1 ; -if ( spdx >= epdx ) -continue ; - bitfield |= spdx; - nodes_used++; -if ( epdx > memtop ) - memtop = epdx; - } -if ( nodes_used <=1 ) - i =BITS_PER_LONG -1 ; -else - i =find_first_bit (&bitfield,sizeof (unsigned long )*8 ); - memnodemapsize = (memtop >> i) +1 ; -return i; -} - -int __init compute_hash_shift (struct node *nodes,int numnodes, -nodeid_t *nodeids) -{ -int shift; - - shift =extract_lsb_from_nodes (nodes, numnodes); -if ( memnodemapsize <=ARRAY_SIZE (_memnodemap) ) - memnodemap = _memnodemap; -else if (allocate_cachealigned_memnodemap () ) -return -1 ; -printk (KERN_DEBUG "NUMA: Using %d for the hash shift.\n" , shift); - -if (populate_memnodemap (nodes, numnodes, shift, nodeids) !=1 ) - { -printk (KERN_INFO "Your memory is not aligned you need to " -"rebuild your hypervisor with a bigger NODEMAPSIZE " -"shift=%d\n" , shift); -return -1 ; - } - -return shift; -} -/* initialize NODE_DATA given nodeid and start/end */ -void __init setup_node_bootmem (nodeid_t nodeid,paddr_t start,paddr_t end) -{ -unsigned long start_pfn, end_pfn; - - start_pfn =paddr_to_pfn (start); - end_pfn =paddr_to_pfn (end); - -NODE_DATA (nodeid)->node_start_pfn = start_pfn; -NODE_DATA (nodeid)->node_spanned_pages = end_pfn - start_pfn; - -node_set_online (nodeid); -} - -void __init numa_init_array (void ) -{ -int rr, i; - -/* There are unfortunately some poorly designed mainboards around -that only connect memory to a single CPU. This breaks the 1:1 cpu->node -mapping. To avoid this fill in the mapping for all possible -CPUs, as the number of CPUs is not known yet. -We round robin the existing nodes. */ - rr =first_node (node_online_map); -for ( i =0 ; i < nr_cpu_ids; i++ ) - { -if ( cpu_to_node[i] !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =cycle_node (rr, node_online_map); - } -} - -#ifdef CONFIG_NUMA_EMU -static int numa_fake__initdata =0 ; - -/* Numa emulation */ -static int __init numa_emulation (unsigned long start_pfn, -unsigned long end_pfn) -{ -int i; -struct node nodes[MAX_NUMNODES ]; -u64 sz =pfn_to_paddr (end_pfn - start_pfn) / numa_fake; - -/* Kludge needed for the hash function */ -if (hweight64 (sz) >1 ) - { -u64 x =1 ; -while ( (x <<1 ) < sz ) - x <<=1 ; -if ( x < sz/2 ) -printk (KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n" ); - sz = x; - } - -memset (&nodes,0 ,sizeof (nodes)); -for ( i =0 ; i < numa_fake; i++ ) - { - nodes[i].start =pfn_to_paddr (start_pfn) + i*sz; -if ( i == numa_fake -1 ) - sz =pfn_to_paddr (end_pfn) - nodes[i].start; - nodes[i].end = nodes[i].start + sz; -printk (KERN_INFO "Faking node %d at %" PRIx64"-%" PRIx64" (%" PRIu64"MB)\n" , - i, - nodes[i].start, nodes[i].end, - (nodes[i].end - nodes[i].start) >>20 ); -node_set_online (i); - } - memnode_shift =compute_hash_shift (nodes, numa_fake,NULL ); -if ( memnode_shift <0 ) - { - memnode_shift =0 ; -printk (KERN_ERR "No NUMA hash function found. Emulation disabled.\n" ); -return -1 ; - } -for_each_online_node ( i ) - setup_node_bootmem(i, nodes[i].start, nodes[i].end); -numa_init_array (); - -return 0 ; -} -#endif - -void __init numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn) -{ -int i; -paddr_t start, end; - -#ifdef CONFIG_NUMA_EMU -if ( numa_fake && !numa_emulation (start_pfn, end_pfn) ) -return ; -#endif - - start =pfn_to_paddr (start_pfn); - end =pfn_to_paddr (end_pfn); - -#ifdef CONFIG_NUMA -if ( !numa_off && !numa_scan_nodes (start, end) ) -return ; -#endif - -printk (KERN_INFO "%s\n" , - numa_off ?"NUMA turned off" :"No NUMA configuration found" ); - -printk (KERN_INFO "Faking a node at %016" PRIpaddr"-%016" PRIpaddr"\n" , - start, end); -/* setup dummy node covering all memory */ - memnode_shift =BITS_PER_LONG -1 ; - memnodemap = _memnodemap; -/* Dummy node only uses 1 slot in reality */ - memnodemap[0 ] =0 ; - memnodemapsize =1 ; - -nodes_clear (node_online_map); -node_set_online (0 ); -for ( i =0 ; i < nr_cpu_ids; i++ ) -numa_set_node (i,0 ); -cpumask_copy (&node_to_cpumask[0 ],cpumask_of (0 )); -setup_node_bootmem (0 , start, end); -} - -void numa_add_cpu (int cpu) -{ -cpumask_set_cpu (cpu, &node_to_cpumask[cpu_to_node (cpu)]); -} - -void numa_set_node (int cpu,nodeid_t node) -{ - cpu_to_node[cpu] = node; -} - -/* [numa=off] */ -static __init int numa_setup (const char *opt) -{ -if ( !strncmp (opt,"off" ,3 ) ) - numa_off = true; -else if ( !strncmp (opt,"on" ,2 ) ) - numa_off = false; -#ifdef CONFIG_NUMA_EMU - else if ( !strncmp(opt, "fake=", 5) ) - { - numa_off = false; - numa_fake =simple_strtoul (opt+5 ,NULL ,0 ); -if ( numa_fake >=MAX_NUMNODES ) - numa_fake =MAX_NUMNODES ; - } -#endif -#ifdef CONFIG_ACPI_NUMA - else if ( !strncmp(opt,"noacpi",6) ) - { - numa_off = false; - fw_numa =-1 ; - } -#endif - else - return -EINVAL; - -return 0 ; -}/* * Setup early cpu_to_node.
@@ -395,146 +96,3 @@ int __init arch_meminfo_get_ram_bank_range(uint32_t bank,return 0 ; } - -static void dump_numa (unsigned char key) -{ -s_time_t now =NOW (); -unsigned int i, j, n; -struct domain *d; -struct page_info *page; -unsigned int page_num_node[MAX_NUMNODES ]; -const struct vnuma_info *vnuma; - -printk ("'%c' pressed -> dumping numa info (now = %" PRI_stime")\n" , key, - now); - -for_each_online_node ( i ) - { -paddr_t pa =pfn_to_paddr (node_start_pfn (i) +1 ); - -printk ("NODE%u start->%lu size->%lu free->%lu\n" , - i,node_start_pfn (i),node_spanned_pages (i), -avail_node_heap_pages (i)); -/* sanity check phys_to_nid() */ -if (phys_to_nid (pa) != i ) -printk ("phys_to_nid(%" PRIpaddr") -> %d should be %u\n" , - pa,phys_to_nid (pa), i); - } - - j =cpumask_first (&cpu_online_map); - n =0 ; -for_each_online_cpu ( i ) - { -if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] ) - { -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - j = i; - n =1 ; - } -else - ++n; - } -if ( n >1 ) -printk ("CPU%u...%u -> NODE%d\n" , j, j + n -1 , cpu_to_node[j]); -else -printk ("CPU%u -> NODE%d\n" , j, cpu_to_node[j]); - -rcu_read_lock (&domlist_read_lock); - -printk ("Memory location of each domain:\n" ); -for_each_domain ( d ) - { -process_pending_softirqs (); - -printk ("Domain %u (total: %u):\n" , d->domain_id,domain_tot_pages (d)); - -for_each_online_node ( i ) - page_num_node[i] =0 ; - -spin_lock (&d->page_alloc_lock); -page_list_for_each (page, &d->page_list) - { - i =phys_to_nid (page_to_maddr (page)); - page_num_node[i]++; - } -spin_unlock (&d->page_alloc_lock); - - for_each_online_node ( i ) - printk(" Node %u: %u\n", i, page_num_node[i]); - -if ( !read_trylock (&d->vnuma_rwlock) ) -continue ; - -if ( !d->vnuma ) - { -read_unlock (&d->vnuma_rwlock); -continue ; - } - - vnuma = d->vnuma; -printk (" %u vnodes, %u vcpus, guest physical layout:\n" , - vnuma->nr_vnodes, d->max_vcpus); -for ( i =0 ; i < vnuma->nr_vnodes; i++ ) - { -unsigned int start_cpu = ~0U ; - -if ( vnuma->vnode_to_pnode[i] ==NUMA_NO_NODE ) -printk (" %3u: pnode ???," , i); -else -printk (" %3u: pnode %3u," , i, vnuma->vnode_to_pnode[i]); - -printk (" vcpus " ); - -for ( j =0 ; j < d->max_vcpus; j++ ) - { -if ( !(j &0x3f ) ) -process_pending_softirqs (); - -if ( vnuma->vcpu_to_vnode[j] == i ) - { -if ( start_cpu == ~0U ) - { -printk ("%d" , j); - start_cpu = j; - } - } -else if ( start_cpu != ~0U ) - { -if ( j -1 != start_cpu ) -printk ("-%d " , j -1 ); -else -printk (" " ); - start_cpu = ~0U ; - } - } - -if ( start_cpu != ~0U && start_cpu != j -1 ) -printk ("-%d" , j -1 ); - -printk ("\n" ); - -for ( j =0 ; j < vnuma->nr_vmemranges; j++ ) - { -if ( vnuma->vmemrange[j].nid == i ) -printk (" %016" PRIx64" - %016" PRIx64"\n" , - vnuma->vmemrange[j].start, - vnuma->vmemrange[j].end); - } - } - -read_unlock (&d->vnuma_rwlock); - } - -rcu_read_unlock (&domlist_read_lock); -} - -static __init int register_numa_trigger (void ) -{ -register_keyhandler ('u' , dump_numa,"dump NUMA info" ,1 ); -return 0 ; -} -__initcall (register_numa_trigger); -
xen/arch/x86/srat.c +2 -251
300 lines not on this page · raw message
xen/common/Makefile +2 -0
@@ -26,6 +26,8 @@ obj-$(CONFIG_MEM_ACCESS) += mem_access.o obj-y += memory.o obj-y += multicall.o obj-y += notifier.o +obj-$(CONFIG_NUMA) += numa.o +obj-$(CONFIG_NUMA) += numa_srat.o obj-y += page_alloc.o obj-$(CONFIG_HAS_PDX) += pdx.o obj-$(CONFIG_PERF_COUNTERS) += perfc.o
xen/common/numa.c new file +450 -0
450 lines not on this page · raw message
xen/common/numa_srat.c new file +264 -0
264 lines not on this page · raw message
xen/include/asm-x86/acpi.h +0 -4
@@ -101,10 +101,6 @@ extern unsigned long acpi_wakeup_address;#define ARCH_HAS_POWER_INIT 1 -extern s8 fw_numa; -extern int numa_scan_nodes (u64 start,u64 end); -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2) -extern struct acpi_sleep_info acpi_sinfo;#define acpi_video_flags bootsym(video_flags)struct xenpf_enter_acpi_sleep ;
xen/include/asm-x86/numa.h +0 -68
85 lines not on this page · raw message
xen/include/asm-x86/setup.h +0 -1
@@ -17,7 +17,6 @@ void early_time_init(void);void set_nr_cpu_ids (unsigned int max_cpus); -void numa_initmem_init (unsigned long start_pfn,unsigned long end_pfn);void arch_init_memory (void );void subarch_init_memory (void );
xen/include/xen/numa.h +82 -0
92 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 19/37 xen/x86: promote VIRTUAL_BUG_ON to ASSERT in
In reply to Wei ChenVIRTUAL_BUG_ON that is using in phys_to_nid is an empty macro. This results in two lines of error-checking code in phys_to_nid are not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is because MAX_NUMNODES is defined in xen/numa.h. But asm/numa.h is a dependent file of xen/numa.h, we can't include xen/numa.h in asm/numa.h. This error has been fixed after we move phys_to_nid to xen/numa.h.
2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But node_data is a data structure variable, it's not a pointer.
So, in this patch, we use ASSERT in VIRTUAL_BUG_ON to enable the two lines of error-checking code. And fix the the left compilation errors by replacing !node_data[nid] to !node_data[nid].node_spanned_pages.
Because when node_spanned_pages is 0, this node has no memory. numa_scan_node will print warning message for such kind of nodes:
"Firmware Bug or mis-configured hardware?". Even Xen allows to online such kind of nodes. I still think it's impossible for phys_to_nid to return a no memory node for a physical address.
--- xen/include/xen/numa.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
xen/include/xen/numa.h +2 -2
@@ -38,7 +38,7 @@ struct node {extern int compute_hash_shift (struct node *nodes,int numnodes,nodeid_t *nodeids); -#define VIRTUAL_BUG_ON (x) +#define VIRTUAL_BUG_ON (x) ASSERT(!(x))extern void numa_add_cpu (int cpu);extern void numa_init_array (void );
@@ -75,7 +75,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)nodeid_t nid;VIRTUAL_BUG_ON ((paddr_to_pdx (addr) >> memnode_shift) >= memnodemapsize); nid = memnodemap[paddr_to_pdx (addr) >> memnode_shift]; -VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid]); +VIRTUAL_BUG_ON (nid >=MAX_NUMNODES || !node_data[nid].node_spanned_pages);return nid; }
Signature · 1 line
2.25.1
Re: [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in
Quoted 8 lines · Wei Chen
VIRTUAL_BUG_ONthat is using inphys_to_nidis an empty macro. This results in two lines of error-checking code inphys_to_nidare not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is becauseMAX_NUMNODESis defined inxen/numa.h. Butasm/numa.his a dependent file ofxen/numa.h, we can't includexen/numa.hinasm/numa.h. This error has been fixed after we movephys_to_nidtoxen/numa.h.
This could easily be taken care of by move MAX_NUMNODES up ahead of the asm/numa.h inclusion point. And then the change here would become independent of the rest of the series (and could hence go in early).
Quoted 6 lines
2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But
node_datais a data structure variable, it's not a pointer.So, in this patch, we use ASSERT in
VIRTUAL_BUG_ONto enable the two lines of error-checking code.
May I suggest to drop VIRTUAL_BUG_ON() and instead use ASSERT() directly?
Jan
RE: [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in
Hi Jan,
Quoted 26 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月18日 0:22
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 19/37] xen/x86: promoteVIRTUAL_BUG_ONto ASSERT inQuoted 8 lines · Wei Chen
VIRTUAL_BUG_ONthat is using inphys_to_nidis an empty macro. This results in two lines of error-checking code inphys_to_nidare not actually working. It also covers up two compilation errors:
1. error: ‘MAX_NUMNODES’ undeclared (first use in this function). This is becauseMAX_NUMNODESis defined inxen/numa.h. Butasm/numa.his a dependent file ofxen/numa.h, we can't includexen/numa.hinasm/numa.h. This error has been fixed after we movephys_to_nidtoxen/numa.h.This could easily be taken care of by move
MAX_NUMNODESup ahead of theasm/numa.hinclusion point. And then the change here would become independent of the rest of the series (and could hence go in early).Quoted 4 lines
2. error: wrong type argument to unary exclamation mark. This is becuase, the error-checking code contains !node_data[nid]. But
node_datais a data structure variable, it's not a pointer.
Quoted 6 lines
Quoted 2 lines
So, in this patch, we use ASSERT in
VIRTUAL_BUG_ONto enable the two lines of error-checking code.May I suggest to drop
VIRTUAL_BUG_ON()and instead useASSERT()directly?
Sure!
Quoted 1 line
Jan
PATCH 20/37 xen: introduce CONFIG_EFI to stub API for non-EFI architecture
In reply to Wei ChenSome architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this Kconfig option to give Xen the ability of stubing EFI API for non-EFI supported architectures.
--- xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 2 +- xen/arch/x86/Kconfig | 1 + xen/common/Kconfig | 11 +++++++++++ xen/include/xen/efi.h | 4 ++++ 5 files changed, 18 insertions(+), 1 deletion(-)
xen/arch/arm/Kconfig +1 -0
@@ -6,6 +6,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT + select EFI select HAS_FAST_MULTIPLY config ARM
xen/arch/arm/Makefile +1 -1
@@ -1,6 +1,6 @@ obj-$(CONFIG_ARM_32) += arm32/ obj-$(CONFIG_ARM_64) += arm64/ -obj-$(CONFIG_ARM_64) += efi/ +obj-$(CONFIG_EFI) += efi/ obj-$(CONFIG_ACPI) += acpi/ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/
xen/arch/x86/Kconfig +1 -0
@@ -10,6 +10,7 @@ config X86 select ALTERNATIVE_CALL select ARCH_SUPPORTS_INT128 select CORE_PARKING + select EFI select HAS_ALTERNATIVE select HAS_COMPAT select HAS_CPUFREQ
xen/common/Kconfig +11 -0
@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware. + config GRANT_TABLE bool "Grant table support" if EXPERT default y
@@ -196,6 +206,7 @@ config KEXEC config EFI_SET_VIRTUAL_ADDRESS_MAP bool "EFI: call SetVirtualAddressMap()" if EXPERT + depends on EFI ---help--- Call EFI SetVirtualAddressMap() runtime service to setup memory map for further runtime services. According to UEFI spec, it isn't strictly
xen/include/xen/efi.h +4 -0
@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;
@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ +#endif /* !__ASSEMBLY__ */ #endif /* __XEN_EFI_H__ */
Signature · 1 line
2.25.1
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 55 lines · Wei Chen
Some architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this
Kconfigoption to give Xen the ability of stubing EFI API for non-EFI supported architectures.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 2 +- xen/arch/x86/Kconfig | 1 + xen/common/Kconfig | 11 +++++++++++ xen/include/xen/efi.h | 4 ++++ 5 files changed, 18 insertions(+), 1 deletion(-)xen/arch/arm/Kconfig +1
-0@@ -6,6 +6,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT + select EFI select HAS_FAST_MULTIPLY config ARMxen/arch/arm/Makefile +1
-1@@ -1,6 +1,6 @@ obj-$(CONFIG_ARM_32) += arm32/ obj-$(CONFIG_ARM_64) += arm64/ -obj-$(CONFIG_ARM_64) += efi/ +obj-$(CONFIG_EFI) += efi/ obj-$(CONFIG_ACPI) += acpi/ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/xen/arch/x86/Kconfig +1
-0@@ -10,6 +10,7 @@ config X86 select ALTERNATIVE_CALL select ARCH_SUPPORTS_INT128 select CORE_PARKING + select EFI select HAS_ALTERNATIVE select HAS_COMPAT select HAS_CPUFREQxen/common/Kconfig +0
-0
xen/common/Kconfig config COMPAT · line 15
Quoted 5 lines
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + bool
Without the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 44 lines
xen/common/Kconfig +9
-0+ ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware. + config GRANT_TABLE bool "Grant table support" if EXPERT default y@@ -196,6 +206,7 @@ config KEXEC config EFI_SET_VIRTUAL_ADDRESS_MAP bool "EFI: call SetVirtualAddressMap()" if EXPERT + depends on EFI ---help--- Call EFI SetVirtualAddressMap() runtime service to setup memory map for further runtime services. According to UEFI spec, it isn't strictlyxen/include/xen/efi.h +4
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ +#endif /* !__ASSEMBLY__ */ #endif /* __XEN_EFI_H__ */ Signature · 1 line
2.25.1
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Stefano,
Quoted 74 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 55 lines · Wei Chen
Some architectures do not support EFI, but EFI API will be used in some common features. Instead of spreading #ifdef ARCH, we introduce this
Kconfigoption to give Xen the ability of stubing EFI API for non-EFI supported architectures.Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/Kconfig | 1 + xen/arch/arm/Makefile | 2 +- xen/arch/x86/Kconfig | 1 + xen/common/Kconfig | 11 +++++++++++ xen/include/xen/efi.h | 4 ++++ 5 files changed, 18 insertions(+), 1 deletion(-)xen/arch/arm/Kconfig +1
-0@@ -6,6 +6,7 @@ config ARM_64 def_bool y depends on !ARM_32 select 64BIT + select EFI select HAS_FAST_MULTIPLY config ARMxen/arch/arm/Makefile +1
-1@@ -1,6 +1,6 @@ obj-$(CONFIG_ARM_32) += arm32/ obj-$(CONFIG_ARM_64) += arm64/ -obj-$(CONFIG_ARM_64) += efi/ +obj-$(CONFIG_EFI) += efi/ obj-$(CONFIG_ACPI) += acpi/ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/xen/arch/x86/Kconfig +1
-0@@ -10,6 +10,7 @@ config X86 select ALTERNATIVE_CALL select ARCH_SUPPORTS_INT128 select CORE_PARKING + select EFI select HAS_ALTERNATIVE select HAS_COMPAT select HAS_CPUFREQxen/common/Kconfig +0
-0xen/common/Kconfig config COMPAT · line 15
Quoted 5 lines
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.
Quoted 7 lines
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
Quoted 47 lines
Quoted 13 lines
xen/common/Kconfig +8
-0+ ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware. + config GRANT_TABLE bool "Grant table support" if EXPERT default yxen/common/Kconfig config KEXEC · line 211
Quoted 6 lines
xen/common/Kconfig +1
-0@@ -196,6 +206,7 @@ config KEXEC config EFI_SET_VIRTUAL_ADDRESS_MAP bool "EFI: call SetVirtualAddressMap()" if EXPERT + depends on EFI ---help--- Call EFI SetVirtualAddressMap() runtime service to setup memorymap for
xen/common/Kconfig config KEXEC · line 212
Quoted 1 line
xen/common/Kconfig +0
-0further runtime services. According to UEFI spec, it isn'tstrictly
Quoted 25 lines
xen/include/xen/efi.h +4
-1@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ +#endif /* !__ASSEMBLY__ */ #endif /* __XEN_EFI_H__ */ --2.25.1
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 16 lines · Wei Chen
Quoted 16 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.
Quoted 8 lines
Quoted 5 lines
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 32 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 22 lines · Wei Chen
Quoted 21 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 8 lines
Quoted 2 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in Kconfig. I agree with you, we can't use Kconfig system to decide to enable EFI build for x86 or not.
So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.
Quoted 1 line
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 35 lines · Wei Chen
Quoted 26 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 22 lines · Wei Chen
Quoted 21 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-selectable). So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 8 lines
Quoted 2 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
Quoted 6 lines
I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in
Kconfig. I agree with you, we can't useKconfigsystem to decide to enable EFI build for x86 or not.So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 51 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Jan Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 14 lines · Wei Chen
Quoted 14 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 10 lines · Wei Chen
Quoted 10 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15Quoted 6 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/common/Kconfig config COMPAT · line 15
Quoted 2 lines
xen/common/Kconfig +2
-0+config EFI + boolWithout the title the option is not user-selectable (or de-
selectable).
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 8 lines
Quoted 2 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in
Kconfig. I agree with you, we can't useKconfigsystem to decide to enable EFI build for x86 or not.
Quoted 6 lines
Quoted 2 lines
So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
In last version, I had introduced an arch-helper to stub EFI_BOOT in Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce a CONFIG_EFI option for non-EFI supported architectures to stub in EFI layer.
[1] https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00808.html
Quoted 2 lines
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Julien, Stefano,
Quoted 72 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureHi Jan,
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf OfJan
Quoted 6 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 43 lines
EFI architecture
Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 10 lines · Wei Chen
Quoted 10 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15Quoted 6 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/common/Kconfig config COMPAT · line 15
Quoted 2 lines
xen/common/Kconfig +2
-0+config EFI + boolWithout the title the option is not user-selectable (or de-
selectable).
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 8 lines
Quoted 2 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in
Kconfig. I agree with you, we can't useKconfigsystem to decide to enable EFI build for x86 or not.Quoted 6 lines
Quoted 2 lines
So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Quoted 8 lines
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.html
As Jan' reminded, x86 doesn't depend on Kconfig to build EFI code. So, if we CONFIG_EFI to stub EFI API's for x86, we will encounter that toolchains enable EFI, but Kconfig disable EFI. Or Kconfig enable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.
If we use CONFIG_EFI for Arm only, that means CONFIG_EFI for x86 is off, this will also cause problem.
So, can we still use previous arch_helpers to stub for Arm32? until x86 can use this selectable option?
Quoted 2 lines
Quoted 2 lines
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 87 lines · Wei Chen
Quoted 72 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureHi Jan,
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf OfJan
Quoted 6 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 43 lines
EFI architecture
Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 10 lines · Wei Chen
Quoted 10 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15Quoted 6 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/common/Kconfig config COMPAT · line 15
Quoted 2 lines
xen/common/Kconfig +2
-0+config EFI + boolWithout the title the option is not user-selectable (or de-
selectable).
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help text.
Quoted 8 lines
Quoted 2 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending on tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this option at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not a selectable option in
Kconfig. I agree with you, we can't useKconfigsystem to decide to enable EFI build for x86 or not.Quoted 6 lines
Quoted 2 lines
So how about we just use this EFI option for Arm only? Because on Arm, we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Quoted 8 lines
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.htmlAs Jan' reminded, x86 doesn't depend on
Kconfigto build EFI code. So, if weCONFIG_EFIto stub EFI API's for x86, we will encounter that toolchains enable EFI, butKconfigdisable EFI. OrKconfigenable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.
Quoted 5 lines
If we use
CONFIG_EFIfor Arm only, that meansCONFIG_EFIfor x86 is off, this will also cause problem.So, can we still use previous
arch_helpersto stub for Arm32? until x86 can use this selectable option?
EFI doesn't have to be necessarily a user-visible option in Kconfig at this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just based CONFIG_ARM64.
On x86 EFI is detected based on compiler support, setting XEN_BUILD_EFI in xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.
On ARM32, XEN_BUILD_EFI should be always unset.
On ARM64 XEN_BUILD_EFI should be always set.
That's it, right? I'd argue that CONFIG_EFI or HAS_EFI are better names than XEN_BUILD_EFI, but that's OK anyway. So for instance you can make XEN_BUILD_EFI an invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Stefano,
Quoted 122 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 2 lines · Wei Chen
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf OfWei
Quoted 6 lines
Quoted 6 lines
Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 6 lines
Quoted 6 lines
EFI architecture
Hi Jan,
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On BehalfOf
Quoted 6 lines
Quoted 6 lines
Jan
Quoted 5 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano StabelliniQuoted 23 lines
Quoted 23 lines
Quoted 1 line
Subject: Re: [PATCH 20/37] xen: introduce
CONFIG_EFIto stub API fornon-
Quoted 21 lines
EFI architecture
Quoted 18 lines · Wei Chen
Quoted 18 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
selectable).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help
text.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending
on
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this
option
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
selectable option in
Kconfig. I agree with you, we can't use
KconfigQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
system to decide to enable EFI build for x86 or not.
So how about we just use this EFI option for Arm only? Because on
Arm,
Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Quoted 8 lines
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.htmlAs Jan' reminded, x86 doesn't depend on
Kconfigto build EFI code. So, if weCONFIG_EFIto stub EFI API's for x86, we will encounter that toolchains enable EFI, butKconfigdisable EFI. OrKconfigenable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.Quoted 5 lines
If we use
CONFIG_EFIfor Arm only, that meansCONFIG_EFIfor x86 is off, this will also cause problem.So, can we still use previous
arch_helpersto stub for Arm32? until x86 can use this selectable option?EFI doesn't have to be necessarily a user-visible option in
Kconfigat this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just basedCONFIG_ARM64.On x86 EFI is detected based on compiler support, setting
XEN_BUILD_EFIin xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.
Quoted 8 lines
On ARM32,
XEN_BUILD_EFIshould be always unset.On ARM64
XEN_BUILD_EFIshould be always set.That's it, right? I'd argue that
CONFIG_EFIorHAS_EFIare better names thanXEN_BUILD_EFI, but that's OK anyway. So for instance you can makeXEN_BUILD_EFIan invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.
Thanks, this is a good approach. But if we place XEN_BUILD_EFI in Kconfig it will be transfer to CONFIG_XEN_BUILD_EFI. How about using another name in Kconfig like ARM_EFI, but use CONFIG_ARM_EFI in config.h to define XEN_BUILD_EFI?
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 131 lines · Wei Chen
Quoted 122 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 2 lines · Wei Chen
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf OfWei
Quoted 6 lines
Quoted 6 lines
Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 6 lines
Quoted 6 lines
EFI architecture
Hi Jan,
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On BehalfOf
Quoted 6 lines
Quoted 6 lines
Jan
Quoted 5 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano StabelliniQuoted 23 lines
Quoted 23 lines
Quoted 1 line
Subject: Re: [PATCH 20/37] xen: introduce
CONFIG_EFIto stub API fornon-
Quoted 21 lines
EFI architecture
Quoted 18 lines · Wei Chen
Quoted 18 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
selectable).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help
text.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending
on
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this
option
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
selectable option in
Kconfig. I agree with you, we can't use
KconfigQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
system to decide to enable EFI build for x86 or not.
So how about we just use this EFI option for Arm only? Because on
Arm,
Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Quoted 8 lines
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.htmlAs Jan' reminded, x86 doesn't depend on
Kconfigto build EFI code. So, if weCONFIG_EFIto stub EFI API's for x86, we will encounter that toolchains enable EFI, butKconfigdisable EFI. OrKconfigenable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.Quoted 5 lines
If we use
CONFIG_EFIfor Arm only, that meansCONFIG_EFIfor x86 is off, this will also cause problem.So, can we still use previous
arch_helpersto stub for Arm32? until x86 can use this selectable option?EFI doesn't have to be necessarily a user-visible option in
Kconfigat this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just basedCONFIG_ARM64.On x86 EFI is detected based on compiler support, setting
XEN_BUILD_EFIin xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.Quoted 8 lines
On ARM32,
XEN_BUILD_EFIshould be always unset.On ARM64
XEN_BUILD_EFIshould be always set.That's it, right? I'd argue that
CONFIG_EFIorHAS_EFIare better names thanXEN_BUILD_EFI, but that's OK anyway. So for instance you can makeXEN_BUILD_EFIan invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.
Quoted 4 lines
Thanks, this is a good approach. But if we place
XEN_BUILD_EFIinKconfigit will be transfer toCONFIG_XEN_BUILD_EFI. How about using another name inKconfiglikeARM_EFI, but useCONFIG_ARM_EFIinconfig.hto defineXEN_BUILD_EFI?
I am OK with that. Another option is to rename XEN_BUILD_EFI to CONFIG_XEN_BUILD_EFI on x86. Either way is fine by me. Jan, do you havea preference?
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 132 lines · Stefano Stabellini
Quoted 131 lines · Wei Chen
Quoted 130 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 2 lines · Wei Chen
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf OfWei
Quoted 6 lines
Quoted 6 lines
Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 6 lines
Quoted 6 lines
EFI architecture
Hi Jan,
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On BehalfOf
Quoted 6 lines
Quoted 6 lines
Jan
Quoted 5 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano StabelliniQuoted 23 lines
Quoted 23 lines
Quoted 1 line
Subject: Re: [PATCH 20/37] xen: introduce
CONFIG_EFIto stub API fornon-
Quoted 21 lines
EFI architecture
Quoted 18 lines · Wei Chen
Quoted 18 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
selectable).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help
text.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending
on
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this
option
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
selectable option in
Kconfig. I agree with you, we can't use
KconfigQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
system to decide to enable EFI build for x86 or not.
So how about we just use this EFI option for Arm only? Because on
Arm,
Quoted 21 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you want to use the option for subsequently.
Quoted 8 lines
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.htmlAs Jan' reminded, x86 doesn't depend on
Kconfigto build EFI code. So, if weCONFIG_EFIto stub EFI API's for x86, we will encounter that toolchains enable EFI, butKconfigdisable EFI. OrKconfigenable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.Quoted 5 lines
If we use
CONFIG_EFIfor Arm only, that meansCONFIG_EFIfor x86 is off, this will also cause problem.So, can we still use previous
arch_helpersto stub for Arm32? until x86 can use this selectable option?EFI doesn't have to be necessarily a user-visible option in
Kconfigat this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just basedCONFIG_ARM64.On x86 EFI is detected based on compiler support, setting
XEN_BUILD_EFIin xen/arch/x86/Makefile. Let's say that we keep using the same name "XEN_BUILD_EFI" on ARM as well.On ARM32,
XEN_BUILD_EFIshould be always unset.On ARM64
XEN_BUILD_EFIshould be always set.That's it, right? I'd argue that
CONFIG_EFIorHAS_EFIare better names thanXEN_BUILD_EFI, but that's OK anyway. So for instance you can makeXEN_BUILD_EFIan invisible symbol in xen/arch/arm/Kconfig and select it only on ARM64.
Quoted 8 lines
Quoted 4 lines
Thanks, this is a good approach. But if we place
XEN_BUILD_EFIinKconfigit will be transfer toCONFIG_XEN_BUILD_EFI. How about using another name inKconfiglikeARM_EFI, but useCONFIG_ARM_EFIinconfig.hto defineXEN_BUILD_EFI?I am OK with that. Another option is to rename
XEN_BUILD_EFItoCONFIG_XEN_BUILD_EFIon x86. Either way is fine by me. Jan, do you havea preference?
Yes, I do: No new CONFIG_* settings please that don't originate from Kconfig. Hence I'm afraid this is a "no" to your suggestion.
Mid-term we should try to get rid of the remaining CONFIG_* which get #define-d in e.g. asm/config.h.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 165 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2021年9月28日 16:03
To: Stefano Stabellini <[email protected]>; Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 9 lines · Stefano Stabellini
Quoted 8 lines · Wei Chen
Quoted 8 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月28日 9:00
To: Wei Chen <[email protected]>
Cc: Jan Beulich <[email protected]>; [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano Stabellini <[email protected]>
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
EFI architecture
Quoted 2 lines · Wei Chen
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On BehalfOf
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Wei
Quoted 5 lines
Quoted 5 lines
Chen
Sent: 2021年9月26日 18:25
To: Jan Beulich <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; Stefano StabelliniQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Subject: RE: [PATCH 20/37] xen: introduce
CONFIG_EFIto stub API fornon-
Quoted 4 lines
Quoted 4 lines
EFI architecture
Hi Jan,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
-----Original Message-----
From: Xen-devel <[email protected]> On BehalfOf
Quoted 5 lines
Quoted 5 lines
Jan
Quoted 4 lines
Beulich
Sent: 2021年9月24日 18:49
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; BertrandMarquis
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
<[email protected]>; Stefano Stabellini
Quoted 1 line
Quoted 1 line
Quoted 1 line
Subject: Re: [PATCH 20/37] xen: introduce
CONFIG_EFIto stub APIfor
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 22 lines
Quoted 22 lines
non-
Quoted 21 lines
EFI architecture
Quoted 18 lines · Wei Chen
Quoted 18 lines
From: Jan Beulich <[email protected]>
Sent: 2021年9月24日 15:59Quoted 14 lines · Wei Chen
Quoted 14 lines
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:15xen/common/Kconfig config COMPAT · line 15
Quoted 8 lines · Wei Chen
xen/common/Kconfig +2
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool +config EFI + boolWithout the title the option is not user-selectable (or de-
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
selectable).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So the help message below can never be seen.
Either add a title, e.g.:
bool "EFI support"
Or fully make the option a silent option by removing the help
text.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
OK, in current Xen code, EFI is unconditionally compiled. Before we change related code, I prefer to remove the help text.
But that's not true: At least on x86 EFI gets compiled depending
on
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
tool chain capabilities. Ultimately we may indeed want a user selectable option here, but until then I'm afraid having this
option
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
at all may be misleading on x86.
I check the build scripts, yes, you're right. For x86, EFI is not
a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
selectable option in
Kconfig. I agree with you, we can't useQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
KconfigQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
system to decide to enable EFI build for x86 or not.
So how about we just use this EFI option for Arm only? Because on
Arm,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
we do not have such toolchain dependency.
To be honest - don't know. That's because I don't know what you
want
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
to use the option for subsequently.
In last version, I had introduced an arch-helper to stub
EFI_BOOTin Arm's common code for Arm32. Because Arm32 doesn't support EFI. So Julien suggested me to introduce aCONFIG_EFIoption for non-EFI supported architectures to stub in EFI layer.[1] https://lists.xenproject.org/archives/html/xen-devel/2021-
08/msg00808.htmlAs Jan' reminded, x86 doesn't depend on
Kconfigto build EFI code. So, if weCONFIG_EFIto stub EFI API's for x86, we will encounter that toolchains enable EFI, butKconfigdisable EFI. OrKconfigenable EFI but toolchain doesn't provide EFI build supports. And then x86 could not work well.If we use
CONFIG_EFIfor Arm only, that meansCONFIG_EFIfor x86 is off, this will also cause problem.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
So, can we still use previous
arch_helpersto stub for Arm32? until x86 can use this selectable option?EFI doesn't have to be necessarily a user-visible option in
Kconfigat this point. I think Julien was just asking to make the #ifdef based on a EFI-related config rather than just basedCONFIG_ARM64.On x86 EFI is detected based on compiler support, setting
XEN_BUILD_EFIQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
in xen/arch/x86/Makefile. Let's say that we keep using the same name "
XEN_BUILD_EFI" on ARM as well.On ARM32,
XEN_BUILD_EFIshould be always unset.On ARM64
XEN_BUILD_EFIshould be always set.That's it, right? I'd argue that
CONFIG_EFIorHAS_EFIare betternames
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
than
XEN_BUILD_EFI, but that's OK anyway. So for instance you can makeXEN_BUILD_EFIan invisible symbol in xen/arch/arm/Kconfig and selectit
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
only on ARM64.
Thanks, this is a good approach. But if we place
XEN_BUILD_EFIin
KconfigQuoted 1 line
Quoted 1 line
it will be transfer to
CONFIG_XEN_BUILD_EFI. How about using anothername
Quoted 6 lines
Quoted 2 lines
in
KconfiglikeARM_EFI, but useCONFIG_ARM_EFIinconfig.hto defineXEN_BUILD_EFI?I am OK with that. Another option is to rename
XEN_BUILD_EFItoCONFIG_XEN_BUILD_EFIon x86. Either way is fine by me. Jan, do you havea preference?
Quoted 6 lines
Yes, I do: No new CONFIG_* settings please that don't originate from
Kconfig. Hence I'm afraid this is a "no" to your suggestion.Mid-term we should try to get rid of the remaining CONFIG_* which get #define-d in e.g.
asm/config.h.
I will do something like this:
- introduce an ARM_EFI invisible symbol in kconfig, selected by ARM64 only
- use CONFIG_ARM_EFI to define XEN_BUILD_EFI in config.h
Quoted 1 line
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
In reply to Wei ChenQuoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING bool
xen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.
The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
Quoted 12 lines
xen/include/xen/efi.h +2
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/
I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 33 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolxen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Quoted 3 lines
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used CONFIG_ARM_EFI to replace this common EFI config in my latest version. This Kconfig option has been removed. And thanks, I will not use --help-- anymore.
Quoted 24 lines
Quoted 12 lines
xen/include/xen/efi.h +2
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 60 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2022年1月27日 16:45
To: Jan Beulich <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureHi Jan,
Quoted 7 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API fornon-
Quoted 26 lines
EFI architecture
Quoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolxen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Quoted 3 lines
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used
CONFIG_ARM_EFIto replace this common EFI config in my latest version. ThisKconfigoption has been removed. And thanks, I will not use --help-- anymore.Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__
Quoted 18 lines
Quoted 6 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call
Quoted 6 lines
xen/include/xen/efi.h +2
-0+#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call*);
Quoted 11 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in efi.h is that, some architectures have implemented stub helpers in their stub.c. If we define stub helpers in efi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function in efi.h. So we use stub file instead of stubing in efi.h
Quoted 1 line
Quoted 1 line
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 63 lines · Wei Chen
Quoted 45 lines
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2022年1月27日 16:45Quoted 27 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35Quoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolxen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Quoted 3 lines
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used
CONFIG_ARM_EFIto replace this common EFI config in my latest version. ThisKconfigoption has been removed. And thanks, I will not use --help-- anymore.Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__Quoted 17 lines
Quoted 6 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call
Quoted 6 lines
xen/include/xen/efi.h +2
-0+#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call*);
Quoted 10 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Quoted 9 lines
I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in
efi.his that, some architectures have implemented stub helpers in theirstub.c. If we define stub helpers inefi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function inefi.h. So we use stub file instead of stubing inefi.h
But you realize we already have such a stub file on x86?
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 84 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 1 line · Wei Chen
Quoted 1 line
From: Xen-devel <[email protected]> On Behalf Of
Wei
Quoted 62 lines
Quoted 44 lines
Chen
Sent: 2022年1月27日 16:45Quoted 27 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35Quoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolxen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Quoted 3 lines
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used
CONFIG_ARM_EFIto replace this common EFI config in my latest version. ThisKconfigoption has been removed. And thanks, I will not use --help-- anymore.Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__Quoted 17 lines
Quoted 6 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call
Quoted 6 lines
xen/include/xen/efi.h +2
-0+#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call*);
Quoted 10 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Quoted 9 lines
I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in
efi.his that, some architectures have implemented stub helpers in theirstub.c. If we define stub helpers inefi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function inefi.h. So we use stub file instead of stubing inefi.h
Quoted 2 lines
But you realize we already have such a stub file on x86?
Yes, we found the redefinition errors that are caused by x86 stub file and new macros in stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS += -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similar stub.c in arch/arm.
Quoted 1 line
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 81 lines · Wei Chen
Quoted 78 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00Quoted 1 line · Wei Chen
Quoted 1 line
From: Xen-devel <[email protected]> On Behalf Of
Wei
Quoted 62 lines
Quoted 44 lines
Chen
Sent: 2022年1月27日 16:45Quoted 27 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35Quoted 5 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolxen/common/Kconfig config COMPAT · line 22
Quoted 10 lines
xen/common/Kconfig +9
-0+config EFI + bool + ---help--- + This option provides support for runtime services provided + by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very least the help text would need to call out the extra dependencies. Plus the help text of course then needs to be generic (i.e. applicable to both Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Quoted 3 lines
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used
CONFIG_ARM_EFIto replace this common EFI config in my latest version. ThisKconfigoption has been removed. And thanks, I will not use --help-- anymore.Quoted 6 lines
Quoted 6 lines
xen/include/xen/efi.h +0
-0@@ -25,6 +25,8 @@ extern struct efi efi;#ifndef __ASSEMBLY__Quoted 17 lines
Quoted 6 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call
Quoted 6 lines
xen/include/xen/efi.h +2
-0+#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct xenpf_efi_runtime_call*);
Quoted 10 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Quoted 9 lines
I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in
efi.his that, some architectures have implemented stub helpers in theirstub.c. If we define stub helpers inefi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function inefi.h. So we use stub file instead of stubing inefi.hQuoted 2 lines
But you realize we already have such a stub file on x86?
Quoted 9 lines
Yes, we found the redefinition errors that are caused by x86 stub file and new macros in
stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS+= -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similarstub.cinarch/arm.
While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why stub.c had to be x86- only.
Jan
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Jan,
Quoted 105 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureQuoted 19 lines · Wei Chen
Quoted 19 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00Quoted 1 line · Wei Chen
Quoted 1 line
From: Xen-devel <[email protected]> On Behalf Of
Wei
Quoted 13 lines
Quoted 13 lines
Chen
Sent: 2022年1月27日 16:45Quoted 10 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:35Quoted 6 lines · Wei Chen
xen/common/Kconfig +0
-0@@ -11,6 +11,16 @@ config COMPAT config CORE_PARKING boolQuoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xen/common/Kconfig +4
-0+config EFI + bool + ---help--- + This option provides support for runtime services providedQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/common/Kconfig config COMPAT · line 22
Quoted 5 lines
xen/common/Kconfig +5
-0+ by UEFI firmware (such as non-volatile variables, realtime + clock, and platform reset). A UEFI stub is also provided to + allow the kernel to be booted as an EFI application. This + is only useful for kernels that may run on systems that have + UEFI firmware.The way enabling of (full) EFI support works on x86, I consider it wrong / misleading to put the option in common code. At the very
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the help text would need to call out the extra dependencies. Plus
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
help text of course then needs to be generic (i.e. applicable to
both
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines
Arm and x86). That's notwithstanding the fact that without a prompt the help text won't ever be seen while configuring Xen.
Also (nit): Indentation. And please don't use ---help--- anymore in new code.
I have used
CONFIG_ARM_EFIto replace this common EFI config in my latest version. ThisKconfigoption has been removed. And thanks, I will not use --help-- anymore.Quoted 4 lines
Quoted 4 lines
xen/include/xen/efi.h +0
-0@@ -25,6 +25,8 @@ extern struct efi efi;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
xen/include/xen/efi.h int efi_runtime_call(struct
Quoted 8 lines
xen/include/xen/efi.h +2
-0#ifndef __ASSEMBLY__ +#ifdef CONFIG_EFI +union xenpf_efi_info ;union compat_pf_efi_info ;@@ -45,6 +47,8 @@ int efi_runtime_call(struct
xenpf_efi_runtime_callQuoted 25 lines
Quoted 22 lines
Quoted 12 lines
Quoted 11 lines
*);
Quoted 10 lines
xen/include/xen/efi.h int efi_runtime_call(struct · line 50
Quoted 4 lines
xen/include/xen/efi.h +1
-0int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#endif /* CONFIG_EFI*/ I can see that in the later patch, when introducing inline stubs, you would need conditionals here, but I don't think you need them right here (or you may want to introduce the stubs right away).
Also (nit): Missing blank in the comment.
Quoted 9 lines
I am sorry, I had missed this comment. In my latest changes, I have introduced a stub file for non-EFI architectures. The reason why we don't use a macro to stub the helpers in
efi.his that, some architectures have implemented stub helpers in theirstub.c. If we define stub helpers inefi.h, this will cause function redefinition error. We need to fix this error for all architectures. And some helpers is not easy to implement as a inline function inefi.h. So we use stub file instead of stubing inefi.hQuoted 2 lines
But you realize we already have such a stub file on x86?
Quoted 9 lines
Yes, we found the redefinition errors that are caused by x86 stub file and new macros in
stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS+= -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similarstub.cinarch/arm.
Quoted 3 lines
While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why
stub.chad to be x86- only.
Some dummy functions in stub.c can be shared by arm or other architectures. But some functions like efi_multiboot2 are architecture dependent.
Quoted 2 lines
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Quoted 20 lines · Wei Chen
Quoted 20 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17Quoted 15 lines · Wei Chen
Quoted 5 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00But you realize we already have such a stub file on x86?
Yes, we found the redefinition errors that are caused by x86 stub file and new macros in
stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS+= -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similarstub.cinarch/arm.
Quoted 6 lines
Quoted 3 lines
While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why
stub.chad to be x86- only.Some dummy functions in
stub.ccan be shared by arm or other architectures. But some functions likeefi_multiboot2are architecture dependent.
Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.
Jan
Re: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi,
Quoted 25 lines · Jan Beulich
Quoted 24 lines · Wei Chen
Quoted 23 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17Quoted 6 lines · Wei Chen
Quoted 5 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00But you realize we already have such a stub file on x86?
Quoted 9 lines
Yes, we found the redefinition errors that are caused by x86 stub file and new macros in
stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS+= -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similarstub.cinarch/arm.While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why
stub.chad to be x86- only.
Quoted 7 lines
Quoted 2 lines
Some dummy functions in
stub.ccan be shared by arm or other architectures. But some functions likeefi_multiboot2are architecture dependent.Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.
If the stubs are mostly the same then they should be common rather than duplicated.
Quoted 2 lines
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me.
But that would be a build risk right? If so, it is quite easy to verify/catch it compare to runtime issue.
Cheers,
Quoted 3 lines
Jan
Signature · 1 line
Julien Grall
RE: [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture
Hi Julien,
Quoted 47 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 2022年1月27日 18:01
To: Jan Beulich <[email protected]>; Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]
Subject: Re: [PATCH 20/37] xen: introduceCONFIG_EFIto stub API for non- EFI architectureHi,
Quoted 4 lines · Jan Beulich
Quoted 3 lines · Wei Chen
Quoted 3 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:17Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Wei Chen
Quoted 7 lines
Quoted 5 lines
From: Jan Beulich <[email protected]>
Sent: 2022年1月27日 17:00But you realize we already have such a stub file on x86?
Yes, we found the redefinition errors that are caused by x86 stub
file
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
and new macros in
stub.h. We had tries to add:
ifeq ($(XEN_BUILD_EFI),y)
CFLAGS-y += -DXEN_BUILD_EFIXEN_CFLAGS+= -DXEN_BUILD_EFI
endif
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me. So I want to introduce a similarstub.cinarch/arm.Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
While that's perhaps fine, ideally common bits would be common. Iirc already back at the time I was wondering why
stub.chad to be x86- only.Some dummy functions in
stub.ccan be shared by arm or otherarchitectures.
Quoted 6 lines
Quoted 1 line
But some functions like
efi_multiboot2are architecture dependent.Right - that's no different from the bulk of the non-stubbed EFI code. I don't really mind you making an Arm-specific stub file if there's not very much duplication, but it doesn't feel very good. In the end it's up to the Arm maintainers anyway.
Quoted 9 lines
If the stubs are mostly the same then they should be common rather than duplicated.
Quoted 2 lines
x86/Makefile to gate these new macros, but it seems that we may need to change EFI build logic for x86. It will cause more risks for me.
But that would be a build risk right? If so, it is quite easy to verify/catch it compare to runtime issue.
Risk may be not very accurate word. I mean, I was afraid that my changes to x86 EFI build would introduce some unforeseen problems to x86. Because I am not very familiar with that. As you and Jan think it's better, I will try to do it.
Quoted 8 lines
Cheers,
Quoted 3 lines
Jan
--
Julien Grall
PATCH 21/37 xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenEFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to prevent early_scan_node to parse memory nodes in EFI boot.
As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent
--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/xen/efi.h | 7 +++++++ 3 files changed, 14 insertions(+), 26 deletions(-)
xen/arch/arm/bootfdt.c +7 -1
@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h>
@@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If Xen has been booted via UEFI, the memory banks will already +* be populated. So we should skip the parsing. +*/ +if ( !efi_enabled (EFI_BOOT ) && +device_tree_node_matches (fdt, node,"memory" )) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )
xen/arch/arm/efi/efi-boot.h +0 -25
@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions.
xen/include/xen/efi.h +7 -0
@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#else + +static inline bool efi_enabled (unsigned int feature) +{ +return false; +} +#endif /* CONFIG_EFI*/ #endif /* !__ASSEMBLY__ */
Signature · 1 line
2.25.1
Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Quoted 12 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
Quoted 7 lines
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to preventearly_scan_nodeto parse memory nodes in EFI boot.As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent
This last sentence is incomplete.
But aside from that, this patch looks good to me.
Quoted 92 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/xen/efi.h | 7 +++++++ 3 files changed, 14 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +7
-1@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ; -if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If Xen has been booted via UEFI, the memory banks will already +* be populated. So we should skip the parsing. +*/ +if ( !efi_enabled (EFI_BOOT ) && +device_tree_node_matches (fdt, node,"memory" )) rc =process_memory_node (fdt, node, name, depth, address_cells, size_cells, &bootinfo.mem);else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )xen/arch/arm/efi/efi-boot.h +0
-25@@ -197,33 +197,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/xen/efi.h +7
-0@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#else + +static inline bool efi_enabled (unsigned int feature) +{ +return false; +} +#endif /* CONFIG_EFI*/ #endif /* !__ASSEMBLY__ */ Signature · 1 line
2.25.1
RE: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Quoted 28 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:23
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIQuoted 17 lines · Wei Chen
EFI can get memory map from EFI system table. But EFI system table doesn't contain memory NUMA information, EFI depends on ACPI SRAT or device tree memory node to parse memory blocks' NUMA mapping.
But in current code, when Xen is booting from EFI, it will delete all memory nodes in device tree. So in UEFI + DTB boot, we don't have numa-node-id for memory blocks any more.
So in this patch, we will keep memory nodes in device tree for NUMA code to parse memory numa-node-id later.
As a side effect, if we still parse boot memory information in
early_scan_node, bootmem.info will calculate memory ranges in memory nodes twice. So we have to preventearly_scan_nodeto parse memory nodes in EFI boot.
Quoted 7 lines
Quoted 2 lines
As EFI APIs only can be used in Arm64, so we introduced a stub API for non-EFI supported Arm32. This will prevent
This last sentence is incomplete.
But aside from that, this patch looks good to me.
Ah, it truncated by accident. I will fix it.
Quoted 95 lines
Quoted 23 lines
Signed-off-by Wei Chen <[email protected]>--- xen/arch/arm/bootfdt.c | 8 +++++++- xen/arch/arm/efi/efi-boot.h | 25 ------------------------- xen/include/xen/efi.h | 7 +++++++ 3 files changed, 14 insertions(+), 26 deletions(-)xen/arch/arm/bootfdt.c +1
-0@@ -11,6 +11,7 @@#include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> +#include <xen/efi.h> #include <xen/device_tree.h> #include <xen/libfdt/libfdt.h> #include <xen/sort.h> @@ -370,7 +371,12 @@ static int __init early_scan_node(const void *fdt, {int rc =0 ;xen/arch/arm/bootfdt.c static int __init early_scan_node(const void *fdt, · line 381
Quoted 10 lines
xen/arch/arm/bootfdt.c +6
-1-if (device_tree_node_matches (fdt, node,"memory" ) ) +/* +* If Xen has been booted via UEFI, the memory banks will already +* be populated. So we should skip the parsing. +*/ +if ( !efi_enabled (EFI_BOOT ) && +device_tree_node_matches (fdt, node,"memory" )) rc = process_memory_node(fdt, node, name, depth, address_cells, size_cells,&bootinfo.mem);
xen/arch/arm/efi/efi-boot.h EFI_STATUS __init
Quoted 6 lines
xen/arch/arm/bootfdt.c +0
-0else if ( depth ==1 && !dt_node_cmp (name,"reserved-memory" ) )xen/arch/arm/efi/efi-boot.h +0
-0@@ -197,33 +197,8 @@ EFI_STATUS __initfdt_add_uefi_nodes(EFI_SYSTEM_TABLE
*sys_table,Quoted 54 lines
xen/arch/arm/efi/efi-boot.h +0
-25int status;u32 fdt_val32;u64 fdt_val64; -int prev;int num_rsv; -/* -* Delete any memory nodes present. The EFI memory map is the only -* memory description provided to Xen. -*/ - prev =0 ; -for (;;) - { -const char *type; -int len; - - node =fdt_next_node (fdt, prev,NULL ); -if ( node <0 ) -break ; - - type =fdt_getprop (fdt, node,"device_type" , &len); -if ( type &&strncmp (type,"memory" , len) ==0 ) - { -fdt_del_node (fdt, node); -continue ; - } - - prev = node; - } -/* * Delete all memory reserve map entries. When booting via UEFI, * kernel will use the UEFI memory map to find reserved regions. xen/include/xen/efi.h +7
-1@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#else + +static inline bool efi_enabled (unsigned int feature) +{ +return false; +} +#endif /* CONFIG_EFI*/ #endif /* !__ASSEMBLY__ */ --2.25.1
Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
In reply to Wei ChenQuoted 4 lines · Wei Chen
xen/include/xen/efi.h +0
-0@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 57
Quoted 10 lines
xen/include/xen/efi.h +7
-0int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *); +#else + +static inline bool efi_enabled (unsigned int feature) +{ +return false; +} +#endif /* CONFIG_EFI*/
As already hinted at for the earlier patch, I think this hunk wants moving ahead in the series.
Jan
RE: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
Hi Jan,
Quoted 16 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 2022年1月25日 18:39
To: Wei Chen <[email protected]>
Cc: Bertrand Marquis <[email protected]>; xen- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFIQuoted 6 lines · Wei Chen
xen/include/xen/efi.h +0
-0@@ -47,6 +47,13 @@ int efi_runtime_call(struct xenpf_efi_runtime_call *);int efi_compat_get_info (uint32_t idx,union compat_pf_efi_info *);int efi_compat_runtime_call (struct compat_pf_efi_runtime_call *);
Quoted 11 lines
xen/include/xen/efi.h int efi_runtime_call(struct xenpf_efi_runtime_call *); · line 57
Quoted 8 lines
xen/include/xen/efi.h +7
-0+#else + +static inline bool efi_enabled (unsigned int feature) +{ +return false; +} +#endif /* CONFIG_EFI*/ As already hinted at for the earlier patch, I think this hunk wants moving ahead in the series.
Yes, I will do it.
Thanks!
Quoted 2 lines
Jan
PATCH 22/37 xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
In reply to Wei ChenAs a memory range described in device tree cannot be split across multiple nodes. So we define NR_NODE_MEMBLKS as NR_MEM_BANKS in arch header. And keep default NR_NODE_MEMBLKS in common header for those architectures NUMA is disabled.
--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)
xen/include/asm-arm/numa.h +7 -1
@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0
xen/include/xen/numa.h +2 -0
@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif +#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor))
Signature · 1 line
2.25.1
Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Quoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.
This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
I am asking because NR_MEM_BANKS is 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so again NR_NODE_MEMBLKS is 128 before this patch.
In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless CONFIG_NR_NUMA_NODES is increased.
So, is the goal to reduce memory usage when CONFIG_NR_NUMA_NODES is higher than 64?
Quoted 2 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.
This last sentence is not accurate: on x86 NUMA is enabled and NR_NODE_MEMBLKS is still defined in xen/include/xen/numa.h (there is no x86 definition of it)
Quoted 44 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +2
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif +#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor))Signature · 1 line
2.25.1
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Hi Stefano,
Quoted 9 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKS
Quoted 8 lines
Quoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't thought so deeply. We just thought a memory range described in DT cannot be split across multiple nodes. So NR_MEM_BANKS should be equal to NR_MEM_BANKS.
https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00974.html
Quoted 10 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless
CONFIG_NR_NUMA_NODESis increased.So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis higher than 64?
I also thought about this problem when I was writing this patch. CONFIG_NR_NUMA_NODES is increasing, but NR_MEM_BANKS is a fixed value, then NR_MEM_BANKS can be smaller than CONFIG_NR_NUMA_NODES at one point.
But I agree with Julien's suggestion, NR_MEM_BANKS and NR_NODE_MEMBLKS must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.
Quoted 8 lines
Quoted 3 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(there is no x86 definition of it)
Yes.
Quoted 45 lines
Quoted 45 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +2
-1@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif +#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif #define vcpu_to_node (v) (cpu_to_node((v)->processor)) --2.25.1
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Quoted 40 lines · Wei Chen
Quoted 9 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKSQuoted 8 lines
Quoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't thought so deeply. We just thought a memory range described in DT cannot be split across multiple nodes. So
NR_MEM_BANKSshould be equal toNR_MEM_BANKS.https://lists.xenproject.org/archives/html/xen-devel/2021-08/msg00974.html
Quoted 10 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless
CONFIG_NR_NUMA_NODESis increased.So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis higher than 64?I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.
Quoted 4 lines
But I agree with Julien's suggestion,
NR_MEM_BANKSandNR_NODE_MEMBLKSmust be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.
OK. In that case I'd say to get rid of the previous definition of NR_NODE_MEMBLKS as it is probably not necessary, see below.
Quoted 47 lines
Quoted 8 lines
Quoted 3 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(there is no x86 definition of it)Yes.
Quoted 36 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif
This one we can remove it completely right?
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Hi Stefano,
Quoted 114 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKSQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 7 lines
Quoted 7 lines
NR_NODE_MEMBLKSQuoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.Quoted 5 lines
Quoted 3 lines
This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't
thought
Quoted 1 line
so deeply. We just thought a memory range described in DT cannot be
split
Quoted 3 lines
across multiple nodes. So
NR_MEM_BANKSshould be equal toNR_MEM_BANKS.08/msg00974.html
Quoted 17 lines
Quoted 11 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless
CONFIG_NR_NUMA_NODESis increased.So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis higher than 64?I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.Quoted 4 lines
But I agree with Julien's suggestion,
NR_MEM_BANKSandNR_NODE_MEMBLKSmust be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.OK. In that case I'd say to get rid of the previous definition of
NR_NODE_MEMBLKSas it is probably not necessary, see below.Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(there isno
Quoted 41 lines
Quoted 2 lines
x86 definition of it)
Yes.
Quoted 36 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif This one we can remove it completely right?
How about define NR_MEM_BANKS to:
#ifdef CONFIG_NR_NUMA_NODES
#define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * 2)
#else
#define NR_MEM_BANKS 128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS 128". And replace all NR_NODE_MEMBLKS in NUMA code to NR_MEM_BANKS to remove NR_NODE_MEMBLKS completely. In this case, NR_MEM_BANKS can be aware of the changes of CONFIG_NR_NUMA_NODES.
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
+x86 maintainers
Quoted 120 lines · Wei Chen
Quoted 114 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKSQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 7 lines
Quoted 7 lines
NR_NODE_MEMBLKSQuoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.Quoted 5 lines
Quoted 3 lines
This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't
thought
Quoted 1 line
so deeply. We just thought a memory range described in DT cannot be
split
Quoted 3 lines
across multiple nodes. So
NR_MEM_BANKSshould be equal toNR_MEM_BANKS.08/msg00974.html
Quoted 17 lines
Quoted 11 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at least doesn't make any difference unless
CONFIG_NR_NUMA_NODESis increased.So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis higher than 64?I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.Quoted 4 lines
But I agree with Julien's suggestion,
NR_MEM_BANKSandNR_NODE_MEMBLKSmust be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.OK. In that case I'd say to get rid of the previous definition of
NR_NODE_MEMBLKSas it is probably not necessary, see below.Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(there isno
Quoted 41 lines
Quoted 2 lines
x86 definition of it)
Yes.
Quoted 36 lines
Quoted 36 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif This one we can remove it completely right?
Quoted 10 lines
How about define
NR_MEM_BANKSto:
#ifdefCONFIG_NR_NUMA_NODES
#defineNR_MEM_BANKS(CONFIG_NR_NUMA_NODES* 2)
#else
#defineNR_MEM_BANKS128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS128". And replace allNR_NODE_MEMBLKSin NUMA code toNR_MEM_BANKSto removeNR_NODE_MEMBLKScompletely. In this case,NR_MEM_BANKScan be aware of the changes ofCONFIG_NR_NUMA_NODES.
x86 doesn't have NR_MEM_BANKS as far as I can tell. I guess you also meant to rename NR_NODE_MEMBLKS to NR_MEM_BANKS?
But NR_MEM_BANKS is not directly related to CONFIG_NR_NUMA_NODES because there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:
#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?
On the other hand, NR_MEM_BANKS and NR_NODE_MEMBLKS seem to be related. In fact, what's the difference?
NR_MEM_BANKS is the max number of memory banks (with or without numa-node-id).
NR_NODE_MEMBLKS is the max number of memory banks with NUMA support (with numa-node-id)?
They are basically the same thing. On ARM I would just do:
#define NR_NODE_MEMBLKS MAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))
And maybe the definition could be common with x86 if we define NR_MEM_BANKS to 128 on x86 too.
Re: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Quoted 162 lines · On Mon, 27 Sep 2021, 06:59 Stefano Stabellini, <[email protected]> wrote:
+x86 maintainers
Quoted 8 lines · Wei Chen
Quoted 8 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 23 lines
Quoted 23 lines
NR_NODE_MEMBLKSQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 2 lines
Quoted 2 lines
Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 11 lines
Quoted 10 lines
NR_NODE_MEMBLKSQuoted 3 lines · Wei Chen
As a memory range described in device tree cannot be split across multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.This statement is true but what is the goal of this patch? Is it to reduce code size and memory consumption?
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
No, when Julien and I discussed this in last version[1], we hadn't
thought
Quoted 1 line
so deeply. We just thought a memory range described in DT cannot be
split
Quoted 1 line
across multiple nodes. So
NR_MEM_BANKSshould be equal to
NR_MEM_BANKS.Quoted 9 lines
Quoted 9 lines
08/msg00974.html
Quoted 6 lines
Quoted 6 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
doesn't make any difference unless
CONFIG_NR_NUMA_NODESisincreased.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis higher than 64?I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.But I agree with Julien's suggestion,
NR_MEM_BANKSand
NR_NODE_MEMBLKSQuoted 10 lines
Quoted 10 lines
Quoted 3 lines
must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.
OK. In that case I'd say to get rid of the previous definition of
NR_NODE_MEMBLKSas it is probably not necessary, see below.Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(thereis
Quoted 7 lines
Quoted 7 lines
no
Quoted 6 lines
Quoted 2 lines
x86 definition of it)
Yes.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Review
Quoted 7 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/xen/include/asm-arm/numa.hQuoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
index
8f1c67e3eb..21569e634b100644xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif This one we can remove it completely right?
Quoted 8 lines
How about define
NR_MEM_BANKSto:
#ifdefCONFIG_NR_NUMA_NODES
#defineNR_MEM_BANKS(CONFIG_NR_NUMA_NODES* 2)
#else
#defineNR_MEM_BANKS128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS128". And replace all
NR_NODE_MEMBLKSQuoted 2 lines
in NUMA code to
NR_MEM_BANKSto removeNR_NODE_MEMBLKScompletely. In this case,NR_MEM_BANKScan be aware of the changes of
CONFIG_NR_NUMA_NODES.x86 doesn't have
NR_MEM_BANKSas far as I can tell. I guess you also meant to renameNR_NODE_MEMBLKStoNR_MEM_BANKS?But
NR_MEM_BANKSis not directly related toCONFIG_NR_NUMA_NODESbecause there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:#define
NR_NODE_MEMBLKS(MAX_NUMNODES*2)Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?
On the other hand,
NR_MEM_BANKSandNR_NODE_MEMBLKSseem to be related. In fact, what's the difference?
NR_MEM_BANKSis the max number of memory banks (with or without numa-node-id).
Quoted 7 lines
NR_NODE_MEMBLKSis the max number of memory banks with NUMA support (with numa-node-id)?They are basically the same thing. On ARM I would just do:
#define
NR_NODE_MEMBLKSMAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))
As you wrote above, the second part of the MAX is totally arbitrary. In fact, it is very likely than if you have more than 64 nodes, you may need a lot more than 2 regions per node.
So, for Arm, I would just define NR_NODE_MEMBLKS as an alias to NR_MEM_BANKS so it can be used by common code.
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
In reply to Stefano StabelliniHi Stefano, Julien,
Quoted 152 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:00
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKS+x86 maintainers
Quoted 8 lines · Wei Chen
Quoted 8 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 8 lines
Quoted 8 lines
NR_NODE_MEMBLKSQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 4 lines
Quoted 4 lines
NR_NODE_MEMBLKSQuoted 1 line · Wei Chen
As a memory range described in device tree cannot be split
across
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin arch header.This statement is true but what is the goal of this patch? Is it
to
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't
thought
Quoted 1 line
so deeply. We just thought a memory range described in DT cannot be
split
Quoted 1 line
across multiple nodes. So
NR_MEM_BANKSshould be equal to
NR_MEM_BANKS.Quoted 9 lines
Quoted 9 lines
08/msg00974.html
Quoted 6 lines
Quoted 6 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
doesn't make any difference unless
CONFIG_NR_NUMA_NODESisincreased.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
higher than 64?
I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.But I agree with Julien's suggestion,
NR_MEM_BANKSand
NR_NODE_MEMBLKSQuoted 10 lines
Quoted 10 lines
Quoted 3 lines
must be aware of each other. I had thought to add some ASSERT check, but I don't know how to do it better. So I post this patch for more suggestion.
OK. In that case I'd say to get rid of the previous definition of
NR_NODE_MEMBLKSas it is probably not necessary, see below.Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(thereis
Quoted 7 lines
Quoted 7 lines
no
Quoted 6 lines
Quoted 2 lines
x86 definition of it)
Yes.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/include/asm-
Quoted 7 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm- renamed from xen/include/asm-arm/numa.h +0
-0
arm/numa.hQuoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
index
8f1c67e3eb..21569e634b100644xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif This one we can remove it completely right?
Quoted 8 lines
How about define
NR_MEM_BANKSto:
#ifdefCONFIG_NR_NUMA_NODES
#defineNR_MEM_BANKS(CONFIG_NR_NUMA_NODES* 2)
#else
#defineNR_MEM_BANKS128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS128". And replace all
Quoted 8 lines
NR_NODE_MEMBLKSQuoted 2 lines
in NUMA code to
NR_MEM_BANKSto removeNR_NODE_MEMBLKScompletely. In this case,NR_MEM_BANKScan be aware of the changes of
CONFIG_NR_NUMA_NODES.x86 doesn't have
NR_MEM_BANKSas far as I can tell. I guess you also meant to renameNR_NODE_MEMBLKStoNR_MEM_BANKS?
Yes.
Quoted 9 lines
But
NR_MEM_BANKSis not directly related toCONFIG_NR_NUMA_NODESbecause there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:#define
NR_NODE_MEMBLKS(MAX_NUMNODES*2)Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?
At that time, this was probably the most cost-effective approach. Enough and easy. But, if more nodes need to be supported in the future, it may bring more memory blocks. And this maximum value might not apply. The maximum may need to support dynamic extension.
Quoted 4 lines
On the other hand,
NR_MEM_BANKSandNR_NODE_MEMBLKSseem to be related. In fact, what's the difference?
Quoted 8 lines
NR_MEM_BANKSis the max number of memory banks (with or without numa-node-id).
NR_NODE_MEMBLKSis the max number of memory banks with NUMA support (with numa-node-id)?They are basically the same thing. On ARM I would just do:
Probably not, NR_MEM_BANKS will count those memory ranges without numa-node-id in boot memory parsing stage (process_memory_node or EFI parser). But NR_NODE_MEMBLKS will only count those memory ranges with numa-node-id.
Quoted 5 lines
#define
NR_NODE_MEMBLKSMAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))And maybe the definition could be common with x86 if we define
NR_MEM_BANKSto 128 on x86 too.
Julien had comment here, I will continue in that email.
RE: [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
Hi Julien,
Quoted 206 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Chen
Sent: 2021年9月27日 14:46
To: Stefano Stabellini <[email protected]>
Cc: [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKSHi Stefano, Julien,
Quoted 32 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 13:00
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto override defaultNR_NODE_MEMBLKS+x86 maintainers
Quoted 8 lines · Wei Chen
Quoted 8 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月27日 11:26
To: Wei Chen <[email protected]>
Cc: Stefano Stabellini <[email protected]>; xen- [email protected]; [email protected]; Bertrand Marquis <[email protected]>
Subject: RE: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 9 lines
Quoted 9 lines
NR_NODE_MEMBLKSQuoted 5 lines · Wei Chen
Quoted 5 lines
-----Original Message-----
From: Stefano Stabellini <[email protected]>
Sent: 2021年9月24日 9:35
To: Wei Chen <[email protected]>
Cc: [email protected]; [email protected];Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Bertrand Marquis <[email protected]>
Subject: Re: [PATCH 22/37] xen/arm: useNR_MEM_BANKSto overridedefault
Quoted 4 lines
Quoted 4 lines
NR_NODE_MEMBLKSQuoted 1 line · Wei Chen
As a memory range described in device tree cannot be split
across
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
multiple nodes. So we define
NR_NODE_MEMBLKSasNR_MEM_BANKSin
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
arch header.
This statement is true but what is the goal of this patch? Is it
to
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
reduce code size and memory consumption?
No, when Julien and I discussed this in last version[1], we hadn't
thought
Quoted 1 line
so deeply. We just thought a memory range described in DT cannot
be
Quoted 21 lines
Quoted 2 lines
Quoted 2 lines
split
Quoted 1 line
across multiple nodes. So
NR_MEM_BANKSshould be equal to
NR_MEM_BANKS.Quoted 9 lines
Quoted 9 lines
08/msg00974.html
Quoted 6 lines
Quoted 6 lines
I am asking because
NR_MEM_BANKSis 128 and NR_NODE_MEMBLKS=2*MAX_NUMNODES which is 64 by default so againNR_NODE_MEMBLKSis 128 before this patch.In other words, this patch alone doesn't make any difference; at
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
doesn't make any difference unless
CONFIG_NR_NUMA_NODESisincreased.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
So, is the goal to reduce memory usage when
CONFIG_NR_NUMA_NODESis
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
higher than 64?
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I also thought about this problem when I was writing this patch.
CONFIG_NR_NUMA_NODESis increasing, butNR_MEM_BANKSis a fixed value, thenNR_MEM_BANKScan be smaller thanCONFIG_NR_NUMA_NODESat one point.But I agree with Julien's suggestion,
NR_MEM_BANKSand
NR_NODE_MEMBLKSQuoted 1 line
Quoted 1 line
Quoted 1 line
must be aware of each other. I had thought to add some ASSERT
check,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
but I don't know how to do it better. So I post this patch for
more
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
suggestion.
OK. In that case I'd say to get rid of the previous definition of
NR_NODE_MEMBLKSas it is probably not necessary, see below.Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
And keep default
NR_NODE_MEMBLKSin common header
for those architectures NUMA is disabled.This last sentence is not accurate: on x86 NUMA is enabled and
NR_NODE_MEMBLKSis still defined inxen/include/xen/numa.h(there
Quoted 58 lines
is
Quoted 7 lines
Quoted 7 lines
no
Quoted 6 lines
Quoted 2 lines
x86 definition of it)
Yes.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/include/asm-
Quoted 7 lines
Signed-off-by Wei Chen <[email protected]>--- xen/include/asm-arm/numa.h | 8 +++++++- xen/include/xen/numa.h | 2 ++ 2 files changed, 9 insertions(+), 1 deletion(-)xen/include/asm- renamed from xen/include/asm-arm/numa.h +0
-0
arm/numa.hQuoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
index
8f1c67e3eb..21569e634b100644xen/include/asm-arm/numa.h +7
-1@@ -3,9 +3,15 @@#include <xen/mm.h> +#include <asm/setup.h> +typedef u8 nodeid_t ; -#ifndef CONFIG_NUMA +#ifdef CONFIG_NUMA + +#define NR_NODE_MEMBLKS NR_MEM_BANKS + +#else /* Fake one node for now. See also node_online_map. */ #define cpu_to_node (cpu) 0xen/include/xen/numa.h +0
-0@@ -12,7 +12,9 @@#define MAX_NUMNODES 1#endif Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
xen/include/xen/numa.h line 17
Quoted 3 lines
xen/include/xen/numa.h +2
-0+#ifndef NR_NODE_MEMBLKS #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) +#endif This one we can remove it completely right?
Quoted 8 lines
How about define
NR_MEM_BANKSto:
#ifdefCONFIG_NR_NUMA_NODES
#defineNR_MEM_BANKS(CONFIG_NR_NUMA_NODES* 2)
#else
#defineNR_MEM_BANKS128
#endif
for both x86 and Arm. For those architectures do not support or enable NUMA, they can still use "NR_MEM_BANKS128". And replace allQuoted 8 lines
NR_NODE_MEMBLKSQuoted 2 lines
in NUMA code to
NR_MEM_BANKSto removeNR_NODE_MEMBLKScompletely. In this case,NR_MEM_BANKScan be aware of the changes of
CONFIG_NR_NUMA_NODES.x86 doesn't have
NR_MEM_BANKSas far as I can tell. I guess you also meant to renameNR_NODE_MEMBLKStoNR_MEM_BANKS?Yes.
Quoted 9 lines
But
NR_MEM_BANKSis not directly related toCONFIG_NR_NUMA_NODESbecause there can be many memory banks for each numa node, certainly more than
2. The existing definition on x86:#define
NR_NODE_MEMBLKS(MAX_NUMNODES*2)Doesn't make a lot of sense to me. Was it just an arbitrary limit for the lack of a better way to set a maximum?
At that time, this was probably the most cost-effective approach. Enough and easy. But, if more nodes need to be supported in the future, it may bring more memory blocks. And this maximum value might not apply. The maximum may need to support dynamic extension.
Quoted 10 lines
On the other hand,
NR_MEM_BANKSandNR_NODE_MEMBLKSseem to be related. In fact, what's the difference?
NR_MEM_BANKSis the max number of memory banks (with or without numa-node-id).
NR_NODE_MEMBLKSis the max number of memory banks with NUMA support (with numa-node-id)?
Quoted 11 lines
Quoted 2 lines
They are basically the same thing. On ARM I would just do:
Probably not,
NR_MEM_BANKSwill count those memory ranges without numa-node-id in boot memory parsing stage (process_memory_nodeor EFI parser). ButNR_NODE_MEMBLKSwill only count those memory ranges with numa-node-id.Quoted 3 lines
#define
NR_NODE_MEMBLKSMAX(NR_MEM_BANKS, (CONFIG_NR_NUMA_NODES * 2))
Quote Julien's comment from HTML email to here:
" As you wrote above, the second part of the MAX is totally arbitrary. In fact, it is very likely than if you have more than 64 nodes, you may need a lot more than 2 regions per node.
So, for Arm, I would just define NR_NODE_MEMBLKS as an alias to NR_MEM_BANKS so it can be used by common code.
"
But here comes the problem:
How can we set the NR_MEM_BANKS maximum value, 128 seems an arbitrary too? If #define NR_MEM_BANKS (CONFIG_NR_NUMA_NODES * N)? And what N should be?
Quoted 4 lines
Quoted 2 lines
And maybe the definition could be common with x86 if we define
NR_MEM_BANKSto 128 on x86 too.Julien had comment here, I will continue in that email.